diff --git a/.gitignore b/.gitignore index 143452b..45f1750 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,6 @@ data/ out/ .env __pycache__/ +data_diff/ +remeasure/ +runs/ diff --git a/README.md b/README.md index 68273aa..01f9712 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,10 @@ The data of the AIMsight stations, in a single copy on the workstation, and their pumping. - `data//`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again. +- `data_diff//`: the differential images, full size, with their index and the patterns, to measure them again, and the metrics, IRON images, weather and `targets.csv` as in `data/` (`extraction/pump_diff.sh`), **out of git**; it never touches `data/`. Shared by every configuration. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`. +- `configs/.yaml`: a configuration, what differs from the cloud (patterns by `sensor_name`: a property, or a window redrawn from a DIFF image; the camera correction: algorithm, mode, focal distance, one measurement at a time or the series), in git; `cloud.yaml`, empty, is the cloud and reproduces `data/` (JUNGFRAU 2026-10-12..16: `corr_x/y` within 0.03 px, the same rejections, the camera pose within 5e-4 deg); `jungfrau_example.yaml`, a model. +- `runs///`: the result of a configuration, **out of git**, the tree of `data/` (`displacement/`, `camera/`, `targets.csv`; `images/`, `metrics/`, `weather/` linked to `data_diff/`): a repository reads it by pointing its `data` link at `~/aimsight-data/runs/` instead of `~/aimsight-data/data`. Beside the stations, `run.yaml` (the configuration as resolved, optical's commit) and `diagnostics/` (the patterns used, the emulator's rows: edge margin, second spot). +- `processing/run_config.sh [SERIAL...] [--from D] [--to E]`: computes `runs//` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 16 s for 5 days of JUNGFRAU); `--check` without data. Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`. A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new): cut it from an image at the start of the period studied. - `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations). - `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data//weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`. - `.env`: the credentials of the database and of S3, at the root, never versioned. diff --git a/configs/cloud.yaml b/configs/cloud.yaml new file mode 100644 index 0000000..7f742ee --- /dev/null +++ b/configs/cloud.yaml @@ -0,0 +1,17 @@ +# The cloud's processing, nothing changed: the patterns of data_diff//images_diff/sensors.json, the camera +# correction of process_shifts (ransac, ransac_threshold 0.99, the unit's focal_distance / pixel_size, else 6.0 / 0.0024), +# one measurement at a time. The reference the other configurations are compared with; it must reproduce data/. +# +# A configuration states only what differs, for example: +# +# camera: # common to the stations of the file +# algorithm: ransac # ransac | umeyama | danish_hybrid | trusted_danish (memory: per_epoch false) +# mode: "off" # off | rotation | rotation_scale | relative (optical's refine_allpoints) +# per_epoch: true # false: the whole series at once (a rejected reference interpolated in time) +# stations: +# JUNGFRAU: +# camera: {focal_distance: 6.3} +# patterns: # by sensor_name: any existing property, or a template redrawn from a DIFF image +# PM_G: {error_threshold: 0.7} +# REF_CT: {pattern_type: target} +# BL_C: {redraw: {image: 20261012T1200Z_9999999.png, corners: [[1700, 929], [1724, 953]]}} diff --git a/configs/jungfrau_example.yaml b/configs/jungfrau_example.yaml new file mode 100644 index 0000000..2f80fe4 --- /dev/null +++ b/configs/jungfrau_example.yaml @@ -0,0 +1,9 @@ +# Example, 2026-11-19: the two findings of the first re-measurement of JUNGFRAU (2026-10-12..16). +# - PM_G is rejected on every image by its threshold of 0.3 (its error is about 0.43): back to the usual 0.7. +# - BL_C is cut by its 20x20 window (edge margin -0.2 px in median): the camera moved by about 2.5 px since the +# patterns were drawn; the window redrawn around where the target is seen now, x_ref kept (same zero). +stations: + JUNGFRAU: + patterns: + PM_G: {error_threshold: 0.7} + BL_C: {redraw: {image: 20261012T1200Z_10163946.png, corners: [[1700, 933], [1720, 953]]}} diff --git a/docs/superpowers/plans/2026-11-19-run-config.md b/docs/superpowers/plans/2026-11-19-run-config.md new file mode 100644 index 0000000..2ff52b4 --- /dev/null +++ b/docs/superpowers/plans/2026-11-19-run-config.md @@ -0,0 +1,53 @@ +# run_config Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** `processing/run_config.sh [SERIAL...]` writes `runs///` with the files of `data/`. + +**Architecture:** one script; optical imported from `$OPTICAL`; re-measurement through `reprocess_images.main`; +correction through `remove_camera_movement`, per epoch by default. + +**Tech Stack:** uv (numpy, pandas, pillow, pyyaml, scipy, scikit-image, opencv-python-headless). + +**Spec:** `docs/superpowers/specs/2026-11-19-run-config-design.md` + +## Global Constraints + +- Writes only under `runs//` (guarded); reads `data_diff/`, never `data/` except for the acceptance comparison. +- Columns of `data/`'s advanced CSV, in that order: `time_utc, sensor_name, x, y, error_value, error_threshold, + relative_x, relative_y, corr_x, corr_y, reproj_err, inlier, sigma0, pattern_type, peak, flux, sharpness, fwhm, + dominance, snr, corr_x_mm, corr_y_mm`. +- Camera CSV columns: `time_utc`, then `median_/min_/max_` of `pan, tilt, scale, rotation, sigma0`. + +## Review Focus + +- A configuration with a typo (`error_treshold`, unknown sensor): error before any work. +- An epoch where only one reference survives: translation only, no camera row (as the cloud). +- A day without images in the range: no file for it. +- Re-running the same configuration: the station folder replaced, other stations untouched. +- optical's working tree dirty: recorded in `run.yaml`. + +--- + +### Task 1: configuration and correction, with `--check` + +**Files:** Create `processing/run_config.py`. + +**Interfaces:** `load_config(path) -> dict`; `camera_settings(config, station, unit_props) -> dict`; +`cloud_properties(sensors) -> {sid: props}`; `apply_patterns(props, overrides, images_dir) -> {sid: props}`; +`correct(meas: DataFrame, props, cam) -> (advanced: DataFrame, camera: DataFrame)`. + +- [ ] `check()`: unknown top / station / camera / pattern key raises; unknown sensor_name raises; camera order station > + common > unit > defaults; synthetic correction: three static references, a static target (corr ~0), a target above + its threshold (corr empty, x kept), relative = x - pattern x, sigma0 on every row, one camera row per minute. +- [ ] Implement; `--check` prints ok. + +### Task 2: the run, the cloud configuration, the acceptance + +**Files:** Modify `processing/run_config.py` (`main`); create `processing/run_config.sh`, `configs/cloud.yaml`; +modify `.gitignore` (`runs/`), `README.md`. + +- [ ] `main`: stations, days, patterns, `reprocess_images.main`, `correct`, per-day files, links, `run.yaml`. +- [ ] `run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17`; `data.py` reads `runs/cloud/JUNGFRAU`. +- [ ] Compare with `data/JUNGFRAU`: corr_x / corr_y (all, and the rejected-reference night of 10-15), camera pose. +- [ ] Docs. diff --git a/docs/superpowers/specs/2026-11-19-run-config-design.md b/docs/superpowers/specs/2026-11-19-run-config-design.md new file mode 100644 index 0000000..c08a369 --- /dev/null +++ b/docs/superpowers/specs/2026-11-19-run-config-design.md @@ -0,0 +1,53 @@ +# run_config: a configuration recomputed into runs// + +2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`. + +## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`) + +Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw +`x`, `y`, `error` (and the metrics) are stored; targets above their `error_threshold` get `x`/`y` blanked; a rejected +`global_ref` gets back the `x`/`y` of the latest stored frame whose error is not above its own (in practice its own +raw position, inserted just before), else its pattern position; `remove_camera_movement(r, focal_distance (6.0), +pixel_size (0.0024), algorithm (ransac, ransac_threshold 0.99))`, which rejects those references from the fit again +(error columns given) but still corrects their rows; `corr -= (x_ref, y_ref)`; `cam_info` (pan, tilt, scale, +rotation, sigma0) and `ref_stats` (reproj_err, inlier) stored. The Advanced export adds `relative = x - pattern x` +and copies `sigma0` onto every row; the camera export is the per-minute median / min / max of `cam_info`. + +## Design + +`processing/run_config.py`, through `processing/run_config.sh` (uv: numpy, pandas, pillow, pyyaml, scipy, +scikit-image, opencv-python-headless; optical's code from `$OPTICAL`, default `~/optical`): + + processing/run_config.sh [SERIAL...] [--from D] [--to E] [--jobs N] + +``: `configs/.yaml` or a path; stations: those named, else those of the configuration; days: those +indexed in `data_diff//images_diff/index/` (within `--from`/`--to`). For each station, `runs///` +is replaced: + +1. **Patterns**: `images_diff/sensors.json` (DIFF type, a pattern, not archived; latest value of each property), then + the configuration by `sensor_name` (an unknown name or key is an error): any property replaced + (`error_threshold`, `pattern_type`, `distance_convertion_factor`, `x`, `y`...); `redraw: {image, corners}` cuts the + template from that DIFF image as the device's pattern editor does (`estimate_camera_pose` from the references, + `saved_anchor_and_reference`: `x_ref` kept, `ImagePatternMono.from_image`). Written to + `diagnostics/_sensor_properties.json`. +2. **Re-measurement**: optical's `reprocess_images.main` (emulator) on the days' images, `--times` from the index; + rows kept in `diagnostics/_remeasured.csv`. +3. **Correction**: as the cloud, one measurement at a time (`per_epoch: true`, default): targets above their + threshold blanked, references kept raw (rejected in the fit by `remove_camera_movement`), `corr -= x_ref`; + `per_epoch: false`: the whole series in one call (a rejected reference interpolated in time, algorithms with + memory). Camera block: `algorithm` (ransac), `ransac_threshold` (0.99, ransac only), `mode` (`off` | `rotation` | + `rotation_scale` | `relative`, optical's `refine_allpoints`), `focal_distance`, `pixel_size` (unit.json, else 6.0 / + 0.0024), `per_epoch`; station block over common block over unit.json over the cloud's defaults. +4. **Output**, the files of `data/`: `displacement/_advanced_.csv` (columns of `data/`: `x, y, + error_value, peak..snr` from the emulator, `relative = x - pattern x`, `corr` from the correction, `reproj_err, + inlier` of the references, `sigma0` of the epoch on every row, `corr_*_mm = corr * distance_convertion_factor`), + `camera/_camera_.csv` (median / min / max per minute), `targets.csv`; `images/`, `metrics/`, + `weather/` as relative symbolic links to `data_diff//`; `run.yaml` at `runs//` (configuration as + resolved per station, optical's commit and state, days, date). + +Acceptance: `configs/cloud.yaml` (empty) on JUNGFRAU 2026-10-12..16 against `data/JUNGFRAU`: positions and errors +within 0.05 px (done by the emulator), `corr_x/y` and the camera pose close to the cloud, the differences counted. + +Check: `run_config.py --check` in the uv environment: the configuration rules (unknown keys, order of the camera +blocks, overrides by name) and a synthetic correction (three references, two targets, one rejected) through +optical's `remove_camera_movement`. diff --git a/processing/run_config.py b/processing/run_config.py new file mode 100644 index 0000000..ba707e9 --- /dev/null +++ b/processing/run_config.py @@ -0,0 +1,361 @@ +"""Recomputes a configuration from the differential images into runs//, in the tree of data/, so that the other +repositories read it as they read data/ (point their `data` link at runs/). Never writes outside runs//. +Design: docs/superpowers/specs/2026-11-19-run-config-design.md. + + processing/run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17 + processing/run_config.sh configs/mine.yaml the stations the configuration names, all their days + +: configs/.yaml (or a path); stations: those given, else those of the configuration; days: those indexed +in data_diff//images_diff/index/ (within --from / --to, --to excluded). For each station, runs/// +is replaced (the other stations of the run are kept): + displacement/_advanced_.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator; + relative = x - pattern x; corr from the camera correction; reproj_err, + inlier of the references; sigma0 of the epoch; corr_*_mm + camera/_camera_.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute + targets.csv the patterns of the configuration + images/ metrics/ weather/ relative symbolic links to data_diff// +and, outside the station folders (the reading tools do not see them): runs//run.yaml (the configuration as +resolved per station, optical's commit, the days) and runs//diagnostics/ (the patterns used, the emulator's rows). + +A configuration (YAML) states what differs from the cloud: the patterns (by sensor_name: any of their properties, or +`redraw: {image: , corners: [[x0, y0], [x1, y1]]}`, cut as the device's pattern editor does) and +the camera correction (`algorithm`, `ransac_threshold`, `mode`: off | rotation | rotation_scale | relative, +`focal_distance`, `pixel_size`, `per_epoch`), common or per station; configs/cloud.yaml, empty, is the cloud. +""" +import argparse +import csv +import json +import os +import shutil +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path + +# Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and +# 8 workers x one BLAS thread per core made a 11 s re-measurement take 2 min (45 min of CPU). +for _var in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"): + os.environ.setdefault(_var, "1") + +import numpy as np # noqa: E402 +import pandas as pd # noqa: E402 +import yaml # noqa: E402 + +ROOT = Path(__file__).resolve().parents[1] +OPTICAL = Path(os.environ.get("OPTICAL", Path.home() / "optical")).resolve() +sys.path[:0] = [str(OPTICAL), str(OPTICAL / "scripts" / "pcc_emulator")] + +MTYPE = "OPTICAL_SHIFT-DIFF" +QUALITY = ["peak", "flux", "sharpness", "fwhm", "dominance", "snr"] +COLUMNS = ["time_utc", "sensor_name", "x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x", + "corr_y", "reproj_err", "inlier", "sigma0", "pattern_type"] + QUALITY + ["corr_x_mm", "corr_y_mm"] +CAM = ["pan", "tilt", "scale", "rotation", "sigma0"] +# the cloud's defaults (process_shifts), then the unit's properties, then the configuration +DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0, + "pixel_size": 0.0024, "per_epoch": True} +UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size") +LINKED = ("images", "metrics", "weather") + + +def _keys(d, allowed, where): + if not isinstance(d, dict): + raise ValueError(f"{where}: a mapping is expected") + bad = set(d) - set(allowed) + if bad: + raise ValueError(f"{where}: unknown key(s) {sorted(bad)}; allowed: {sorted(allowed)}") + + +def load_config(path): + """The configuration, its keys checked (a typo is an error, not a setting silently ignored).""" + c = yaml.safe_load(Path(path).read_text()) or {} + _keys(c, {"stations", "camera"}, "the configuration") + _keys(c.get("camera") or {}, DEFAULTS, "camera") + for s, block in (c.get("stations") or {}).items(): + _keys(block or {}, {"patterns", "camera"}, f"stations.{s}") + _keys((block or {}).get("camera") or {}, DEFAULTS, f"stations.{s}.camera") + return c + + +def latest(rows): + """{name: value} of property rows ordered by name then time (sensors.json, unit.json): the latest wins.""" + return {r["name"]: r["value"] for r in rows} + + +def camera_settings(config, station, unit_props): + cam = dict(DEFAULTS) + cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props}) + cam.update(config.get("camera") or {}) + cam.update(((config.get("stations") or {}).get(station) or {}).get("camera") or {}) + return cam + + +def cloud_properties(sensors): + """{sensor_id: properties} of the patterns the cloud measures the DIFF images with (process_shifts): DIFF type, a + pattern, not archived.""" + out = {} + for s in sensors: + p = latest(s["properties"]) + if s["type"] == MTYPE and p.get("pattern") and not p.get("archive"): + out[str(s["id"])] = p + return out + + +def apply_patterns(props, overrides, images_dir): + """The configuration's patterns over the cloud's, by sensor_name: a property replaced (only an existing one: a + typo is an error), or the template redrawn from an image.""" + props = {sid: dict(p) for sid, p in props.items()} + by_name = {p.get("sensor_name"): sid for sid, p in props.items()} + redraws = {} + for name, ov in (overrides or {}).items(): + if name not in by_name: + raise ValueError(f"patterns.{name}: no such pattern; known: {sorted(map(str, by_name))}") + sid = by_name[name] + _keys(ov, set(props[sid]) - {"pattern"} | {"redraw"}, f"patterns.{name}") + for k, v in ov.items(): + if k == "redraw": + _keys(v, {"image", "corners"}, f"patterns.{name}.redraw") + redraws[sid] = v + else: + props[sid][k] = v + if redraws: + redraw(props, redraws, images_dir) + return props + + +def redraw(props, redraws, images_dir): + """As the device's pattern editor saves a drawn box (optical_server pages/patternCreation.py): the camera pose from + the references found in the image, the anchor and x_ref (kept) of saved_anchor_and_reference, the template cut + at the box.""" + from aimsight_optical.imagePatternMono import ImagePatternMono, get_center, saved_anchor_and_reference + from aimsight_optical.pattern_converter import load_pattern_from_params + from aimsight_optical.pattern_pose import estimate_camera_pose + from reprocess_images import read_image + + for image in sorted({r["image"] for r in redraws.values()}): + im = read_image(images_dir / image) + stored = {sid: load_pattern_from_params(**dict(p)) for sid, p in props.items()} + mine = {sid: r for sid, r in redraws.items() if r["image"] == image} + drawn = {sid: get_center(r["corners"]) for sid, r in mine.items() + if props[sid].get("pattern_type") == "global_ref"} + pose, seen = estimate_camera_pose(stored, im, drawn=drawn) + for sid, r in mine.items(): + old, corners = stored[sid], [[int(v) for v in c] for c in r["corners"]] + anchor, (x_ref, y_ref) = saved_anchor_and_reference(old, corners, old.extra_attr.get("pattern_type", ""), + pose=pose, seen=seen.get(sid)) + kept = {k: v for k, v in old.extra_attr.items() if k not in ("median", "p99", "x_ref", "y_ref")} + new = ImagePatternMono.from_image(im, corners, center=anchor, x_ref=x_ref, y_ref=y_ref, **kept) + props[sid] = json.loads(json.dumps(new.to_dict(), default=_plain)) + + +def _plain(o): + return o.item() if hasattr(o, "item") else o.tolist() if hasattr(o, "tolist") else str(o) + + +def correct(meas, props, cam): + """The emulator's rows (time_utc, pattern_id, x, y, error, metrics) -> (advanced rows in the columns of data/, + camera rows per minute), as the cloud's process_shifts with per_epoch, else over the whole series.""" + from aimsight_optical.imagePatternMono import get_center, reference_or_anchor + from aimsight_optical.pattern_converter import remove_camera_movement + + m = meas.copy() + m["sensor_id"] = m["pattern_id"].astype(str) + m["time_utc"] = pd.to_datetime(m["time_utc"], utc=True, format="mixed") + p = {sid: props[sid] for sid in m["sensor_id"].unique()} + anchor = {sid: (float(q["x"]), float(q["y"])) if np.isfinite([q.get("x", np.nan), q.get("y", np.nan)]).all() + else get_center(q["corners"]) for sid, q in p.items()} + ref = {sid: reference_or_anchor(q.get("x_ref"), q.get("y_ref"), anchor[sid]) for sid, q in p.items()} + sid = m["sensor_id"] + m["sensor_name"] = sid.map({s: str(q.get("sensor_name", s)) for s, q in p.items()}) + m["error_threshold"] = pd.to_numeric(sid.map({s: q.get("error_threshold", 0.7) for s, q in p.items()})) + m["sensor_type"] = sid.map({s: q.get("pattern_type", "") or "" for s, q in p.items()}) + m["x_ref"], m["y_ref"] = sid.map({s: r[0] for s, r in ref.items()}), sid.map({s: r[1] for s, r in ref.items()}) + + inp = m[["time_utc", "sensor_id", "x", "y", "error", "error_threshold", "sensor_type", "x_ref", "y_ref"]].copy() + # as the cloud: a target above its threshold has no position; a reference keeps its raw one (its row is corrected, + # remove_camera_movement leaves it out of the fit on its error) + inp.loc[(inp["error"] > inp["error_threshold"]) & (inp["sensor_type"] != "global_ref"), ["x", "y"]] = np.nan + kw = {"algorithm": cam["algorithm"]} + if str(cam["algorithm"]).upper() == "RANSAC": + kw["ransac_threshold"] = cam["ransac_threshold"] + if cam["mode"] != "off": + kw["refine_allpoints"] = cam["mode"] + frames = [g for _, g in inp.groupby("time_utc")] if cam["per_epoch"] else [inp] + corrs, cams, stats = [], [], [] + for g in frames: + c, cp, rs = remove_camera_movement(g.copy(), cam["focal_distance"], cam["pixel_size"], + timeserie_column="time_utc", **kw) + corrs.append(c[["time_utc", "sensor_id", "corr_x", "corr_y"]]) + if not cp.empty: + cams.append(cp[CAM].rename_axis("time_utc").reset_index()) + if not rs.empty: + stats.append(rs[["time_utc", "sensor_id", "reproj_err", "inlier"]]) + m = m.merge(pd.concat(corrs), on=["time_utc", "sensor_id"], how="left") + m = m.merge(pd.concat(stats) if stats else pd.DataFrame(columns=["time_utc", "sensor_id", "reproj_err", "inlier"]), + on=["time_utc", "sensor_id"], how="left") + cp = pd.concat(cams).dropna(subset=CAM, how="all") if cams else pd.DataFrame(columns=["time_utc"] + CAM) + cp["time_utc"] = pd.to_datetime(cp["time_utc"], utc=True) + m["sigma0"] = m["time_utc"].map(cp.groupby("time_utc")["sigma0"].median()) + + factor = pd.to_numeric(sid.map({s: q.get("distance_convertion_factor") for s, q in p.items()}), errors="coerce") + adv = pd.DataFrame({ + "time_utc": m["time_utc"], "sensor_name": m["sensor_name"], "x": m["x"], "y": m["y"], + "error_value": m["error"], "error_threshold": m["error_threshold"], + "relative_x": m["x"] - sid.map({s: a[0] for s, a in anchor.items()}), + "relative_y": m["y"] - sid.map({s: a[1] for s, a in anchor.items()}), + "corr_x": m["corr_x"] - m["x_ref"], "corr_y": m["corr_y"] - m["y_ref"], + "reproj_err": m["reproj_err"], "inlier": m["inlier"], "sigma0": m["sigma0"], "pattern_type": m["sensor_type"], + **{q: m[q] if q in m else np.nan for q in QUALITY}}) + adv["corr_x_mm"], adv["corr_y_mm"] = adv["corr_x"] * factor.to_numpy(), adv["corr_y"] * factor.to_numpy() + adv = adv.sort_values(["time_utc", "sensor_name"], kind="stable").reset_index(drop=True)[COLUMNS] + + cp["time_utc"] = cp["time_utc"].dt.floor("1min") + agg = cp.groupby("time_utc")[CAM].agg(["median", "min", "max"]) + agg.columns = [f"{stat}_{c}" for c, stat in agg.columns] + return adv, agg.reset_index() + + +def write_station(adv, camera, props, dest, station): + for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")): + (dest / folder).mkdir(parents=True, exist_ok=True) + for day, d in df.groupby(df["time_utc"].dt.date): + d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False) + t = pd.DataFrame([{"sensor_name": q.get("sensor_name"), "pattern_type": q.get("pattern_type"), + "distance_mm": q.get("distance_mm"), "mm_per_px": q.get("distance_convertion_factor")} + for q in props.values()]).sort_values("sensor_name") + t.to_csv(dest / "targets.csv", index=False) + + +def optical_state(): + git = lambda *a: subprocess.run(["git", "-C", str(OPTICAL), *a], capture_output=True, text=True).stdout.strip() # noqa: E731 + return {"path": str(OPTICAL), "commit": git("rev-parse", "HEAD"), "branch": git("rev-parse", "--abbrev-ref", "HEAD"), + "dirty": bool(git("status", "--porcelain", "--untracked-files=no"))} + + +def main(): + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("config", help="name of configs/.yaml, or a path") + p.add_argument("stations", nargs="*", help="serial numbers; none: those of the configuration") + p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD") + p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD") + p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)") + p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff") + p.add_argument("--out", type=Path, default=ROOT / "runs") + a = p.parse_args() + path = Path(a.config) if a.config.endswith((".yaml", ".yml")) or os.sep in a.config \ + else ROOT / "configs" / f"{a.config}.yaml" + config, name = load_config(path), path.stem + stations = a.stations or list(config.get("stations") or {}) + if not stations: + sys.exit(f"No station: name one ({path} names none)") + run = a.out / name + (run / "diagnostics").mkdir(parents=True, exist_ok=True) + record_path = run / "run.yaml" + record = yaml.safe_load(record_path.read_text()) if record_path.exists() else {} + record.update({"configuration": name, "source": str(path), "optical": optical_state()}) + record.setdefault("stations", {}) + + import reprocess_images + + for station in stations: + src = a.data_diff / station + days = sorted(f for f in (src / "images_diff" / "index").glob(f"{station}_diff_*.csv") + if (not a.start or f.stem[-8:] >= a.start.replace("-", "")) + and (not a.end or f.stem[-8:] < a.end.replace("-", ""))) + rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]] + if not rows: + print(f"{station}: no differential image indexed in the range, nothing done") + continue + sensors = json.loads((src / "images_diff" / "sensors.json").read_text()) + unit = json.loads((src / "images_diff" / "unit.json").read_text()) + cam = camera_settings(config, station, latest(unit["properties"])) + overrides = ((config.get("stations") or {}).get(station) or {}).get("patterns") + props = apply_patterns(cloud_properties(sensors), overrides, src / "images_diff") + print(f"{station}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} patterns, " + f"camera {cam}") + + diag = run / "diagnostics" + sp, times, remeasured = (diag / f"{station}_sensor_properties.json", diag / f"{station}_times.csv", + diag / f"{station}_remeasured.csv") + sp.write_text(json.dumps(props)) + with open(times, "w", newline="") as fp: + w = csv.writer(fp) + w.writerow(["file", "time_utc"]) + w.writerows((r["file"], r["time_utc"]) for r in rows) + reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured), + "--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows]) + adv, camera = correct(pd.read_csv(remeasured), props, cam) + + dest = run / station + if dest.exists(): + assert dest.resolve().parent == run.resolve(), dest # only runs// is ever removed + shutil.rmtree(dest) + write_station(adv, camera, props, dest, station) + for sub in LINKED: + if (src / sub).exists(): + os.symlink(os.path.relpath(src / sub, dest), dest / sub) + record["stations"][station] = { + "days": [days[0].stem[-8:], days[-1].stem[-8:]], "images": len(rows), "camera": cam, + "patterns": overrides or {}, "date": datetime.now(timezone.utc).isoformat(timespec="seconds")} + print(f"{station}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows -> {dest}") + record_path.write_text(yaml.safe_dump(record, sort_keys=False, allow_unicode=True)) + + +def check(): + """`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical.""" + import tempfile + + with tempfile.TemporaryDirectory() as tmp: + f = Path(tmp) / "c.yaml" + for text in ("camera: {algoritm: ransac}", "stations: {S: {pattern: {}}}", "stations: {S: {camera: {mod: x}}}", + "other: 1"): + f.write_text(text) + try: + load_config(f) + raise AssertionError(f"accepted: {text}") + except ValueError: + pass + f.write_text("camera: {mode: rotation, focal_distance: 6.1}\nstations: {S: {camera: {focal_distance: 6.3}}}") + c = load_config(f) + cam = camera_settings(c, "S", {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1}) + assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation", + "ransac"), cam + assert camera_settings({}, "S", {})["focal_distance"] == 6.0 + + pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)} + props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y, + "error_threshold": 0.7, "distance_convertion_factor": 2.0, "corners": [[0, 0], [10, 10]]} + for i, (n, (x, y)) in enumerate(pos.items())} + sensors = [{"id": int(i), "type": MTYPE, "properties": [{"name": k, "value": v} for k, v in {**q, "pattern": "b"} + .items()]} for i, q in props.items()] + [{"id": 9, "type": "camera_movement", "properties": []}] + assert set(cloud_properties(sensors)) == set(props) + for bad in ({"NOPE": {"error_threshold": 1}}, {"T1": {"error_treshold": 1}}): + try: + apply_patterns(props, bad, Path(".")) + raise AssertionError(f"accepted: {bad}") + except ValueError: + pass + assert apply_patterns(props, {"T1": {"error_threshold": 0.9}}, Path("."))["3"]["error_threshold"] == 0.9 + + t0, t1 = "2026-11-03T10:00:07.25+00:00", "2026-11-03T10:15:07.25+00:00" + rows = [] + for t, (dx, dy) in ((t0, (0.0, 0.0)), (t1, (1.0, 0.5))): # the whole camera shifts by (1, 0.5) px at t1 + for i, q in props.items(): + rows.append({"time_utc": t, "pattern_id": int(i), "x": q["x"] + dx, "y": q["y"] + dy, "peak": 1.0, + "error": 0.9 if (t == t1 and q["sensor_name"] == "T2") else 0.2}) + for per_epoch in (True, False): + adv, camera = correct(pd.DataFrame(rows), props, {**DEFAULTS, "per_epoch": per_epoch}) + assert list(adv.columns) == COLUMNS and len(adv) == 10, adv.columns + a = adv.set_index(["time_utc", "sensor_name"]) + t1x = pd.Timestamp(t1) + assert a.loc[(t1x, "T1"), ["corr_x", "corr_y"]].abs().max() < 1e-3, a # static target: no displacement + assert a.loc[(t1x, "T1"), "relative_x"] == 1.0 and a.loc[(t1x, "T1"), "corr_x_mm"] == \ + 2.0 * a.loc[(t1x, "T1"), "corr_x"] + assert np.isnan(a.loc[(t1x, "T2"), "corr_x"]) and a.loc[(t1x, "T2"), "x"] == 701.0 # rejected: raw x kept + assert a["sigma0"].notna().all() and a.loc[(t1x, "R1"), "inlier"] == a.loc[(t1x, "R1"), "inlier"], a + assert len(camera) == 2 and str(camera["time_utc"][1]) == "2026-11-03 10:15:00+00:00", camera + assert list(camera.columns)[:4] == ["time_utc", "median_pan", "min_pan", "max_pan"], camera.columns + print("ok") + + +if __name__ == "__main__": + check() if sys.argv[1:] == ["--check"] else main() diff --git a/processing/run_config.sh b/processing/run_config.sh new file mode 100755 index 0000000..3f57201 --- /dev/null +++ b/processing/run_config.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +# Recomputes a configuration into runs// (processing/run_config.py), in a throwaway uv environment with what +# optical's emulator and camera correction need; optical's code from $OPTICAL (default ~/optical). No credentials. +# processing/run_config.sh [SERIAL...] [--from D] [--to E] [--jobs N] +# processing/run_config.sh --check +set -u +export PATH="$HOME/.local/bin:$PATH" +here=$(cd "$(dirname "$0")" && pwd) +exec uv run --no-project --quiet --python 3.12 \ + --with numpy --with pandas --with pillow --with pyyaml --with scipy --with scikit-image \ + --with opencv-python-headless \ + python "$here/run_config.py" "$@"