From ca366c150e7204558ab8170c2d83edfbe20cd9d5 Mon Sep 17 00:00:00 2001 From: maxime Date: Thu, 19 Nov 2026 10:52:46 +0100 Subject: [PATCH] Runs in data_diff//runs//: new_run.sh creates one, run_config.sh computes it - The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/, weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv, patterns.json. - processing/new_run.sh : config.yaml of a new, empty run, the cloud's processing written in full (the patterns of sensors.json, the camera, optical's current commit); never over an existing one. - processing/run_config.sh : reads the run's config.yaml (patterns, camera, emulator.version, checked against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns) and patterns.json (the patterns as given to the emulator). No configs/, no runs// outside data_diff/. - pump_diff no longer writes targets.csv: it depends on the run. JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within 0.034 px; a run created by new_run.sh and run as is gives run01 exactly. Co-Authored-By: Claude Opus 5.5 --- README.md | 6 +- .../specs/2026-11-19-configurations-design.md | 18 ++ .../specs/2026-11-19-run-config-design.md | 10 +- extraction/README.md | 6 +- extraction/pump_diff.py | 29 +-- processing/new_run.sh | 6 + processing/run_config.py | 228 +++++++++++------- processing/run_config.sh | 5 +- 8 files changed, 185 insertions(+), 123 deletions(-) create mode 100755 processing/new_run.sh diff --git a/README.md b/README.md index 01f9712..4ee9425 100644 --- a/README.md +++ b/README.md @@ -3,10 +3,8 @@ The data of the AIMsight stations, in a single copy on the workstation, and their pumping. - `data//`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again. -- `data_diff//`: the differential images, full size, with their index and the patterns, to measure them again, and the metrics, IRON images, weather and `targets.csv` as in `data/` (`extraction/pump_diff.sh`), **out of git**; it never touches `data/`. Shared by every configuration. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`. -- `configs/.yaml`: a configuration, what differs from the cloud (patterns by `sensor_name`: a property, or a window redrawn from a DIFF image; the camera correction: algorithm, mode, focal distance, one measurement at a time or the series), in git; `cloud.yaml`, empty, is the cloud and reproduces `data/` (JUNGFRAU 2026-10-12..16: `corr_x/y` within 0.03 px, the same rejections, the camera pose within 5e-4 deg); `jungfrau_example.yaml`, a model. -- `runs///`: the result of a configuration, **out of git**, the tree of `data/` (`displacement/`, `camera/`, `targets.csv`; `images/`, `metrics/`, `weather/` linked to `data_diff/`): a repository reads it by pointing its `data` link at `~/aimsight-data/runs/` instead of `~/aimsight-data/data`. Beside the stations, `run.yaml` (the configuration as resolved, optical's commit) and `diagnostics/` (the patterns used, the emulator's rows: edge margin, second spot). -- `processing/run_config.sh [SERIAL...] [--from D] [--to E]`: computes `runs//` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 16 s for 5 days of JUNGFRAU); `--check` without data. Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`. A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new): cut it from an image at the start of the period studied. +- `data_diff//`, **out of git**: pumped by `extraction/pump_diff.sh` (never touching `data/`): the differential images full size under `images_diff/` (with their index, the patterns `sensors.json` and the unit `unit.json`), `images/` (IRON), `metrics/`, `weather/`, as in `data/`: only what is the same for every run. Its runs in `runs//`, with everything that depends on the run: `config.yaml` (the run's configuration), `displacement/`, `camera/` and `targets.csv` (its result, the files of `data/`), `patterns.json`. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`. +- `processing/run_config.sh [--from D] [--to E]`: runs `data_diff//runs//config.yaml` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 18 s for 6 days of JUNGFRAU) and writes the run's `displacement/`, `camera/`, `targets.csv` (from its patterns) and `patterns.json` (the patterns exactly as given to the emulator, templates included); `--check` without data. `config.yaml` states what differs from the cloud's processing (an empty file: the cloud's): `patterns` by `sensor_name` (a property, or a window redrawn from a DIFF image), `camera` (algorithm, ransac_threshold, mode, focal_distance, pixel_size, per_epoch) and `emulator.version` (optical's commit: the run stops if `$OPTICAL` is at another one or has local changes); its keys are checked. `processing/new_run.sh ` creates a run, empty: its `config.yaml` only, the cloud's processing written in full (every pattern of `sensors.json`, the camera, optical's current commit), to edit before running it; it never overwrites an existing `config.yaml`. The standard configuration reproduces `data/` (JUNGFRAU, 6 days: `corr_x/y` within 0.034 px, the same rejections). A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new). Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`. - `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations). - `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data//weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`. - `.env`: the credentials of the database and of S3, at the root, never versioned. diff --git a/docs/superpowers/specs/2026-11-19-configurations-design.md b/docs/superpowers/specs/2026-11-19-configurations-design.md index 9216390..b1ef73a 100644 --- a/docs/superpowers/specs/2026-11-19-configurations-design.md +++ b/docs/superpowers/specs/2026-11-19-configurations-design.md @@ -3,6 +3,24 @@ 2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes: section "data_diff"). +**Replaced the same day by the user's own layout** (the layout below, `configs/` and `runs//`, is not used): + +``` +data_diff// + images_diff/ (index/, sensors.json, unit.json) images/ metrics/ weather/ pumped, the same for every run + runs// one run: whatever depends on it + config.yaml the run's configuration: patterns (by sensor_name), camera and emulator.version (optical's + commit, checked), over the cloud's processing + displacement/ camera/ targets.csv its result, the files of data/ (targets.csv from its patterns) + patterns.json the patterns exactly as given to the emulator, templates included +``` + +`processing/run_config.sh ` reads the run's `config.yaml` and writes its `displacement/`, `camera/`, +`targets.csv` and `patterns.json`, nothing else. Rule: what is not the same for every possible run is in the run's +folder. `processing/new_run.sh ` creates a run, empty: its `config.yaml`, the cloud's processing written +in full (patterns of `sensors.json`, camera, optical's current commit); never over an existing one. Note: the reading tools (aimsight-tools `data.find_files`) read a station folder whole, subfolders +included: pointed at `data_diff//`, they see every run's curves together. + ## Intent Recompute a station's data from its differential images under configurations of our own (patterns redrawn, diff --git a/docs/superpowers/specs/2026-11-19-run-config-design.md b/docs/superpowers/specs/2026-11-19-run-config-design.md index c08a369..9ef56fd 100644 --- a/docs/superpowers/specs/2026-11-19-run-config-design.md +++ b/docs/superpowers/specs/2026-11-19-run-config-design.md @@ -1,7 +1,15 @@ -# run_config: a configuration recomputed into runs// +# run_config: a run of a station recomputed from its differential images 2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`. +**Amended the same day (the user's layout, see the configurations design):** `processing/run_config.sh ` +reads `data_diff//runs//config.yaml` (`patterns`, `camera`, `emulator.version`: optical's commit, the run +stops if `$OPTICAL` is at another one or has local changes; no `stations` level) and writes only that run's +`displacement/`, `camera/`, `targets.csv` (from its patterns; none at the station's root any more) and `patterns.json` +(the patterns as given to the emulator); the emulator's other files go through a temporary folder. No `runs//`, +no links, no `run.yaml`. The rest below holds (patterns, correction as +the cloud, columns, acceptance). + ## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`) Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw diff --git a/extraction/README.md b/extraction/README.md index bea53e7..6c99d61 100644 --- a/extraction/README.md +++ b/extraction/README.md @@ -58,7 +58,7 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small ## The differential images, into `data_diff/` (`pump_diff.sh`) -`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff//` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the configurations, `runs//`, read like `data/`). +`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff//` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the runs, `data_diff//runs//`). - `images_diff/TZ_.png`: the S3 object as is; - `images_diff/index/_diff_.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON); @@ -66,7 +66,7 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small - `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm); - (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder) - `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`; -- `targets.csv`: as in `data/`, for the patterns the DIFF images are measured with (not archived); `displacement/` and `camera/` come only from the recomputation of a configuration (`runs//`), never from the cloud. +- `displacement/`, `camera/` and `targets.csv` are not pumped: they depend on the run (its patterns, its settings), each run writes its own in `runs//` (`processing/run_config.sh`), never from the cloud; a pumping never touches `runs/`. ```bash extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here @@ -74,6 +74,6 @@ extraction/pump_diff.sh JUNGFRAU # from its extraction/pump_diff.sh # the same for every station of data_diff/ ``` -Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`; `targets.csv` the same without the two non-DIFF sensors `target_01`, `target_02`. +Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`. Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not. diff --git a/extraction/pump_diff.py b/extraction/pump_diff.py index fe4a881..392d570 100644 --- a/extraction/pump_diff.py +++ b/extraction/pump_diff.py @@ -18,8 +18,9 @@ read it the same way (docs/superpowers/specs/2026-11-19-configurations-design.md time_utc, status}] (every property row as stored), rewritten at each run /images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties (under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder) - /metrics/, images/ (IRON, one per --every minutes), targets.csv (the DIFF patterns), weather/: as pump.py - and weather.py write them into data/, with their own functions + /metrics/, images/ (IRON, one per --every minutes), weather/: as pump.py and weather.py write them into + data/, with their own functions. Not targets.csv: it depends on the run (its patterns), each run writes its own + (processing/run_config.py) Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index @@ -138,17 +139,6 @@ def unit_json(conn, unit): "properties": [prop(*r) for r in rows]} -def diff_patterns(sensors): - """{sensor_name: sensor_id} of the patterns the DIFF images are measured with (sensors_json rows): type - OPTICAL_SHIFT-DIFF, a pattern, not archived; for targets.csv.""" - out = {} - for s in sensors: - p = {r["name"]: r["value"] for r in s["properties"]} - if s["type"] == MTYPE and p.get("pattern") and not p.get("archive"): - out[p.get("sensor_name") or s["name"]] = s["id"] - return out - - def to_json(obj): return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode() @@ -172,7 +162,7 @@ def main(): from sqlalchemy import create_engine, text from sqlalchemy.engine import URL - from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause, sensors, write_targets + from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause tomorrow = datetime.now(UTC).date() + timedelta(days=1) start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None @@ -224,11 +214,8 @@ def main(): sens = sensors_json(conn, unit) write_atomic(folder / "images_diff" / "sensors.json", to_json(sens)) write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit))) - t = write_targets(folder, sensors(conn, unit), diff_patterns(sens)) - print(f"{station}: targets.csv, {len(t)} sensors") except Exception as e: # the next run writes them - print(f"{station} images range, sensors.json, unit.json, targets.csv: failed ({root_cause(e)}), " - "run again") + print(f"{station} images range, sensors.json, unit.json: failed ({root_cause(e)}), run again") finally: conn.rollback() print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}") @@ -313,12 +300,6 @@ def check(): assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}' assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index assert indexed_days(folder) == {d(3)} - row = lambda name, typ, **p: {"name": name, "type": typ, "id": len(p), # noqa: E731 - "properties": [{"name": k, "value": v} for k, v in p.items()]} - assert diff_patterns([row("u1", MTYPE, pattern="b64", sensor_name="T1"), row("u2", MTYPE, pattern="b64"), - row("u3", MTYPE, pattern="b64", sensor_name="OLD", archive="2026-01-01"), - row("u4", "camera_movement", sensor_name="CAM"), row("u5", MTYPE, sensor_name="NOPAT")]) \ - == {"T1": 2, "u2": 1}, "DIFF patterns: by sensor_name, else the sensor's name; archived and others left out" print("ok") diff --git a/processing/new_run.sh b/processing/new_run.sh new file mode 100755 index 0000000..82bac6f --- /dev/null +++ b/processing/new_run.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +# Creates a new, empty run: data_diff//runs//config.yaml with the cloud's processing written in full (the +# patterns of images_diff/sensors.json, the camera, optical's current commit); no result until it is run. +# processing/new_run.sh JUNGFRAU run02 +# processing/run_config.sh JUNGFRAU run02 then, once config.yaml is edited +exec "$(dirname "$0")/run_config.sh" "$@" --new diff --git a/processing/run_config.py b/processing/run_config.py index ba707e9..7909da0 100644 --- a/processing/run_config.py +++ b/processing/run_config.py @@ -1,35 +1,42 @@ -"""Recomputes a configuration from the differential images into runs//, in the tree of data/, so that the other -repositories read it as they read data/ (point their `data` link at runs/). Never writes outside runs//. -Design: docs/superpowers/specs/2026-11-19-run-config-design.md. +"""Runs one configuration of a station: data_diff//runs//config.yaml in, the same folder's displacement/, +camera/ and targets.csv out, in the files of data/. The station's pumped data, the same for every run (images_diff/, +images/, metrics/, weather/), stay at data_diff// and are only read; whatever depends on the run is in its folder. - processing/run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17 - processing/run_config.sh configs/mine.yaml the stations the configuration names, all their days + processing/run_config.sh JUNGFRAU run02 --from 2026-10-12 --to 2026-10-17 + processing/run_config.sh JUNGFRAU run02 every indexed day -: configs/.yaml (or a path); stations: those given, else those of the configuration; days: those indexed -in data_diff//images_diff/index/ (within --from / --to, --to excluded). For each station, runs/// -is replaced (the other stations of the run are kept): +Days: those indexed in data_diff//images_diff/index/ (within --from / --to, --to excluded). Running a run again +replaces its displacement/, camera/, targets.csv and patterns.json; nothing else is written: displacement/_advanced_.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator; relative = x - pattern x; corr from the camera correction; reproj_err, inlier of the references; sigma0 of the epoch; corr_*_mm camera/_camera_.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute - targets.csv the patterns of the configuration - images/ metrics/ weather/ relative symbolic links to data_diff// -and, outside the station folders (the reading tools do not see them): runs//run.yaml (the configuration as -resolved per station, optical's commit, the days) and runs//diagnostics/ (the patterns used, the emulator's rows). + targets.csv as in data/, from the run's patterns: pattern_type, distance_mm, mm_per_px + patterns.json the patterns exactly as given to the emulator (templates included) -A configuration (YAML) states what differs from the cloud: the patterns (by sensor_name: any of their properties, or -`redraw: {image: , corners: [[x0, y0], [x1, y1]]}`, cut as the device's pattern editor does) and -the camera correction (`algorithm`, `ransac_threshold`, `mode`: off | rotation | rotation_scale | relative, -`focal_distance`, `pixel_size`, `per_epoch`), common or per station; configs/cloud.yaml, empty, is the cloud. +config.yaml states what differs from the cloud's processing (an empty file: the cloud's), checked key by key: + patterns: by sensor_name, over images_diff/sensors.json: any existing property, or + PM_G: {error_threshold: 0.7} a template redrawn from a DIFF image of images_diff/, as the + BL_C: {redraw: {image: , corners: [[x0, y0], [x1, y1]]}} device's pattern editor does (x_ref kept) + camera: over images_diff/unit.json, over the cloud's defaults + algorithm: ransac ransac | umeyama | danish_hybrid | trusted_danish + ransac_threshold: 0.99 + mode: "off" off | rotation | rotation_scale | relative (optical's refine_allpoints) + focal_distance: 6.0 pixel_size: 0.0024 + per_epoch: true one measurement at a time, as the cloud; false: the whole series at once + emulator: + version: "65a4c70" optical's commit: the run stops if $OPTICAL is at another one or has local changes +Design: docs/superpowers/specs/2026-11-19-run-config-design.md. """ import argparse import csv +import datetime import json import os import shutil import subprocess import sys -from datetime import datetime, timezone +import tempfile from pathlib import Path # Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and @@ -54,7 +61,7 @@ CAM = ["pan", "tilt", "scale", "rotation", "sigma0"] DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0, "pixel_size": 0.0024, "per_epoch": True} UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size") -LINKED = ("images", "metrics", "weather") +OUTPUTS = ("displacement", "camera") # what a run writes into its folder, and replaces def _keys(d, allowed, where): @@ -68,11 +75,10 @@ def _keys(d, allowed, where): def load_config(path): """The configuration, its keys checked (a typo is an error, not a setting silently ignored).""" c = yaml.safe_load(Path(path).read_text()) or {} - _keys(c, {"stations", "camera"}, "the configuration") - _keys(c.get("camera") or {}, DEFAULTS, "camera") - for s, block in (c.get("stations") or {}).items(): - _keys(block or {}, {"patterns", "camera"}, f"stations.{s}") - _keys((block or {}).get("camera") or {}, DEFAULTS, f"stations.{s}.camera") + _keys(c, {"patterns", "camera", "emulator"}, str(path)) + _keys(c.get("camera") or {}, DEFAULTS, f"{path}: camera") + _keys(c.get("emulator") or {}, {"version"}, f"{path}: emulator") + _keys(c.get("patterns") or {}, set(c.get("patterns") or {}), f"{path}: patterns") # a mapping (names: later) return c @@ -81,11 +87,10 @@ def latest(rows): return {r["name"]: r["value"] for r in rows} -def camera_settings(config, station, unit_props): +def camera_settings(config, unit_props): cam = dict(DEFAULTS) cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props}) cam.update(config.get("camera") or {}) - cam.update(((config.get("stations") or {}).get(station) or {}).get("camera") or {}) return cam @@ -214,15 +219,14 @@ def correct(meas, props, cam): return adv, agg.reset_index() -def write_station(adv, camera, props, dest, station): +def write_run(adv, camera, dest, station): + """displacement/ and camera/ of a run, one file per UTC day, replacing the run's previous ones.""" for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")): - (dest / folder).mkdir(parents=True, exist_ok=True) + if (dest / folder).exists(): + shutil.rmtree(dest / folder) + (dest / folder).mkdir() for day, d in df.groupby(df["time_utc"].dt.date): d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False) - t = pd.DataFrame([{"sensor_name": q.get("sensor_name"), "pattern_type": q.get("pattern_type"), - "distance_mm": q.get("distance_mm"), "mm_per_px": q.get("distance_convertion_factor")} - for q in props.values()]).sort_values("sensor_name") - t.to_csv(dest / "targets.csv", index=False) def optical_state(): @@ -231,52 +235,107 @@ def optical_state(): "dirty": bool(git("status", "--porcelain", "--untracked-files=no"))} +PATTERN_KEYS = ["pattern_type", "corners", "x", "y", "error_threshold", "resolution", "distance_convertion_factor"] +CAMERA_NOTES = {"algorithm": "ransac | umeyama | danish_hybrid | trusted_danish", "ransac_threshold": "px, ransac only", + "mode": "off | rotation | rotation_scale | relative", "focal_distance": "mm; pan, tilt, sigma0 only", + "pixel_size": "mm; pan, tilt, sigma0 only", + "per_epoch": "one measurement at a time, as the cloud; false: the whole series at once"} + + +def new_run(src, name): + """runs//config.yaml of a new, empty run (no result yet): the cloud's processing written in full, the patterns + of images_diff/sensors.json, the camera of unit.json or the cloud's defaults, optical's current commit.""" + run = src / "runs" / name + if (run / "config.yaml").exists(): + raise FileExistsError(f"{run / 'config.yaml'} exists: choose another name, or edit that file") + props = sorted(cloud_properties(json.loads((src / "images_diff" / "sensors.json").read_text())).values(), + key=lambda p: (p.get("pattern_type") != "global_ref", str(p.get("sensor_name")))) + unit = latest(json.loads((src / "images_diff" / "unit.json").read_text())["properties"]) + cam, state = camera_settings({}, unit), optical_state() + width = max(len(str(p.get("sensor_name"))) for p in props) + 1 + line = lambda p: f" {str(p.get('sensor_name')) + ':':{width}s} {{" + ", ".join( # noqa: E731 + f"{k}: {json.dumps(p[k])}" for k in PATTERN_KEYS if k in p) + "}" + refs = [line(p) for p in props if p.get("pattern_type") == "global_ref"] + targets = [line(p) for p in props if p.get("pattern_type") != "global_ref"] + text = f"""# {name}: created on {datetime.date.today()} with the cloud's processing. Change a value, then run it: +# processing/run_config.sh {src.name} {name} +# The run also writes here targets.csv (as in data/, from these patterns) and patterns.json (the patterns exactly as +# given to the emulator, templates included). +emulator: + version: "{state['commit'][:7]}" # optical's commit (branch {state['branch']}); the run stops if $OPTICAL is + # at another one or has local changes +patterns: + # The {len(props)} patterns of the DIFF images, as in images_diff/sensors.json on {datetime.date.today()} (frozen here: a + # later pumping does not change them). The template stays in sensors.json; to change it: + # : {{redraw: {{image: , corners: [[x0, y0], [x1, y1]]}}}} + # pattern_type: global_ref = a reference of the camera fit, anything else a target. corners and x, y move together + # (same window size as the template). resolution: 1/100 px. distance_convertion_factor: mm per px (corr_*_mm only). + # references +""" + "\n".join(refs) + "\n # targets\n" + "\n".join(targets) + "\ncamera:" + "".join( + f"\n {f'{k}: {json.dumps(v)}':26s} # {CAMERA_NOTES[k]}" for k, v in cam.items()) + ( + "\n # focal_distance, pixel_size: the unit has none on the platform, the cloud's values" + if not {"focal_distance", "pixel_size"} & set(unit) else "") + "\n" + run.mkdir(parents=True, exist_ok=True) + (run / "config.yaml").write_text(text) + return run / "config.yaml", state + + def main(): p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - p.add_argument("config", help="name of configs/.yaml, or a path") - p.add_argument("stations", nargs="*", help="serial numbers; none: those of the configuration") + p.add_argument("station", help="serial number: the folder data_diff//") + p.add_argument("run", help="the folder data_diff//runs//, holding config.yaml") p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD") p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD") p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)") p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff") - p.add_argument("--out", type=Path, default=ROOT / "runs") + p.add_argument("--new", action="store_true", help="create the run, empty: its config.yaml only (processing/new_run.sh)") a = p.parse_args() - path = Path(a.config) if a.config.endswith((".yaml", ".yml")) or os.sep in a.config \ - else ROOT / "configs" / f"{a.config}.yaml" - config, name = load_config(path), path.stem - stations = a.stations or list(config.get("stations") or {}) - if not stations: - sys.exit(f"No station: name one ({path} names none)") - run = a.out / name - (run / "diagnostics").mkdir(parents=True, exist_ok=True) - record_path = run / "run.yaml" - record = yaml.safe_load(record_path.read_text()) if record_path.exists() else {} - record.update({"configuration": name, "source": str(path), "optical": optical_state()}) - record.setdefault("stations", {}) + src = a.data_diff / a.station + run = src / "runs" / a.run + if a.new: + try: + path, state = new_run(src, a.run) + except FileExistsError as e: + sys.exit(str(e)) + print(f"{path}: written{', but optical has local changes: the run will stop until they are committed' if state['dirty'] else ''}" + f"\nRun it: processing/run_config.sh {a.station} {a.run}") + return + if not (run / "config.yaml").is_file(): + sys.exit(f"No {run / 'config.yaml'}: create the run first, processing/new_run.sh {a.station} {a.run}") + config = load_config(run / "config.yaml") + version = (config.get("emulator") or {}).get("version") + if version is not None: # the code that makes the run is the one config.yaml names + state = optical_state() + want = subprocess.run(["git", "-C", str(OPTICAL), "rev-parse", "--verify", "--quiet", f"{version}^{{commit}}"], + capture_output=True, text=True).stdout.strip() + if not want: + sys.exit(f"emulator.version {version}: not a commit of {OPTICAL}") + if want != state["commit"] or state["dirty"]: + sys.exit(f"emulator.version {version}, but {OPTICAL} is at {state['commit'][:7]}" + f"{' with local changes' if state['dirty'] else ''}: check {version} out, or change the version") + + days = sorted(f for f in (src / "images_diff" / "index").glob(f"{a.station}_diff_*.csv") + if (not a.start or f.stem[-8:] >= a.start.replace("-", "")) + and (not a.end or f.stem[-8:] < a.end.replace("-", ""))) + rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]] + if not rows: + sys.exit(f"{a.station}: no differential image indexed in the range, nothing done") + sensors = json.loads((src / "images_diff" / "sensors.json").read_text()) + unit = json.loads((src / "images_diff" / "unit.json").read_text()) + cam = camera_settings(config, latest(unit["properties"])) + props = apply_patterns(cloud_properties(sensors), config.get("patterns"), src / "images_diff") + print(f"{a.station} {a.run}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} " + f"patterns, camera {cam}, optical {optical_state()}") import reprocess_images - for station in stations: - src = a.data_diff / station - days = sorted(f for f in (src / "images_diff" / "index").glob(f"{station}_diff_*.csv") - if (not a.start or f.stem[-8:] >= a.start.replace("-", "")) - and (not a.end or f.stem[-8:] < a.end.replace("-", ""))) - rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]] - if not rows: - print(f"{station}: no differential image indexed in the range, nothing done") - continue - sensors = json.loads((src / "images_diff" / "sensors.json").read_text()) - unit = json.loads((src / "images_diff" / "unit.json").read_text()) - cam = camera_settings(config, station, latest(unit["properties"])) - overrides = ((config.get("stations") or {}).get(station) or {}).get("patterns") - props = apply_patterns(cloud_properties(sensors), overrides, src / "images_diff") - print(f"{station}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} patterns, " - f"camera {cam}") - - diag = run / "diagnostics" - sp, times, remeasured = (diag / f"{station}_sensor_properties.json", diag / f"{station}_times.csv", - diag / f"{station}_remeasured.csv") - sp.write_text(json.dumps(props)) + sp = run / "patterns.json" # kept: the target configuration the run was made with, templates included + sp.write_text(json.dumps(props, indent=1)) + pd.DataFrame([{"sensor_name": p.get("sensor_name"), "pattern_type": p.get("pattern_type"), # pump.py's columns + "distance_mm": p.get("distance_mm"), "mm_per_px": p.get("distance_convertion_factor")} + for p in props.values()]).sort_values("sensor_name").to_csv(run / "targets.csv", index=False) + with tempfile.TemporaryDirectory() as tmp: # the emulator's other files: not kept + times, remeasured = Path(tmp) / "times.csv", Path(tmp) / "rows.csv" with open(times, "w", newline="") as fp: w = csv.writer(fp) w.writerow(["file", "time_utc"]) @@ -284,42 +343,33 @@ def main(): reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured), "--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows]) adv, camera = correct(pd.read_csv(remeasured), props, cam) - - dest = run / station - if dest.exists(): - assert dest.resolve().parent == run.resolve(), dest # only runs// is ever removed - shutil.rmtree(dest) - write_station(adv, camera, props, dest, station) - for sub in LINKED: - if (src / sub).exists(): - os.symlink(os.path.relpath(src / sub, dest), dest / sub) - record["stations"][station] = { - "days": [days[0].stem[-8:], days[-1].stem[-8:]], "images": len(rows), "camera": cam, - "patterns": overrides or {}, "date": datetime.now(timezone.utc).isoformat(timespec="seconds")} - print(f"{station}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows -> {dest}") - record_path.write_text(yaml.safe_dump(record, sort_keys=False, allow_unicode=True)) + write_run(adv, camera, run, a.station) + print(f"{a.station} {a.run}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows " + f"-> {run}") def check(): """`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical.""" - import tempfile - with tempfile.TemporaryDirectory() as tmp: - f = Path(tmp) / "c.yaml" - for text in ("camera: {algoritm: ransac}", "stations: {S: {pattern: {}}}", "stations: {S: {camera: {mod: x}}}", - "other: 1"): + f = Path(tmp) / "config.yaml" + for text in ("camera: {algoritm: ransac}", "pattern: {}", "stations: {S: {}}", "patterns: [PM_G]", + "emulator: {versio: 65a4c70}"): f.write_text(text) try: load_config(f) raise AssertionError(f"accepted: {text}") except ValueError: pass - f.write_text("camera: {mode: rotation, focal_distance: 6.1}\nstations: {S: {camera: {focal_distance: 6.3}}}") + f.write_text("") + assert load_config(f) == {} # an empty file: the cloud's processing + f.write_text("camera: {mode: rotation, focal_distance: 6.3}\npatterns: {T1: {error_threshold: 0.9}}\n" + "emulator: {version: '65a4c70'}") c = load_config(f) - cam = camera_settings(c, "S", {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1}) + assert c["emulator"]["version"] == "65a4c70" + cam = camera_settings(c, {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1}) assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation", "ransac"), cam - assert camera_settings({}, "S", {})["focal_distance"] == 6.0 + assert camera_settings({}, {})["focal_distance"] == 6.0 pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)} props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y, diff --git a/processing/run_config.sh b/processing/run_config.sh index 3f57201..56eec72 100755 --- a/processing/run_config.sh +++ b/processing/run_config.sh @@ -1,7 +1,8 @@ #!/usr/bin/env bash -# Recomputes a configuration into runs// (processing/run_config.py), in a throwaway uv environment with what +# Runs data_diff//runs//config.yaml (processing/run_config.py), in a throwaway uv environment with what # optical's emulator and camera correction need; optical's code from $OPTICAL (default ~/optical). No credentials. -# processing/run_config.sh [SERIAL...] [--from D] [--to E] [--jobs N] +# processing/run_config.sh [--from D] [--to E] [--jobs N] +# processing/run_config.sh --new (processing/new_run.sh): the run created, empty # processing/run_config.sh --check set -u export PATH="$HOME/.local/bin:$PATH"