Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it

- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/,
  weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv,
  patterns.json.
- processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the
  patterns of sensors.json, the camera, optical's current commit); never over an existing one.
- processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked
  against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns)
  and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/.
- pump_diff no longer writes targets.csv: it depends on the run.

JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within
0.034 px; a run created by new_run.sh and run as is gives run01 exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-19 10:52:46 +01:00
1 parent d3bf4b6860
commit ca366c150e
8 files changed
+185 -123

No files matched your search

+2 -4
View File
@@ -3,10 +3,8 @@
The data of the AIMsight stations, in a single copy on the workstation, and their pumping. The data of the AIMsight stations, in a single copy on the workstation, and their pumping.
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again. - `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again.
- `data_diff/<SERIAL>/`: the differential images, full size, with their index and the patterns, to measure them again, and the metrics, IRON images, weather and `targets.csv` as in `data/` (`extraction/pump_diff.sh`), **out of git**; it never touches `data/`. Shared by every configuration. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`. - `data_diff/<SERIAL>/`, **out of git**: pumped by `extraction/pump_diff.sh` (never touching `data/`): the differential images full size under `images_diff/` (with their index, the patterns `sensors.json` and the unit `unit.json`), `images/` (IRON), `metrics/`, `weather/`, as in `data/`: only what is the same for every run. Its runs in `runs/<run>/`, with everything that depends on the run: `config.yaml` (the run's configuration), `displacement/`, `camera/` and `targets.csv` (its result, the files of `data/`), `patterns.json`. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`.
- `configs/<name>.yaml`: a configuration, what differs from the cloud (patterns by `sensor_name`: a property, or a window redrawn from a DIFF image; the camera correction: algorithm, mode, focal distance, one measurement at a time or the series), in git; `cloud.yaml`, empty, is the cloud and reproduces `data/` (JUNGFRAU 2026-10-12..16: `corr_x/y` within 0.03 px, the same rejections, the camera pose within 5e-4 deg); `jungfrau_example.yaml`, a model. - `processing/run_config.sh <SERIAL> <run> [--from D] [--to E]`: runs `data_diff/<SERIAL>/runs/<run>/config.yaml` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 18 s for 6 days of JUNGFRAU) and writes the run's `displacement/`, `camera/`, `targets.csv` (from its patterns) and `patterns.json` (the patterns exactly as given to the emulator, templates included); `--check` without data. `config.yaml` states what differs from the cloud's processing (an empty file: the cloud's): `patterns` by `sensor_name` (a property, or a window redrawn from a DIFF image), `camera` (algorithm, ransac_threshold, mode, focal_distance, pixel_size, per_epoch) and `emulator.version` (optical's commit: the run stops if `$OPTICAL` is at another one or has local changes); its keys are checked. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml` only, the cloud's processing written in full (every pattern of `sensors.json`, the camera, optical's current commit), to edit before running it; it never overwrites an existing `config.yaml`. The standard configuration reproduces `data/` (JUNGFRAU, 6 days: `corr_x/y` within 0.034 px, the same rejections). A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new). Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`.
- `runs/<name>/<SERIAL>/`: the result of a configuration, **out of git**, the tree of `data/` (`displacement/`, `camera/`, `targets.csv`; `images/`, `metrics/`, `weather/` linked to `data_diff/`): a repository reads it by pointing its `data` link at `~/aimsight-data/runs/<name>` instead of `~/aimsight-data/data`. Beside the stations, `run.yaml` (the configuration as resolved, optical's commit) and `diagnostics/` (the patterns used, the emulator's rows: edge margin, second spot).
- `processing/run_config.sh <config> [SERIAL...] [--from D] [--to E]`: computes `runs/<name>/` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 16 s for 5 days of JUNGFRAU); `--check` without data. Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`. A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new): cut it from an image at the start of the period studied.
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations). - `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations).
- `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data/<SERIAL>/weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`. - `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data/<SERIAL>/weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`.
- `.env`: the credentials of the database and of S3, at the root, never versioned. - `.env`: the credentials of the database and of S3, at the root, never versioned.
@@ -3,6 +3,24 @@
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes: 2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
section "data_diff"). section "data_diff").
**Replaced the same day by the user's own layout** (the layout below, `configs/` and `runs/<name>/`, is not used):
```
data_diff/<SERIAL>/
images_diff/ (index/, sensors.json, unit.json) images/ metrics/ weather/ pumped, the same for every run
runs/<run>/ one run: whatever depends on it
config.yaml the run's configuration: patterns (by sensor_name), camera and emulator.version (optical's
commit, checked), over the cloud's processing
displacement/ camera/ targets.csv its result, the files of data/ (targets.csv from its patterns)
patterns.json the patterns exactly as given to the emulator, templates included
```
`processing/run_config.sh <SERIAL> <run>` reads the run's `config.yaml` and writes its `displacement/`, `camera/`,
`targets.csv` and `patterns.json`, nothing else. Rule: what is not the same for every possible run is in the run's
folder. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml`, the cloud's processing written
in full (patterns of `sensors.json`, camera, optical's current commit); never over an existing one. Note: the reading tools (aimsight-tools `data.find_files`) read a station folder whole, subfolders
included: pointed at `data_diff/<SERIAL>/`, they see every run's curves together.
## Intent ## Intent
Recompute a station's data from its differential images under configurations of our own (patterns redrawn, Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
@@ -1,7 +1,15 @@
# run_config: a configuration recomputed into runs/<name>/ # run_config: a run of a station recomputed from its differential images
2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`. 2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`.
**Amended the same day (the user's layout, see the configurations design):** `processing/run_config.sh <SERIAL> <run>`
reads `data_diff/<SERIAL>/runs/<run>/config.yaml` (`patterns`, `camera`, `emulator.version`: optical's commit, the run
stops if `$OPTICAL` is at another one or has local changes; no `stations` level) and writes only that run's
`displacement/`, `camera/`, `targets.csv` (from its patterns; none at the station's root any more) and `patterns.json`
(the patterns as given to the emulator); the emulator's other files go through a temporary folder. No `runs/<name>/`,
no links, no `run.yaml`. The rest below holds (patterns, correction as
the cloud, columns, acceptance).
## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`) ## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`)
Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw
+3 -3
View File
@@ -58,7 +58,7 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small
## The differential images, into `data_diff/` (`pump_diff.sh`) ## The differential images, into `data_diff/` (`pump_diff.sh`)
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the configurations, `runs/<name>/`, read like `data/`). `pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the runs, `data_diff/<SERIAL>/runs/<run>/`).
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is; - `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is;
- `images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON); - `images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON);
@@ -66,7 +66,7 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small
- `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm); - `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm);
- (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder) - (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder)
- `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`; - `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`;
- `targets.csv`: as in `data/`, for the patterns the DIFF images are measured with (not archived); `displacement/` and `camera/` come only from the recomputation of a configuration (`runs/<name>/`), never from the cloud. - `displacement/`, `camera/` and `targets.csv` are not pumped: they depend on the run (its patterns, its settings), each run writes its own in `runs/<run>/` (`processing/run_config.sh`), never from the cloud; a pumping never touches `runs/`.
```bash ```bash
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here
@@ -74,6 +74,6 @@ extraction/pump_diff.sh JUNGFRAU # from its
extraction/pump_diff.sh # the same for every station of data_diff/ extraction/pump_diff.sh # the same for every station of data_diff/
``` ```
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`; `targets.csv` the same without the two non-DIFF sensors `target_01`, `target_02`. Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`.
Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not. Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.
+5 -24
View File
@@ -18,8 +18,9 @@ read it the same way (docs/superpowers/specs/2026-11-19-configurations-design.md
time_utc, status}] (every property row as stored), rewritten at each run time_utc, status}] (every property row as stored), rewritten at each run
<STATION>/images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties <STATION>/images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties
(under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder) (under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder)
<STATION>/metrics/, images/ (IRON, one per --every minutes), targets.csv (the DIFF patterns), weather/: as pump.py <STATION>/metrics/, images/ (IRON, one per --every minutes), weather/: as pump.py and weather.py write them into
and weather.py write them into data/, with their own functions data/, with their own functions. Not targets.csv: it depends on the run (its patterns), each run writes its own
(processing/run_config.py)
Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and
the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index
@@ -138,17 +139,6 @@ def unit_json(conn, unit):
"properties": [prop(*r) for r in rows]} "properties": [prop(*r) for r in rows]}
def diff_patterns(sensors):
"""{sensor_name: sensor_id} of the patterns the DIFF images are measured with (sensors_json rows): type
OPTICAL_SHIFT-DIFF, a pattern, not archived; for targets.csv."""
out = {}
for s in sensors:
p = {r["name"]: r["value"] for r in s["properties"]}
if s["type"] == MTYPE and p.get("pattern") and not p.get("archive"):
out[p.get("sensor_name") or s["name"]] = s["id"]
return out
def to_json(obj): def to_json(obj):
return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode() return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode()
@@ -172,7 +162,7 @@ def main():
from sqlalchemy import create_engine, text from sqlalchemy import create_engine, text
from sqlalchemy.engine import URL from sqlalchemy.engine import URL
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause, sensors, write_targets from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause
tomorrow = datetime.now(UTC).date() + timedelta(days=1) tomorrow = datetime.now(UTC).date() + timedelta(days=1)
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
@@ -224,11 +214,8 @@ def main():
sens = sensors_json(conn, unit) sens = sensors_json(conn, unit)
write_atomic(folder / "images_diff" / "sensors.json", to_json(sens)) write_atomic(folder / "images_diff" / "sensors.json", to_json(sens))
write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit))) write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit)))
t = write_targets(folder, sensors(conn, unit), diff_patterns(sens))
print(f"{station}: targets.csv, {len(t)} sensors")
except Exception as e: # the next run writes them except Exception as e: # the next run writes them
print(f"{station} images range, sensors.json, unit.json, targets.csv: failed ({root_cause(e)}), " print(f"{station} images range, sensors.json, unit.json: failed ({root_cause(e)}), run again")
"run again")
finally: finally:
conn.rollback() conn.rollback()
print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}") print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}")
@@ -313,12 +300,6 @@ def check():
assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}' assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}'
assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index
assert indexed_days(folder) == {d(3)} assert indexed_days(folder) == {d(3)}
row = lambda name, typ, **p: {"name": name, "type": typ, "id": len(p), # noqa: E731
"properties": [{"name": k, "value": v} for k, v in p.items()]}
assert diff_patterns([row("u1", MTYPE, pattern="b64", sensor_name="T1"), row("u2", MTYPE, pattern="b64"),
row("u3", MTYPE, pattern="b64", sensor_name="OLD", archive="2026-01-01"),
row("u4", "camera_movement", sensor_name="CAM"), row("u5", MTYPE, sensor_name="NOPAT")]) \
== {"T1": 2, "u2": 1}, "DIFF patterns: by sensor_name, else the sensor's name; archived and others left out"
print("ok") print("ok")
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Creates a new, empty run: data_diff/<SERIAL>/runs/<run>/config.yaml with the cloud's processing written in full (the
# patterns of images_diff/sensors.json, the camera, optical's current commit); no result until it is run.
# processing/new_run.sh JUNGFRAU run02
# processing/run_config.sh JUNGFRAU run02 then, once config.yaml is edited
exec "$(dirname "$0")/run_config.sh" "$@" --new
+139 -89
View File
@@ -1,35 +1,42 @@
"""Recomputes a configuration from the differential images into runs/<name>/, in the tree of data/, so that the other """Runs one configuration of a station: data_diff/<SERIAL>/runs/<run>/config.yaml in, the same folder's displacement/,
repositories read it as they read data/ (point their `data` link at runs/<name>). Never writes outside runs/<name>/. camera/ and targets.csv out, in the files of data/. The station's pumped data, the same for every run (images_diff/,
Design: docs/superpowers/specs/2026-11-19-run-config-design.md. images/, metrics/, weather/), stay at data_diff/<SERIAL>/ and are only read; whatever depends on the run is in its folder.
processing/run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17 processing/run_config.sh JUNGFRAU run02 --from 2026-10-12 --to 2026-10-17
processing/run_config.sh configs/mine.yaml the stations the configuration names, all their days processing/run_config.sh JUNGFRAU run02 every indexed day
<config>: configs/<name>.yaml (or a path); stations: those given, else those of the configuration; days: those indexed Days: those indexed in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). Running a run again
in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). For each station, runs/<name>/<SERIAL>/ replaces its displacement/, camera/, targets.csv and patterns.json; nothing else is written:
is replaced (the other stations of the run are kept):
displacement/<S>_advanced_<YYYYMMDD>.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator; displacement/<S>_advanced_<YYYYMMDD>.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator;
relative = x - pattern x; corr from the camera correction; reproj_err, relative = x - pattern x; corr from the camera correction; reproj_err,
inlier of the references; sigma0 of the epoch; corr_*_mm inlier of the references; sigma0 of the epoch; corr_*_mm
camera/<S>_camera_<YYYYMMDD>.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute camera/<S>_camera_<YYYYMMDD>.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute
targets.csv the patterns of the configuration targets.csv as in data/, from the run's patterns: pattern_type, distance_mm, mm_per_px
images/ metrics/ weather/ relative symbolic links to data_diff/<SERIAL>/ patterns.json the patterns exactly as given to the emulator (templates included)
and, outside the station folders (the reading tools do not see them): runs/<name>/run.yaml (the configuration as
resolved per station, optical's commit, the days) and runs/<name>/diagnostics/ (the patterns used, the emulator's rows).
A configuration (YAML) states what differs from the cloud: the patterns (by sensor_name: any of their properties, or config.yaml states what differs from the cloud's processing (an empty file: the cloud's), checked key by key:
`redraw: {image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}`, cut as the device's pattern editor does) and patterns: by sensor_name, over images_diff/sensors.json: any existing property, or
the camera correction (`algorithm`, `ransac_threshold`, `mode`: off | rotation | rotation_scale | relative, PM_G: {error_threshold: 0.7} a template redrawn from a DIFF image of images_diff/, as the
`focal_distance`, `pixel_size`, `per_epoch`), common or per station; configs/cloud.yaml, empty, is the cloud. BL_C: {redraw: {image: <file>, corners: [[x0, y0], [x1, y1]]}} device's pattern editor does (x_ref kept)
camera: over images_diff/unit.json, over the cloud's defaults
algorithm: ransac ransac | umeyama | danish_hybrid | trusted_danish
ransac_threshold: 0.99
mode: "off" off | rotation | rotation_scale | relative (optical's refine_allpoints)
focal_distance: 6.0 pixel_size: 0.0024
per_epoch: true one measurement at a time, as the cloud; false: the whole series at once
emulator:
version: "65a4c70" optical's commit: the run stops if $OPTICAL is at another one or has local changes
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
""" """
import argparse import argparse
import csv import csv
import datetime
import json import json
import os import os
import shutil import shutil
import subprocess import subprocess
import sys import sys
from datetime import datetime, timezone import tempfile
from pathlib import Path from pathlib import Path
# Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and # Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and
@@ -54,7 +61,7 @@ CAM = ["pan", "tilt", "scale", "rotation", "sigma0"]
DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0, DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0,
"pixel_size": 0.0024, "per_epoch": True} "pixel_size": 0.0024, "per_epoch": True}
UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size") UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size")
LINKED = ("images", "metrics", "weather") OUTPUTS = ("displacement", "camera") # what a run writes into its folder, and replaces
def _keys(d, allowed, where): def _keys(d, allowed, where):
@@ -68,11 +75,10 @@ def _keys(d, allowed, where):
def load_config(path): def load_config(path):
"""The configuration, its keys checked (a typo is an error, not a setting silently ignored).""" """The configuration, its keys checked (a typo is an error, not a setting silently ignored)."""
c = yaml.safe_load(Path(path).read_text()) or {} c = yaml.safe_load(Path(path).read_text()) or {}
_keys(c, {"stations", "camera"}, "the configuration") _keys(c, {"patterns", "camera", "emulator"}, str(path))
_keys(c.get("camera") or {}, DEFAULTS, "camera") _keys(c.get("camera") or {}, DEFAULTS, f"{path}: camera")
for s, block in (c.get("stations") or {}).items(): _keys(c.get("emulator") or {}, {"version"}, f"{path}: emulator")
_keys(block or {}, {"patterns", "camera"}, f"stations.{s}") _keys(c.get("patterns") or {}, set(c.get("patterns") or {}), f"{path}: patterns") # a mapping (names: later)
_keys((block or {}).get("camera") or {}, DEFAULTS, f"stations.{s}.camera")
return c return c
@@ -81,11 +87,10 @@ def latest(rows):
return {r["name"]: r["value"] for r in rows} return {r["name"]: r["value"] for r in rows}
def camera_settings(config, station, unit_props): def camera_settings(config, unit_props):
cam = dict(DEFAULTS) cam = dict(DEFAULTS)
cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props}) cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props})
cam.update(config.get("camera") or {}) cam.update(config.get("camera") or {})
cam.update(((config.get("stations") or {}).get(station) or {}).get("camera") or {})
return cam return cam
@@ -214,15 +219,14 @@ def correct(meas, props, cam):
return adv, agg.reset_index() return adv, agg.reset_index()
def write_station(adv, camera, props, dest, station): def write_run(adv, camera, dest, station):
"""displacement/ and camera/ of a run, one file per UTC day, replacing the run's previous ones."""
for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")): for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")):
(dest / folder).mkdir(parents=True, exist_ok=True) if (dest / folder).exists():
shutil.rmtree(dest / folder)
(dest / folder).mkdir()
for day, d in df.groupby(df["time_utc"].dt.date): for day, d in df.groupby(df["time_utc"].dt.date):
d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False) d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False)
t = pd.DataFrame([{"sensor_name": q.get("sensor_name"), "pattern_type": q.get("pattern_type"),
"distance_mm": q.get("distance_mm"), "mm_per_px": q.get("distance_convertion_factor")}
for q in props.values()]).sort_values("sensor_name")
t.to_csv(dest / "targets.csv", index=False)
def optical_state(): def optical_state():
@@ -231,52 +235,107 @@ def optical_state():
"dirty": bool(git("status", "--porcelain", "--untracked-files=no"))} "dirty": bool(git("status", "--porcelain", "--untracked-files=no"))}
PATTERN_KEYS = ["pattern_type", "corners", "x", "y", "error_threshold", "resolution", "distance_convertion_factor"]
CAMERA_NOTES = {"algorithm": "ransac | umeyama | danish_hybrid | trusted_danish", "ransac_threshold": "px, ransac only",
"mode": "off | rotation | rotation_scale | relative", "focal_distance": "mm; pan, tilt, sigma0 only",
"pixel_size": "mm; pan, tilt, sigma0 only",
"per_epoch": "one measurement at a time, as the cloud; false: the whole series at once"}
def new_run(src, name):
"""runs/<name>/config.yaml of a new, empty run (no result yet): the cloud's processing written in full, the patterns
of images_diff/sensors.json, the camera of unit.json or the cloud's defaults, optical's current commit."""
run = src / "runs" / name
if (run / "config.yaml").exists():
raise FileExistsError(f"{run / 'config.yaml'} exists: choose another name, or edit that file")
props = sorted(cloud_properties(json.loads((src / "images_diff" / "sensors.json").read_text())).values(),
key=lambda p: (p.get("pattern_type") != "global_ref", str(p.get("sensor_name"))))
unit = latest(json.loads((src / "images_diff" / "unit.json").read_text())["properties"])
cam, state = camera_settings({}, unit), optical_state()
width = max(len(str(p.get("sensor_name"))) for p in props) + 1
line = lambda p: f" {str(p.get('sensor_name')) + ':':{width}s} {{" + ", ".join( # noqa: E731
f"{k}: {json.dumps(p[k])}" for k in PATTERN_KEYS if k in p) + "}"
refs = [line(p) for p in props if p.get("pattern_type") == "global_ref"]
targets = [line(p) for p in props if p.get("pattern_type") != "global_ref"]
text = f"""# {name}: created on {datetime.date.today()} with the cloud's processing. Change a value, then run it:
# processing/run_config.sh {src.name} {name}
# The run also writes here targets.csv (as in data/, from these patterns) and patterns.json (the patterns exactly as
# given to the emulator, templates included).
emulator:
version: "{state['commit'][:7]}" # optical's commit (branch {state['branch']}); the run stops if $OPTICAL is
# at another one or has local changes
patterns:
# The {len(props)} patterns of the DIFF images, as in images_diff/sensors.json on {datetime.date.today()} (frozen here: a
# later pumping does not change them). The template stays in sensors.json; to change it:
# <name>: {{redraw: {{image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}}}}
# pattern_type: global_ref = a reference of the camera fit, anything else a target. corners and x, y move together
# (same window size as the template). resolution: 1/100 px. distance_convertion_factor: mm per px (corr_*_mm only).
# references
""" + "\n".join(refs) + "\n # targets\n" + "\n".join(targets) + "\ncamera:" + "".join(
f"\n {f'{k}: {json.dumps(v)}':26s} # {CAMERA_NOTES[k]}" for k, v in cam.items()) + (
"\n # focal_distance, pixel_size: the unit has none on the platform, the cloud's values"
if not {"focal_distance", "pixel_size"} & set(unit) else "") + "\n"
run.mkdir(parents=True, exist_ok=True)
(run / "config.yaml").write_text(text)
return run / "config.yaml", state
def main(): def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("config", help="name of configs/<name>.yaml, or a path") p.add_argument("station", help="serial number: the folder data_diff/<SERIAL>/")
p.add_argument("stations", nargs="*", help="serial numbers; none: those of the configuration") p.add_argument("run", help="the folder data_diff/<SERIAL>/runs/<run>/, holding config.yaml")
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD") p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD")
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD") p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD")
p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)") p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)")
p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff") p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff")
p.add_argument("--out", type=Path, default=ROOT / "runs") p.add_argument("--new", action="store_true", help="create the run, empty: its config.yaml only (processing/new_run.sh)")
a = p.parse_args() a = p.parse_args()
path = Path(a.config) if a.config.endswith((".yaml", ".yml")) or os.sep in a.config \ src = a.data_diff / a.station
else ROOT / "configs" / f"{a.config}.yaml" run = src / "runs" / a.run
config, name = load_config(path), path.stem if a.new:
stations = a.stations or list(config.get("stations") or {}) try:
if not stations: path, state = new_run(src, a.run)
sys.exit(f"No station: name one ({path} names none)") except FileExistsError as e:
run = a.out / name sys.exit(str(e))
(run / "diagnostics").mkdir(parents=True, exist_ok=True) print(f"{path}: written{', but optical has local changes: the run will stop until they are committed' if state['dirty'] else ''}"
record_path = run / "run.yaml" f"\nRun it: processing/run_config.sh {a.station} {a.run}")
record = yaml.safe_load(record_path.read_text()) if record_path.exists() else {} return
record.update({"configuration": name, "source": str(path), "optical": optical_state()}) if not (run / "config.yaml").is_file():
record.setdefault("stations", {}) sys.exit(f"No {run / 'config.yaml'}: create the run first, processing/new_run.sh {a.station} {a.run}")
config = load_config(run / "config.yaml")
version = (config.get("emulator") or {}).get("version")
if version is not None: # the code that makes the run is the one config.yaml names
state = optical_state()
want = subprocess.run(["git", "-C", str(OPTICAL), "rev-parse", "--verify", "--quiet", f"{version}^{{commit}}"],
capture_output=True, text=True).stdout.strip()
if not want:
sys.exit(f"emulator.version {version}: not a commit of {OPTICAL}")
if want != state["commit"] or state["dirty"]:
sys.exit(f"emulator.version {version}, but {OPTICAL} is at {state['commit'][:7]}"
f"{' with local changes' if state['dirty'] else ''}: check {version} out, or change the version")
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{a.station}_diff_*.csv")
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
if not rows:
sys.exit(f"{a.station}: no differential image indexed in the range, nothing done")
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
unit = json.loads((src / "images_diff" / "unit.json").read_text())
cam = camera_settings(config, latest(unit["properties"]))
props = apply_patterns(cloud_properties(sensors), config.get("patterns"), src / "images_diff")
print(f"{a.station} {a.run}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} "
f"patterns, camera {cam}, optical {optical_state()}")
import reprocess_images import reprocess_images
for station in stations: sp = run / "patterns.json" # kept: the target configuration the run was made with, templates included
src = a.data_diff / station sp.write_text(json.dumps(props, indent=1))
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{station}_diff_*.csv") pd.DataFrame([{"sensor_name": p.get("sensor_name"), "pattern_type": p.get("pattern_type"), # pump.py's columns
if (not a.start or f.stem[-8:] >= a.start.replace("-", "")) "distance_mm": p.get("distance_mm"), "mm_per_px": p.get("distance_convertion_factor")}
and (not a.end or f.stem[-8:] < a.end.replace("-", ""))) for p in props.values()]).sort_values("sensor_name").to_csv(run / "targets.csv", index=False)
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]] with tempfile.TemporaryDirectory() as tmp: # the emulator's other files: not kept
if not rows: times, remeasured = Path(tmp) / "times.csv", Path(tmp) / "rows.csv"
print(f"{station}: no differential image indexed in the range, nothing done")
continue
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
unit = json.loads((src / "images_diff" / "unit.json").read_text())
cam = camera_settings(config, station, latest(unit["properties"]))
overrides = ((config.get("stations") or {}).get(station) or {}).get("patterns")
props = apply_patterns(cloud_properties(sensors), overrides, src / "images_diff")
print(f"{station}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} patterns, "
f"camera {cam}")
diag = run / "diagnostics"
sp, times, remeasured = (diag / f"{station}_sensor_properties.json", diag / f"{station}_times.csv",
diag / f"{station}_remeasured.csv")
sp.write_text(json.dumps(props))
with open(times, "w", newline="") as fp: with open(times, "w", newline="") as fp:
w = csv.writer(fp) w = csv.writer(fp)
w.writerow(["file", "time_utc"]) w.writerow(["file", "time_utc"])
@@ -284,42 +343,33 @@ def main():
reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured), reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured),
"--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows]) "--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows])
adv, camera = correct(pd.read_csv(remeasured), props, cam) adv, camera = correct(pd.read_csv(remeasured), props, cam)
write_run(adv, camera, run, a.station)
dest = run / station print(f"{a.station} {a.run}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows "
if dest.exists(): f"-> {run}")
assert dest.resolve().parent == run.resolve(), dest # only runs/<name>/<SERIAL> is ever removed
shutil.rmtree(dest)
write_station(adv, camera, props, dest, station)
for sub in LINKED:
if (src / sub).exists():
os.symlink(os.path.relpath(src / sub, dest), dest / sub)
record["stations"][station] = {
"days": [days[0].stem[-8:], days[-1].stem[-8:]], "images": len(rows), "camera": cam,
"patterns": overrides or {}, "date": datetime.now(timezone.utc).isoformat(timespec="seconds")}
print(f"{station}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows -> {dest}")
record_path.write_text(yaml.safe_dump(record, sort_keys=False, allow_unicode=True))
def check(): def check():
"""`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical.""" """`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical."""
import tempfile
with tempfile.TemporaryDirectory() as tmp: with tempfile.TemporaryDirectory() as tmp:
f = Path(tmp) / "c.yaml" f = Path(tmp) / "config.yaml"
for text in ("camera: {algoritm: ransac}", "stations: {S: {pattern: {}}}", "stations: {S: {camera: {mod: x}}}", for text in ("camera: {algoritm: ransac}", "pattern: {}", "stations: {S: {}}", "patterns: [PM_G]",
"other: 1"): "emulator: {versio: 65a4c70}"):
f.write_text(text) f.write_text(text)
try: try:
load_config(f) load_config(f)
raise AssertionError(f"accepted: {text}") raise AssertionError(f"accepted: {text}")
except ValueError: except ValueError:
pass pass
f.write_text("camera: {mode: rotation, focal_distance: 6.1}\nstations: {S: {camera: {focal_distance: 6.3}}}") f.write_text("")
assert load_config(f) == {} # an empty file: the cloud's processing
f.write_text("camera: {mode: rotation, focal_distance: 6.3}\npatterns: {T1: {error_threshold: 0.9}}\n"
"emulator: {version: '65a4c70'}")
c = load_config(f) c = load_config(f)
cam = camera_settings(c, "S", {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1}) assert c["emulator"]["version"] == "65a4c70"
cam = camera_settings(c, {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation", assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation",
"ransac"), cam "ransac"), cam
assert camera_settings({}, "S", {})["focal_distance"] == 6.0 assert camera_settings({}, {})["focal_distance"] == 6.0
pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)} pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)}
props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y, props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y,
+3 -2
View File
@@ -1,7 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Recomputes a configuration into runs/<name>/ (processing/run_config.py), in a throwaway uv environment with what # Runs data_diff/<SERIAL>/runs/<run>/config.yaml (processing/run_config.py), in a throwaway uv environment with what
# optical's emulator and camera correction need; optical's code from $OPTICAL (default ~/optical). No credentials. # optical's emulator and camera correction need; optical's code from $OPTICAL (default ~/optical). No credentials.
# processing/run_config.sh <config> [SERIAL...] [--from D] [--to E] [--jobs N] # processing/run_config.sh <SERIAL> <run> [--from D] [--to E] [--jobs N]
# processing/run_config.sh <SERIAL> <run> --new (processing/new_run.sh): the run created, empty
# processing/run_config.sh --check # processing/run_config.sh --check
set -u set -u
export PATH="$HOME/.local/bin:$PATH" export PATH="$HOME/.local/bin:$PATH"