Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it

- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/,
  weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv,
  patterns.json.
- processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the
  patterns of sensors.json, the camera, optical's current commit); never over an existing one.
- processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked
  against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns)
  and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/.
- pump_diff no longer writes targets.csv: it depends on the run.

JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within
0.034 px; a run created by new_run.sh and run as is gives run01 exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-19 10:52:46 +01:00
1 parent d3bf4b6860
commit ca366c150e
8 files changed
+185 -123

No files matched your search

+2 -4
View File
@@ -3,10 +3,8 @@
The data of the AIMsight stations, in a single copy on the workstation, and their pumping.
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again.
- `data_diff/<SERIAL>/`: the differential images, full size, with their index and the patterns, to measure them again, and the metrics, IRON images, weather and `targets.csv` as in `data/` (`extraction/pump_diff.sh`), **out of git**; it never touches `data/`. Shared by every configuration. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`.
- `configs/<name>.yaml`: a configuration, what differs from the cloud (patterns by `sensor_name`: a property, or a window redrawn from a DIFF image; the camera correction: algorithm, mode, focal distance, one measurement at a time or the series), in git; `cloud.yaml`, empty, is the cloud and reproduces `data/` (JUNGFRAU 2026-10-12..16: `corr_x/y` within 0.03 px, the same rejections, the camera pose within 5e-4 deg); `jungfrau_example.yaml`, a model.
- `runs/<name>/<SERIAL>/`: the result of a configuration, **out of git**, the tree of `data/` (`displacement/`, `camera/`, `targets.csv`; `images/`, `metrics/`, `weather/` linked to `data_diff/`): a repository reads it by pointing its `data` link at `~/aimsight-data/runs/<name>` instead of `~/aimsight-data/data`. Beside the stations, `run.yaml` (the configuration as resolved, optical's commit) and `diagnostics/` (the patterns used, the emulator's rows: edge margin, second spot).
- `processing/run_config.sh <config> [SERIAL...] [--from D] [--to E]`: computes `runs/<name>/` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 16 s for 5 days of JUNGFRAU); `--check` without data. Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`. A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new): cut it from an image at the start of the period studied.
- `data_diff/<SERIAL>/`, **out of git**: pumped by `extraction/pump_diff.sh` (never touching `data/`): the differential images full size under `images_diff/` (with their index, the patterns `sensors.json` and the unit `unit.json`), `images/` (IRON), `metrics/`, `weather/`, as in `data/`: only what is the same for every run. Its runs in `runs/<run>/`, with everything that depends on the run: `config.yaml` (the run's configuration), `displacement/`, `camera/` and `targets.csv` (its result, the files of `data/`), `patterns.json`. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`.
- `processing/run_config.sh <SERIAL> <run> [--from D] [--to E]`: runs `data_diff/<SERIAL>/runs/<run>/config.yaml` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 18 s for 6 days of JUNGFRAU) and writes the run's `displacement/`, `camera/`, `targets.csv` (from its patterns) and `patterns.json` (the patterns exactly as given to the emulator, templates included); `--check` without data. `config.yaml` states what differs from the cloud's processing (an empty file: the cloud's): `patterns` by `sensor_name` (a property, or a window redrawn from a DIFF image), `camera` (algorithm, ransac_threshold, mode, focal_distance, pixel_size, per_epoch) and `emulator.version` (optical's commit: the run stops if `$OPTICAL` is at another one or has local changes); its keys are checked. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml` only, the cloud's processing written in full (every pattern of `sensors.json`, the camera, optical's current commit), to edit before running it; it never overwrites an existing `config.yaml`. The standard configuration reproduces `data/` (JUNGFRAU, 6 days: `corr_x/y` within 0.034 px, the same rejections). A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new). Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`.
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations).
- `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data/<SERIAL>/weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`.
- `.env`: the credentials of the database and of S3, at the root, never versioned.
@@ -3,6 +3,24 @@
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
section "data_diff").
**Replaced the same day by the user's own layout** (the layout below, `configs/` and `runs/<name>/`, is not used):
```
data_diff/<SERIAL>/
images_diff/ (index/, sensors.json, unit.json) images/ metrics/ weather/ pumped, the same for every run
runs/<run>/ one run: whatever depends on it
config.yaml the run's configuration: patterns (by sensor_name), camera and emulator.version (optical's
commit, checked), over the cloud's processing
displacement/ camera/ targets.csv its result, the files of data/ (targets.csv from its patterns)
patterns.json the patterns exactly as given to the emulator, templates included
```
`processing/run_config.sh <SERIAL> <run>` reads the run's `config.yaml` and writes its `displacement/`, `camera/`,
`targets.csv` and `patterns.json`, nothing else. Rule: what is not the same for every possible run is in the run's
folder. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml`, the cloud's processing written
in full (patterns of `sensors.json`, camera, optical's current commit); never over an existing one. Note: the reading tools (aimsight-tools `data.find_files`) read a station folder whole, subfolders
included: pointed at `data_diff/<SERIAL>/`, they see every run's curves together.
## Intent
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
@@ -1,7 +1,15 @@
# run_config: a configuration recomputed into runs/<name>/
# run_config: a run of a station recomputed from its differential images
2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`.
**Amended the same day (the user's layout, see the configurations design):** `processing/run_config.sh <SERIAL> <run>`
reads `data_diff/<SERIAL>/runs/<run>/config.yaml` (`patterns`, `camera`, `emulator.version`: optical's commit, the run
stops if `$OPTICAL` is at another one or has local changes; no `stations` level) and writes only that run's
`displacement/`, `camera/`, `targets.csv` (from its patterns; none at the station's root any more) and `patterns.json`
(the patterns as given to the emulator); the emulator's other files go through a temporary folder. No `runs/<name>/`,
no links, no `run.yaml`. The rest below holds (patterns, correction as
the cloud, columns, acceptance).
## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`)
Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw
+3 -3
View File
@@ -58,7 +58,7 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small
## The differential images, into `data_diff/` (`pump_diff.sh`)
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the configurations, `runs/<name>/`, read like `data/`).
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the runs, `data_diff/<SERIAL>/runs/<run>/`).
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is;
- `images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON);
@@ -66,7 +66,7 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small
- `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm);
- (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder)
- `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`;
- `targets.csv`: as in `data/`, for the patterns the DIFF images are measured with (not archived); `displacement/` and `camera/` come only from the recomputation of a configuration (`runs/<name>/`), never from the cloud.
- `displacement/`, `camera/` and `targets.csv` are not pumped: they depend on the run (its patterns, its settings), each run writes its own in `runs/<run>/` (`processing/run_config.sh`), never from the cloud; a pumping never touches `runs/`.
```bash
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here
@@ -74,6 +74,6 @@ extraction/pump_diff.sh JUNGFRAU # from its
extraction/pump_diff.sh # the same for every station of data_diff/
```
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`; `targets.csv` the same without the two non-DIFF sensors `target_01`, `target_02`.
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`.
Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.
+5 -24
View File
@@ -18,8 +18,9 @@ read it the same way (docs/superpowers/specs/2026-11-19-configurations-design.md
time_utc, status}] (every property row as stored), rewritten at each run
<STATION>/images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties
(under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder)
<STATION>/metrics/, images/ (IRON, one per --every minutes), targets.csv (the DIFF patterns), weather/: as pump.py
and weather.py write them into data/, with their own functions
<STATION>/metrics/, images/ (IRON, one per --every minutes), weather/: as pump.py and weather.py write them into
data/, with their own functions. Not targets.csv: it depends on the run (its patterns), each run writes its own
(processing/run_config.py)
Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and
the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index
@@ -138,17 +139,6 @@ def unit_json(conn, unit):
"properties": [prop(*r) for r in rows]}
def diff_patterns(sensors):
"""{sensor_name: sensor_id} of the patterns the DIFF images are measured with (sensors_json rows): type
OPTICAL_SHIFT-DIFF, a pattern, not archived; for targets.csv."""
out = {}
for s in sensors:
p = {r["name"]: r["value"] for r in s["properties"]}
if s["type"] == MTYPE and p.get("pattern") and not p.get("archive"):
out[p.get("sensor_name") or s["name"]] = s["id"]
return out
def to_json(obj):
return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode()
@@ -172,7 +162,7 @@ def main():
from sqlalchemy import create_engine, text
from sqlalchemy.engine import URL
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause, sensors, write_targets
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
@@ -224,11 +214,8 @@ def main():
sens = sensors_json(conn, unit)
write_atomic(folder / "images_diff" / "sensors.json", to_json(sens))
write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit)))
t = write_targets(folder, sensors(conn, unit), diff_patterns(sens))
print(f"{station}: targets.csv, {len(t)} sensors")
except Exception as e: # the next run writes them
print(f"{station} images range, sensors.json, unit.json, targets.csv: failed ({root_cause(e)}), "
"run again")
print(f"{station} images range, sensors.json, unit.json: failed ({root_cause(e)}), run again")
finally:
conn.rollback()
print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}")
@@ -313,12 +300,6 @@ def check():
assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}'
assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index
assert indexed_days(folder) == {d(3)}
row = lambda name, typ, **p: {"name": name, "type": typ, "id": len(p), # noqa: E731
"properties": [{"name": k, "value": v} for k, v in p.items()]}
assert diff_patterns([row("u1", MTYPE, pattern="b64", sensor_name="T1"), row("u2", MTYPE, pattern="b64"),
row("u3", MTYPE, pattern="b64", sensor_name="OLD", archive="2026-01-01"),
row("u4", "camera_movement", sensor_name="CAM"), row("u5", MTYPE, sensor_name="NOPAT")]) \
== {"T1": 2, "u2": 1}, "DIFF patterns: by sensor_name, else the sensor's name; archived and others left out"
print("ok")
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Creates a new, empty run: data_diff/<SERIAL>/runs/<run>/config.yaml with the cloud's processing written in full (the
# patterns of images_diff/sensors.json, the camera, optical's current commit); no result until it is run.
# processing/new_run.sh JUNGFRAU run02
# processing/run_config.sh JUNGFRAU run02 then, once config.yaml is edited
exec "$(dirname "$0")/run_config.sh" "$@" --new
+139 -89
View File
@@ -1,35 +1,42 @@
"""Recomputes a configuration from the differential images into runs/<name>/, in the tree of data/, so that the other
repositories read it as they read data/ (point their `data` link at runs/<name>). Never writes outside runs/<name>/.
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
"""Runs one configuration of a station: data_diff/<SERIAL>/runs/<run>/config.yaml in, the same folder's displacement/,
camera/ and targets.csv out, in the files of data/. The station's pumped data, the same for every run (images_diff/,
images/, metrics/, weather/), stay at data_diff/<SERIAL>/ and are only read; whatever depends on the run is in its folder.
processing/run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17
processing/run_config.sh configs/mine.yaml the stations the configuration names, all their days
processing/run_config.sh JUNGFRAU run02 --from 2026-10-12 --to 2026-10-17
processing/run_config.sh JUNGFRAU run02 every indexed day
<config>: configs/<name>.yaml (or a path); stations: those given, else those of the configuration; days: those indexed
in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). For each station, runs/<name>/<SERIAL>/
is replaced (the other stations of the run are kept):
Days: those indexed in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). Running a run again
replaces its displacement/, camera/, targets.csv and patterns.json; nothing else is written:
displacement/<S>_advanced_<YYYYMMDD>.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator;
relative = x - pattern x; corr from the camera correction; reproj_err,
inlier of the references; sigma0 of the epoch; corr_*_mm
camera/<S>_camera_<YYYYMMDD>.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute
targets.csv the patterns of the configuration
images/ metrics/ weather/ relative symbolic links to data_diff/<SERIAL>/
and, outside the station folders (the reading tools do not see them): runs/<name>/run.yaml (the configuration as
resolved per station, optical's commit, the days) and runs/<name>/diagnostics/ (the patterns used, the emulator's rows).
targets.csv as in data/, from the run's patterns: pattern_type, distance_mm, mm_per_px
patterns.json the patterns exactly as given to the emulator (templates included)
A configuration (YAML) states what differs from the cloud: the patterns (by sensor_name: any of their properties, or
`redraw: {image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}`, cut as the device's pattern editor does) and
the camera correction (`algorithm`, `ransac_threshold`, `mode`: off | rotation | rotation_scale | relative,
`focal_distance`, `pixel_size`, `per_epoch`), common or per station; configs/cloud.yaml, empty, is the cloud.
config.yaml states what differs from the cloud's processing (an empty file: the cloud's), checked key by key:
patterns: by sensor_name, over images_diff/sensors.json: any existing property, or
PM_G: {error_threshold: 0.7} a template redrawn from a DIFF image of images_diff/, as the
BL_C: {redraw: {image: <file>, corners: [[x0, y0], [x1, y1]]}} device's pattern editor does (x_ref kept)
camera: over images_diff/unit.json, over the cloud's defaults
algorithm: ransac ransac | umeyama | danish_hybrid | trusted_danish
ransac_threshold: 0.99
mode: "off" off | rotation | rotation_scale | relative (optical's refine_allpoints)
focal_distance: 6.0 pixel_size: 0.0024
per_epoch: true one measurement at a time, as the cloud; false: the whole series at once
emulator:
version: "65a4c70" optical's commit: the run stops if $OPTICAL is at another one or has local changes
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
"""
import argparse
import csv
import datetime
import json
import os
import shutil
import subprocess
import sys
from datetime import datetime, timezone
import tempfile
from pathlib import Path
# Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and
@@ -54,7 +61,7 @@ CAM = ["pan", "tilt", "scale", "rotation", "sigma0"]
DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0,
"pixel_size": 0.0024, "per_epoch": True}
UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size")
LINKED = ("images", "metrics", "weather")
OUTPUTS = ("displacement", "camera") # what a run writes into its folder, and replaces
def _keys(d, allowed, where):
@@ -68,11 +75,10 @@ def _keys(d, allowed, where):
def load_config(path):
"""The configuration, its keys checked (a typo is an error, not a setting silently ignored)."""
c = yaml.safe_load(Path(path).read_text()) or {}
_keys(c, {"stations", "camera"}, "the configuration")
_keys(c.get("camera") or {}, DEFAULTS, "camera")
for s, block in (c.get("stations") or {}).items():
_keys(block or {}, {"patterns", "camera"}, f"stations.{s}")
_keys((block or {}).get("camera") or {}, DEFAULTS, f"stations.{s}.camera")
_keys(c, {"patterns", "camera", "emulator"}, str(path))
_keys(c.get("camera") or {}, DEFAULTS, f"{path}: camera")
_keys(c.get("emulator") or {}, {"version"}, f"{path}: emulator")
_keys(c.get("patterns") or {}, set(c.get("patterns") or {}), f"{path}: patterns") # a mapping (names: later)
return c
@@ -81,11 +87,10 @@ def latest(rows):
return {r["name"]: r["value"] for r in rows}
def camera_settings(config, station, unit_props):
def camera_settings(config, unit_props):
cam = dict(DEFAULTS)
cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props})
cam.update(config.get("camera") or {})
cam.update(((config.get("stations") or {}).get(station) or {}).get("camera") or {})
return cam
@@ -214,15 +219,14 @@ def correct(meas, props, cam):
return adv, agg.reset_index()
def write_station(adv, camera, props, dest, station):
def write_run(adv, camera, dest, station):
"""displacement/ and camera/ of a run, one file per UTC day, replacing the run's previous ones."""
for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")):
(dest / folder).mkdir(parents=True, exist_ok=True)
if (dest / folder).exists():
shutil.rmtree(dest / folder)
(dest / folder).mkdir()
for day, d in df.groupby(df["time_utc"].dt.date):
d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False)
t = pd.DataFrame([{"sensor_name": q.get("sensor_name"), "pattern_type": q.get("pattern_type"),
"distance_mm": q.get("distance_mm"), "mm_per_px": q.get("distance_convertion_factor")}
for q in props.values()]).sort_values("sensor_name")
t.to_csv(dest / "targets.csv", index=False)
def optical_state():
@@ -231,52 +235,107 @@ def optical_state():
"dirty": bool(git("status", "--porcelain", "--untracked-files=no"))}
PATTERN_KEYS = ["pattern_type", "corners", "x", "y", "error_threshold", "resolution", "distance_convertion_factor"]
CAMERA_NOTES = {"algorithm": "ransac | umeyama | danish_hybrid | trusted_danish", "ransac_threshold": "px, ransac only",
"mode": "off | rotation | rotation_scale | relative", "focal_distance": "mm; pan, tilt, sigma0 only",
"pixel_size": "mm; pan, tilt, sigma0 only",
"per_epoch": "one measurement at a time, as the cloud; false: the whole series at once"}
def new_run(src, name):
"""runs/<name>/config.yaml of a new, empty run (no result yet): the cloud's processing written in full, the patterns
of images_diff/sensors.json, the camera of unit.json or the cloud's defaults, optical's current commit."""
run = src / "runs" / name
if (run / "config.yaml").exists():
raise FileExistsError(f"{run / 'config.yaml'} exists: choose another name, or edit that file")
props = sorted(cloud_properties(json.loads((src / "images_diff" / "sensors.json").read_text())).values(),
key=lambda p: (p.get("pattern_type") != "global_ref", str(p.get("sensor_name"))))
unit = latest(json.loads((src / "images_diff" / "unit.json").read_text())["properties"])
cam, state = camera_settings({}, unit), optical_state()
width = max(len(str(p.get("sensor_name"))) for p in props) + 1
line = lambda p: f" {str(p.get('sensor_name')) + ':':{width}s} {{" + ", ".join( # noqa: E731
f"{k}: {json.dumps(p[k])}" for k in PATTERN_KEYS if k in p) + "}"
refs = [line(p) for p in props if p.get("pattern_type") == "global_ref"]
targets = [line(p) for p in props if p.get("pattern_type") != "global_ref"]
text = f"""# {name}: created on {datetime.date.today()} with the cloud's processing. Change a value, then run it:
# processing/run_config.sh {src.name} {name}
# The run also writes here targets.csv (as in data/, from these patterns) and patterns.json (the patterns exactly as
# given to the emulator, templates included).
emulator:
version: "{state['commit'][:7]}" # optical's commit (branch {state['branch']}); the run stops if $OPTICAL is
# at another one or has local changes
patterns:
# The {len(props)} patterns of the DIFF images, as in images_diff/sensors.json on {datetime.date.today()} (frozen here: a
# later pumping does not change them). The template stays in sensors.json; to change it:
# <name>: {{redraw: {{image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}}}}
# pattern_type: global_ref = a reference of the camera fit, anything else a target. corners and x, y move together
# (same window size as the template). resolution: 1/100 px. distance_convertion_factor: mm per px (corr_*_mm only).
# references
""" + "\n".join(refs) + "\n # targets\n" + "\n".join(targets) + "\ncamera:" + "".join(
f"\n {f'{k}: {json.dumps(v)}':26s} # {CAMERA_NOTES[k]}" for k, v in cam.items()) + (
"\n # focal_distance, pixel_size: the unit has none on the platform, the cloud's values"
if not {"focal_distance", "pixel_size"} & set(unit) else "") + "\n"
run.mkdir(parents=True, exist_ok=True)
(run / "config.yaml").write_text(text)
return run / "config.yaml", state
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("config", help="name of configs/<name>.yaml, or a path")
p.add_argument("stations", nargs="*", help="serial numbers; none: those of the configuration")
p.add_argument("station", help="serial number: the folder data_diff/<SERIAL>/")
p.add_argument("run", help="the folder data_diff/<SERIAL>/runs/<run>/, holding config.yaml")
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD")
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD")
p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)")
p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff")
p.add_argument("--out", type=Path, default=ROOT / "runs")
p.add_argument("--new", action="store_true", help="create the run, empty: its config.yaml only (processing/new_run.sh)")
a = p.parse_args()
path = Path(a.config) if a.config.endswith((".yaml", ".yml")) or os.sep in a.config \
else ROOT / "configs" / f"{a.config}.yaml"
config, name = load_config(path), path.stem
stations = a.stations or list(config.get("stations") or {})
if not stations:
sys.exit(f"No station: name one ({path} names none)")
run = a.out / name
(run / "diagnostics").mkdir(parents=True, exist_ok=True)
record_path = run / "run.yaml"
record = yaml.safe_load(record_path.read_text()) if record_path.exists() else {}
record.update({"configuration": name, "source": str(path), "optical": optical_state()})
record.setdefault("stations", {})
src = a.data_diff / a.station
run = src / "runs" / a.run
if a.new:
try:
path, state = new_run(src, a.run)
except FileExistsError as e:
sys.exit(str(e))
print(f"{path}: written{', but optical has local changes: the run will stop until they are committed' if state['dirty'] else ''}"
f"\nRun it: processing/run_config.sh {a.station} {a.run}")
return
if not (run / "config.yaml").is_file():
sys.exit(f"No {run / 'config.yaml'}: create the run first, processing/new_run.sh {a.station} {a.run}")
config = load_config(run / "config.yaml")
version = (config.get("emulator") or {}).get("version")
if version is not None: # the code that makes the run is the one config.yaml names
state = optical_state()
want = subprocess.run(["git", "-C", str(OPTICAL), "rev-parse", "--verify", "--quiet", f"{version}^{{commit}}"],
capture_output=True, text=True).stdout.strip()
if not want:
sys.exit(f"emulator.version {version}: not a commit of {OPTICAL}")
if want != state["commit"] or state["dirty"]:
sys.exit(f"emulator.version {version}, but {OPTICAL} is at {state['commit'][:7]}"
f"{' with local changes' if state['dirty'] else ''}: check {version} out, or change the version")
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{a.station}_diff_*.csv")
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
if not rows:
sys.exit(f"{a.station}: no differential image indexed in the range, nothing done")
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
unit = json.loads((src / "images_diff" / "unit.json").read_text())
cam = camera_settings(config, latest(unit["properties"]))
props = apply_patterns(cloud_properties(sensors), config.get("patterns"), src / "images_diff")
print(f"{a.station} {a.run}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} "
f"patterns, camera {cam}, optical {optical_state()}")
import reprocess_images
for station in stations:
src = a.data_diff / station
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{station}_diff_*.csv")
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
if not rows:
print(f"{station}: no differential image indexed in the range, nothing done")
continue
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
unit = json.loads((src / "images_diff" / "unit.json").read_text())
cam = camera_settings(config, station, latest(unit["properties"]))
overrides = ((config.get("stations") or {}).get(station) or {}).get("patterns")
props = apply_patterns(cloud_properties(sensors), overrides, src / "images_diff")
print(f"{station}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} patterns, "
f"camera {cam}")
diag = run / "diagnostics"
sp, times, remeasured = (diag / f"{station}_sensor_properties.json", diag / f"{station}_times.csv",
diag / f"{station}_remeasured.csv")
sp.write_text(json.dumps(props))
sp = run / "patterns.json" # kept: the target configuration the run was made with, templates included
sp.write_text(json.dumps(props, indent=1))
pd.DataFrame([{"sensor_name": p.get("sensor_name"), "pattern_type": p.get("pattern_type"), # pump.py's columns
"distance_mm": p.get("distance_mm"), "mm_per_px": p.get("distance_convertion_factor")}
for p in props.values()]).sort_values("sensor_name").to_csv(run / "targets.csv", index=False)
with tempfile.TemporaryDirectory() as tmp: # the emulator's other files: not kept
times, remeasured = Path(tmp) / "times.csv", Path(tmp) / "rows.csv"
with open(times, "w", newline="") as fp:
w = csv.writer(fp)
w.writerow(["file", "time_utc"])
@@ -284,42 +343,33 @@ def main():
reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured),
"--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows])
adv, camera = correct(pd.read_csv(remeasured), props, cam)
dest = run / station
if dest.exists():
assert dest.resolve().parent == run.resolve(), dest # only runs/<name>/<SERIAL> is ever removed
shutil.rmtree(dest)
write_station(adv, camera, props, dest, station)
for sub in LINKED:
if (src / sub).exists():
os.symlink(os.path.relpath(src / sub, dest), dest / sub)
record["stations"][station] = {
"days": [days[0].stem[-8:], days[-1].stem[-8:]], "images": len(rows), "camera": cam,
"patterns": overrides or {}, "date": datetime.now(timezone.utc).isoformat(timespec="seconds")}
print(f"{station}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows -> {dest}")
record_path.write_text(yaml.safe_dump(record, sort_keys=False, allow_unicode=True))
write_run(adv, camera, run, a.station)
print(f"{a.station} {a.run}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows "
f"-> {run}")
def check():
"""`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical."""
import tempfile
with tempfile.TemporaryDirectory() as tmp:
f = Path(tmp) / "c.yaml"
for text in ("camera: {algoritm: ransac}", "stations: {S: {pattern: {}}}", "stations: {S: {camera: {mod: x}}}",
"other: 1"):
f = Path(tmp) / "config.yaml"
for text in ("camera: {algoritm: ransac}", "pattern: {}", "stations: {S: {}}", "patterns: [PM_G]",
"emulator: {versio: 65a4c70}"):
f.write_text(text)
try:
load_config(f)
raise AssertionError(f"accepted: {text}")
except ValueError:
pass
f.write_text("camera: {mode: rotation, focal_distance: 6.1}\nstations: {S: {camera: {focal_distance: 6.3}}}")
f.write_text("")
assert load_config(f) == {} # an empty file: the cloud's processing
f.write_text("camera: {mode: rotation, focal_distance: 6.3}\npatterns: {T1: {error_threshold: 0.9}}\n"
"emulator: {version: '65a4c70'}")
c = load_config(f)
cam = camera_settings(c, "S", {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
assert c["emulator"]["version"] == "65a4c70"
cam = camera_settings(c, {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation",
"ransac"), cam
assert camera_settings({}, "S", {})["focal_distance"] == 6.0
assert camera_settings({}, {})["focal_distance"] == 6.0
pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)}
props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y,
+3 -2
View File
@@ -1,7 +1,8 @@
#!/usr/bin/env bash
# Recomputes a configuration into runs/<name>/ (processing/run_config.py), in a throwaway uv environment with what
# Runs data_diff/<SERIAL>/runs/<run>/config.yaml (processing/run_config.py), in a throwaway uv environment with what
# optical's emulator and camera correction need; optical's code from $OPTICAL (default ~/optical). No credentials.
# processing/run_config.sh <config> [SERIAL...] [--from D] [--to E] [--jobs N]
# processing/run_config.sh <SERIAL> <run> [--from D] [--to E] [--jobs N]
# processing/run_config.sh <SERIAL> <run> --new (processing/new_run.sh): the run created, empty
# processing/run_config.sh --check
set -u
export PATH="$HOME/.local/bin:$PATH"