Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it

- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/,
  weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv,
  patterns.json.
- processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the
  patterns of sensors.json, the camera, optical's current commit); never over an existing one.
- processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked
  against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns)
  and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/.
- pump_diff no longer writes targets.csv: it depends on the run.

JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within
0.034 px; a run created by new_run.sh and run as is gives run01 exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-19 10:52:46 +01:00
1 parent d3bf4b6860
commit ca366c150e
8 files changed
+185 -123

No files matched your search

+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Creates a new, empty run: data_diff/<SERIAL>/runs/<run>/config.yaml with the cloud's processing written in full (the
# patterns of images_diff/sensors.json, the camera, optical's current commit); no result until it is run.
# processing/new_run.sh JUNGFRAU run02
# processing/run_config.sh JUNGFRAU run02 then, once config.yaml is edited
exec "$(dirname "$0")/run_config.sh" "$@" --new
+139 -89
View File
@@ -1,35 +1,42 @@
"""Recomputes a configuration from the differential images into runs/<name>/, in the tree of data/, so that the other
repositories read it as they read data/ (point their `data` link at runs/<name>). Never writes outside runs/<name>/.
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
"""Runs one configuration of a station: data_diff/<SERIAL>/runs/<run>/config.yaml in, the same folder's displacement/,
camera/ and targets.csv out, in the files of data/. The station's pumped data, the same for every run (images_diff/,
images/, metrics/, weather/), stay at data_diff/<SERIAL>/ and are only read; whatever depends on the run is in its folder.
processing/run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17
processing/run_config.sh configs/mine.yaml the stations the configuration names, all their days
processing/run_config.sh JUNGFRAU run02 --from 2026-10-12 --to 2026-10-17
processing/run_config.sh JUNGFRAU run02 every indexed day
<config>: configs/<name>.yaml (or a path); stations: those given, else those of the configuration; days: those indexed
in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). For each station, runs/<name>/<SERIAL>/
is replaced (the other stations of the run are kept):
Days: those indexed in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). Running a run again
replaces its displacement/, camera/, targets.csv and patterns.json; nothing else is written:
displacement/<S>_advanced_<YYYYMMDD>.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator;
relative = x - pattern x; corr from the camera correction; reproj_err,
inlier of the references; sigma0 of the epoch; corr_*_mm
camera/<S>_camera_<YYYYMMDD>.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute
targets.csv the patterns of the configuration
images/ metrics/ weather/ relative symbolic links to data_diff/<SERIAL>/
and, outside the station folders (the reading tools do not see them): runs/<name>/run.yaml (the configuration as
resolved per station, optical's commit, the days) and runs/<name>/diagnostics/ (the patterns used, the emulator's rows).
targets.csv as in data/, from the run's patterns: pattern_type, distance_mm, mm_per_px
patterns.json the patterns exactly as given to the emulator (templates included)
A configuration (YAML) states what differs from the cloud: the patterns (by sensor_name: any of their properties, or
`redraw: {image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}`, cut as the device's pattern editor does) and
the camera correction (`algorithm`, `ransac_threshold`, `mode`: off | rotation | rotation_scale | relative,
`focal_distance`, `pixel_size`, `per_epoch`), common or per station; configs/cloud.yaml, empty, is the cloud.
config.yaml states what differs from the cloud's processing (an empty file: the cloud's), checked key by key:
patterns: by sensor_name, over images_diff/sensors.json: any existing property, or
PM_G: {error_threshold: 0.7} a template redrawn from a DIFF image of images_diff/, as the
BL_C: {redraw: {image: <file>, corners: [[x0, y0], [x1, y1]]}} device's pattern editor does (x_ref kept)
camera: over images_diff/unit.json, over the cloud's defaults
algorithm: ransac ransac | umeyama | danish_hybrid | trusted_danish
ransac_threshold: 0.99
mode: "off" off | rotation | rotation_scale | relative (optical's refine_allpoints)
focal_distance: 6.0 pixel_size: 0.0024
per_epoch: true one measurement at a time, as the cloud; false: the whole series at once
emulator:
version: "65a4c70" optical's commit: the run stops if $OPTICAL is at another one or has local changes
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
"""
import argparse
import csv
import datetime
import json
import os
import shutil
import subprocess
import sys
from datetime import datetime, timezone
import tempfile
from pathlib import Path
# Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and
@@ -54,7 +61,7 @@ CAM = ["pan", "tilt", "scale", "rotation", "sigma0"]
DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0,
"pixel_size": 0.0024, "per_epoch": True}
UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size")
LINKED = ("images", "metrics", "weather")
OUTPUTS = ("displacement", "camera") # what a run writes into its folder, and replaces
def _keys(d, allowed, where):
@@ -68,11 +75,10 @@ def _keys(d, allowed, where):
def load_config(path):
"""The configuration, its keys checked (a typo is an error, not a setting silently ignored)."""
c = yaml.safe_load(Path(path).read_text()) or {}
_keys(c, {"stations", "camera"}, "the configuration")
_keys(c.get("camera") or {}, DEFAULTS, "camera")
for s, block in (c.get("stations") or {}).items():
_keys(block or {}, {"patterns", "camera"}, f"stations.{s}")
_keys((block or {}).get("camera") or {}, DEFAULTS, f"stations.{s}.camera")
_keys(c, {"patterns", "camera", "emulator"}, str(path))
_keys(c.get("camera") or {}, DEFAULTS, f"{path}: camera")
_keys(c.get("emulator") or {}, {"version"}, f"{path}: emulator")
_keys(c.get("patterns") or {}, set(c.get("patterns") or {}), f"{path}: patterns") # a mapping (names: later)
return c
@@ -81,11 +87,10 @@ def latest(rows):
return {r["name"]: r["value"] for r in rows}
def camera_settings(config, station, unit_props):
def camera_settings(config, unit_props):
cam = dict(DEFAULTS)
cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props})
cam.update(config.get("camera") or {})
cam.update(((config.get("stations") or {}).get(station) or {}).get("camera") or {})
return cam
@@ -214,15 +219,14 @@ def correct(meas, props, cam):
return adv, agg.reset_index()
def write_station(adv, camera, props, dest, station):
def write_run(adv, camera, dest, station):
"""displacement/ and camera/ of a run, one file per UTC day, replacing the run's previous ones."""
for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")):
(dest / folder).mkdir(parents=True, exist_ok=True)
if (dest / folder).exists():
shutil.rmtree(dest / folder)
(dest / folder).mkdir()
for day, d in df.groupby(df["time_utc"].dt.date):
d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False)
t = pd.DataFrame([{"sensor_name": q.get("sensor_name"), "pattern_type": q.get("pattern_type"),
"distance_mm": q.get("distance_mm"), "mm_per_px": q.get("distance_convertion_factor")}
for q in props.values()]).sort_values("sensor_name")
t.to_csv(dest / "targets.csv", index=False)
def optical_state():
@@ -231,52 +235,107 @@ def optical_state():
"dirty": bool(git("status", "--porcelain", "--untracked-files=no"))}
PATTERN_KEYS = ["pattern_type", "corners", "x", "y", "error_threshold", "resolution", "distance_convertion_factor"]
CAMERA_NOTES = {"algorithm": "ransac | umeyama | danish_hybrid | trusted_danish", "ransac_threshold": "px, ransac only",
"mode": "off | rotation | rotation_scale | relative", "focal_distance": "mm; pan, tilt, sigma0 only",
"pixel_size": "mm; pan, tilt, sigma0 only",
"per_epoch": "one measurement at a time, as the cloud; false: the whole series at once"}
def new_run(src, name):
"""runs/<name>/config.yaml of a new, empty run (no result yet): the cloud's processing written in full, the patterns
of images_diff/sensors.json, the camera of unit.json or the cloud's defaults, optical's current commit."""
run = src / "runs" / name
if (run / "config.yaml").exists():
raise FileExistsError(f"{run / 'config.yaml'} exists: choose another name, or edit that file")
props = sorted(cloud_properties(json.loads((src / "images_diff" / "sensors.json").read_text())).values(),
key=lambda p: (p.get("pattern_type") != "global_ref", str(p.get("sensor_name"))))
unit = latest(json.loads((src / "images_diff" / "unit.json").read_text())["properties"])
cam, state = camera_settings({}, unit), optical_state()
width = max(len(str(p.get("sensor_name"))) for p in props) + 1
line = lambda p: f" {str(p.get('sensor_name')) + ':':{width}s} {{" + ", ".join( # noqa: E731
f"{k}: {json.dumps(p[k])}" for k in PATTERN_KEYS if k in p) + "}"
refs = [line(p) for p in props if p.get("pattern_type") == "global_ref"]
targets = [line(p) for p in props if p.get("pattern_type") != "global_ref"]
text = f"""# {name}: created on {datetime.date.today()} with the cloud's processing. Change a value, then run it:
# processing/run_config.sh {src.name} {name}
# The run also writes here targets.csv (as in data/, from these patterns) and patterns.json (the patterns exactly as
# given to the emulator, templates included).
emulator:
version: "{state['commit'][:7]}" # optical's commit (branch {state['branch']}); the run stops if $OPTICAL is
# at another one or has local changes
patterns:
# The {len(props)} patterns of the DIFF images, as in images_diff/sensors.json on {datetime.date.today()} (frozen here: a
# later pumping does not change them). The template stays in sensors.json; to change it:
# <name>: {{redraw: {{image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}}}}
# pattern_type: global_ref = a reference of the camera fit, anything else a target. corners and x, y move together
# (same window size as the template). resolution: 1/100 px. distance_convertion_factor: mm per px (corr_*_mm only).
# references
""" + "\n".join(refs) + "\n # targets\n" + "\n".join(targets) + "\ncamera:" + "".join(
f"\n {f'{k}: {json.dumps(v)}':26s} # {CAMERA_NOTES[k]}" for k, v in cam.items()) + (
"\n # focal_distance, pixel_size: the unit has none on the platform, the cloud's values"
if not {"focal_distance", "pixel_size"} & set(unit) else "") + "\n"
run.mkdir(parents=True, exist_ok=True)
(run / "config.yaml").write_text(text)
return run / "config.yaml", state
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("config", help="name of configs/<name>.yaml, or a path")
p.add_argument("stations", nargs="*", help="serial numbers; none: those of the configuration")
p.add_argument("station", help="serial number: the folder data_diff/<SERIAL>/")
p.add_argument("run", help="the folder data_diff/<SERIAL>/runs/<run>/, holding config.yaml")
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD")
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD")
p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)")
p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff")
p.add_argument("--out", type=Path, default=ROOT / "runs")
p.add_argument("--new", action="store_true", help="create the run, empty: its config.yaml only (processing/new_run.sh)")
a = p.parse_args()
path = Path(a.config) if a.config.endswith((".yaml", ".yml")) or os.sep in a.config \
else ROOT / "configs" / f"{a.config}.yaml"
config, name = load_config(path), path.stem
stations = a.stations or list(config.get("stations") or {})
if not stations:
sys.exit(f"No station: name one ({path} names none)")
run = a.out / name
(run / "diagnostics").mkdir(parents=True, exist_ok=True)
record_path = run / "run.yaml"
record = yaml.safe_load(record_path.read_text()) if record_path.exists() else {}
record.update({"configuration": name, "source": str(path), "optical": optical_state()})
record.setdefault("stations", {})
src = a.data_diff / a.station
run = src / "runs" / a.run
if a.new:
try:
path, state = new_run(src, a.run)
except FileExistsError as e:
sys.exit(str(e))
print(f"{path}: written{', but optical has local changes: the run will stop until they are committed' if state['dirty'] else ''}"
f"\nRun it: processing/run_config.sh {a.station} {a.run}")
return
if not (run / "config.yaml").is_file():
sys.exit(f"No {run / 'config.yaml'}: create the run first, processing/new_run.sh {a.station} {a.run}")
config = load_config(run / "config.yaml")
version = (config.get("emulator") or {}).get("version")
if version is not None: # the code that makes the run is the one config.yaml names
state = optical_state()
want = subprocess.run(["git", "-C", str(OPTICAL), "rev-parse", "--verify", "--quiet", f"{version}^{{commit}}"],
capture_output=True, text=True).stdout.strip()
if not want:
sys.exit(f"emulator.version {version}: not a commit of {OPTICAL}")
if want != state["commit"] or state["dirty"]:
sys.exit(f"emulator.version {version}, but {OPTICAL} is at {state['commit'][:7]}"
f"{' with local changes' if state['dirty'] else ''}: check {version} out, or change the version")
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{a.station}_diff_*.csv")
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
if not rows:
sys.exit(f"{a.station}: no differential image indexed in the range, nothing done")
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
unit = json.loads((src / "images_diff" / "unit.json").read_text())
cam = camera_settings(config, latest(unit["properties"]))
props = apply_patterns(cloud_properties(sensors), config.get("patterns"), src / "images_diff")
print(f"{a.station} {a.run}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} "
f"patterns, camera {cam}, optical {optical_state()}")
import reprocess_images
for station in stations:
src = a.data_diff / station
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{station}_diff_*.csv")
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
if not rows:
print(f"{station}: no differential image indexed in the range, nothing done")
continue
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
unit = json.loads((src / "images_diff" / "unit.json").read_text())
cam = camera_settings(config, station, latest(unit["properties"]))
overrides = ((config.get("stations") or {}).get(station) or {}).get("patterns")
props = apply_patterns(cloud_properties(sensors), overrides, src / "images_diff")
print(f"{station}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} patterns, "
f"camera {cam}")
diag = run / "diagnostics"
sp, times, remeasured = (diag / f"{station}_sensor_properties.json", diag / f"{station}_times.csv",
diag / f"{station}_remeasured.csv")
sp.write_text(json.dumps(props))
sp = run / "patterns.json" # kept: the target configuration the run was made with, templates included
sp.write_text(json.dumps(props, indent=1))
pd.DataFrame([{"sensor_name": p.get("sensor_name"), "pattern_type": p.get("pattern_type"), # pump.py's columns
"distance_mm": p.get("distance_mm"), "mm_per_px": p.get("distance_convertion_factor")}
for p in props.values()]).sort_values("sensor_name").to_csv(run / "targets.csv", index=False)
with tempfile.TemporaryDirectory() as tmp: # the emulator's other files: not kept
times, remeasured = Path(tmp) / "times.csv", Path(tmp) / "rows.csv"
with open(times, "w", newline="") as fp:
w = csv.writer(fp)
w.writerow(["file", "time_utc"])
@@ -284,42 +343,33 @@ def main():
reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured),
"--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows])
adv, camera = correct(pd.read_csv(remeasured), props, cam)
dest = run / station
if dest.exists():
assert dest.resolve().parent == run.resolve(), dest # only runs/<name>/<SERIAL> is ever removed
shutil.rmtree(dest)
write_station(adv, camera, props, dest, station)
for sub in LINKED:
if (src / sub).exists():
os.symlink(os.path.relpath(src / sub, dest), dest / sub)
record["stations"][station] = {
"days": [days[0].stem[-8:], days[-1].stem[-8:]], "images": len(rows), "camera": cam,
"patterns": overrides or {}, "date": datetime.now(timezone.utc).isoformat(timespec="seconds")}
print(f"{station}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows -> {dest}")
record_path.write_text(yaml.safe_dump(record, sort_keys=False, allow_unicode=True))
write_run(adv, camera, run, a.station)
print(f"{a.station} {a.run}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows "
f"-> {run}")
def check():
"""`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical."""
import tempfile
with tempfile.TemporaryDirectory() as tmp:
f = Path(tmp) / "c.yaml"
for text in ("camera: {algoritm: ransac}", "stations: {S: {pattern: {}}}", "stations: {S: {camera: {mod: x}}}",
"other: 1"):
f = Path(tmp) / "config.yaml"
for text in ("camera: {algoritm: ransac}", "pattern: {}", "stations: {S: {}}", "patterns: [PM_G]",
"emulator: {versio: 65a4c70}"):
f.write_text(text)
try:
load_config(f)
raise AssertionError(f"accepted: {text}")
except ValueError:
pass
f.write_text("camera: {mode: rotation, focal_distance: 6.1}\nstations: {S: {camera: {focal_distance: 6.3}}}")
f.write_text("")
assert load_config(f) == {} # an empty file: the cloud's processing
f.write_text("camera: {mode: rotation, focal_distance: 6.3}\npatterns: {T1: {error_threshold: 0.9}}\n"
"emulator: {version: '65a4c70'}")
c = load_config(f)
cam = camera_settings(c, "S", {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
assert c["emulator"]["version"] == "65a4c70"
cam = camera_settings(c, {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation",
"ransac"), cam
assert camera_settings({}, "S", {})["focal_distance"] == 6.0
assert camera_settings({}, {})["focal_distance"] == 6.0
pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)}
props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y,
+3 -2
View File
@@ -1,7 +1,8 @@
#!/usr/bin/env bash
# Recomputes a configuration into runs/<name>/ (processing/run_config.py), in a throwaway uv environment with what
# Runs data_diff/<SERIAL>/runs/<run>/config.yaml (processing/run_config.py), in a throwaway uv environment with what
# optical's emulator and camera correction need; optical's code from $OPTICAL (default ~/optical). No credentials.
# processing/run_config.sh <config> [SERIAL...] [--from D] [--to E] [--jobs N]
# processing/run_config.sh <SERIAL> <run> [--from D] [--to E] [--jobs N]
# processing/run_config.sh <SERIAL> <run> --new (processing/new_run.sh): the run created, empty
# processing/run_config.sh --check
set -u
export PATH="$HOME/.local/bin:$PATH"