Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it
- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/, weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv, patterns.json. - processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the patterns of sensors.json, the camera, optical's current commit); never over an existing one. - processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns) and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/. - pump_diff no longer writes targets.csv: it depends on the run. JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within 0.034 px; a run created by new_run.sh and run as is gives run01 exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d3bf4b6860
commit
ca366c150e
8 files changed
+185
-123
No files matched your search
+139
-89
@@ -1,35 +1,42 @@
|
||||
"""Recomputes a configuration from the differential images into runs/<name>/, in the tree of data/, so that the other
|
||||
repositories read it as they read data/ (point their `data` link at runs/<name>). Never writes outside runs/<name>/.
|
||||
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
|
||||
"""Runs one configuration of a station: data_diff/<SERIAL>/runs/<run>/config.yaml in, the same folder's displacement/,
|
||||
camera/ and targets.csv out, in the files of data/. The station's pumped data, the same for every run (images_diff/,
|
||||
images/, metrics/, weather/), stay at data_diff/<SERIAL>/ and are only read; whatever depends on the run is in its folder.
|
||||
|
||||
processing/run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17
|
||||
processing/run_config.sh configs/mine.yaml the stations the configuration names, all their days
|
||||
processing/run_config.sh JUNGFRAU run02 --from 2026-10-12 --to 2026-10-17
|
||||
processing/run_config.sh JUNGFRAU run02 every indexed day
|
||||
|
||||
<config>: configs/<name>.yaml (or a path); stations: those given, else those of the configuration; days: those indexed
|
||||
in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). For each station, runs/<name>/<SERIAL>/
|
||||
is replaced (the other stations of the run are kept):
|
||||
Days: those indexed in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). Running a run again
|
||||
replaces its displacement/, camera/, targets.csv and patterns.json; nothing else is written:
|
||||
displacement/<S>_advanced_<YYYYMMDD>.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator;
|
||||
relative = x - pattern x; corr from the camera correction; reproj_err,
|
||||
inlier of the references; sigma0 of the epoch; corr_*_mm
|
||||
camera/<S>_camera_<YYYYMMDD>.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute
|
||||
targets.csv the patterns of the configuration
|
||||
images/ metrics/ weather/ relative symbolic links to data_diff/<SERIAL>/
|
||||
and, outside the station folders (the reading tools do not see them): runs/<name>/run.yaml (the configuration as
|
||||
resolved per station, optical's commit, the days) and runs/<name>/diagnostics/ (the patterns used, the emulator's rows).
|
||||
targets.csv as in data/, from the run's patterns: pattern_type, distance_mm, mm_per_px
|
||||
patterns.json the patterns exactly as given to the emulator (templates included)
|
||||
|
||||
A configuration (YAML) states what differs from the cloud: the patterns (by sensor_name: any of their properties, or
|
||||
`redraw: {image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}`, cut as the device's pattern editor does) and
|
||||
the camera correction (`algorithm`, `ransac_threshold`, `mode`: off | rotation | rotation_scale | relative,
|
||||
`focal_distance`, `pixel_size`, `per_epoch`), common or per station; configs/cloud.yaml, empty, is the cloud.
|
||||
config.yaml states what differs from the cloud's processing (an empty file: the cloud's), checked key by key:
|
||||
patterns: by sensor_name, over images_diff/sensors.json: any existing property, or
|
||||
PM_G: {error_threshold: 0.7} a template redrawn from a DIFF image of images_diff/, as the
|
||||
BL_C: {redraw: {image: <file>, corners: [[x0, y0], [x1, y1]]}} device's pattern editor does (x_ref kept)
|
||||
camera: over images_diff/unit.json, over the cloud's defaults
|
||||
algorithm: ransac ransac | umeyama | danish_hybrid | trusted_danish
|
||||
ransac_threshold: 0.99
|
||||
mode: "off" off | rotation | rotation_scale | relative (optical's refine_allpoints)
|
||||
focal_distance: 6.0 pixel_size: 0.0024
|
||||
per_epoch: true one measurement at a time, as the cloud; false: the whole series at once
|
||||
emulator:
|
||||
version: "65a4c70" optical's commit: the run stops if $OPTICAL is at another one or has local changes
|
||||
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
|
||||
"""
|
||||
import argparse
|
||||
import csv
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
# Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and
|
||||
@@ -54,7 +61,7 @@ CAM = ["pan", "tilt", "scale", "rotation", "sigma0"]
|
||||
DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0,
|
||||
"pixel_size": 0.0024, "per_epoch": True}
|
||||
UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size")
|
||||
LINKED = ("images", "metrics", "weather")
|
||||
OUTPUTS = ("displacement", "camera") # what a run writes into its folder, and replaces
|
||||
|
||||
|
||||
def _keys(d, allowed, where):
|
||||
@@ -68,11 +75,10 @@ def _keys(d, allowed, where):
|
||||
def load_config(path):
|
||||
"""The configuration, its keys checked (a typo is an error, not a setting silently ignored)."""
|
||||
c = yaml.safe_load(Path(path).read_text()) or {}
|
||||
_keys(c, {"stations", "camera"}, "the configuration")
|
||||
_keys(c.get("camera") or {}, DEFAULTS, "camera")
|
||||
for s, block in (c.get("stations") or {}).items():
|
||||
_keys(block or {}, {"patterns", "camera"}, f"stations.{s}")
|
||||
_keys((block or {}).get("camera") or {}, DEFAULTS, f"stations.{s}.camera")
|
||||
_keys(c, {"patterns", "camera", "emulator"}, str(path))
|
||||
_keys(c.get("camera") or {}, DEFAULTS, f"{path}: camera")
|
||||
_keys(c.get("emulator") or {}, {"version"}, f"{path}: emulator")
|
||||
_keys(c.get("patterns") or {}, set(c.get("patterns") or {}), f"{path}: patterns") # a mapping (names: later)
|
||||
return c
|
||||
|
||||
|
||||
@@ -81,11 +87,10 @@ def latest(rows):
|
||||
return {r["name"]: r["value"] for r in rows}
|
||||
|
||||
|
||||
def camera_settings(config, station, unit_props):
|
||||
def camera_settings(config, unit_props):
|
||||
cam = dict(DEFAULTS)
|
||||
cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props})
|
||||
cam.update(config.get("camera") or {})
|
||||
cam.update(((config.get("stations") or {}).get(station) or {}).get("camera") or {})
|
||||
return cam
|
||||
|
||||
|
||||
@@ -214,15 +219,14 @@ def correct(meas, props, cam):
|
||||
return adv, agg.reset_index()
|
||||
|
||||
|
||||
def write_station(adv, camera, props, dest, station):
|
||||
def write_run(adv, camera, dest, station):
|
||||
"""displacement/ and camera/ of a run, one file per UTC day, replacing the run's previous ones."""
|
||||
for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")):
|
||||
(dest / folder).mkdir(parents=True, exist_ok=True)
|
||||
if (dest / folder).exists():
|
||||
shutil.rmtree(dest / folder)
|
||||
(dest / folder).mkdir()
|
||||
for day, d in df.groupby(df["time_utc"].dt.date):
|
||||
d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False)
|
||||
t = pd.DataFrame([{"sensor_name": q.get("sensor_name"), "pattern_type": q.get("pattern_type"),
|
||||
"distance_mm": q.get("distance_mm"), "mm_per_px": q.get("distance_convertion_factor")}
|
||||
for q in props.values()]).sort_values("sensor_name")
|
||||
t.to_csv(dest / "targets.csv", index=False)
|
||||
|
||||
|
||||
def optical_state():
|
||||
@@ -231,52 +235,107 @@ def optical_state():
|
||||
"dirty": bool(git("status", "--porcelain", "--untracked-files=no"))}
|
||||
|
||||
|
||||
PATTERN_KEYS = ["pattern_type", "corners", "x", "y", "error_threshold", "resolution", "distance_convertion_factor"]
|
||||
CAMERA_NOTES = {"algorithm": "ransac | umeyama | danish_hybrid | trusted_danish", "ransac_threshold": "px, ransac only",
|
||||
"mode": "off | rotation | rotation_scale | relative", "focal_distance": "mm; pan, tilt, sigma0 only",
|
||||
"pixel_size": "mm; pan, tilt, sigma0 only",
|
||||
"per_epoch": "one measurement at a time, as the cloud; false: the whole series at once"}
|
||||
|
||||
|
||||
def new_run(src, name):
|
||||
"""runs/<name>/config.yaml of a new, empty run (no result yet): the cloud's processing written in full, the patterns
|
||||
of images_diff/sensors.json, the camera of unit.json or the cloud's defaults, optical's current commit."""
|
||||
run = src / "runs" / name
|
||||
if (run / "config.yaml").exists():
|
||||
raise FileExistsError(f"{run / 'config.yaml'} exists: choose another name, or edit that file")
|
||||
props = sorted(cloud_properties(json.loads((src / "images_diff" / "sensors.json").read_text())).values(),
|
||||
key=lambda p: (p.get("pattern_type") != "global_ref", str(p.get("sensor_name"))))
|
||||
unit = latest(json.loads((src / "images_diff" / "unit.json").read_text())["properties"])
|
||||
cam, state = camera_settings({}, unit), optical_state()
|
||||
width = max(len(str(p.get("sensor_name"))) for p in props) + 1
|
||||
line = lambda p: f" {str(p.get('sensor_name')) + ':':{width}s} {{" + ", ".join( # noqa: E731
|
||||
f"{k}: {json.dumps(p[k])}" for k in PATTERN_KEYS if k in p) + "}"
|
||||
refs = [line(p) for p in props if p.get("pattern_type") == "global_ref"]
|
||||
targets = [line(p) for p in props if p.get("pattern_type") != "global_ref"]
|
||||
text = f"""# {name}: created on {datetime.date.today()} with the cloud's processing. Change a value, then run it:
|
||||
# processing/run_config.sh {src.name} {name}
|
||||
# The run also writes here targets.csv (as in data/, from these patterns) and patterns.json (the patterns exactly as
|
||||
# given to the emulator, templates included).
|
||||
emulator:
|
||||
version: "{state['commit'][:7]}" # optical's commit (branch {state['branch']}); the run stops if $OPTICAL is
|
||||
# at another one or has local changes
|
||||
patterns:
|
||||
# The {len(props)} patterns of the DIFF images, as in images_diff/sensors.json on {datetime.date.today()} (frozen here: a
|
||||
# later pumping does not change them). The template stays in sensors.json; to change it:
|
||||
# <name>: {{redraw: {{image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}}}}
|
||||
# pattern_type: global_ref = a reference of the camera fit, anything else a target. corners and x, y move together
|
||||
# (same window size as the template). resolution: 1/100 px. distance_convertion_factor: mm per px (corr_*_mm only).
|
||||
# references
|
||||
""" + "\n".join(refs) + "\n # targets\n" + "\n".join(targets) + "\ncamera:" + "".join(
|
||||
f"\n {f'{k}: {json.dumps(v)}':26s} # {CAMERA_NOTES[k]}" for k, v in cam.items()) + (
|
||||
"\n # focal_distance, pixel_size: the unit has none on the platform, the cloud's values"
|
||||
if not {"focal_distance", "pixel_size"} & set(unit) else "") + "\n"
|
||||
run.mkdir(parents=True, exist_ok=True)
|
||||
(run / "config.yaml").write_text(text)
|
||||
return run / "config.yaml", state
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("config", help="name of configs/<name>.yaml, or a path")
|
||||
p.add_argument("stations", nargs="*", help="serial numbers; none: those of the configuration")
|
||||
p.add_argument("station", help="serial number: the folder data_diff/<SERIAL>/")
|
||||
p.add_argument("run", help="the folder data_diff/<SERIAL>/runs/<run>/, holding config.yaml")
|
||||
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD")
|
||||
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD")
|
||||
p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)")
|
||||
p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff")
|
||||
p.add_argument("--out", type=Path, default=ROOT / "runs")
|
||||
p.add_argument("--new", action="store_true", help="create the run, empty: its config.yaml only (processing/new_run.sh)")
|
||||
a = p.parse_args()
|
||||
path = Path(a.config) if a.config.endswith((".yaml", ".yml")) or os.sep in a.config \
|
||||
else ROOT / "configs" / f"{a.config}.yaml"
|
||||
config, name = load_config(path), path.stem
|
||||
stations = a.stations or list(config.get("stations") or {})
|
||||
if not stations:
|
||||
sys.exit(f"No station: name one ({path} names none)")
|
||||
run = a.out / name
|
||||
(run / "diagnostics").mkdir(parents=True, exist_ok=True)
|
||||
record_path = run / "run.yaml"
|
||||
record = yaml.safe_load(record_path.read_text()) if record_path.exists() else {}
|
||||
record.update({"configuration": name, "source": str(path), "optical": optical_state()})
|
||||
record.setdefault("stations", {})
|
||||
src = a.data_diff / a.station
|
||||
run = src / "runs" / a.run
|
||||
if a.new:
|
||||
try:
|
||||
path, state = new_run(src, a.run)
|
||||
except FileExistsError as e:
|
||||
sys.exit(str(e))
|
||||
print(f"{path}: written{', but optical has local changes: the run will stop until they are committed' if state['dirty'] else ''}"
|
||||
f"\nRun it: processing/run_config.sh {a.station} {a.run}")
|
||||
return
|
||||
if not (run / "config.yaml").is_file():
|
||||
sys.exit(f"No {run / 'config.yaml'}: create the run first, processing/new_run.sh {a.station} {a.run}")
|
||||
config = load_config(run / "config.yaml")
|
||||
version = (config.get("emulator") or {}).get("version")
|
||||
if version is not None: # the code that makes the run is the one config.yaml names
|
||||
state = optical_state()
|
||||
want = subprocess.run(["git", "-C", str(OPTICAL), "rev-parse", "--verify", "--quiet", f"{version}^{{commit}}"],
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
if not want:
|
||||
sys.exit(f"emulator.version {version}: not a commit of {OPTICAL}")
|
||||
if want != state["commit"] or state["dirty"]:
|
||||
sys.exit(f"emulator.version {version}, but {OPTICAL} is at {state['commit'][:7]}"
|
||||
f"{' with local changes' if state['dirty'] else ''}: check {version} out, or change the version")
|
||||
|
||||
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{a.station}_diff_*.csv")
|
||||
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
|
||||
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
|
||||
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
|
||||
if not rows:
|
||||
sys.exit(f"{a.station}: no differential image indexed in the range, nothing done")
|
||||
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
|
||||
unit = json.loads((src / "images_diff" / "unit.json").read_text())
|
||||
cam = camera_settings(config, latest(unit["properties"]))
|
||||
props = apply_patterns(cloud_properties(sensors), config.get("patterns"), src / "images_diff")
|
||||
print(f"{a.station} {a.run}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} "
|
||||
f"patterns, camera {cam}, optical {optical_state()}")
|
||||
|
||||
import reprocess_images
|
||||
|
||||
for station in stations:
|
||||
src = a.data_diff / station
|
||||
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{station}_diff_*.csv")
|
||||
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
|
||||
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
|
||||
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
|
||||
if not rows:
|
||||
print(f"{station}: no differential image indexed in the range, nothing done")
|
||||
continue
|
||||
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
|
||||
unit = json.loads((src / "images_diff" / "unit.json").read_text())
|
||||
cam = camera_settings(config, station, latest(unit["properties"]))
|
||||
overrides = ((config.get("stations") or {}).get(station) or {}).get("patterns")
|
||||
props = apply_patterns(cloud_properties(sensors), overrides, src / "images_diff")
|
||||
print(f"{station}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} patterns, "
|
||||
f"camera {cam}")
|
||||
|
||||
diag = run / "diagnostics"
|
||||
sp, times, remeasured = (diag / f"{station}_sensor_properties.json", diag / f"{station}_times.csv",
|
||||
diag / f"{station}_remeasured.csv")
|
||||
sp.write_text(json.dumps(props))
|
||||
sp = run / "patterns.json" # kept: the target configuration the run was made with, templates included
|
||||
sp.write_text(json.dumps(props, indent=1))
|
||||
pd.DataFrame([{"sensor_name": p.get("sensor_name"), "pattern_type": p.get("pattern_type"), # pump.py's columns
|
||||
"distance_mm": p.get("distance_mm"), "mm_per_px": p.get("distance_convertion_factor")}
|
||||
for p in props.values()]).sort_values("sensor_name").to_csv(run / "targets.csv", index=False)
|
||||
with tempfile.TemporaryDirectory() as tmp: # the emulator's other files: not kept
|
||||
times, remeasured = Path(tmp) / "times.csv", Path(tmp) / "rows.csv"
|
||||
with open(times, "w", newline="") as fp:
|
||||
w = csv.writer(fp)
|
||||
w.writerow(["file", "time_utc"])
|
||||
@@ -284,42 +343,33 @@ def main():
|
||||
reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured),
|
||||
"--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows])
|
||||
adv, camera = correct(pd.read_csv(remeasured), props, cam)
|
||||
|
||||
dest = run / station
|
||||
if dest.exists():
|
||||
assert dest.resolve().parent == run.resolve(), dest # only runs/<name>/<SERIAL> is ever removed
|
||||
shutil.rmtree(dest)
|
||||
write_station(adv, camera, props, dest, station)
|
||||
for sub in LINKED:
|
||||
if (src / sub).exists():
|
||||
os.symlink(os.path.relpath(src / sub, dest), dest / sub)
|
||||
record["stations"][station] = {
|
||||
"days": [days[0].stem[-8:], days[-1].stem[-8:]], "images": len(rows), "camera": cam,
|
||||
"patterns": overrides or {}, "date": datetime.now(timezone.utc).isoformat(timespec="seconds")}
|
||||
print(f"{station}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows -> {dest}")
|
||||
record_path.write_text(yaml.safe_dump(record, sort_keys=False, allow_unicode=True))
|
||||
write_run(adv, camera, run, a.station)
|
||||
print(f"{a.station} {a.run}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows "
|
||||
f"-> {run}")
|
||||
|
||||
|
||||
def check():
|
||||
"""`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical."""
|
||||
import tempfile
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
f = Path(tmp) / "c.yaml"
|
||||
for text in ("camera: {algoritm: ransac}", "stations: {S: {pattern: {}}}", "stations: {S: {camera: {mod: x}}}",
|
||||
"other: 1"):
|
||||
f = Path(tmp) / "config.yaml"
|
||||
for text in ("camera: {algoritm: ransac}", "pattern: {}", "stations: {S: {}}", "patterns: [PM_G]",
|
||||
"emulator: {versio: 65a4c70}"):
|
||||
f.write_text(text)
|
||||
try:
|
||||
load_config(f)
|
||||
raise AssertionError(f"accepted: {text}")
|
||||
except ValueError:
|
||||
pass
|
||||
f.write_text("camera: {mode: rotation, focal_distance: 6.1}\nstations: {S: {camera: {focal_distance: 6.3}}}")
|
||||
f.write_text("")
|
||||
assert load_config(f) == {} # an empty file: the cloud's processing
|
||||
f.write_text("camera: {mode: rotation, focal_distance: 6.3}\npatterns: {T1: {error_threshold: 0.9}}\n"
|
||||
"emulator: {version: '65a4c70'}")
|
||||
c = load_config(f)
|
||||
cam = camera_settings(c, "S", {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
|
||||
assert c["emulator"]["version"] == "65a4c70"
|
||||
cam = camera_settings(c, {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
|
||||
assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation",
|
||||
"ransac"), cam
|
||||
assert camera_settings({}, "S", {})["focal_distance"] == 6.0
|
||||
assert camera_settings({}, {})["focal_distance"] == 6.0
|
||||
|
||||
pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)}
|
||||
props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y,
|
||||
|
||||
Reference in new issue
Block a user