Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it

- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/,
  weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv,
  patterns.json.
- processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the
  patterns of sensors.json, the camera, optical's current commit); never over an existing one.
- processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked
  against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns)
  and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/.
- pump_diff no longer writes targets.csv: it depends on the run.

JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within
0.034 px; a run created by new_run.sh and run as is gives run01 exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-19 10:52:46 +01:00
1 parent d3bf4b6860
commit ca366c150e
8 files changed
+185 -123

No files matched your search

+5 -24
View File
@@ -18,8 +18,9 @@ read it the same way (docs/superpowers/specs/2026-11-19-configurations-design.md
time_utc, status}] (every property row as stored), rewritten at each run
<STATION>/images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties
(under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder)
<STATION>/metrics/, images/ (IRON, one per --every minutes), targets.csv (the DIFF patterns), weather/: as pump.py
and weather.py write them into data/, with their own functions
<STATION>/metrics/, images/ (IRON, one per --every minutes), weather/: as pump.py and weather.py write them into
data/, with their own functions. Not targets.csv: it depends on the run (its patterns), each run writes its own
(processing/run_config.py)
Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and
the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index
@@ -138,17 +139,6 @@ def unit_json(conn, unit):
"properties": [prop(*r) for r in rows]}
def diff_patterns(sensors):
"""{sensor_name: sensor_id} of the patterns the DIFF images are measured with (sensors_json rows): type
OPTICAL_SHIFT-DIFF, a pattern, not archived; for targets.csv."""
out = {}
for s in sensors:
p = {r["name"]: r["value"] for r in s["properties"]}
if s["type"] == MTYPE and p.get("pattern") and not p.get("archive"):
out[p.get("sensor_name") or s["name"]] = s["id"]
return out
def to_json(obj):
return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode()
@@ -172,7 +162,7 @@ def main():
from sqlalchemy import create_engine, text
from sqlalchemy.engine import URL
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause, sensors, write_targets
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
@@ -224,11 +214,8 @@ def main():
sens = sensors_json(conn, unit)
write_atomic(folder / "images_diff" / "sensors.json", to_json(sens))
write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit)))
t = write_targets(folder, sensors(conn, unit), diff_patterns(sens))
print(f"{station}: targets.csv, {len(t)} sensors")
except Exception as e: # the next run writes them
print(f"{station} images range, sensors.json, unit.json, targets.csv: failed ({root_cause(e)}), "
"run again")
print(f"{station} images range, sensors.json, unit.json: failed ({root_cause(e)}), run again")
finally:
conn.rollback()
print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}")
@@ -313,12 +300,6 @@ def check():
assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}'
assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index
assert indexed_days(folder) == {d(3)}
row = lambda name, typ, **p: {"name": name, "type": typ, "id": len(p), # noqa: E731
"properties": [{"name": k, "value": v} for k, v in p.items()]}
assert diff_patterns([row("u1", MTYPE, pattern="b64", sensor_name="T1"), row("u2", MTYPE, pattern="b64"),
row("u3", MTYPE, pattern="b64", sensor_name="OLD", archive="2026-01-01"),
row("u4", "camera_movement", sensor_name="CAM"), row("u5", MTYPE, sensor_name="NOPAT")]) \
== {"T1": 2, "u2": 1}, "DIFF patterns: by sensor_name, else the sensor's name; archived and others left out"
print("ok")