Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it
- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/, weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv, patterns.json. - processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the patterns of sensors.json, the camera, optical's current commit); never over an existing one. - processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns) and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/. - pump_diff no longer writes targets.csv: it depends on the run. JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within 0.034 px; a run created by new_run.sh and run as is gives run01 exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d3bf4b6860
commit
ca366c150e
8 files changed
+185
-123
No files matched your search
+5
-24
@@ -18,8 +18,9 @@ read it the same way (docs/superpowers/specs/2026-11-19-configurations-design.md
|
||||
time_utc, status}] (every property row as stored), rewritten at each run
|
||||
<STATION>/images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties
|
||||
(under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder)
|
||||
<STATION>/metrics/, images/ (IRON, one per --every minutes), targets.csv (the DIFF patterns), weather/: as pump.py
|
||||
and weather.py write them into data/, with their own functions
|
||||
<STATION>/metrics/, images/ (IRON, one per --every minutes), weather/: as pump.py and weather.py write them into
|
||||
data/, with their own functions. Not targets.csv: it depends on the run (its patterns), each run writes its own
|
||||
(processing/run_config.py)
|
||||
|
||||
Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and
|
||||
the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index
|
||||
@@ -138,17 +139,6 @@ def unit_json(conn, unit):
|
||||
"properties": [prop(*r) for r in rows]}
|
||||
|
||||
|
||||
def diff_patterns(sensors):
|
||||
"""{sensor_name: sensor_id} of the patterns the DIFF images are measured with (sensors_json rows): type
|
||||
OPTICAL_SHIFT-DIFF, a pattern, not archived; for targets.csv."""
|
||||
out = {}
|
||||
for s in sensors:
|
||||
p = {r["name"]: r["value"] for r in s["properties"]}
|
||||
if s["type"] == MTYPE and p.get("pattern") and not p.get("archive"):
|
||||
out[p.get("sensor_name") or s["name"]] = s["id"]
|
||||
return out
|
||||
|
||||
|
||||
def to_json(obj):
|
||||
return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode()
|
||||
|
||||
@@ -172,7 +162,7 @@ def main():
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.engine import URL
|
||||
|
||||
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause, sensors, write_targets
|
||||
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause
|
||||
|
||||
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
|
||||
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
|
||||
@@ -224,11 +214,8 @@ def main():
|
||||
sens = sensors_json(conn, unit)
|
||||
write_atomic(folder / "images_diff" / "sensors.json", to_json(sens))
|
||||
write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit)))
|
||||
t = write_targets(folder, sensors(conn, unit), diff_patterns(sens))
|
||||
print(f"{station}: targets.csv, {len(t)} sensors")
|
||||
except Exception as e: # the next run writes them
|
||||
print(f"{station} images range, sensors.json, unit.json, targets.csv: failed ({root_cause(e)}), "
|
||||
"run again")
|
||||
print(f"{station} images range, sensors.json, unit.json: failed ({root_cause(e)}), run again")
|
||||
finally:
|
||||
conn.rollback()
|
||||
print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}")
|
||||
@@ -313,12 +300,6 @@ def check():
|
||||
assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}'
|
||||
assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index
|
||||
assert indexed_days(folder) == {d(3)}
|
||||
row = lambda name, typ, **p: {"name": name, "type": typ, "id": len(p), # noqa: E731
|
||||
"properties": [{"name": k, "value": v} for k, v in p.items()]}
|
||||
assert diff_patterns([row("u1", MTYPE, pattern="b64", sensor_name="T1"), row("u2", MTYPE, pattern="b64"),
|
||||
row("u3", MTYPE, pattern="b64", sensor_name="OLD", archive="2026-01-01"),
|
||||
row("u4", "camera_movement", sensor_name="CAM"), row("u5", MTYPE, sensor_name="NOPAT")]) \
|
||||
== {"T1": 2, "u2": 1}, "DIFF patterns: by sensor_name, else the sensor's name; archived and others left out"
|
||||
print("ok")
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user