Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it
- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/, weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv, patterns.json. - processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the patterns of sensors.json, the camera, optical's current commit); never over an existing one. - processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns) and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/. - pump_diff no longer writes targets.csv: it depends on the run. JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within 0.034 px; a run created by new_run.sh and run as is gives run01 exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d3bf4b6860
commit
ca366c150e
8 files changed
+185
-123
No files matched your search
@@ -58,7 +58,7 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small
|
||||
|
||||
## The differential images, into `data_diff/` (`pump_diff.sh`)
|
||||
|
||||
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the configurations, `runs/<name>/`, read like `data/`).
|
||||
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the runs, `data_diff/<SERIAL>/runs/<run>/`).
|
||||
|
||||
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is;
|
||||
- `images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON);
|
||||
@@ -66,7 +66,7 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small
|
||||
- `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm);
|
||||
- (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder)
|
||||
- `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`;
|
||||
- `targets.csv`: as in `data/`, for the patterns the DIFF images are measured with (not archived); `displacement/` and `camera/` come only from the recomputation of a configuration (`runs/<name>/`), never from the cloud.
|
||||
- `displacement/`, `camera/` and `targets.csv` are not pumped: they depend on the run (its patterns, its settings), each run writes its own in `runs/<run>/` (`processing/run_config.sh`), never from the cloud; a pumping never touches `runs/`.
|
||||
|
||||
```bash
|
||||
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here
|
||||
@@ -74,6 +74,6 @@ extraction/pump_diff.sh JUNGFRAU # from its
|
||||
extraction/pump_diff.sh # the same for every station of data_diff/
|
||||
```
|
||||
|
||||
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`; `targets.csv` the same without the two non-DIFF sensors `target_01`, `target_02`.
|
||||
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`.
|
||||
|
||||
Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.
|
||||
+5
-24
@@ -18,8 +18,9 @@ read it the same way (docs/superpowers/specs/2026-11-19-configurations-design.md
|
||||
time_utc, status}] (every property row as stored), rewritten at each run
|
||||
<STATION>/images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties
|
||||
(under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder)
|
||||
<STATION>/metrics/, images/ (IRON, one per --every minutes), targets.csv (the DIFF patterns), weather/: as pump.py
|
||||
and weather.py write them into data/, with their own functions
|
||||
<STATION>/metrics/, images/ (IRON, one per --every minutes), weather/: as pump.py and weather.py write them into
|
||||
data/, with their own functions. Not targets.csv: it depends on the run (its patterns), each run writes its own
|
||||
(processing/run_config.py)
|
||||
|
||||
Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and
|
||||
the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index
|
||||
@@ -138,17 +139,6 @@ def unit_json(conn, unit):
|
||||
"properties": [prop(*r) for r in rows]}
|
||||
|
||||
|
||||
def diff_patterns(sensors):
|
||||
"""{sensor_name: sensor_id} of the patterns the DIFF images are measured with (sensors_json rows): type
|
||||
OPTICAL_SHIFT-DIFF, a pattern, not archived; for targets.csv."""
|
||||
out = {}
|
||||
for s in sensors:
|
||||
p = {r["name"]: r["value"] for r in s["properties"]}
|
||||
if s["type"] == MTYPE and p.get("pattern") and not p.get("archive"):
|
||||
out[p.get("sensor_name") or s["name"]] = s["id"]
|
||||
return out
|
||||
|
||||
|
||||
def to_json(obj):
|
||||
return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode()
|
||||
|
||||
@@ -172,7 +162,7 @@ def main():
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.engine import URL
|
||||
|
||||
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause, sensors, write_targets
|
||||
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause
|
||||
|
||||
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
|
||||
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
|
||||
@@ -224,11 +214,8 @@ def main():
|
||||
sens = sensors_json(conn, unit)
|
||||
write_atomic(folder / "images_diff" / "sensors.json", to_json(sens))
|
||||
write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit)))
|
||||
t = write_targets(folder, sensors(conn, unit), diff_patterns(sens))
|
||||
print(f"{station}: targets.csv, {len(t)} sensors")
|
||||
except Exception as e: # the next run writes them
|
||||
print(f"{station} images range, sensors.json, unit.json, targets.csv: failed ({root_cause(e)}), "
|
||||
"run again")
|
||||
print(f"{station} images range, sensors.json, unit.json: failed ({root_cause(e)}), run again")
|
||||
finally:
|
||||
conn.rollback()
|
||||
print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}")
|
||||
@@ -313,12 +300,6 @@ def check():
|
||||
assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}'
|
||||
assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index
|
||||
assert indexed_days(folder) == {d(3)}
|
||||
row = lambda name, typ, **p: {"name": name, "type": typ, "id": len(p), # noqa: E731
|
||||
"properties": [{"name": k, "value": v} for k, v in p.items()]}
|
||||
assert diff_patterns([row("u1", MTYPE, pattern="b64", sensor_name="T1"), row("u2", MTYPE, pattern="b64"),
|
||||
row("u3", MTYPE, pattern="b64", sensor_name="OLD", archive="2026-01-01"),
|
||||
row("u4", "camera_movement", sensor_name="CAM"), row("u5", MTYPE, sensor_name="NOPAT")]) \
|
||||
== {"T1": 2, "u2": 1}, "DIFF patterns: by sensor_name, else the sensor's name; archived and others left out"
|
||||
print("ok")
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user