The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.2 KiB
Pumping the differential images Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: extraction/pump_diff.py + pump_diff.sh pump a station's OPTICAL_SHIFT-DIFF images, their index and
the pattern / unit properties into data_diff/<SERIAL>/, read-only, without touching data/ or pump.py.
Architecture: One script beside pump.py, importing acquisition_unit, root_cause, DB_NAMES, TYPES, UTC
from it; pure range logic and a per-day function testable through stubs; a wrapper copied from pump.sh minus
absolute.py / weather.py.
Tech Stack: Python 3.12 in uv (sqlalchemy, psycopg2-binary, pandas, boto3, as pump.sh), standard csv /
json.
Spec: docs/superpowers/specs/2026-11-19-pump-diff-design.md
Global Constraints
- Read-only:
default_transaction_read_only=on,statement_timeout=--delay(default 60 s), aspump.py. - Never read or print the
.env; variable names only. - Image names
<YYYYMMDD>T<HHMM>Z_<measurement_id>.<ext>, aspump.py. --min-free-gbdefault 50, checked before each day.- Lock
out/pump_diff.lock, logout/pump_diff.log.
Review Focus
- A crash in the middle of a day: no index file for it (temporary file +
os.replace), so the next run reads it again. - Two measurements in the same minute (bursts): distinct files thanks to the id; both in the index.
- A station without acquisition unit or unknown serial: a message, the next station goes on.
- No
--fromand nothing indexed: the station is skipped with a message, nothing downloaded. - Disk below the limit: the run stops before the day, with the free space printed.
Task 1: range logic and one day, with --check
Files:
- Create:
extraction/pump_diff.py
Interfaces:
-
Produces:
indexed_days(folder: Path) -> set[date];days_to_pump(indexed: set[date], start: date | None, end: date) -> list[date] | None(None: no--fromand nothing indexed);pump_day(rows, client, bucket: str, ext: str, folder: Path, station: str, day: date) -> Counter(rows: the day's measurements, queried bymain; counts per status, plusnewandbytes);check(). -
Step 1: write
check()with these assertions (days 1..19 of 2026-11):days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)];days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)](before the last: missing only; the last and after: again);days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)];days_to_pump(set(), None, d(5)) is None;pump_dayon stubconn/clientwith 4 measurements (one image to fetch, one already on disk, onemeasurement_uuidNone, one whereget_objectraises): returns{"ok": 2, "no_uuid": 1, "s3_missing": 1}, writes one new file, leaves the existing one untouched, and the index has 4 rows with the rightstatus,fileempty for the two missing,propertiesas JSON.
-
Step 2:
python3 extraction/pump_diff.py --checkfails (functions missing). -
Step 3: implement the three functions.
pump_dayqueriesid, time_utc, measurement_group_id, measurement_uuid, propertiesofmeasurementfor the unit, type and UTC day, ordered by time; index columnsmeasurement_id, time_utc, measurement_group_id, file, status, properties; written toindex/<S>_diff_<YYYYMMDD>.csvthrough<name>.tmpandos.replaceafter the images. -
Step 4:
--checkprintsok.
Task 2: database wiring, wrapper, docs, real trial
Files:
- Modify:
extraction/pump_diff.py(addsensors_json,unit_json,main) - Create:
extraction/pump_diff.sh - Modify:
extraction/README.md,README.md,.gitignore(data_diff/)
Interfaces:
-
Consumes: Task 1's functions.
-
Produces:
sensors_json(conn, unit: int) -> list[dict](id, name, type, time_utc, properties: [{name, value, time_utc, status}], every sensor of the unit, archived included);unit_json(conn, unit: int) -> dict(id, serial_number, product_number, properties: [...]). -
Step 1:
main(): argumentsstations*,--from,--to,--delay,--min-free-gb,--out(default<repo>/data_diff); without stations, the folders of--out; engine and S3 client aspump.py'smain; per station: system, unit, measurement typeOPTICAL_SHIFT-DIFF,sensors.json/unit.jsonwritten, then each day ofdays_to_pumpafter the disk check, one line printed per day (counts, MB). -
Step 2:
pump_diff.shfrompump.sh: same.envload anduvpackages, its own lock and log, noabsolute.py/weather.py, no default arguments. -
Step 3:
--checkstillok;pump_diff.sh -hshows the help. -
Step 4: real trial:
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-02 JUNGFRAU; expected: 94 images, byte-identical todata/JUNGFRAU/images_diff/20260901*(cmp), index of the day,sensors.json,unit.json;data/unchanged (find data -newerthe start mark finds nothing). -
Step 5: docs: a
pump_diffsection inextraction/README.md, one line inREADME.md,data_diff/in.gitignore.