Files
aimsight-data/docs/superpowers/plans/2026-11-19-pump-diff.md
T
maximeandClaude Opus 5.5 81908ca370 Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)
The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every
measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived
included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into
data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now
returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF
images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 09:59:50 +01:00

5.2 KiB

Pumping the differential images Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: extraction/pump_diff.py + pump_diff.sh pump a station's OPTICAL_SHIFT-DIFF images, their index and the pattern / unit properties into data_diff/<SERIAL>/, read-only, without touching data/ or pump.py.

Architecture: One script beside pump.py, importing acquisition_unit, root_cause, DB_NAMES, TYPES, UTC from it; pure range logic and a per-day function testable through stubs; a wrapper copied from pump.sh minus absolute.py / weather.py.

Tech Stack: Python 3.12 in uv (sqlalchemy, psycopg2-binary, pandas, boto3, as pump.sh), standard csv / json.

Spec: docs/superpowers/specs/2026-11-19-pump-diff-design.md

Global Constraints

  • Read-only: default_transaction_read_only=on, statement_timeout=--delay (default 60 s), as pump.py.
  • Never read or print the .env; variable names only.
  • Image names <YYYYMMDD>T<HHMM>Z_<measurement_id>.<ext>, as pump.py.
  • --min-free-gb default 50, checked before each day.
  • Lock out/pump_diff.lock, log out/pump_diff.log.

Review Focus

  • A crash in the middle of a day: no index file for it (temporary file + os.replace), so the next run reads it again.
  • Two measurements in the same minute (bursts): distinct files thanks to the id; both in the index.
  • A station without acquisition unit or unknown serial: a message, the next station goes on.
  • No --from and nothing indexed: the station is skipped with a message, nothing downloaded.
  • Disk below the limit: the run stops before the day, with the free space printed.

Task 1: range logic and one day, with --check

Files:

  • Create: extraction/pump_diff.py

Interfaces:

  • Produces: indexed_days(folder: Path) -> set[date]; days_to_pump(indexed: set[date], start: date | None, end: date) -> list[date] | None (None: no --from and nothing indexed); pump_day(rows, client, bucket: str, ext: str, folder: Path, station: str, day: date) -> Counter (rows: the day's measurements, queried by main; counts per status, plus new and bytes); check().

  • Step 1: write check() with these assertions (days 1..19 of 2026-11):

    • days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)];
    • days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)] (before the last: missing only; the last and after: again);
    • days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)]; days_to_pump(set(), None, d(5)) is None;
    • pump_day on stub conn / client with 4 measurements (one image to fetch, one already on disk, one measurement_uuid None, one where get_object raises): returns {"ok": 2, "no_uuid": 1, "s3_missing": 1}, writes one new file, leaves the existing one untouched, and the index has 4 rows with the right status, file empty for the two missing, properties as JSON.
  • Step 2: python3 extraction/pump_diff.py --check fails (functions missing).

  • Step 3: implement the three functions. pump_day queries id, time_utc, measurement_group_id, measurement_uuid, properties of measurement for the unit, type and UTC day, ordered by time; index columns measurement_id, time_utc, measurement_group_id, file, status, properties; written to index/<S>_diff_<YYYYMMDD>.csv through <name>.tmp and os.replace after the images.

  • Step 4: --check prints ok.

Task 2: database wiring, wrapper, docs, real trial

Files:

  • Modify: extraction/pump_diff.py (add sensors_json, unit_json, main)
  • Create: extraction/pump_diff.sh
  • Modify: extraction/README.md, README.md, .gitignore (data_diff/)

Interfaces:

  • Consumes: Task 1's functions.

  • Produces: sensors_json(conn, unit: int) -> list[dict] (id, name, type, time_utc, properties: [{name, value, time_utc, status}], every sensor of the unit, archived included); unit_json(conn, unit: int) -> dict (id, serial_number, product_number, properties: [...]).

  • Step 1: main(): arguments stations*, --from, --to, --delay, --min-free-gb, --out (default <repo>/data_diff); without stations, the folders of --out; engine and S3 client as pump.py's main; per station: system, unit, measurement type OPTICAL_SHIFT-DIFF, sensors.json / unit.json written, then each day of days_to_pump after the disk check, one line printed per day (counts, MB).

  • Step 2: pump_diff.sh from pump.sh: same .env load and uv packages, its own lock and log, no absolute.py / weather.py, no default arguments.

  • Step 3: --check still ok; pump_diff.sh -h shows the help.

  • Step 4: real trial: extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-02 JUNGFRAU; expected: 94 images, byte-identical to data/JUNGFRAU/images_diff/20260901* (cmp), index of the day, sensors.json, unit.json; data/ unchanged (find data -newer the start mark finds nothing).

  • Step 5: docs: a pump_diff section in extraction/README.md, one line in README.md, data_diff/ in .gitignore.