Files
aimsight-data/docs/superpowers/plans/2026-11-19-pump-diff.md
T
maximeandClaude Opus 5.5 81908ca370 Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)
The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every
measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived
included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into
data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now
returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF
images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 09:59:50 +01:00

87 lines
5.2 KiB
Markdown

# Pumping the differential images Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** `extraction/pump_diff.py` + `pump_diff.sh` pump a station's `OPTICAL_SHIFT-DIFF` images, their index and
the pattern / unit properties into `data_diff/<SERIAL>/`, read-only, without touching `data/` or `pump.py`.
**Architecture:** One script beside `pump.py`, importing `acquisition_unit`, `root_cause`, `DB_NAMES`, `TYPES`, `UTC`
from it; pure range logic and a per-day function testable through stubs; a wrapper copied from `pump.sh` minus
`absolute.py` / `weather.py`.
**Tech Stack:** Python 3.12 in `uv` (sqlalchemy, psycopg2-binary, pandas, boto3, as `pump.sh`), standard `csv` /
`json`.
**Spec:** `docs/superpowers/specs/2026-11-19-pump-diff-design.md`
## Global Constraints
- Read-only: `default_transaction_read_only=on`, `statement_timeout=--delay` (default 60 s), as `pump.py`.
- Never read or print the `.env`; variable names only.
- Image names `<YYYYMMDD>T<HHMM>Z_<measurement_id>.<ext>`, as `pump.py`.
- `--min-free-gb` default 50, checked before each day.
- Lock `out/pump_diff.lock`, log `out/pump_diff.log`.
## Review Focus
- A crash in the middle of a day: no index file for it (temporary file + `os.replace`), so the next run reads it again.
- Two measurements in the same minute (bursts): distinct files thanks to the id; both in the index.
- A station without acquisition unit or unknown serial: a message, the next station goes on.
- No `--from` and nothing indexed: the station is skipped with a message, nothing downloaded.
- Disk below the limit: the run stops before the day, with the free space printed.
---
### Task 1: range logic and one day, with `--check`
**Files:**
- Create: `extraction/pump_diff.py`
**Interfaces:**
- Produces: `indexed_days(folder: Path) -> set[date]`; `days_to_pump(indexed: set[date], start: date | None, end: date)
-> list[date] | None` (None: no `--from` and nothing indexed); `pump_day(rows, client, bucket: str, ext: str,
folder: Path, station: str, day: date) -> Counter` (rows: the day's measurements, queried by `main`; counts per
status, plus `new` and `bytes`); `check()`.
- [ ] **Step 1:** write `check()` with these assertions (days 1..19 of 2026-11):
- `days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)]`;
- `days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)]` (before the last: missing only; the
last and after: again);
- `days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)]`; `days_to_pump(set(), None, d(5)) is None`;
- `pump_day` on stub `conn` / `client` with 4 measurements (one image to fetch, one already on disk, one
`measurement_uuid` None, one where `get_object` raises): returns `{"ok": 2, "no_uuid": 1, "s3_missing": 1}`, writes
one new file, leaves the existing one untouched, and the index has 4 rows with the right `status`, `file` empty for
the two missing, `properties` as JSON.
- [ ] **Step 2:** `python3 extraction/pump_diff.py --check` fails (functions missing).
- [ ] **Step 3:** implement the three functions. `pump_day` queries `id, time_utc, measurement_group_id,
measurement_uuid, properties` of `measurement` for the unit, type and UTC day, ordered by time; index columns
`measurement_id, time_utc, measurement_group_id, file, status, properties`; written to `index/<S>_diff_<YYYYMMDD>.csv`
through `<name>.tmp` and `os.replace` after the images.
- [ ] **Step 4:** `--check` prints `ok`.
### Task 2: database wiring, wrapper, docs, real trial
**Files:**
- Modify: `extraction/pump_diff.py` (add `sensors_json`, `unit_json`, `main`)
- Create: `extraction/pump_diff.sh`
- Modify: `extraction/README.md`, `README.md`, `.gitignore` (`data_diff/`)
**Interfaces:**
- Consumes: Task 1's functions.
- Produces: `sensors_json(conn, unit: int) -> list[dict]` (`id, name, type, time_utc, properties: [{name, value,
time_utc, status}]`, every sensor of the unit, archived included); `unit_json(conn, unit: int) -> dict` (`id,
serial_number, product_number, properties: [...]`).
- [ ] **Step 1:** `main()`: arguments `stations*`, `--from`, `--to`, `--delay`, `--min-free-gb`, `--out` (default
`<repo>/data_diff`); without stations, the folders of `--out`; engine and S3 client as `pump.py`'s `main`; per
station: system, unit, measurement type `OPTICAL_SHIFT-DIFF`, `sensors.json` / `unit.json` written, then each day of
`days_to_pump` after the disk check, one line printed per day (counts, MB).
- [ ] **Step 2:** `pump_diff.sh` from `pump.sh`: same `.env` load and `uv` packages, its own lock and log, no
`absolute.py` / `weather.py`, no default arguments.
- [ ] **Step 3:** `--check` still `ok`; `pump_diff.sh -h` shows the help.
- [ ] **Step 4:** real trial: `extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-02 JUNGFRAU`; expected: 94
images, byte-identical to `data/JUNGFRAU/images_diff/20260901*` (`cmp`), index of the day, `sensors.json`,
`unit.json`; `data/` unchanged (`find data -newer` the start mark finds nothing).
- [ ] **Step 5:** docs: a `pump_diff` section in `extraction/README.md`, one line in `README.md`, `data_diff/` in
`.gitignore`.