# Pumping the differential images Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** `extraction/pump_diff.py` + `pump_diff.sh` pump a station's `OPTICAL_SHIFT-DIFF` images, their index and the pattern / unit properties into `data_diff//`, read-only, without touching `data/` or `pump.py`. **Architecture:** One script beside `pump.py`, importing `acquisition_unit`, `root_cause`, `DB_NAMES`, `TYPES`, `UTC` from it; pure range logic and a per-day function testable through stubs; a wrapper copied from `pump.sh` minus `absolute.py` / `weather.py`. **Tech Stack:** Python 3.12 in `uv` (sqlalchemy, psycopg2-binary, pandas, boto3, as `pump.sh`), standard `csv` / `json`. **Spec:** `docs/superpowers/specs/2026-11-19-pump-diff-design.md` ## Global Constraints - Read-only: `default_transaction_read_only=on`, `statement_timeout=--delay` (default 60 s), as `pump.py`. - Never read or print the `.env`; variable names only. - Image names `TZ_.`, as `pump.py`. - `--min-free-gb` default 50, checked before each day. - Lock `out/pump_diff.lock`, log `out/pump_diff.log`. ## Review Focus - A crash in the middle of a day: no index file for it (temporary file + `os.replace`), so the next run reads it again. - Two measurements in the same minute (bursts): distinct files thanks to the id; both in the index. - A station without acquisition unit or unknown serial: a message, the next station goes on. - No `--from` and nothing indexed: the station is skipped with a message, nothing downloaded. - Disk below the limit: the run stops before the day, with the free space printed. --- ### Task 1: range logic and one day, with `--check` **Files:** - Create: `extraction/pump_diff.py` **Interfaces:** - Produces: `indexed_days(folder: Path) -> set[date]`; `days_to_pump(indexed: set[date], start: date | None, end: date) -> list[date] | None` (None: no `--from` and nothing indexed); `pump_day(rows, client, bucket: str, ext: str, folder: Path, station: str, day: date) -> Counter` (rows: the day's measurements, queried by `main`; counts per status, plus `new` and `bytes`); `check()`. - [ ] **Step 1:** write `check()` with these assertions (days 1..19 of 2026-11): - `days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)]`; - `days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)]` (before the last: missing only; the last and after: again); - `days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)]`; `days_to_pump(set(), None, d(5)) is None`; - `pump_day` on stub `conn` / `client` with 4 measurements (one image to fetch, one already on disk, one `measurement_uuid` None, one where `get_object` raises): returns `{"ok": 2, "no_uuid": 1, "s3_missing": 1}`, writes one new file, leaves the existing one untouched, and the index has 4 rows with the right `status`, `file` empty for the two missing, `properties` as JSON. - [ ] **Step 2:** `python3 extraction/pump_diff.py --check` fails (functions missing). - [ ] **Step 3:** implement the three functions. `pump_day` queries `id, time_utc, measurement_group_id, measurement_uuid, properties` of `measurement` for the unit, type and UTC day, ordered by time; index columns `measurement_id, time_utc, measurement_group_id, file, status, properties`; written to `index/_diff_.csv` through `.tmp` and `os.replace` after the images. - [ ] **Step 4:** `--check` prints `ok`. ### Task 2: database wiring, wrapper, docs, real trial **Files:** - Modify: `extraction/pump_diff.py` (add `sensors_json`, `unit_json`, `main`) - Create: `extraction/pump_diff.sh` - Modify: `extraction/README.md`, `README.md`, `.gitignore` (`data_diff/`) **Interfaces:** - Consumes: Task 1's functions. - Produces: `sensors_json(conn, unit: int) -> list[dict]` (`id, name, type, time_utc, properties: [{name, value, time_utc, status}]`, every sensor of the unit, archived included); `unit_json(conn, unit: int) -> dict` (`id, serial_number, product_number, properties: [...]`). - [ ] **Step 1:** `main()`: arguments `stations*`, `--from`, `--to`, `--delay`, `--min-free-gb`, `--out` (default `/data_diff`); without stations, the folders of `--out`; engine and S3 client as `pump.py`'s `main`; per station: system, unit, measurement type `OPTICAL_SHIFT-DIFF`, `sensors.json` / `unit.json` written, then each day of `days_to_pump` after the disk check, one line printed per day (counts, MB). - [ ] **Step 2:** `pump_diff.sh` from `pump.sh`: same `.env` load and `uv` packages, its own lock and log, no `absolute.py` / `weather.py`, no default arguments. - [ ] **Step 3:** `--check` still `ok`; `pump_diff.sh -h` shows the help. - [ ] **Step 4:** real trial: `extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-02 JUNGFRAU`; expected: 94 images, byte-identical to `data/JUNGFRAU/images_diff/20260901*` (`cmp`), index of the day, `sensors.json`, `unit.json`; `data/` unchanged (`find data -newer` the start mark finds nothing). - [ ] **Step 5:** docs: a `pump_diff` section in `extraction/README.md`, one line in `README.md`, `data_diff/` in `.gitignore`.