Files

86 lines
5.2 KiB
Markdown
Raw Permalink Normal View History

# Pumping the differential images Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** `extraction/pump_diff.py` + `pump_diff.sh` pump a station's `OPTICAL_SHIFT-DIFF` images, their index and
the pattern / unit properties into `data_diff/<SERIAL>/`, read-only, without touching `data/` or `pump.py`.
**Architecture:** One script beside `pump.py`, importing `acquisition_unit`, `root_cause`, `DB_NAMES`, `TYPES`, `UTC`
from it; pure range logic and a per-day function testable through stubs; a wrapper copied from `pump.sh` minus
`absolute.py` / `weather.py`.
**Tech Stack:** Python 3.12 in `uv` (sqlalchemy, psycopg2-binary, pandas, boto3, as `pump.sh`), standard `csv` /
`json`.
**Spec:** `docs/superpowers/specs/2026-11-19-pump-diff-design.md`
## Global Constraints
- Read-only: `default_transaction_read_only=on`, `statement_timeout=--delay` (default 60 s), as `pump.py`.
- Never read or print the `.env`; variable names only.
- Image names `<YYYYMMDD>T<HHMM>Z_<measurement_id>.<ext>`, as `pump.py`.
- `--min-free-gb` default 50, checked before each day.
- Lock `out/pump_diff.lock`, log `out/pump_diff.log`.
## Review Focus
- A crash in the middle of a day: no index file for it (temporary file + `os.replace`), so the next run reads it again.
- Two measurements in the same minute (bursts): distinct files thanks to the id; both in the index.
- A station without acquisition unit or unknown serial: a message, the next station goes on.
- No `--from` and nothing indexed: the station is skipped with a message, nothing downloaded.
- Disk below the limit: the run stops before the day, with the free space printed.
---
### Task 1: range logic and one day, with `--check`
**Files:**
- Create: `extraction/pump_diff.py`
**Interfaces:**
- Produces: `indexed_days(folder: Path) -> set[date]`; `days_to_pump(indexed: set[date], start: date | None, end: date)
-> list[date] | None` (None: no `--from` and nothing indexed); `pump_day(rows, client, bucket: str, ext: str,
folder: Path, station: str, day: date) -> Counter` (rows: the day's measurements, queried by `main`; counts per
status, plus `new` and `bytes`); `check()`.
- [ ] **Step 1:** write `check()` with these assertions (days 1..19 of 2026-11):
- `days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)]`;
- `days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)]` (before the last: missing only; the
last and after: again);
- `days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)]`; `days_to_pump(set(), None, d(5)) is None`;
- `pump_day` on stub `conn` / `client` with 4 measurements (one image to fetch, one already on disk, one
`measurement_uuid` None, one where `get_object` raises): returns `{"ok": 2, "no_uuid": 1, "s3_missing": 1}`, writes
one new file, leaves the existing one untouched, and the index has 4 rows with the right `status`, `file` empty for
the two missing, `properties` as JSON.
- [ ] **Step 2:** `python3 extraction/pump_diff.py --check` fails (functions missing).
- [ ] **Step 3:** implement the three functions. `pump_day` queries `id, time_utc, measurement_group_id,
measurement_uuid, properties` of `measurement` for the unit, type and UTC day, ordered by time; index columns
`measurement_id, time_utc, measurement_group_id, file, status, properties`; written to `index/<S>_diff_<YYYYMMDD>.csv`
through `<name>.tmp` and `os.replace` after the images.
- [ ] **Step 4:** `--check` prints `ok`.
### Task 2: database wiring, wrapper, docs, real trial
**Files:**
- Modify: `extraction/pump_diff.py` (add `sensors_json`, `unit_json`, `main`)
- Create: `extraction/pump_diff.sh`
- Modify: `extraction/README.md`, `README.md`, `.gitignore` (`data_diff/`)
**Interfaces:**
- Consumes: Task 1's functions.
- Produces: `sensors_json(conn, unit: int) -> list[dict]` (`id, name, type, time_utc, properties: [{name, value,
time_utc, status}]`, every sensor of the unit, archived included); `unit_json(conn, unit: int) -> dict` (`id,
serial_number, product_number, properties: [...]`).
- [ ] **Step 1:** `main()`: arguments `stations*`, `--from`, `--to`, `--delay`, `--min-free-gb`, `--out` (default
`<repo>/data_diff`); without stations, the folders of `--out`; engine and S3 client as `pump.py`'s `main`; per
station: system, unit, measurement type `OPTICAL_SHIFT-DIFF`, `sensors.json` / `unit.json` written, then each day of
`days_to_pump` after the disk check, one line printed per day (counts, MB).
- [ ] **Step 2:** `pump_diff.sh` from `pump.sh`: same `.env` load and `uv` packages, its own lock and log, no
`absolute.py` / `weather.py`, no default arguments.
- [ ] **Step 3:** `--check` still `ok`; `pump_diff.sh -h` shows the help.
- [ ] **Step 4:** real trial: `extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-02 JUNGFRAU`; expected: 94
images, byte-identical to `data/JUNGFRAU/images_diff/20260901*` (`cmp`), index of the day, `sensors.json`,
`unit.json`; `data/` unchanged (`find data -newer` the start mark finds nothing).
- [ ] **Step 5:** docs: a `pump_diff` section in `extraction/README.md`, one line in `README.md`, `data_diff/` in
`.gitignore`.