86 lines
5.2 KiB
Markdown
86 lines
5.2 KiB
Markdown
# Pumping the differential images Implementation Plan
|
|||
|
|
|
||
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
|
|
||
|
|
**Goal:** `extraction/pump_diff.py` + `pump_diff.sh` pump a station's `OPTICAL_SHIFT-DIFF` images, their index and
|
||
|
|
the pattern / unit properties into `data_diff/<SERIAL>/`, read-only, without touching `data/` or `pump.py`.
|
||
|
|
|
||
|
|
**Architecture:** One script beside `pump.py`, importing `acquisition_unit`, `root_cause`, `DB_NAMES`, `TYPES`, `UTC`
|
||
|
|
from it; pure range logic and a per-day function testable through stubs; a wrapper copied from `pump.sh` minus
|
||
|
|
`absolute.py` / `weather.py`.
|
||
|
|
|
||
|
|
**Tech Stack:** Python 3.12 in `uv` (sqlalchemy, psycopg2-binary, pandas, boto3, as `pump.sh`), standard `csv` /
|
||
|
|
`json`.
|
||
|
|
|
||
|
|
**Spec:** `docs/superpowers/specs/2026-11-19-pump-diff-design.md`
|
||
|
|
|
||
|
|
## Global Constraints
|
||
|
|
|
||
|
|
- Read-only: `default_transaction_read_only=on`, `statement_timeout=--delay` (default 60 s), as `pump.py`.
|
||
|
|
- Never read or print the `.env`; variable names only.
|
||
|
|
- Image names `<YYYYMMDD>T<HHMM>Z_<measurement_id>.<ext>`, as `pump.py`.
|
||
|
|
- `--min-free-gb` default 50, checked before each day.
|
||
|
|
- Lock `out/pump_diff.lock`, log `out/pump_diff.log`.
|
||
|
|
|
||
|
|
## Review Focus
|
||
|
|
|
||
|
|
- A crash in the middle of a day: no index file for it (temporary file + `os.replace`), so the next run reads it again.
|
||
|
|
- Two measurements in the same minute (bursts): distinct files thanks to the id; both in the index.
|
||
|
|
- A station without acquisition unit or unknown serial: a message, the next station goes on.
|
||
|
|
- No `--from` and nothing indexed: the station is skipped with a message, nothing downloaded.
|
||
|
|
- Disk below the limit: the run stops before the day, with the free space printed.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Task 1: range logic and one day, with `--check`
|
||
|
|
|
||
|
|
**Files:**
|
||
|
|
- Create: `extraction/pump_diff.py`
|
||
|
|
|
||
|
|
**Interfaces:**
|
||
|
|
- Produces: `indexed_days(folder: Path) -> set[date]`; `days_to_pump(indexed: set[date], start: date | None, end: date)
|
||
|
|
-> list[date] | None` (None: no `--from` and nothing indexed); `pump_day(rows, client, bucket: str, ext: str,
|
||
|
|
folder: Path, station: str, day: date) -> Counter` (rows: the day's measurements, queried by `main`; counts per
|
||
|
|
status, plus `new` and `bytes`); `check()`.
|
||
|
|
|
||
|
|
- [ ] **Step 1:** write `check()` with these assertions (days 1..19 of 2026-11):
|
||
|
|
- `days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)]`;
|
||
|
|
- `days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)]` (before the last: missing only; the
|
||
|
|
last and after: again);
|
||
|
|
- `days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)]`; `days_to_pump(set(), None, d(5)) is None`;
|
||
|
|
- `pump_day` on stub `conn` / `client` with 4 measurements (one image to fetch, one already on disk, one
|
||
|
|
`measurement_uuid` None, one where `get_object` raises): returns `{"ok": 2, "no_uuid": 1, "s3_missing": 1}`, writes
|
||
|
|
one new file, leaves the existing one untouched, and the index has 4 rows with the right `status`, `file` empty for
|
||
|
|
the two missing, `properties` as JSON.
|
||
|
|
- [ ] **Step 2:** `python3 extraction/pump_diff.py --check` fails (functions missing).
|
||
|
|
- [ ] **Step 3:** implement the three functions. `pump_day` queries `id, time_utc, measurement_group_id,
|
||
|
|
measurement_uuid, properties` of `measurement` for the unit, type and UTC day, ordered by time; index columns
|
||
|
|
`measurement_id, time_utc, measurement_group_id, file, status, properties`; written to `index/<S>_diff_<YYYYMMDD>.csv`
|
||
|
|
through `<name>.tmp` and `os.replace` after the images.
|
||
|
|
- [ ] **Step 4:** `--check` prints `ok`.
|
||
|
|
|
||
|
|
### Task 2: database wiring, wrapper, docs, real trial
|
||
|
|
|
||
|
|
**Files:**
|
||
|
|
- Modify: `extraction/pump_diff.py` (add `sensors_json`, `unit_json`, `main`)
|
||
|
|
- Create: `extraction/pump_diff.sh`
|
||
|
|
- Modify: `extraction/README.md`, `README.md`, `.gitignore` (`data_diff/`)
|
||
|
|
|
||
|
|
**Interfaces:**
|
||
|
|
- Consumes: Task 1's functions.
|
||
|
|
- Produces: `sensors_json(conn, unit: int) -> list[dict]` (`id, name, type, time_utc, properties: [{name, value,
|
||
|
|
time_utc, status}]`, every sensor of the unit, archived included); `unit_json(conn, unit: int) -> dict` (`id,
|
||
|
|
serial_number, product_number, properties: [...]`).
|
||
|
|
|
||
|
|
- [ ] **Step 1:** `main()`: arguments `stations*`, `--from`, `--to`, `--delay`, `--min-free-gb`, `--out` (default
|
||
|
|
`<repo>/data_diff`); without stations, the folders of `--out`; engine and S3 client as `pump.py`'s `main`; per
|
||
|
|
station: system, unit, measurement type `OPTICAL_SHIFT-DIFF`, `sensors.json` / `unit.json` written, then each day of
|
||
|
|
`days_to_pump` after the disk check, one line printed per day (counts, MB).
|
||
|
|
- [ ] **Step 2:** `pump_diff.sh` from `pump.sh`: same `.env` load and `uv` packages, its own lock and log, no
|
||
|
|
`absolute.py` / `weather.py`, no default arguments.
|
||
|
|
- [ ] **Step 3:** `--check` still `ok`; `pump_diff.sh -h` shows the help.
|
||
|
|
- [ ] **Step 4:** real trial: `extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-02 JUNGFRAU`; expected: 94
|
||
|
|
images, byte-identical to `data/JUNGFRAU/images_diff/20260901*` (`cmp`), index of the day, `sensors.json`,
|
||
|
|
`unit.json`; `data/` unchanged (`find data -newer` the start mark finds nothing).
|
||
|
|
- [ ] **Step 5:** docs: a `pump_diff` section in `extraction/README.md`, one line in `README.md`, `data_diff/` in
|
||
|
|
`.gitignore`.
|