Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)

The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every
measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived
included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into
data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now
returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF
images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-19 09:59:50 +01:00
1 parent 1b918facd9
commit 81908ca370
9 files changed
+660 -8

No files matched your search

@@ -0,0 +1,48 @@
# pump_diff: the layout of data/ Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** `data_diff/<SERIAL>/` read by the other repositories as `data/<SERIAL>/`: images, metrics, weather,
targets.csv as in `data/`; the DIFF images' extras under `images_diff/`.
**Architecture:** `pump_diff.py` calls `pump.py`'s `pump_metrics`, `pump_images`, `sensors`, `write_targets` per
day / station, then `weather.main` on its `--out`; `weather.py` gets a root folder parameter.
**Tech Stack:** as `pump_diff.sh` (uv, sqlalchemy, psycopg2-binary, pandas, boto3).
**Spec:** `docs/superpowers/specs/2026-11-19-configurations-design.md` (section "Work, in order", 1)
## Global Constraints
- `data/` never written; `pump.py`'s behaviour unchanged for `pump.sh` (only a return value added).
- Extras paths: `images_diff/index/<S>_diff_<YYYYMMDD>.csv`, `images_diff/sensors.json`, `images_diff/unit.json`.
- IRON `--every` default 60; metrics replaced for every day pumped.
## Review Focus
- Metrics failed for a day: no index, the day is read again next run.
- `weather.py` without `--data`: writes into `data/` as before (`pump.sh` unchanged).
- A station without position: no weather, the pumping goes on.
- Weather raising something unexpected (network): printed, the pumping's result kept.
- Old layout on disk (`index/` at the station root): migrated by hand once, then regenerated.
---
### Task 1: pump_diff and weather
**Files:** Modify `extraction/pump_diff.py`, `extraction/pump.py` (`pump_metrics` returns `bool`),
`extraction/weather.py` (`main(serials, data=DATA)`, `station(..., data)`, `--data DIR`), `extraction/README.md`,
`README.md`.
- [ ] **Step 1:** `check()`: the index and `indexed_days` under `images_diff/index/`; fails.
- [ ] **Step 2:** move the extras; `--every`; per day `pump_metrics` (skip the day when it returns False),
`pump_images(..., "images", every, out)`, then the DIFF day; per station `write_targets(folder, sensors(conn, unit),
seen)` with `seen` = {sensor_name: id} of the non-archived DIFF sensors with a pattern; at the end `weather.main(
stations, a.out)` in a try.
- [ ] **Step 3:** `--check` ok; `python3 extraction/pump.py --check` ok (uv); `python3 extraction/weather.py --data
/nonexistent` prints nothing and exits 0.
- [ ] **Step 4:** migrate `data_diff/` (old `index/` removed, `sensors.json` / `unit.json` moved; LENZSPITZE, empty,
removed); `pump_diff.sh --from 2026-10-12 --to 2026-10-17 JUNGFRAU` and `--from 2026-09-01 --to 2026-09-02`;
expected: no DIFF image downloaded again, metrics / images / weather / targets.csv present, `data/` unchanged.
- [ ] **Step 5:** `data.py` on `data_diff/JUNGFRAU`: `find_files` without `others` from the extras; metrics,
temperature and weather loaded; same counts as `data/JUNGFRAU` on those days.
@@ -0,0 +1,86 @@
# Pumping the differential images Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** `extraction/pump_diff.py` + `pump_diff.sh` pump a station's `OPTICAL_SHIFT-DIFF` images, their index and
the pattern / unit properties into `data_diff/<SERIAL>/`, read-only, without touching `data/` or `pump.py`.
**Architecture:** One script beside `pump.py`, importing `acquisition_unit`, `root_cause`, `DB_NAMES`, `TYPES`, `UTC`
from it; pure range logic and a per-day function testable through stubs; a wrapper copied from `pump.sh` minus
`absolute.py` / `weather.py`.
**Tech Stack:** Python 3.12 in `uv` (sqlalchemy, psycopg2-binary, pandas, boto3, as `pump.sh`), standard `csv` /
`json`.
**Spec:** `docs/superpowers/specs/2026-11-19-pump-diff-design.md`
## Global Constraints
- Read-only: `default_transaction_read_only=on`, `statement_timeout=--delay` (default 60 s), as `pump.py`.
- Never read or print the `.env`; variable names only.
- Image names `<YYYYMMDD>T<HHMM>Z_<measurement_id>.<ext>`, as `pump.py`.
- `--min-free-gb` default 50, checked before each day.
- Lock `out/pump_diff.lock`, log `out/pump_diff.log`.
## Review Focus
- A crash in the middle of a day: no index file for it (temporary file + `os.replace`), so the next run reads it again.
- Two measurements in the same minute (bursts): distinct files thanks to the id; both in the index.
- A station without acquisition unit or unknown serial: a message, the next station goes on.
- No `--from` and nothing indexed: the station is skipped with a message, nothing downloaded.
- Disk below the limit: the run stops before the day, with the free space printed.
---
### Task 1: range logic and one day, with `--check`
**Files:**
- Create: `extraction/pump_diff.py`
**Interfaces:**
- Produces: `indexed_days(folder: Path) -> set[date]`; `days_to_pump(indexed: set[date], start: date | None, end: date)
-> list[date] | None` (None: no `--from` and nothing indexed); `pump_day(rows, client, bucket: str, ext: str,
folder: Path, station: str, day: date) -> Counter` (rows: the day's measurements, queried by `main`; counts per
status, plus `new` and `bytes`); `check()`.
- [ ] **Step 1:** write `check()` with these assertions (days 1..19 of 2026-11):
- `days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)]`;
- `days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)]` (before the last: missing only; the
last and after: again);
- `days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)]`; `days_to_pump(set(), None, d(5)) is None`;
- `pump_day` on stub `conn` / `client` with 4 measurements (one image to fetch, one already on disk, one
`measurement_uuid` None, one where `get_object` raises): returns `{"ok": 2, "no_uuid": 1, "s3_missing": 1}`, writes
one new file, leaves the existing one untouched, and the index has 4 rows with the right `status`, `file` empty for
the two missing, `properties` as JSON.
- [ ] **Step 2:** `python3 extraction/pump_diff.py --check` fails (functions missing).
- [ ] **Step 3:** implement the three functions. `pump_day` queries `id, time_utc, measurement_group_id,
measurement_uuid, properties` of `measurement` for the unit, type and UTC day, ordered by time; index columns
`measurement_id, time_utc, measurement_group_id, file, status, properties`; written to `index/<S>_diff_<YYYYMMDD>.csv`
through `<name>.tmp` and `os.replace` after the images.
- [ ] **Step 4:** `--check` prints `ok`.
### Task 2: database wiring, wrapper, docs, real trial
**Files:**
- Modify: `extraction/pump_diff.py` (add `sensors_json`, `unit_json`, `main`)
- Create: `extraction/pump_diff.sh`
- Modify: `extraction/README.md`, `README.md`, `.gitignore` (`data_diff/`)
**Interfaces:**
- Consumes: Task 1's functions.
- Produces: `sensors_json(conn, unit: int) -> list[dict]` (`id, name, type, time_utc, properties: [{name, value,
time_utc, status}]`, every sensor of the unit, archived included); `unit_json(conn, unit: int) -> dict` (`id,
serial_number, product_number, properties: [...]`).
- [ ] **Step 1:** `main()`: arguments `stations*`, `--from`, `--to`, `--delay`, `--min-free-gb`, `--out` (default
`<repo>/data_diff`); without stations, the folders of `--out`; engine and S3 client as `pump.py`'s `main`; per
station: system, unit, measurement type `OPTICAL_SHIFT-DIFF`, `sensors.json` / `unit.json` written, then each day of
`days_to_pump` after the disk check, one line printed per day (counts, MB).
- [ ] **Step 2:** `pump_diff.sh` from `pump.sh`: same `.env` load and `uv` packages, its own lock and log, no
`absolute.py` / `weather.py`, no default arguments.
- [ ] **Step 3:** `--check` still `ok`; `pump_diff.sh -h` shows the help.
- [ ] **Step 4:** real trial: `extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-02 JUNGFRAU`; expected: 94
images, byte-identical to `data/JUNGFRAU/images_diff/20260901*` (`cmp`), index of the day, `sensors.json`,
`unit.json`; `data/` unchanged (`find data -newer` the start mark finds nothing).
- [ ] **Step 5:** docs: a `pump_diff` section in `extraction/README.md`, one line in `README.md`, `data_diff/` in
`.gitignore`.