The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2.9 KiB
pump_diff: the layout of data/ Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: data_diff/<SERIAL>/ read by the other repositories as data/<SERIAL>/: images, metrics, weather,
targets.csv as in data/; the DIFF images' extras under images_diff/.
Architecture: pump_diff.py calls pump.py's pump_metrics, pump_images, sensors, write_targets per
day / station, then weather.main on its --out; weather.py gets a root folder parameter.
Tech Stack: as pump_diff.sh (uv, sqlalchemy, psycopg2-binary, pandas, boto3).
Spec: docs/superpowers/specs/2026-11-19-configurations-design.md (section "Work, in order", 1)
Global Constraints
data/never written;pump.py's behaviour unchanged forpump.sh(only a return value added).- Extras paths:
images_diff/index/<S>_diff_<YYYYMMDD>.csv,images_diff/sensors.json,images_diff/unit.json. - IRON
--everydefault 60; metrics replaced for every day pumped.
Review Focus
- Metrics failed for a day: no index, the day is read again next run.
weather.pywithout--data: writes intodata/as before (pump.shunchanged).- A station without position: no weather, the pumping goes on.
- Weather raising something unexpected (network): printed, the pumping's result kept.
- Old layout on disk (
index/at the station root): migrated by hand once, then regenerated.
Task 1: pump_diff and weather
Files: Modify extraction/pump_diff.py, extraction/pump.py (pump_metrics returns bool),
extraction/weather.py (main(serials, data=DATA), station(..., data), --data DIR), extraction/README.md,
README.md.
- Step 1:
check(): the index andindexed_daysunderimages_diff/index/; fails. - Step 2: move the extras;
--every; per daypump_metrics(skip the day when it returns False),pump_images(..., "images", every, out), then the DIFF day; per stationwrite_targets(folder, sensors(conn, unit), seen)withseen= {sensor_name: id} of the non-archived DIFF sensors with a pattern; at the endweather.main( stations, a.out)in a try. - Step 3:
--checkok;python3 extraction/pump.py --checkok (uv);python3 extraction/weather.py --data /nonexistentprints nothing and exits 0. - Step 4: migrate
data_diff/(oldindex/removed,sensors.json/unit.jsonmoved; LENZSPITZE, empty, removed);pump_diff.sh --from 2026-10-12 --to 2026-10-17 JUNGFRAUand--from 2026-09-01 --to 2026-09-02; expected: no DIFF image downloaded again, metrics / images / weather / targets.csv present,data/unchanged. - Step 5:
data.pyondata_diff/JUNGFRAU:find_fileswithoutothersfrom the extras; metrics, temperature and weather loaded; same counts asdata/JUNGFRAUon those days.