Files
aimsight-data/docs/superpowers/plans/2026-11-19-pump-diff-data-layout.md
maximeandClaude Opus 5.5 81908ca370 Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)
The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every
measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived
included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into
data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now
returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF
images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 09:59:50 +01:00

2.9 KiB

pump_diff: the layout of data/ Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: data_diff/<SERIAL>/ read by the other repositories as data/<SERIAL>/: images, metrics, weather, targets.csv as in data/; the DIFF images' extras under images_diff/.

Architecture: pump_diff.py calls pump.py's pump_metrics, pump_images, sensors, write_targets per day / station, then weather.main on its --out; weather.py gets a root folder parameter.

Tech Stack: as pump_diff.sh (uv, sqlalchemy, psycopg2-binary, pandas, boto3).

Spec: docs/superpowers/specs/2026-11-19-configurations-design.md (section "Work, in order", 1)

Global Constraints

  • data/ never written; pump.py's behaviour unchanged for pump.sh (only a return value added).
  • Extras paths: images_diff/index/<S>_diff_<YYYYMMDD>.csv, images_diff/sensors.json, images_diff/unit.json.
  • IRON --every default 60; metrics replaced for every day pumped.

Review Focus

  • Metrics failed for a day: no index, the day is read again next run.
  • weather.py without --data: writes into data/ as before (pump.sh unchanged).
  • A station without position: no weather, the pumping goes on.
  • Weather raising something unexpected (network): printed, the pumping's result kept.
  • Old layout on disk (index/ at the station root): migrated by hand once, then regenerated.

Task 1: pump_diff and weather

Files: Modify extraction/pump_diff.py, extraction/pump.py (pump_metrics returns bool), extraction/weather.py (main(serials, data=DATA), station(..., data), --data DIR), extraction/README.md, README.md.

  • Step 1: check(): the index and indexed_days under images_diff/index/; fails.
  • Step 2: move the extras; --every; per day pump_metrics (skip the day when it returns False), pump_images(..., "images", every, out), then the DIFF day; per station write_targets(folder, sensors(conn, unit), seen) with seen = {sensor_name: id} of the non-archived DIFF sensors with a pattern; at the end weather.main( stations, a.out) in a try.
  • Step 3: --check ok; python3 extraction/pump.py --check ok (uv); python3 extraction/weather.py --data /nonexistent prints nothing and exits 0.
  • Step 4: migrate data_diff/ (old index/ removed, sensors.json / unit.json moved; LENZSPITZE, empty, removed); pump_diff.sh --from 2026-10-12 --to 2026-10-17 JUNGFRAU and --from 2026-09-01 --to 2026-09-02; expected: no DIFF image downloaded again, metrics / images / weather / targets.csv present, data/ unchanged.
  • Step 5: data.py on data_diff/JUNGFRAU: find_files without others from the extras; metrics, temperature and weather loaded; same counts as data/JUNGFRAU on those days.