# pump_diff: the layout of data/ Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** `data_diff//` read by the other repositories as `data//`: images, metrics, weather, targets.csv as in `data/`; the DIFF images' extras under `images_diff/`. **Architecture:** `pump_diff.py` calls `pump.py`'s `pump_metrics`, `pump_images`, `sensors`, `write_targets` per day / station, then `weather.main` on its `--out`; `weather.py` gets a root folder parameter. **Tech Stack:** as `pump_diff.sh` (uv, sqlalchemy, psycopg2-binary, pandas, boto3). **Spec:** `docs/superpowers/specs/2026-11-19-configurations-design.md` (section "Work, in order", 1) ## Global Constraints - `data/` never written; `pump.py`'s behaviour unchanged for `pump.sh` (only a return value added). - Extras paths: `images_diff/index/_diff_.csv`, `images_diff/sensors.json`, `images_diff/unit.json`. - IRON `--every` default 60; metrics replaced for every day pumped. ## Review Focus - Metrics failed for a day: no index, the day is read again next run. - `weather.py` without `--data`: writes into `data/` as before (`pump.sh` unchanged). - A station without position: no weather, the pumping goes on. - Weather raising something unexpected (network): printed, the pumping's result kept. - Old layout on disk (`index/` at the station root): migrated by hand once, then regenerated. --- ### Task 1: pump_diff and weather **Files:** Modify `extraction/pump_diff.py`, `extraction/pump.py` (`pump_metrics` returns `bool`), `extraction/weather.py` (`main(serials, data=DATA)`, `station(..., data)`, `--data DIR`), `extraction/README.md`, `README.md`. - [ ] **Step 1:** `check()`: the index and `indexed_days` under `images_diff/index/`; fails. - [ ] **Step 2:** move the extras; `--every`; per day `pump_metrics` (skip the day when it returns False), `pump_images(..., "images", every, out)`, then the DIFF day; per station `write_targets(folder, sensors(conn, unit), seen)` with `seen` = {sensor_name: id} of the non-archived DIFF sensors with a pattern; at the end `weather.main( stations, a.out)` in a try. - [ ] **Step 3:** `--check` ok; `python3 extraction/pump.py --check` ok (uv); `python3 extraction/weather.py --data /nonexistent` prints nothing and exits 0. - [ ] **Step 4:** migrate `data_diff/` (old `index/` removed, `sensors.json` / `unit.json` moved; LENZSPITZE, empty, removed); `pump_diff.sh --from 2026-10-12 --to 2026-10-17 JUNGFRAU` and `--from 2026-09-01 --to 2026-09-02`; expected: no DIFF image downloaded again, metrics / images / weather / targets.csv present, `data/` unchanged. - [ ] **Step 5:** `data.py` on `data_diff/JUNGFRAU`: `find_files` without `others` from the extras; metrics, temperature and weather loaded; same counts as `data/JUNGFRAU` on those days.