The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
49 lines
2.9 KiB
Markdown
49 lines
2.9 KiB
Markdown
# pump_diff: the layout of data/ Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** `data_diff/<SERIAL>/` read by the other repositories as `data/<SERIAL>/`: images, metrics, weather,
|
|
targets.csv as in `data/`; the DIFF images' extras under `images_diff/`.
|
|
|
|
**Architecture:** `pump_diff.py` calls `pump.py`'s `pump_metrics`, `pump_images`, `sensors`, `write_targets` per
|
|
day / station, then `weather.main` on its `--out`; `weather.py` gets a root folder parameter.
|
|
|
|
**Tech Stack:** as `pump_diff.sh` (uv, sqlalchemy, psycopg2-binary, pandas, boto3).
|
|
|
|
**Spec:** `docs/superpowers/specs/2026-11-19-configurations-design.md` (section "Work, in order", 1)
|
|
|
|
## Global Constraints
|
|
|
|
- `data/` never written; `pump.py`'s behaviour unchanged for `pump.sh` (only a return value added).
|
|
- Extras paths: `images_diff/index/<S>_diff_<YYYYMMDD>.csv`, `images_diff/sensors.json`, `images_diff/unit.json`.
|
|
- IRON `--every` default 60; metrics replaced for every day pumped.
|
|
|
|
## Review Focus
|
|
|
|
- Metrics failed for a day: no index, the day is read again next run.
|
|
- `weather.py` without `--data`: writes into `data/` as before (`pump.sh` unchanged).
|
|
- A station without position: no weather, the pumping goes on.
|
|
- Weather raising something unexpected (network): printed, the pumping's result kept.
|
|
- Old layout on disk (`index/` at the station root): migrated by hand once, then regenerated.
|
|
|
|
---
|
|
|
|
### Task 1: pump_diff and weather
|
|
|
|
**Files:** Modify `extraction/pump_diff.py`, `extraction/pump.py` (`pump_metrics` returns `bool`),
|
|
`extraction/weather.py` (`main(serials, data=DATA)`, `station(..., data)`, `--data DIR`), `extraction/README.md`,
|
|
`README.md`.
|
|
|
|
- [ ] **Step 1:** `check()`: the index and `indexed_days` under `images_diff/index/`; fails.
|
|
- [ ] **Step 2:** move the extras; `--every`; per day `pump_metrics` (skip the day when it returns False),
|
|
`pump_images(..., "images", every, out)`, then the DIFF day; per station `write_targets(folder, sensors(conn, unit),
|
|
seen)` with `seen` = {sensor_name: id} of the non-archived DIFF sensors with a pattern; at the end `weather.main(
|
|
stations, a.out)` in a try.
|
|
- [ ] **Step 3:** `--check` ok; `python3 extraction/pump.py --check` ok (uv); `python3 extraction/weather.py --data
|
|
/nonexistent` prints nothing and exits 0.
|
|
- [ ] **Step 4:** migrate `data_diff/` (old `index/` removed, `sensors.json` / `unit.json` moved; LENZSPITZE, empty,
|
|
removed); `pump_diff.sh --from 2026-10-12 --to 2026-10-17 JUNGFRAU` and `--from 2026-09-01 --to 2026-09-02`;
|
|
expected: no DIFF image downloaded again, metrics / images / weather / targets.csv present, `data/` unchanged.
|
|
- [ ] **Step 5:** `data.py` on `data_diff/JUNGFRAU`: `find_files` without `others` from the extras; metrics,
|
|
temperature and weather loaded; same counts as `data/JUNGFRAU` on those days.
|