Files

48 lines
2.9 KiB
Markdown
Raw Permalink Normal View History

# pump_diff: the layout of data/ Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** `data_diff/<SERIAL>/` read by the other repositories as `data/<SERIAL>/`: images, metrics, weather,
targets.csv as in `data/`; the DIFF images' extras under `images_diff/`.
**Architecture:** `pump_diff.py` calls `pump.py`'s `pump_metrics`, `pump_images`, `sensors`, `write_targets` per
day / station, then `weather.main` on its `--out`; `weather.py` gets a root folder parameter.
**Tech Stack:** as `pump_diff.sh` (uv, sqlalchemy, psycopg2-binary, pandas, boto3).
**Spec:** `docs/superpowers/specs/2026-11-19-configurations-design.md` (section "Work, in order", 1)
## Global Constraints
- `data/` never written; `pump.py`'s behaviour unchanged for `pump.sh` (only a return value added).
- Extras paths: `images_diff/index/<S>_diff_<YYYYMMDD>.csv`, `images_diff/sensors.json`, `images_diff/unit.json`.
- IRON `--every` default 60; metrics replaced for every day pumped.
## Review Focus
- Metrics failed for a day: no index, the day is read again next run.
- `weather.py` without `--data`: writes into `data/` as before (`pump.sh` unchanged).
- A station without position: no weather, the pumping goes on.
- Weather raising something unexpected (network): printed, the pumping's result kept.
- Old layout on disk (`index/` at the station root): migrated by hand once, then regenerated.
---
### Task 1: pump_diff and weather
**Files:** Modify `extraction/pump_diff.py`, `extraction/pump.py` (`pump_metrics` returns `bool`),
`extraction/weather.py` (`main(serials, data=DATA)`, `station(..., data)`, `--data DIR`), `extraction/README.md`,
`README.md`.
- [ ] **Step 1:** `check()`: the index and `indexed_days` under `images_diff/index/`; fails.
- [ ] **Step 2:** move the extras; `--every`; per day `pump_metrics` (skip the day when it returns False),
`pump_images(..., "images", every, out)`, then the DIFF day; per station `write_targets(folder, sensors(conn, unit),
seen)` with `seen` = {sensor_name: id} of the non-archived DIFF sensors with a pattern; at the end `weather.main(
stations, a.out)` in a try.
- [ ] **Step 3:** `--check` ok; `python3 extraction/pump.py --check` ok (uv); `python3 extraction/weather.py --data
/nonexistent` prints nothing and exits 0.
- [ ] **Step 4:** migrate `data_diff/` (old `index/` removed, `sensors.json` / `unit.json` moved; LENZSPITZE, empty,
removed); `pump_diff.sh --from 2026-10-12 --to 2026-10-17 JUNGFRAU` and `--from 2026-09-01 --to 2026-09-02`;
expected: no DIFF image downloaded again, metrics / images / weather / targets.csv present, `data/` unchanged.
- [ ] **Step 5:** `data.py` on `data_diff/JUNGFRAU`: `find_files` without `others` from the extras; metrics,
temperature and weather loaded; same counts as `data/JUNGFRAU` on those days.