48 lines
2.9 KiB
Markdown
48 lines
2.9 KiB
Markdown
# pump_diff: the layout of data/ Implementation Plan
|
|||
|
|
|
||
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
|
|
||
|
|
**Goal:** `data_diff/<SERIAL>/` read by the other repositories as `data/<SERIAL>/`: images, metrics, weather,
|
||
|
|
targets.csv as in `data/`; the DIFF images' extras under `images_diff/`.
|
||
|
|
|
||
|
|
**Architecture:** `pump_diff.py` calls `pump.py`'s `pump_metrics`, `pump_images`, `sensors`, `write_targets` per
|
||
|
|
day / station, then `weather.main` on its `--out`; `weather.py` gets a root folder parameter.
|
||
|
|
|
||
|
|
**Tech Stack:** as `pump_diff.sh` (uv, sqlalchemy, psycopg2-binary, pandas, boto3).
|
||
|
|
|
||
|
|
**Spec:** `docs/superpowers/specs/2026-11-19-configurations-design.md` (section "Work, in order", 1)
|
||
|
|
|
||
|
|
## Global Constraints
|
||
|
|
|
||
|
|
- `data/` never written; `pump.py`'s behaviour unchanged for `pump.sh` (only a return value added).
|
||
|
|
- Extras paths: `images_diff/index/<S>_diff_<YYYYMMDD>.csv`, `images_diff/sensors.json`, `images_diff/unit.json`.
|
||
|
|
- IRON `--every` default 60; metrics replaced for every day pumped.
|
||
|
|
|
||
|
|
## Review Focus
|
||
|
|
|
||
|
|
- Metrics failed for a day: no index, the day is read again next run.
|
||
|
|
- `weather.py` without `--data`: writes into `data/` as before (`pump.sh` unchanged).
|
||
|
|
- A station without position: no weather, the pumping goes on.
|
||
|
|
- Weather raising something unexpected (network): printed, the pumping's result kept.
|
||
|
|
- Old layout on disk (`index/` at the station root): migrated by hand once, then regenerated.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### Task 1: pump_diff and weather
|
||
|
|
|
||
|
|
**Files:** Modify `extraction/pump_diff.py`, `extraction/pump.py` (`pump_metrics` returns `bool`),
|
||
|
|
`extraction/weather.py` (`main(serials, data=DATA)`, `station(..., data)`, `--data DIR`), `extraction/README.md`,
|
||
|
|
`README.md`.
|
||
|
|
|
||
|
|
- [ ] **Step 1:** `check()`: the index and `indexed_days` under `images_diff/index/`; fails.
|
||
|
|
- [ ] **Step 2:** move the extras; `--every`; per day `pump_metrics` (skip the day when it returns False),
|
||
|
|
`pump_images(..., "images", every, out)`, then the DIFF day; per station `write_targets(folder, sensors(conn, unit),
|
||
|
|
seen)` with `seen` = {sensor_name: id} of the non-archived DIFF sensors with a pattern; at the end `weather.main(
|
||
|
|
stations, a.out)` in a try.
|
||
|
|
- [ ] **Step 3:** `--check` ok; `python3 extraction/pump.py --check` ok (uv); `python3 extraction/weather.py --data
|
||
|
|
/nonexistent` prints nothing and exits 0.
|
||
|
|
- [ ] **Step 4:** migrate `data_diff/` (old `index/` removed, `sensors.json` / `unit.json` moved; LENZSPITZE, empty,
|
||
|
|
removed); `pump_diff.sh --from 2026-10-12 --to 2026-10-17 JUNGFRAU` and `--from 2026-09-01 --to 2026-09-02`;
|
||
|
|
expected: no DIFF image downloaded again, metrics / images / weather / targets.csv present, `data/` unchanged.
|
||
|
|
- [ ] **Step 5:** `data.py` on `data_diff/JUNGFRAU`: `find_files` without `others` from the extras; metrics,
|
||
|
|
temperature and weather loaded; same counts as `data/JUNGFRAU` on those days.
|