Files
aimsight-data/docs/superpowers/specs/2026-11-19-pump-diff-design.md
T

67 lines
4.2 KiB
Markdown
Raw Normal View History

# Pumping the differential images into `data_diff/`
2026-11-19. Short design, approved in conversation ("A", "go with your recommendations").
## Intent
Recompute every station's data from the differential images with configurations of our own (patterns redrawn,
thresholds, camera-correction modes), recover the frames the cloud rejected or left uncorrected, and in the end
replace `data/`. This design covers only the first of four steps: **pump the raw material**. The next ones
(re-measure the images with the device chain of optical, correct the camera motion, write the result in the format of
`data/`) get their own designs.
## Goals, non-goals
- Every `OPTICAL_SHIFT-DIFF` image of a station over a range, full size, as stored on S3 (patterns may be redrawn later,
so no crop).
- Everything needed to redo the measurement from them: all the patterns, archived ones included, with every property
row as stored (value, time, status); the acquisition unit's properties (focal distance, pixel size, algorithm...);
each measurement's exact time, group and properties.
- Gaps explained, not only seen: every measurement of the database is listed, with its file or why it has none.
- Non-goal: touching `data/` or `pump.py` (the nightly pump stays as it is). Non-goal: scheduling (later, once the
history is in).
- Anti-goal: filling the disk (about 0.6 GB per station-day at 15 min; 802 GB free on 2026-11-19).
## Approaches
- **A, chosen: a separate script `extraction/pump_diff.py` + `pump_diff.sh`.** Imports what it needs from `pump.py`
(acquisition unit, error formatting); own lock and log, so a backfill of days does not block the nightly pump; does
not run `absolute.py` / `weather.py`.
- B: a "diff only" mode in `pump.py`: least code, but touches the nightly pump and shares its lock.
- C: optical's `scripts/pcc_emulator/fetch_cloud_images.py`: needs monitoring-dashboard's code and environment,
against this repository's rule of no dependency on the app.
## Design
Output, `data_diff/<SERIAL>/` (out of git):
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is, named as in `data/`.
- `index/<SERIAL>_diff_<YYYYMMDD>.csv`, one per UTC day: `measurement_id, time_utc` (exact, ISO),
`measurement_group_id, file` (empty without image), `status` (`ok`, `no_uuid`: never reached S3, `s3_missing`: S3
returned nothing), `properties` (the measurement's JSON). Written after the day's images, through a temporary file:
an index on disk means its day was read through.
- `sensors.json`: every sensor of the acquisition unit (any type, archived included): `id, name, type, time_utc` and
all its property rows `{name, value, time_utc, status}`. Rewritten at each run (the database keeps archived
sensors, so nothing is lost by rewriting).
- `unit.json`: the acquisition unit (`id, serial_number, product_number`) and its property rows, same shape.
Runs:
- `pump_diff.sh --from D [--to E] SERIAL...`: the days of `[D, E)`. A day whose index exists and is before the last
indexed day is skipped; the last indexed day and the later ones are read again (a day pumped while it was running
is incomplete). An image already on disk is never downloaded again.
- `pump_diff.sh [SERIAL...]` without `--from`: from the last indexed day of each station up to now. Without station:
the stations already in `data_diff/`. A station with nothing indexed is skipped with a message (its history starts
only on an explicit `--from`, so no 2.6 TB by accident).
- `--min-free-gb` (default 50): checked before each day; below it, the run stops cleanly.
- `--delay` as in `pump.py`; read-only connection; one query per day for the measurements; images one at a time.
- Lock `out/pump_diff.lock`, log appended to `out/pump_diff.log`. Same `.env` and `uv` environment as `pump.sh`.
Check: `python3 extraction/pump_diff.py --check`, without database or S3: the range logic and one day pumped through
stubs (an image downloaded, one already there, one without uuid, one refused by S3, the index written).
## Left to the builder
- The SQL of the sensors and unit properties (`property` joined to `datatype`, as `pump.py`'s `sensors`).
- Whether a pattern's property has several rows over time is to be seen on the first real run; the format keeps them
all either way.