# Pumping the differential images into `data_diff/` 2026-11-19. Short design, approved in conversation ("A", "go with your recommendations"). ## Intent Recompute every station's data from the differential images with configurations of our own (patterns redrawn, thresholds, camera-correction modes), recover the frames the cloud rejected or left uncorrected, and in the end replace `data/`. This design covers only the first of four steps: **pump the raw material**. The next ones (re-measure the images with the device chain of optical, correct the camera motion, write the result in the format of `data/`) get their own designs. ## Goals, non-goals - Every `OPTICAL_SHIFT-DIFF` image of a station over a range, full size, as stored on S3 (patterns may be redrawn later, so no crop). - Everything needed to redo the measurement from them: all the patterns, archived ones included, with every property row as stored (value, time, status); the acquisition unit's properties (focal distance, pixel size, algorithm...); each measurement's exact time, group and properties. - Gaps explained, not only seen: every measurement of the database is listed, with its file or why it has none. - Non-goal: touching `data/` or `pump.py` (the nightly pump stays as it is). Non-goal: scheduling (later, once the history is in). - Anti-goal: filling the disk (about 0.6 GB per station-day at 15 min; 802 GB free on 2026-11-19). ## Approaches - **A, chosen: a separate script `extraction/pump_diff.py` + `pump_diff.sh`.** Imports what it needs from `pump.py` (acquisition unit, error formatting); own lock and log, so a backfill of days does not block the nightly pump; does not run `absolute.py` / `weather.py`. - B: a "diff only" mode in `pump.py`: least code, but touches the nightly pump and shares its lock. - C: optical's `scripts/pcc_emulator/fetch_cloud_images.py`: needs monitoring-dashboard's code and environment, against this repository's rule of no dependency on the app. ## Design Output, `data_diff//` (out of git): - `images_diff/TZ_.png`: the S3 object as is, named as in `data/`. - `index/_diff_.csv`, one per UTC day: `measurement_id, time_utc` (exact, ISO), `measurement_group_id, file` (empty without image), `status` (`ok`, `no_uuid`: never reached S3, `s3_missing`: S3 returned nothing), `properties` (the measurement's JSON). Written after the day's images, through a temporary file: an index on disk means its day was read through. - `sensors.json`: every sensor of the acquisition unit (any type, archived included): `id, name, type, time_utc` and all its property rows `{name, value, time_utc, status}`. Rewritten at each run (the database keeps archived sensors, so nothing is lost by rewriting). - `unit.json`: the acquisition unit (`id, serial_number, product_number`) and its property rows, same shape. Runs: - `pump_diff.sh --from D [--to E] SERIAL...`: the days of `[D, E)`. A day whose index exists and is before the last indexed day is skipped; the last indexed day and the later ones are read again (a day pumped while it was running is incomplete). An image already on disk is never downloaded again. - `pump_diff.sh [SERIAL...]` without `--from`: from the last indexed day of each station up to now. Without station: the stations already in `data_diff/`. A station with nothing indexed is skipped with a message (its history starts only on an explicit `--from`, so no 2.6 TB by accident). - `--min-free-gb` (default 50): checked before each day; below it, the run stops cleanly. - `--delay` as in `pump.py`; read-only connection; one query per day for the measurements; images one at a time. - Lock `out/pump_diff.lock`, log appended to `out/pump_diff.log`. Same `.env` and `uv` environment as `pump.sh`. Check: `python3 extraction/pump_diff.py --check`, without database or S3: the range logic and one day pumped through stubs (an image downloaded, one already there, one without uuid, one refused by S3, the index written). ## Left to the builder - The SQL of the sensors and unit properties (`property` joined to `datatype`, as `pump.py`'s `sensors`). - Whether a pattern's property has several rows over time is to be seen on the first real run; the format keeps them all either way.