67 lines
4.2 KiB
Markdown
67 lines
4.2 KiB
Markdown
# Pumping the differential images into `data_diff/`
|
|||
|
|
|
||
|
|
2026-11-19. Short design, approved in conversation ("A", "go with your recommendations").
|
||
|
|
|
||
|
|
## Intent
|
||
|
|
|
||
|
|
Recompute every station's data from the differential images with configurations of our own (patterns redrawn,
|
||
|
|
thresholds, camera-correction modes), recover the frames the cloud rejected or left uncorrected, and in the end
|
||
|
|
replace `data/`. This design covers only the first of four steps: **pump the raw material**. The next ones
|
||
|
|
(re-measure the images with the device chain of optical, correct the camera motion, write the result in the format of
|
||
|
|
`data/`) get their own designs.
|
||
|
|
|
||
|
|
## Goals, non-goals
|
||
|
|
|
||
|
|
- Every `OPTICAL_SHIFT-DIFF` image of a station over a range, full size, as stored on S3 (patterns may be redrawn later,
|
||
|
|
so no crop).
|
||
|
|
- Everything needed to redo the measurement from them: all the patterns, archived ones included, with every property
|
||
|
|
row as stored (value, time, status); the acquisition unit's properties (focal distance, pixel size, algorithm...);
|
||
|
|
each measurement's exact time, group and properties.
|
||
|
|
- Gaps explained, not only seen: every measurement of the database is listed, with its file or why it has none.
|
||
|
|
- Non-goal: touching `data/` or `pump.py` (the nightly pump stays as it is). Non-goal: scheduling (later, once the
|
||
|
|
history is in).
|
||
|
|
- Anti-goal: filling the disk (about 0.6 GB per station-day at 15 min; 802 GB free on 2026-11-19).
|
||
|
|
|
||
|
|
## Approaches
|
||
|
|
|
||
|
|
- **A, chosen: a separate script `extraction/pump_diff.py` + `pump_diff.sh`.** Imports what it needs from `pump.py`
|
||
|
|
(acquisition unit, error formatting); own lock and log, so a backfill of days does not block the nightly pump; does
|
||
|
|
not run `absolute.py` / `weather.py`.
|
||
|
|
- B: a "diff only" mode in `pump.py`: least code, but touches the nightly pump and shares its lock.
|
||
|
|
- C: optical's `scripts/pcc_emulator/fetch_cloud_images.py`: needs monitoring-dashboard's code and environment,
|
||
|
|
against this repository's rule of no dependency on the app.
|
||
|
|
|
||
|
|
## Design
|
||
|
|
|
||
|
|
Output, `data_diff/<SERIAL>/` (out of git):
|
||
|
|
|
||
|
|
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is, named as in `data/`.
|
||
|
|
- `index/<SERIAL>_diff_<YYYYMMDD>.csv`, one per UTC day: `measurement_id, time_utc` (exact, ISO),
|
||
|
|
`measurement_group_id, file` (empty without image), `status` (`ok`, `no_uuid`: never reached S3, `s3_missing`: S3
|
||
|
|
returned nothing), `properties` (the measurement's JSON). Written after the day's images, through a temporary file:
|
||
|
|
an index on disk means its day was read through.
|
||
|
|
- `sensors.json`: every sensor of the acquisition unit (any type, archived included): `id, name, type, time_utc` and
|
||
|
|
all its property rows `{name, value, time_utc, status}`. Rewritten at each run (the database keeps archived
|
||
|
|
sensors, so nothing is lost by rewriting).
|
||
|
|
- `unit.json`: the acquisition unit (`id, serial_number, product_number`) and its property rows, same shape.
|
||
|
|
|
||
|
|
Runs:
|
||
|
|
|
||
|
|
- `pump_diff.sh --from D [--to E] SERIAL...`: the days of `[D, E)`. A day whose index exists and is before the last
|
||
|
|
indexed day is skipped; the last indexed day and the later ones are read again (a day pumped while it was running
|
||
|
|
is incomplete). An image already on disk is never downloaded again.
|
||
|
|
- `pump_diff.sh [SERIAL...]` without `--from`: from the last indexed day of each station up to now. Without station:
|
||
|
|
the stations already in `data_diff/`. A station with nothing indexed is skipped with a message (its history starts
|
||
|
|
only on an explicit `--from`, so no 2.6 TB by accident).
|
||
|
|
- `--min-free-gb` (default 50): checked before each day; below it, the run stops cleanly.
|
||
|
|
- `--delay` as in `pump.py`; read-only connection; one query per day for the measurements; images one at a time.
|
||
|
|
- Lock `out/pump_diff.lock`, log appended to `out/pump_diff.log`. Same `.env` and `uv` environment as `pump.sh`.
|
||
|
|
|
||
|
|
Check: `python3 extraction/pump_diff.py --check`, without database or S3: the range logic and one day pumped through
|
||
|
|
stubs (an image downloaded, one already there, one without uuid, one refused by S3, the index written).
|
||
|
|
|
||
|
|
## Left to the builder
|
||
|
|
|
||
|
|
- The SQL of the sensors and unit properties (`property` joined to `datatype`, as `pump.py`'s `sensors`).
|
||
|
|
- Whether a pattern's property has several rows over time is to be seen on the first real run; the format keeps them
|
||
|
|
all either way.
|