Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)

The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every
measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived
included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into
data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now
returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF
images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-19 09:59:50 +01:00
1 parent 1b918facd9
commit 81908ca370
9 files changed
+660 -8

No files matched your search

@@ -0,0 +1,70 @@
# Configurations: recomputing the stations' data from the differential images
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
section "data_diff").
## Intent
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
thresholds, references, camera correction), compare them, and let every other repository (aimsight-app, tools,
detectors, models, optimizer) read the result exactly as it reads `data/`. `data/` stays as it is: the cloud's
version, the reference to compare with.
## Layout
```
aimsight-data/
data/ the cloud, unchanged (pump.sh)
data_diff/<SERIAL>/ raw material shared by every configuration (pump_diff.sh), out of git
images_diff/ the DIFF images; and, invisible to the tools (data.find_files skips images_diff/):
index/<SERIAL>_diff_<YYYYMMDD>.csv, sensors.json, unit.json
images/ metrics/ weather/ targets.csv as in data/ (targets.csv: the cloud's patterns)
configs/<name>.yaml one configuration per file, in git
runs/<name>/ the result of a configuration, out of git, the same tree as data/
<SERIAL>/
displacement/ camera/ targets.csv recomputed, the columns of data/
images/ metrics/ weather/ symbolic links to data_diff/<SERIAL>/ (nothing copied)
run.yaml the configuration used, optical's commit, the date (outside the station folders:
invisible to the tools)
diagnostics/ the emulator's raw rows (edge margin, second spot)
```
A component reads a configuration by pointing its `data` link at `~/aimsight-data/runs/<name>` instead of
`~/aimsight-data/data`. Checked on 2026-11-19: `data.py` (aimsight-tools) reads through symbolic links (images,
metrics, temperature, weather).
## Configuration
A configuration states only what differs from the cloud's patterns (`data_diff/<SERIAL>/images_diff/sensors.json`)
and from the unit's properties (`unit.json`); the configuration `cloud`, with nothing in it, must reproduce `data/`
(the reference: on JUNGFRAU 2026-10-12..16 the emulator matched the cloud within 0.05 px, errors identical).
```yaml
stations:
JUNGFRAU:
patterns: # by sensor_name, over sensors.json
PM_G: {error_threshold: 0.7}
BL_C: {corners: [[1700, 929], [1724, 953]]}
REF_CT: {pattern_type: target}
camera: {focal_distance: 6.3} # over unit.json and the common camera block
camera: # common to the stations of the file
algorithm: ransac # ransac | danish_hybrid | trusted_danish
mode: "off" # off | rotation | rotation_scale | relative
```
Running a configuration again replaces `runs/<name>/`: to compare two settings, two files. No cache between the
re-measurement and the correction (431 images in 11 s; about 15 min for a year of one station): added if it gets slow.
Rejected: a full copy per configuration (the DIFF images duplicated, 0.6 GB per station-day); a configuration
suffix in file names (the tools recognise files by their columns and would mix configurations).
## Work, in order
1. **`pump_diff`** (this repository's `extraction/`): the extras of the DIFF images move under `images_diff/`; per day,
before the DIFF images and their index, the metrics (`pump.py`'s `pump_metrics`) and the IRON images
(`pump_images`, `--every`, default 60 min); per station, `targets.csv` (`write_targets`, on the DIFF patterns);
at the end, the weather (`weather.py`, given the root folder). A day whose metrics failed gets no index: the next
run reads it again. `pump.py`'s only change: `pump_metrics` returns whether every day succeeded.
2. **`run_config`**: re-measurement (optical's emulator), camera correction (optical's `remove_camera_movement` and
the all-point modes), written in the format of `data/`. Its own design once optical's correction outputs are mapped
to the columns of `data/` (`relative_x/y, corr_x/y, reproj_err, inlier, sigma0`, the camera pose).
@@ -0,0 +1,67 @@
# Pumping the differential images into `data_diff/`
2026-11-19. Short design, approved in conversation ("A", "go with your recommendations").
## Intent
Recompute every station's data from the differential images with configurations of our own (patterns redrawn,
thresholds, camera-correction modes), recover the frames the cloud rejected or left uncorrected, and in the end
replace `data/`. This design covers only the first of four steps: **pump the raw material**. The next ones
(re-measure the images with the device chain of optical, correct the camera motion, write the result in the format of
`data/`) get their own designs.
## Goals, non-goals
- Every `OPTICAL_SHIFT-DIFF` image of a station over a range, full size, as stored on S3 (patterns may be redrawn later,
so no crop).
- Everything needed to redo the measurement from them: all the patterns, archived ones included, with every property
row as stored (value, time, status); the acquisition unit's properties (focal distance, pixel size, algorithm...);
each measurement's exact time, group and properties.
- Gaps explained, not only seen: every measurement of the database is listed, with its file or why it has none.
- Non-goal: touching `data/` or `pump.py` (the nightly pump stays as it is). Non-goal: scheduling (later, once the
history is in).
- Anti-goal: filling the disk (about 0.6 GB per station-day at 15 min; 802 GB free on 2026-11-19).
## Approaches
- **A, chosen: a separate script `extraction/pump_diff.py` + `pump_diff.sh`.** Imports what it needs from `pump.py`
(acquisition unit, error formatting); own lock and log, so a backfill of days does not block the nightly pump; does
not run `absolute.py` / `weather.py`.
- B: a "diff only" mode in `pump.py`: least code, but touches the nightly pump and shares its lock.
- C: optical's `scripts/pcc_emulator/fetch_cloud_images.py`: needs monitoring-dashboard's code and environment,
against this repository's rule of no dependency on the app.
## Design
Output, `data_diff/<SERIAL>/` (out of git):
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is, named as in `data/`.
- `index/<SERIAL>_diff_<YYYYMMDD>.csv`, one per UTC day: `measurement_id, time_utc` (exact, ISO),
`measurement_group_id, file` (empty without image), `status` (`ok`, `no_uuid`: never reached S3, `s3_missing`: S3
returned nothing), `properties` (the measurement's JSON). Written after the day's images, through a temporary file:
an index on disk means its day was read through.
- `sensors.json`: every sensor of the acquisition unit (any type, archived included): `id, name, type, time_utc` and
all its property rows `{name, value, time_utc, status}`. Rewritten at each run (the database keeps archived
sensors, so nothing is lost by rewriting).
- `unit.json`: the acquisition unit (`id, serial_number, product_number`) and its property rows, same shape.
Runs:
- `pump_diff.sh --from D [--to E] SERIAL...`: the days of `[D, E)`. A day whose index exists and is before the last
indexed day is skipped; the last indexed day and the later ones are read again (a day pumped while it was running
is incomplete). An image already on disk is never downloaded again.
- `pump_diff.sh [SERIAL...]` without `--from`: from the last indexed day of each station up to now. Without station:
the stations already in `data_diff/`. A station with nothing indexed is skipped with a message (its history starts
only on an explicit `--from`, so no 2.6 TB by accident).
- `--min-free-gb` (default 50): checked before each day; below it, the run stops cleanly.
- `--delay` as in `pump.py`; read-only connection; one query per day for the measurements; images one at a time.
- Lock `out/pump_diff.lock`, log appended to `out/pump_diff.log`. Same `.env` and `uv` environment as `pump.sh`.
Check: `python3 extraction/pump_diff.py --check`, without database or S3: the range logic and one day pumped through
stubs (an image downloaded, one already there, one without uuid, one refused by S3, the index written).
## Left to the builder
- The SQL of the sensors and unit properties (`property` joined to `datatype`, as `pump.py`'s `sensors`).
- Whether a pattern's property has several rows over time is to be seen on the first real run; the format keeps them
all either way.