Files
aimsight-data/docs/superpowers/specs/2026-11-19-configurations-design.md
T
maximeandClaude Opus 5.5 81908ca370 Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)
The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every
measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived
included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into
data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now
returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF
images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 09:59:50 +01:00

4.1 KiB

Configurations: recomputing the stations' data from the differential images

2026-11-19. Approved in conversation. Extends 2026-11-19-pump-diff-design.md (its layout of data_diff/ changes: section "data_diff").

Intent

Recompute a station's data from its differential images under configurations of our own (patterns redrawn, thresholds, references, camera correction), compare them, and let every other repository (aimsight-app, tools, detectors, models, optimizer) read the result exactly as it reads data/. data/ stays as it is: the cloud's version, the reference to compare with.

Layout

aimsight-data/
  data/                     the cloud, unchanged (pump.sh)
  data_diff/<SERIAL>/       raw material shared by every configuration (pump_diff.sh), out of git
    images_diff/            the DIFF images; and, invisible to the tools (data.find_files skips images_diff/):
      index/<SERIAL>_diff_<YYYYMMDD>.csv, sensors.json, unit.json
    images/ metrics/ weather/ targets.csv      as in data/ (targets.csv: the cloud's patterns)
  configs/<name>.yaml       one configuration per file, in git
  runs/<name>/              the result of a configuration, out of git, the same tree as data/
    <SERIAL>/
      displacement/ camera/ targets.csv        recomputed, the columns of data/
      images/ metrics/ weather/                symbolic links to data_diff/<SERIAL>/ (nothing copied)
    run.yaml                the configuration used, optical's commit, the date (outside the station folders:
                            invisible to the tools)
    diagnostics/            the emulator's raw rows (edge margin, second spot)

A component reads a configuration by pointing its data link at ~/aimsight-data/runs/<name> instead of ~/aimsight-data/data. Checked on 2026-11-19: data.py (aimsight-tools) reads through symbolic links (images, metrics, temperature, weather).

Configuration

A configuration states only what differs from the cloud's patterns (data_diff/<SERIAL>/images_diff/sensors.json) and from the unit's properties (unit.json); the configuration cloud, with nothing in it, must reproduce data/ (the reference: on JUNGFRAU 2026-10-12..16 the emulator matched the cloud within 0.05 px, errors identical).

stations:
  JUNGFRAU:
    patterns:                       # by sensor_name, over sensors.json
      PM_G:   {error_threshold: 0.7}
      BL_C:   {corners: [[1700, 929], [1724, 953]]}
      REF_CT: {pattern_type: target}
    camera: {focal_distance: 6.3}   # over unit.json and the common camera block
camera:                             # common to the stations of the file
  algorithm: ransac                 # ransac | danish_hybrid | trusted_danish
  mode: "off"                       # off | rotation | rotation_scale | relative

Running a configuration again replaces runs/<name>/: to compare two settings, two files. No cache between the re-measurement and the correction (431 images in 11 s; about 15 min for a year of one station): added if it gets slow.

Rejected: a full copy per configuration (the DIFF images duplicated, 0.6 GB per station-day); a configuration suffix in file names (the tools recognise files by their columns and would mix configurations).

Work, in order

  1. pump_diff (this repository's extraction/): the extras of the DIFF images move under images_diff/; per day, before the DIFF images and their index, the metrics (pump.py's pump_metrics) and the IRON images (pump_images, --every, default 60 min); per station, targets.csv (write_targets, on the DIFF patterns); at the end, the weather (weather.py, given the root folder). A day whose metrics failed gets no index: the next run reads it again. pump.py's only change: pump_metrics returns whether every day succeeded.
  2. run_config: re-measurement (optical's emulator), camera correction (optical's remove_camera_movement and the all-point modes), written in the format of data/. Its own design once optical's correction outputs are mapped to the columns of data/ (relative_x/y, corr_x/y, reproj_err, inlier, sigma0, the camera pose).