Files
aimsight-data/docs/superpowers/specs/2026-11-19-configurations-design.md
T
maximeandClaude Opus 5.5 ca366c150e Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it
- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/,
  weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv,
  patterns.json.
- processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the
  patterns of sensors.json, the camera, optical's current commit); never over an existing one.
- processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked
  against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns)
  and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/.
- pump_diff no longer writes targets.csv: it depends on the run.

JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within
0.034 px; a run created by new_run.sh and run as is gives run01 exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:52:46 +01:00

5.4 KiB

Configurations: recomputing the stations' data from the differential images

2026-11-19. Approved in conversation. Extends 2026-11-19-pump-diff-design.md (its layout of data_diff/ changes: section "data_diff").

Replaced the same day by the user's own layout (the layout below, configs/ and runs/<name>/, is not used):

data_diff/<SERIAL>/
  images_diff/ (index/, sensors.json, unit.json)  images/  metrics/  weather/    pumped, the same for every run
  runs/<run>/                                                                   one run: whatever depends on it
    config.yaml       the run's configuration: patterns (by sensor_name), camera and emulator.version (optical's
                      commit, checked), over the cloud's processing
    displacement/  camera/  targets.csv      its result, the files of data/ (targets.csv from its patterns)
    patterns.json     the patterns exactly as given to the emulator, templates included

processing/run_config.sh <SERIAL> <run> reads the run's config.yaml and writes its displacement/, camera/, targets.csv and patterns.json, nothing else. Rule: what is not the same for every possible run is in the run's folder. processing/new_run.sh <SERIAL> <run> creates a run, empty: its config.yaml, the cloud's processing written in full (patterns of sensors.json, camera, optical's current commit); never over an existing one. Note: the reading tools (aimsight-tools data.find_files) read a station folder whole, subfolders included: pointed at data_diff/<SERIAL>/, they see every run's curves together.

Intent

Recompute a station's data from its differential images under configurations of our own (patterns redrawn, thresholds, references, camera correction), compare them, and let every other repository (aimsight-app, tools, detectors, models, optimizer) read the result exactly as it reads data/. data/ stays as it is: the cloud's version, the reference to compare with.

Layout

aimsight-data/
  data/                     the cloud, unchanged (pump.sh)
  data_diff/<SERIAL>/       raw material shared by every configuration (pump_diff.sh), out of git
    images_diff/            the DIFF images; and, invisible to the tools (data.find_files skips images_diff/):
      index/<SERIAL>_diff_<YYYYMMDD>.csv, sensors.json, unit.json
    images/ metrics/ weather/ targets.csv      as in data/ (targets.csv: the cloud's patterns)
  configs/<name>.yaml       one configuration per file, in git
  runs/<name>/              the result of a configuration, out of git, the same tree as data/
    <SERIAL>/
      displacement/ camera/ targets.csv        recomputed, the columns of data/
      images/ metrics/ weather/                symbolic links to data_diff/<SERIAL>/ (nothing copied)
    run.yaml                the configuration used, optical's commit, the date (outside the station folders:
                            invisible to the tools)
    diagnostics/            the emulator's raw rows (edge margin, second spot)

A component reads a configuration by pointing its data link at ~/aimsight-data/runs/<name> instead of ~/aimsight-data/data. Checked on 2026-11-19: data.py (aimsight-tools) reads through symbolic links (images, metrics, temperature, weather).

Configuration

A configuration states only what differs from the cloud's patterns (data_diff/<SERIAL>/images_diff/sensors.json) and from the unit's properties (unit.json); the configuration cloud, with nothing in it, must reproduce data/ (the reference: on JUNGFRAU 2026-10-12..16 the emulator matched the cloud within 0.05 px, errors identical).

stations:
  JUNGFRAU:
    patterns:                       # by sensor_name, over sensors.json
      PM_G:   {error_threshold: 0.7}
      BL_C:   {corners: [[1700, 929], [1724, 953]]}
      REF_CT: {pattern_type: target}
    camera: {focal_distance: 6.3}   # over unit.json and the common camera block
camera:                             # common to the stations of the file
  algorithm: ransac                 # ransac | danish_hybrid | trusted_danish
  mode: "off"                       # off | rotation | rotation_scale | relative

Running a configuration again replaces runs/<name>/: to compare two settings, two files. No cache between the re-measurement and the correction (431 images in 11 s; about 15 min for a year of one station): added if it gets slow.

Rejected: a full copy per configuration (the DIFF images duplicated, 0.6 GB per station-day); a configuration suffix in file names (the tools recognise files by their columns and would mix configurations).

Work, in order

  1. pump_diff (this repository's extraction/): the extras of the DIFF images move under images_diff/; per day, before the DIFF images and their index, the metrics (pump.py's pump_metrics) and the IRON images (pump_images, --every, default 60 min); per station, targets.csv (write_targets, on the DIFF patterns); at the end, the weather (weather.py, given the root folder). A day whose metrics failed gets no index: the next run reads it again. pump.py's only change: pump_metrics returns whether every day succeeded.
  2. run_config: re-measurement (optical's emulator), camera correction (optical's remove_camera_movement and the all-point modes), written in the format of data/. Its own design once optical's correction outputs are mapped to the columns of data/ (relative_x/y, corr_x/y, reproj_err, inlier, sigma0, the camera pose).