Files
aimsight-data/docs/superpowers/specs/2026-11-19-run-config-design.md
T
maximeandClaude Opus 5.5 ca366c150e Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it
- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/,
  weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv,
  patterns.json.
- processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the
  patterns of sensors.json, the camera, optical's current commit); never over an existing one.
- processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked
  against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns)
  and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/.
- pump_diff no longer writes targets.csv: it depends on the run.

JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within
0.034 px; a run created by new_run.sh and run as is gives run01 exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:52:46 +01:00

4.8 KiB

run_config: a run of a station recomputed from its differential images

2026-11-19. Approved in conversation. Step 2 of 2026-11-19-configurations-design.md.

Amended the same day (the user's layout, see the configurations design): processing/run_config.sh <SERIAL> <run> reads data_diff/<SERIAL>/runs/<run>/config.yaml (patterns, camera, emulator.version: optical's commit, the run stops if $OPTICAL is at another one or has local changes; no stations level) and writes only that run's displacement/, camera/, targets.csv (from its patterns; none at the station's root any more) and patterns.json (the patterns as given to the emulator); the emulator's other files go through a temporary folder. No runs/<name>/, no links, no run.yaml. The rest below holds (patterns, correction as the cloud, columns, acceptance).

What the cloud does (monitoring-dashboard app/processing/process_shifts.py, branch explo)

Per DIFF measurement, alone: the non-archived DIFF patterns (load_pattern_from_params) are found in the image; raw x, y, error (and the metrics) are stored; targets above their error_threshold get x/y blanked; a rejected global_ref gets back the x/y of the latest stored frame whose error is not above its own (in practice its own raw position, inserted just before), else its pattern position; remove_camera_movement(r, focal_distance (6.0), pixel_size (0.0024), algorithm (ransac, ransac_threshold 0.99)), which rejects those references from the fit again (error columns given) but still corrects their rows; corr -= (x_ref, y_ref); cam_info (pan, tilt, scale, rotation, sigma0) and ref_stats (reproj_err, inlier) stored. The Advanced export adds relative = x - pattern x and copies sigma0 onto every row; the camera export is the per-minute median / min / max of cam_info.

Design

processing/run_config.py, through processing/run_config.sh (uv: numpy, pandas, pillow, pyyaml, scipy, scikit-image, opencv-python-headless; optical's code from $OPTICAL, default ~/optical):

processing/run_config.sh <config> [SERIAL...] [--from D] [--to E] [--jobs N]

<config>: configs/<name>.yaml or a path; stations: those named, else those of the configuration; days: those indexed in data_diff/<SERIAL>/images_diff/index/ (within --from/--to). For each station, runs/<name>/<SERIAL>/ is replaced:

  1. Patterns: images_diff/sensors.json (DIFF type, a pattern, not archived; latest value of each property), then the configuration by sensor_name (an unknown name or key is an error): any property replaced (error_threshold, pattern_type, distance_convertion_factor, x, y...); redraw: {image, corners} cuts the template from that DIFF image as the device's pattern editor does (estimate_camera_pose from the references, saved_anchor_and_reference: x_ref kept, ImagePatternMono.from_image). Written to diagnostics/<SERIAL>_sensor_properties.json.
  2. Re-measurement: optical's reprocess_images.main (emulator) on the days' images, --times from the index; rows kept in diagnostics/<SERIAL>_remeasured.csv.
  3. Correction: as the cloud, one measurement at a time (per_epoch: true, default): targets above their threshold blanked, references kept raw (rejected in the fit by remove_camera_movement), corr -= x_ref; per_epoch: false: the whole series in one call (a rejected reference interpolated in time, algorithms with memory). Camera block: algorithm (ransac), ransac_threshold (0.99, ransac only), mode (off | rotation | rotation_scale | relative, optical's refine_allpoints), focal_distance, pixel_size (unit.json, else 6.0 / 0.0024), per_epoch; station block over common block over unit.json over the cloud's defaults.
  4. Output, the files of data/: displacement/<S>_advanced_<YYYYMMDD>.csv (columns of data/: x, y, error_value, peak..snr from the emulator, relative = x - pattern x, corr from the correction, reproj_err, inlier of the references, sigma0 of the epoch on every row, corr_*_mm = corr * distance_convertion_factor), camera/<S>_camera_<YYYYMMDD>.csv (median / min / max per minute), targets.csv; images/, metrics/, weather/ as relative symbolic links to data_diff/<SERIAL>/; run.yaml at runs/<name>/ (configuration as resolved per station, optical's commit and state, days, date).

Acceptance: configs/cloud.yaml (empty) on JUNGFRAU 2026-10-12..16 against data/JUNGFRAU: positions and errors within 0.05 px (done by the emulator), corr_x/y and the camera pose close to the cloud, the differences counted.

Check: run_config.py --check in the uv environment: the configuration rules (unknown keys, order of the camera blocks, overrides by name) and a synthetic correction (three references, two targets, one rejected) through optical's remove_camera_movement.