Files
aimsight-data/docs/superpowers/specs/2026-11-19-run-config-design.md
T
maximeandClaude Opus 5.5 d3bf4b6860 Configurations: processing/run_config.sh recomputes configs/<name>.yaml from data_diff/ into runs/<name>/, read like data/
Patterns by sensor_name (a property, or a window redrawn from a DIFF image as the device's pattern editor does),
re-measured with optical's emulator, camera correction as the cloud's process_shifts (one measurement at a time, or
the series), written in the files of data/ (displacement/, camera/, targets.csv; images/, metrics/, weather/ linked to
data_diff/), with run.yaml and diagnostics/ beside the stations. configs/cloud.yaml, empty, reproduces data/JUNGFRAU
2026-10-12..16: corr_x/y within 0.03 px, the same rejections, the camera pose within 5e-4 deg; 16 s.
configs/jungfrau_example.yaml: PM_G back to 0.7, BL_C's window redrawn. .gitignore: data_diff/, runs/, remeasure/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:00:02 +01:00

4.1 KiB

run_config: a configuration recomputed into runs//

2026-11-19. Approved in conversation. Step 2 of 2026-11-19-configurations-design.md.

What the cloud does (monitoring-dashboard app/processing/process_shifts.py, branch explo)

Per DIFF measurement, alone: the non-archived DIFF patterns (load_pattern_from_params) are found in the image; raw x, y, error (and the metrics) are stored; targets above their error_threshold get x/y blanked; a rejected global_ref gets back the x/y of the latest stored frame whose error is not above its own (in practice its own raw position, inserted just before), else its pattern position; remove_camera_movement(r, focal_distance (6.0), pixel_size (0.0024), algorithm (ransac, ransac_threshold 0.99)), which rejects those references from the fit again (error columns given) but still corrects their rows; corr -= (x_ref, y_ref); cam_info (pan, tilt, scale, rotation, sigma0) and ref_stats (reproj_err, inlier) stored. The Advanced export adds relative = x - pattern x and copies sigma0 onto every row; the camera export is the per-minute median / min / max of cam_info.

Design

processing/run_config.py, through processing/run_config.sh (uv: numpy, pandas, pillow, pyyaml, scipy, scikit-image, opencv-python-headless; optical's code from $OPTICAL, default ~/optical):

processing/run_config.sh <config> [SERIAL...] [--from D] [--to E] [--jobs N]

<config>: configs/<name>.yaml or a path; stations: those named, else those of the configuration; days: those indexed in data_diff/<SERIAL>/images_diff/index/ (within --from/--to). For each station, runs/<name>/<SERIAL>/ is replaced:

  1. Patterns: images_diff/sensors.json (DIFF type, a pattern, not archived; latest value of each property), then the configuration by sensor_name (an unknown name or key is an error): any property replaced (error_threshold, pattern_type, distance_convertion_factor, x, y...); redraw: {image, corners} cuts the template from that DIFF image as the device's pattern editor does (estimate_camera_pose from the references, saved_anchor_and_reference: x_ref kept, ImagePatternMono.from_image). Written to diagnostics/<SERIAL>_sensor_properties.json.
  2. Re-measurement: optical's reprocess_images.main (emulator) on the days' images, --times from the index; rows kept in diagnostics/<SERIAL>_remeasured.csv.
  3. Correction: as the cloud, one measurement at a time (per_epoch: true, default): targets above their threshold blanked, references kept raw (rejected in the fit by remove_camera_movement), corr -= x_ref; per_epoch: false: the whole series in one call (a rejected reference interpolated in time, algorithms with memory). Camera block: algorithm (ransac), ransac_threshold (0.99, ransac only), mode (off | rotation | rotation_scale | relative, optical's refine_allpoints), focal_distance, pixel_size (unit.json, else 6.0 / 0.0024), per_epoch; station block over common block over unit.json over the cloud's defaults.
  4. Output, the files of data/: displacement/<S>_advanced_<YYYYMMDD>.csv (columns of data/: x, y, error_value, peak..snr from the emulator, relative = x - pattern x, corr from the correction, reproj_err, inlier of the references, sigma0 of the epoch on every row, corr_*_mm = corr * distance_convertion_factor), camera/<S>_camera_<YYYYMMDD>.csv (median / min / max per minute), targets.csv; images/, metrics/, weather/ as relative symbolic links to data_diff/<SERIAL>/; run.yaml at runs/<name>/ (configuration as resolved per station, optical's commit and state, days, date).

Acceptance: configs/cloud.yaml (empty) on JUNGFRAU 2026-10-12..16 against data/JUNGFRAU: positions and errors within 0.05 px (done by the emulator), corr_x/y and the camera pose close to the cloud, the differences counted.

Check: run_config.py --check in the uv environment: the configuration rules (unknown keys, order of the camera blocks, overrides by name) and a synthetic correction (three references, two targets, one rejected) through optical's remove_camera_movement.