- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/, weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv, patterns.json. - processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the patterns of sensors.json, the camera, optical's current commit); never over an existing one. - processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns) and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/. - pump_diff no longer writes targets.csv: it depends on the run. JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within 0.034 px; a run created by new_run.sh and run as is gives run01 exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4.8 KiB
run_config: a run of a station recomputed from its differential images
2026-11-19. Approved in conversation. Step 2 of 2026-11-19-configurations-design.md.
Amended the same day (the user's layout, see the configurations design): processing/run_config.sh <SERIAL> <run>
reads data_diff/<SERIAL>/runs/<run>/config.yaml (patterns, camera, emulator.version: optical's commit, the run
stops if $OPTICAL is at another one or has local changes; no stations level) and writes only that run's
displacement/, camera/, targets.csv (from its patterns; none at the station's root any more) and patterns.json
(the patterns as given to the emulator); the emulator's other files go through a temporary folder. No runs/<name>/,
no links, no run.yaml. The rest below holds (patterns, correction as
the cloud, columns, acceptance).
What the cloud does (monitoring-dashboard app/processing/process_shifts.py, branch explo)
Per DIFF measurement, alone: the non-archived DIFF patterns (load_pattern_from_params) are found in the image; raw
x, y, error (and the metrics) are stored; targets above their error_threshold get x/y blanked; a rejected
global_ref gets back the x/y of the latest stored frame whose error is not above its own (in practice its own
raw position, inserted just before), else its pattern position; remove_camera_movement(r, focal_distance (6.0), pixel_size (0.0024), algorithm (ransac, ransac_threshold 0.99)), which rejects those references from the fit again
(error columns given) but still corrects their rows; corr -= (x_ref, y_ref); cam_info (pan, tilt, scale,
rotation, sigma0) and ref_stats (reproj_err, inlier) stored. The Advanced export adds relative = x - pattern x
and copies sigma0 onto every row; the camera export is the per-minute median / min / max of cam_info.
Design
processing/run_config.py, through processing/run_config.sh (uv: numpy, pandas, pillow, pyyaml, scipy,
scikit-image, opencv-python-headless; optical's code from $OPTICAL, default ~/optical):
processing/run_config.sh <config> [SERIAL...] [--from D] [--to E] [--jobs N]
<config>: configs/<name>.yaml or a path; stations: those named, else those of the configuration; days: those
indexed in data_diff/<SERIAL>/images_diff/index/ (within --from/--to). For each station, runs/<name>/<SERIAL>/
is replaced:
- Patterns:
images_diff/sensors.json(DIFF type, a pattern, not archived; latest value of each property), then the configuration bysensor_name(an unknown name or key is an error): any property replaced (error_threshold,pattern_type,distance_convertion_factor,x,y...);redraw: {image, corners}cuts the template from that DIFF image as the device's pattern editor does (estimate_camera_posefrom the references,saved_anchor_and_reference:x_refkept,ImagePatternMono.from_image). Written todiagnostics/<SERIAL>_sensor_properties.json. - Re-measurement: optical's
reprocess_images.main(emulator) on the days' images,--timesfrom the index; rows kept indiagnostics/<SERIAL>_remeasured.csv. - Correction: as the cloud, one measurement at a time (
per_epoch: true, default): targets above their threshold blanked, references kept raw (rejected in the fit byremove_camera_movement),corr -= x_ref;per_epoch: false: the whole series in one call (a rejected reference interpolated in time, algorithms with memory). Camera block:algorithm(ransac),ransac_threshold(0.99, ransac only),mode(off|rotation|rotation_scale|relative, optical'srefine_allpoints),focal_distance,pixel_size(unit.json, else 6.0 / 0.0024),per_epoch; station block over common block over unit.json over the cloud's defaults. - Output, the files of
data/:displacement/<S>_advanced_<YYYYMMDD>.csv(columns ofdata/:x, y, error_value, peak..snrfrom the emulator,relative = x - pattern x,corrfrom the correction,reproj_err, inlierof the references,sigma0of the epoch on every row,corr_*_mm = corr * distance_convertion_factor),camera/<S>_camera_<YYYYMMDD>.csv(median / min / max per minute),targets.csv;images/,metrics/,weather/as relative symbolic links todata_diff/<SERIAL>/;run.yamlatruns/<name>/(configuration as resolved per station, optical's commit and state, days, date).
Acceptance: configs/cloud.yaml (empty) on JUNGFRAU 2026-10-12..16 against data/JUNGFRAU: positions and errors
within 0.05 px (done by the emulator), corr_x/y and the camera pose close to the cloud, the differences counted.
Check: run_config.py --check in the uv environment: the configuration rules (unknown keys, order of the camera
blocks, overrides by name) and a synthetic correction (three references, two targets, one rejected) through
optical's remove_camera_movement.