- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/, weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv, patterns.json. - processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the patterns of sensors.json, the camera, optical's current commit); never over an existing one. - processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns) and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/. - pump_diff no longer writes targets.csv: it depends on the run. JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within 0.034 px; a run created by new_run.sh and run as is gives run01 exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.4 KiB
Configurations: recomputing the stations' data from the differential images
2026-11-19. Approved in conversation. Extends 2026-11-19-pump-diff-design.md (its layout of data_diff/ changes:
section "data_diff").
Replaced the same day by the user's own layout (the layout below, configs/ and runs/<name>/, is not used):
data_diff/<SERIAL>/
images_diff/ (index/, sensors.json, unit.json) images/ metrics/ weather/ pumped, the same for every run
runs/<run>/ one run: whatever depends on it
config.yaml the run's configuration: patterns (by sensor_name), camera and emulator.version (optical's
commit, checked), over the cloud's processing
displacement/ camera/ targets.csv its result, the files of data/ (targets.csv from its patterns)
patterns.json the patterns exactly as given to the emulator, templates included
processing/run_config.sh <SERIAL> <run> reads the run's config.yaml and writes its displacement/, camera/,
targets.csv and patterns.json, nothing else. Rule: what is not the same for every possible run is in the run's
folder. processing/new_run.sh <SERIAL> <run> creates a run, empty: its config.yaml, the cloud's processing written
in full (patterns of sensors.json, camera, optical's current commit); never over an existing one. Note: the reading tools (aimsight-tools data.find_files) read a station folder whole, subfolders
included: pointed at data_diff/<SERIAL>/, they see every run's curves together.
Intent
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
thresholds, references, camera correction), compare them, and let every other repository (aimsight-app, tools,
detectors, models, optimizer) read the result exactly as it reads data/. data/ stays as it is: the cloud's
version, the reference to compare with.
Layout
aimsight-data/
data/ the cloud, unchanged (pump.sh)
data_diff/<SERIAL>/ raw material shared by every configuration (pump_diff.sh), out of git
images_diff/ the DIFF images; and, invisible to the tools (data.find_files skips images_diff/):
index/<SERIAL>_diff_<YYYYMMDD>.csv, sensors.json, unit.json
images/ metrics/ weather/ targets.csv as in data/ (targets.csv: the cloud's patterns)
configs/<name>.yaml one configuration per file, in git
runs/<name>/ the result of a configuration, out of git, the same tree as data/
<SERIAL>/
displacement/ camera/ targets.csv recomputed, the columns of data/
images/ metrics/ weather/ symbolic links to data_diff/<SERIAL>/ (nothing copied)
run.yaml the configuration used, optical's commit, the date (outside the station folders:
invisible to the tools)
diagnostics/ the emulator's raw rows (edge margin, second spot)
A component reads a configuration by pointing its data link at ~/aimsight-data/runs/<name> instead of
~/aimsight-data/data. Checked on 2026-11-19: data.py (aimsight-tools) reads through symbolic links (images,
metrics, temperature, weather).
Configuration
A configuration states only what differs from the cloud's patterns (data_diff/<SERIAL>/images_diff/sensors.json)
and from the unit's properties (unit.json); the configuration cloud, with nothing in it, must reproduce data/
(the reference: on JUNGFRAU 2026-10-12..16 the emulator matched the cloud within 0.05 px, errors identical).
stations:
JUNGFRAU:
patterns: # by sensor_name, over sensors.json
PM_G: {error_threshold: 0.7}
BL_C: {corners: [[1700, 929], [1724, 953]]}
REF_CT: {pattern_type: target}
camera: {focal_distance: 6.3} # over unit.json and the common camera block
camera: # common to the stations of the file
algorithm: ransac # ransac | danish_hybrid | trusted_danish
mode: "off" # off | rotation | rotation_scale | relative
Running a configuration again replaces runs/<name>/: to compare two settings, two files. No cache between the
re-measurement and the correction (431 images in 11 s; about 15 min for a year of one station): added if it gets slow.
Rejected: a full copy per configuration (the DIFF images duplicated, 0.6 GB per station-day); a configuration suffix in file names (the tools recognise files by their columns and would mix configurations).
Work, in order
pump_diff(this repository'sextraction/): the extras of the DIFF images move underimages_diff/; per day, before the DIFF images and their index, the metrics (pump.py'spump_metrics) and the IRON images (pump_images,--every, default 60 min); per station,targets.csv(write_targets, on the DIFF patterns); at the end, the weather (weather.py, given the root folder). A day whose metrics failed gets no index: the next run reads it again.pump.py's only change:pump_metricsreturns whether every day succeeded.run_config: re-measurement (optical's emulator), camera correction (optical'sremove_camera_movementand the all-point modes), written in the format ofdata/. Its own design once optical's correction outputs are mapped to the columns ofdata/(relative_x/y, corr_x/y, reproj_err, inlier, sigma0, the camera pose).