2026-11-19 09:59:50 +01:00
# Configurations: recomputing the stations' data from the differential images
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
section "data_diff").
2026-11-19 10:52:46 +01:00
**Replaced the same day by the user's own layout** (the layout below, `configs/` and `runs/<name>/` , is not used):
```
data_diff/<SERIAL>/
images_diff/ (index/, sensors.json, unit.json) images/ metrics/ weather/ pumped, the same for every run
runs/<run>/ one run: whatever depends on it
config.yaml the run's configuration: patterns (by sensor_name), camera and emulator.version (optical's
commit, checked), over the cloud's processing
displacement/ camera/ targets.csv its result, the files of data/ (targets.csv from its patterns)
patterns.json the patterns exactly as given to the emulator, templates included
```
`processing/run_config.sh <SERIAL> <run>` reads the run's `config.yaml` and writes its `displacement/` , `camera/` ,
`targets.csv` and `patterns.json` , nothing else. Rule: what is not the same for every possible run is in the run's
folder. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml` , the cloud's processing written
in full (patterns of `sensors.json` , camera, optical's current commit); never over an existing one. Note: the reading tools (aimsight-tools `data.find_files` ) read a station folder whole, subfolders
included: pointed at `data_diff/<SERIAL>/` , they see every run's curves together.
2026-11-19 09:59:50 +01:00
## Intent
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
thresholds, references, camera correction), compare them, and let every other repository (aimsight-app, tools,
detectors, models, optimizer) read the result exactly as it reads `data/` . `data/` stays as it is: the cloud's
version, the reference to compare with.
## Layout
```
aimsight-data/
data/ the cloud, unchanged (pump.sh)
data_diff/<SERIAL>/ raw material shared by every configuration (pump_diff.sh), out of git
images_diff/ the DIFF images; and, invisible to the tools (data.find_files skips images_diff/):
index/<SERIAL>_diff_<YYYYMMDD>.csv, sensors.json, unit.json
images/ metrics/ weather/ targets.csv as in data/ (targets.csv: the cloud's patterns)
configs/<name>.yaml one configuration per file, in git
runs/<name>/ the result of a configuration, out of git, the same tree as data/
<SERIAL>/
displacement/ camera/ targets.csv recomputed, the columns of data/
images/ metrics/ weather/ symbolic links to data_diff/<SERIAL>/ (nothing copied)
run.yaml the configuration used, optical's commit, the date (outside the station folders:
invisible to the tools)
diagnostics/ the emulator's raw rows (edge margin, second spot)
```
A component reads a configuration by pointing its `data` link at `~/aimsight-data/runs/<name>` instead of
`~/aimsight-data/data` . Checked on 2026-11-19: `data.py` (aimsight-tools) reads through symbolic links (images,
metrics, temperature, weather).
## Configuration
A configuration states only what differs from the cloud's patterns (`data_diff/<SERIAL>/images_diff/sensors.json` )
and from the unit's properties (`unit.json` ); the configuration `cloud` , with nothing in it, must reproduce `data/`
(the reference: on JUNGFRAU 2026-10-12..16 the emulator matched the cloud within 0.05 px, errors identical).
```yaml
stations :
JUNGFRAU :
patterns : # by sensor_name, over sensors.json
PM_G : {error_threshold : 0.7 }
BL_C : {corners : [[ 1700 , 929 ], [ 1724 , 953 ]] }
REF_CT : {pattern_type : target}
camera : {focal_distance : 6.3 } # over unit.json and the common camera block
camera : # common to the stations of the file
algorithm : ransac # ransac | danish_hybrid | trusted_danish
mode : "off" # off | rotation | rotation_scale | relative
```
Running a configuration again replaces `runs/<name>/` : to compare two settings, two files. No cache between the
re-measurement and the correction (431 images in 11 s; about 15 min for a year of one station): added if it gets slow.
Rejected: a full copy per configuration (the DIFF images duplicated, 0.6 GB per station-day); a configuration
suffix in file names (the tools recognise files by their columns and would mix configurations).
## Work, in order
1. ** `pump_diff` ** (this repository's `extraction/` ): the extras of the DIFF images move under `images_diff/` ; per day,
before the DIFF images and their index, the metrics (`pump.py` 's `pump_metrics` ) and the IRON images
(`pump_images` , `--every` , default 60 min); per station, `targets.csv` (`write_targets` , on the DIFF patterns);
at the end, the weather (`weather.py` , given the root folder). A day whose metrics failed gets no index: the next
run reads it again. `pump.py` 's only change: `pump_metrics` returns whether every day succeeded.
2. ** `run_config` **: re-measurement (optical's emulator), camera correction (optical's `remove_camera_movement` and
the all-point modes), written in the format of `data/` . Its own design once optical's correction outputs are mapped
to the columns of `data/` (`relative_x/y, corr_x/y, reproj_err, inlier, sigma0` , the camera pose).