Files
aimsight-data/docs/superpowers/specs/2026-11-19-configurations-design.md
T

70 lines
4.1 KiB
Markdown
Raw Normal View History

# Configurations: recomputing the stations' data from the differential images
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
section "data_diff").
## Intent
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
thresholds, references, camera correction), compare them, and let every other repository (aimsight-app, tools,
detectors, models, optimizer) read the result exactly as it reads `data/`. `data/` stays as it is: the cloud's
version, the reference to compare with.
## Layout
```
aimsight-data/
data/ the cloud, unchanged (pump.sh)
data_diff/<SERIAL>/ raw material shared by every configuration (pump_diff.sh), out of git
images_diff/ the DIFF images; and, invisible to the tools (data.find_files skips images_diff/):
index/<SERIAL>_diff_<YYYYMMDD>.csv, sensors.json, unit.json
images/ metrics/ weather/ targets.csv as in data/ (targets.csv: the cloud's patterns)
configs/<name>.yaml one configuration per file, in git
runs/<name>/ the result of a configuration, out of git, the same tree as data/
<SERIAL>/
displacement/ camera/ targets.csv recomputed, the columns of data/
images/ metrics/ weather/ symbolic links to data_diff/<SERIAL>/ (nothing copied)
run.yaml the configuration used, optical's commit, the date (outside the station folders:
invisible to the tools)
diagnostics/ the emulator's raw rows (edge margin, second spot)
```
A component reads a configuration by pointing its `data` link at `~/aimsight-data/runs/<name>` instead of
`~/aimsight-data/data`. Checked on 2026-11-19: `data.py` (aimsight-tools) reads through symbolic links (images,
metrics, temperature, weather).
## Configuration
A configuration states only what differs from the cloud's patterns (`data_diff/<SERIAL>/images_diff/sensors.json`)
and from the unit's properties (`unit.json`); the configuration `cloud`, with nothing in it, must reproduce `data/`
(the reference: on JUNGFRAU 2026-10-12..16 the emulator matched the cloud within 0.05 px, errors identical).
```yaml
stations:
JUNGFRAU:
patterns: # by sensor_name, over sensors.json
PM_G: {error_threshold: 0.7}
BL_C: {corners: [[1700, 929], [1724, 953]]}
REF_CT: {pattern_type: target}
camera: {focal_distance: 6.3} # over unit.json and the common camera block
camera: # common to the stations of the file
algorithm: ransac # ransac | danish_hybrid | trusted_danish
mode: "off" # off | rotation | rotation_scale | relative
```
Running a configuration again replaces `runs/<name>/`: to compare two settings, two files. No cache between the
re-measurement and the correction (431 images in 11 s; about 15 min for a year of one station): added if it gets slow.
Rejected: a full copy per configuration (the DIFF images duplicated, 0.6 GB per station-day); a configuration
suffix in file names (the tools recognise files by their columns and would mix configurations).
## Work, in order
1. **`pump_diff`** (this repository's `extraction/`): the extras of the DIFF images move under `images_diff/`; per day,
before the DIFF images and their index, the metrics (`pump.py`'s `pump_metrics`) and the IRON images
(`pump_images`, `--every`, default 60 min); per station, `targets.csv` (`write_targets`, on the DIFF patterns);
at the end, the weather (`weather.py`, given the root folder). A day whose metrics failed gets no index: the next
run reads it again. `pump.py`'s only change: `pump_metrics` returns whether every day succeeded.
2. **`run_config`**: re-measurement (optical's emulator), camera correction (optical's `remove_camera_movement` and
the all-point modes), written in the format of `data/`. Its own design once optical's correction outputs are mapped
to the columns of `data/` (`relative_x/y, corr_x/y, reproj_err, inlier, sigma0`, the camera pose).