70 lines
4.1 KiB
Markdown
70 lines
4.1 KiB
Markdown
# Configurations: recomputing the stations' data from the differential images
|
|||
|
|
|
||
|
|
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
|
||
|
|
section "data_diff").
|
||
|
|
|
||
|
|
## Intent
|
||
|
|
|
||
|
|
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
|
||
|
|
thresholds, references, camera correction), compare them, and let every other repository (aimsight-app, tools,
|
||
|
|
detectors, models, optimizer) read the result exactly as it reads `data/`. `data/` stays as it is: the cloud's
|
||
|
|
version, the reference to compare with.
|
||
|
|
|
||
|
|
## Layout
|
||
|
|
|
||
|
|
```
|
||
|
|
aimsight-data/
|
||
|
|
data/ the cloud, unchanged (pump.sh)
|
||
|
|
data_diff/<SERIAL>/ raw material shared by every configuration (pump_diff.sh), out of git
|
||
|
|
images_diff/ the DIFF images; and, invisible to the tools (data.find_files skips images_diff/):
|
||
|
|
index/<SERIAL>_diff_<YYYYMMDD>.csv, sensors.json, unit.json
|
||
|
|
images/ metrics/ weather/ targets.csv as in data/ (targets.csv: the cloud's patterns)
|
||
|
|
configs/<name>.yaml one configuration per file, in git
|
||
|
|
runs/<name>/ the result of a configuration, out of git, the same tree as data/
|
||
|
|
<SERIAL>/
|
||
|
|
displacement/ camera/ targets.csv recomputed, the columns of data/
|
||
|
|
images/ metrics/ weather/ symbolic links to data_diff/<SERIAL>/ (nothing copied)
|
||
|
|
run.yaml the configuration used, optical's commit, the date (outside the station folders:
|
||
|
|
invisible to the tools)
|
||
|
|
diagnostics/ the emulator's raw rows (edge margin, second spot)
|
||
|
|
```
|
||
|
|
|
||
|
|
A component reads a configuration by pointing its `data` link at `~/aimsight-data/runs/<name>` instead of
|
||
|
|
`~/aimsight-data/data`. Checked on 2026-11-19: `data.py` (aimsight-tools) reads through symbolic links (images,
|
||
|
|
metrics, temperature, weather).
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
|
||
|
|
A configuration states only what differs from the cloud's patterns (`data_diff/<SERIAL>/images_diff/sensors.json`)
|
||
|
|
and from the unit's properties (`unit.json`); the configuration `cloud`, with nothing in it, must reproduce `data/`
|
||
|
|
(the reference: on JUNGFRAU 2026-10-12..16 the emulator matched the cloud within 0.05 px, errors identical).
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
stations:
|
||
|
|
JUNGFRAU:
|
||
|
|
patterns: # by sensor_name, over sensors.json
|
||
|
|
PM_G: {error_threshold: 0.7}
|
||
|
|
BL_C: {corners: [[1700, 929], [1724, 953]]}
|
||
|
|
REF_CT: {pattern_type: target}
|
||
|
|
camera: {focal_distance: 6.3} # over unit.json and the common camera block
|
||
|
|
camera: # common to the stations of the file
|
||
|
|
algorithm: ransac # ransac | danish_hybrid | trusted_danish
|
||
|
|
mode: "off" # off | rotation | rotation_scale | relative
|
||
|
|
```
|
||
|
|
|
||
|
|
Running a configuration again replaces `runs/<name>/`: to compare two settings, two files. No cache between the
|
||
|
|
re-measurement and the correction (431 images in 11 s; about 15 min for a year of one station): added if it gets slow.
|
||
|
|
|
||
|
|
Rejected: a full copy per configuration (the DIFF images duplicated, 0.6 GB per station-day); a configuration
|
||
|
|
suffix in file names (the tools recognise files by their columns and would mix configurations).
|
||
|
|
|
||
|
|
## Work, in order
|
||
|
|
|
||
|
|
1. **`pump_diff`** (this repository's `extraction/`): the extras of the DIFF images move under `images_diff/`; per day,
|
||
|
|
before the DIFF images and their index, the metrics (`pump.py`'s `pump_metrics`) and the IRON images
|
||
|
|
(`pump_images`, `--every`, default 60 min); per station, `targets.csv` (`write_targets`, on the DIFF patterns);
|
||
|
|
at the end, the weather (`weather.py`, given the root folder). A day whose metrics failed gets no index: the next
|
||
|
|
run reads it again. `pump.py`'s only change: `pump_metrics` returns whether every day succeeded.
|
||
|
|
2. **`run_config`**: re-measurement (optical's emulator), camera correction (optical's `remove_camera_movement` and
|
||
|
|
the all-point modes), written in the format of `data/`. Its own design once optical's correction outputs are mapped
|
||
|
|
to the columns of `data/` (`relative_x/y, corr_x/y, reproj_err, inlier, sigma0`, the camera pose).
|