m2.py: one thread per process, --test to share the stations left out between processes, --out on the time split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aimsight-models
Learned detectors for AIMsight: models trained on the reviewed labels to tell, at each measurement instant of a station, which families are present (labeling/families.md), and which sensors carry them. A pilot, not yet in use: the detectors in production are the rules of aimsight-detectors.
The pilot
- Families: Visibility and Obstruction; the others stay rules for now.
- Model: multiple-instance learning, since a label names a station and a range, not its sensors, and each station has its own number of sensors: one small MLP shared by every sensor, attention pooling to a score per family for the station; the attention weights give the sensors.
- Inputs: each sensor against its own usual level (median and IQR over the 3 days before), on windows in hours (the step goes from 5 min to 4 h); the station context (share of high and rejected sensors,
sigma0, references, camera pose, housing temperature, common mode as the median over the targets); the image quality metrics when present, with a mask. - Training: pre-training on the outputs of the current detectors over all the data, then fine-tuning on the reviewed labels of
labeling/auto/. - Evaluation: the measure of
detectors/bench/score.py(found, correct, per family and per instant), against the rules on the same instants. - The criterion (decided 2026-11-16): on the stations seen, the way the model will run (every station followed keeps being labelled in the app): trained on the start of every station, tested on its end (a cut in time, never at random). The model beats the rules by at least 5 points of found and of correct, in total, for Visibility and for Obstruction, with no regression as
detectors/bench/compare.pycounts them (a family of a station losing more than 5 points and more than 20 instants; since 2026-11-16, before: 2 points). A new station stays under the rules until it has a few weeks of reviewed labels; the score with a station left out is followed at every step, as an indicator, not a gate. - Steps: dataset, evaluator (reproducing the rules' figures first), gradient boosting baseline, the PyTorch model, pre-training compared with and without, decoding into episodes.
Use
uv venv .venv && uv pip install -p .venv -r requirements.txt
.venv/bin/python test_dataset.py ~/aimsight-detectors && .venv/bin/python test_evaluate.py && .venv/bin/python test_decode.py
.venv/bin/python dataset.py --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --out cache
.venv/bin/python evaluate.py rules --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --cache cache
.venv/bin/python pretrain.py --detectors ~/aimsight-detectors --data ~/aimsight-data/data --out cache/pre
dataset.py(step 1, ~1 min): onecache/<STATION>.npzper labelled station, over its labelled range as the benchmark reads it:xs[instant, sensor, feature],xc[instant, feature],y[instant, family] with the conventions ofbench/score.py(its labelled instants per family, exactly). NaN where a value does not exist (a 1 h window at a 4 h step, no temperature, no image quality outside LENZSPITZE).evaluate.py rules(step 2, ~1 min, after every change of the detectors or the labels): the rules' predictions per instant incache/rules/, checked against the counts ofbench/score.pystation by station (exit 1 otherwise).evaluate.py score <folder>: predictions per station against the labels, beside the rules, with the regressions ofbench/compare.py.pretrain.py(step 5, ~3 min): every station of the data folder, labelled or not, over its whole range, in chunks of two weeks: the features and the families the rules flag (cache/pre/).
.venv/bin/python m1.py --cache cache --out - --split-time --balance
.venv/bin/python m2.py --cache cache --pretrain cache/pre --out runs/m2_split_pre
for g in BREITHORN,CLARIDEN,FLORIDA JUNGFRAU,LENZSPITZE,MICHIGAN NEBRASKA,NIEDERHORN,NORDEND SNCF01,WILDSTRUBEL; do
.venv/bin/python m2.py --cache cache --pretrain cache/pre --leave-out --seeds 3 --test $g --out runs/m2_pre & done; wait
.venv/bin/python evaluate.py score runs/m2_pre --cache cache
.venv/bin/python decode.py runs/m2_split_pre --cache cache --labels ~/aimsight-labeling/auto --out runs/episodes
m1.py(step 3, ~4 min): gradient boosting on the station's features and the aggregates of its sensors' (median, q90, min, max over the targets and over the references);--split-timeis the criterion, without it a station left out.m2.py(steps 4 and 5, PyTorch, CPU, one thread per process): multiple-instance learning, one MLP shared by every sensor and an attention per family over the sensors. One model alone varies by 10 to 20 points with its seed: 5 averaged (3 with a station left out).--pretrain: first the six families the rules flag on every station and month (never the instants tested), then the fine-tuning on the reviewed labels. Time split ~40 min; a station left out, ~1 h in four processes (memory: ~0.6 GB each with the pre-training set).decode.py(step 6): the flagged instants into episodes (a gap of one instant bridged), scored per episode beside the rules decoded the same way; the sensors of the attention against the sensors the labels name;--out, the episodes in the detectors' format.
Results
On 2026-11-17, after the simplification of the detectors (2026-11-16), which raised the rules a lot (on the last 30 %, Visibility found 49 → 58 %, Obstruction 15 → 33 %). Found / correct per instant, in total:
| Visibility | Obstruction | |
|---|---|---|
| rules, last 30 % of every station | 58 / 78 % | 33 / 41 % |
M1, last 30 % (--balance) |
58 / 85 % | 33 / 75 % |
| M2, last 30 % | 58 / 85 % | 28 / 79 % |
| M2 pre-trained, last 30 % | 58 / 84 % | 56 / 74 % |
| rules, all stations | 57 / 72 % | 40 / 34 % |
| M2, station left out | 57 / 56 % | 13 / 14 % |
| M2 pre-trained, station left out | 57 / 60 % | 9 / 17 % |
Against the criterion (stations seen, the last 30 %), the pre-trained M2:
- Obstruction passes the totals by far: found +23 points, correct +33. The pre-training is what does it (28 → 56 % found).
- Visibility does not: found +0, correct +6. No model, no threshold passes both: lowering M1's threshold to 0.3 gives 63 / 80 % (+5, +2), to 0.2, 67 / 77 %. With these inputs, Visibility is at a ceiling.
- Six regressions: NORDEND Obstruction correct 99 → 81 % (245 instants, flagging twice as much: found 33 → 52 %); NORDEND Visibility found 51 → 43 % (94); JUNGFRAU Visibility found 81 → 69 % (26); LENZSPITZE Visibility correct 98 → 79 % (28); WILDSTRUBEL Visibility found 94 → 89 % (21) and correct 97 → 90 % (28).
A new station stays under the rules, with or without pre-training (Obstruction is labelled on three stations; leaving NORDEND out removes half of its labels): as decided, a new station keeps the rules until it has a few weeks of reviewed labels.
Episodes (pre-trained M2, last 30 %, decode.py): the model finds more labels than the rules (Visibility 72/86 against 58/86, Obstruction 77/90 against 65/90), but its Visibility episodes are cut up: 64 % correct against 85 %, bounds 0.55 against 0.68 (shared instants over the instants of either). Obstruction: 79 % correct against 77 %, bounds 0.33 against 0.17. Bridging two instants instead of one gives 68 % correct for Visibility.
Sensors: for the labels naming 1 to 3 sensors, a named sensor among the first three of the attention: Visibility 13/16, Obstruction 28/32; first: 9/16 and 11/32. A plain ranking by error_value against its usual level (ev_z) does as well for Obstruction (27/32 among three, 17/32 first) and worse for Visibility (10/16, 3/16). The attention does not add much to the plain ranking.
Tried and dropped: the rules' flags as inputs of M1 (--with-rules): found falls (Visibility 56 %, Obstruction 30 %). The weather (MeteoSwiss, ~/aimsight-data/weather/): 4 of the 11 labelled stations have a position in extraction/stations.txt, none of BREITHORN, CLARIDEN, FLORIDA, SNCF01, WILDSTRUBEL.
What comes next, by expected gain:
- Obstruction: the pre-trained M2 is a candidate beside the rules, on the stations already labelled; NORDEND's 245 instants to read with the analyst first (wrongly flagged, or Obstruction the labels missed).
- Visibility needs new information rather than another model: the weather (the positions of the stations in
stations.txt), the image quality metrics on the labelled stations. - Labels where the model hesitates or disagrees with the rules (active learning through the analyst and the review in the app), on more stations for Obstruction.
Contracts
- Reads the measurements through
scripts/data.pyofaimsight-tools(on the path throughdetectors/scripts/categories.py), the reading of the benchmark; the labels fromlabeling/auto/; the data fromdata/(read-only). Paths are passed as arguments. - Never changes
aimsight-detectors(neitherscripts/norbench/): the benchmark is the measure. - Episodes in the format of the detectors' contract (
start,end,kind,label,text,ignore,sensors), so that a model can later sit behinddetect(measurements).
Rules
- Every time is in UTC. No rule specific to a station, a target or a date.
- English for code, identifiers and docs.