m2.py: one thread per process, --test to share the stations left out between processes, --out on the time split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
84 lines
9.2 KiB
Markdown
84 lines
9.2 KiB
Markdown
# aimsight-models
|
|
|
|
Learned detectors for AIMsight: models trained on the reviewed labels to tell, at each measurement instant of a station, which families are present (`labeling/families.md`), and which sensors carry them. A pilot, not yet in use: the detectors in production are the rules of `aimsight-detectors`.
|
|
|
|
## The pilot
|
|
|
|
- **Families**: Visibility and Obstruction; the others stay rules for now.
|
|
- **Model**: multiple-instance learning, since a label names a station and a range, not its sensors, and each station has its own number of sensors: one small MLP shared by every sensor, attention pooling to a score per family for the station; the attention weights give the sensors.
|
|
- **Inputs**: each sensor against its own usual level (median and IQR over the 3 days before), on windows in hours (the step goes from 5 min to 4 h); the station context (share of high and rejected sensors, `sigma0`, references, camera pose, housing temperature, common mode as the median over the targets); the image quality metrics when present, with a mask.
|
|
- **Training**: pre-training on the outputs of the current detectors over all the data, then fine-tuning on the reviewed labels of `labeling/auto/`.
|
|
- **Evaluation**: the measure of `detectors/bench/score.py` (found, correct, per family and per instant), against the rules on the same instants.
|
|
- **The criterion** (decided 2026-11-16): on the stations seen, the way the model will run (every station followed keeps being labelled in the app): trained on the start of every station, tested on its end (a cut in time, never at random). The model beats the rules by at least 5 points of found and of correct, in total, for Visibility and for Obstruction, with no regression as `detectors/bench/compare.py` counts them (a family of a station losing more than 5 points and more than 20 instants; since 2026-11-16, before: 2 points). A new station stays under the rules until it has a few weeks of reviewed labels; the score with a station left out is followed at every step, as an indicator, not a gate.
|
|
- **Steps**: dataset, evaluator (reproducing the rules' figures first), gradient boosting baseline, the PyTorch model, pre-training compared with and without, decoding into episodes.
|
|
|
|
## Use
|
|
|
|
```bash
|
|
uv venv .venv && uv pip install -p .venv -r requirements.txt
|
|
.venv/bin/python test_dataset.py ~/aimsight-detectors && .venv/bin/python test_evaluate.py && .venv/bin/python test_decode.py
|
|
.venv/bin/python dataset.py --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --out cache
|
|
.venv/bin/python evaluate.py rules --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --cache cache
|
|
.venv/bin/python pretrain.py --detectors ~/aimsight-detectors --data ~/aimsight-data/data --out cache/pre
|
|
```
|
|
|
|
- `dataset.py` (step 1, ~1 min): one `cache/<STATION>.npz` per labelled station, over its labelled range as the benchmark reads it: `xs` [instant, sensor, feature], `xc` [instant, feature], `y` [instant, family] with the conventions of `bench/score.py` (its labelled instants per family, exactly). NaN where a value does not exist (a 1 h window at a 4 h step, no temperature, no image quality outside LENZSPITZE).
|
|
- `evaluate.py rules` (step 2, ~1 min, after every change of the detectors or the labels): the rules' predictions per instant in `cache/rules/`, checked against the counts of `bench/score.py` station by station (exit 1 otherwise). `evaluate.py score <folder>`: predictions per station against the labels, beside the rules, with the regressions of `bench/compare.py`.
|
|
- `pretrain.py` (step 5, ~3 min): every station of the data folder, labelled or not, over its whole range, in chunks of two weeks: the features and the families the rules flag (`cache/pre/`).
|
|
|
|
```bash
|
|
.venv/bin/python m1.py --cache cache --out - --split-time --balance
|
|
.venv/bin/python m2.py --cache cache --pretrain cache/pre --out runs/m2_split_pre
|
|
for g in BREITHORN,CLARIDEN,FLORIDA JUNGFRAU,LENZSPITZE,MICHIGAN NEBRASKA,NIEDERHORN,NORDEND SNCF01,WILDSTRUBEL; do
|
|
.venv/bin/python m2.py --cache cache --pretrain cache/pre --leave-out --seeds 3 --test $g --out runs/m2_pre & done; wait
|
|
.venv/bin/python evaluate.py score runs/m2_pre --cache cache
|
|
.venv/bin/python decode.py runs/m2_split_pre --cache cache --labels ~/aimsight-labeling/auto --out runs/episodes
|
|
```
|
|
|
|
- `m1.py` (step 3, ~4 min): gradient boosting on the station's features and the aggregates of its sensors' (median, q90, min, max over the targets and over the references); `--split-time` is the criterion, without it a station left out.
|
|
- `m2.py` (steps 4 and 5, PyTorch, CPU, one thread per process): multiple-instance learning, one MLP shared by every sensor and an attention per family over the sensors. One model alone varies by 10 to 20 points with its seed: 5 averaged (3 with a station left out). `--pretrain`: first the six families the rules flag on every station and month (never the instants tested), then the fine-tuning on the reviewed labels. Time split ~40 min; a station left out, ~1 h in four processes (memory: ~0.6 GB each with the pre-training set).
|
|
- `decode.py` (step 6): the flagged instants into episodes (a gap of one instant bridged), scored per episode beside the rules decoded the same way; the sensors of the attention against the sensors the labels name; `--out`, the episodes in the detectors' format.
|
|
|
|
## Results
|
|
|
|
On 2026-11-17, after the simplification of the detectors (2026-11-16), which raised the rules a lot (on the last 30 %, Visibility found 49 → 58 %, Obstruction 15 → 33 %). Found / correct per instant, in total:
|
|
|
|
| | Visibility | Obstruction |
|
|
|---|---|---|
|
|
| rules, last 30 % of every station | 58 / 78 % | 33 / 41 % |
|
|
| M1, last 30 % (`--balance`) | 58 / 85 % | 33 / 75 % |
|
|
| M2, last 30 % | 58 / 85 % | 28 / 79 % |
|
|
| **M2 pre-trained, last 30 %** | **58 / 84 %** | **56 / 74 %** |
|
|
| rules, all stations | 57 / 72 % | 40 / 34 % |
|
|
| M2, station left out | 57 / 56 % | 13 / 14 % |
|
|
| M2 pre-trained, station left out | 57 / 60 % | 9 / 17 % |
|
|
|
|
**Against the criterion** (stations seen, the last 30 %), the pre-trained M2:
|
|
- Obstruction passes the totals by far: found +23 points, correct +33. The pre-training is what does it (28 → 56 % found).
|
|
- Visibility does not: found +0, correct +6. No model, no threshold passes both: lowering M1's threshold to 0.3 gives 63 / 80 % (+5, +2), to 0.2, 67 / 77 %. With these inputs, Visibility is at a ceiling.
|
|
- Six regressions: NORDEND Obstruction correct 99 → 81 % (245 instants, flagging twice as much: found 33 → 52 %); NORDEND Visibility found 51 → 43 % (94); JUNGFRAU Visibility found 81 → 69 % (26); LENZSPITZE Visibility correct 98 → 79 % (28); WILDSTRUBEL Visibility found 94 → 89 % (21) and correct 97 → 90 % (28).
|
|
|
|
**A new station** stays under the rules, with or without pre-training (Obstruction is labelled on three stations; leaving NORDEND out removes half of its labels): as decided, a new station keeps the rules until it has a few weeks of reviewed labels.
|
|
|
|
**Episodes** (pre-trained M2, last 30 %, `decode.py`): the model finds more labels than the rules (Visibility 72/86 against 58/86, Obstruction 77/90 against 65/90), but its Visibility episodes are cut up: 64 % correct against 85 %, bounds 0.55 against 0.68 (shared instants over the instants of either). Obstruction: 79 % correct against 77 %, bounds 0.33 against 0.17. Bridging two instants instead of one gives 68 % correct for Visibility.
|
|
|
|
**Sensors**: for the labels naming 1 to 3 sensors, a named sensor among the first three of the attention: Visibility 13/16, Obstruction 28/32; first: 9/16 and 11/32. A plain ranking by error_value against its usual level (`ev_z`) does as well for Obstruction (27/32 among three, 17/32 first) and worse for Visibility (10/16, 3/16). The attention does not add much to the plain ranking.
|
|
|
|
**Tried and dropped**: the rules' flags as inputs of M1 (`--with-rules`): found falls (Visibility 56 %, Obstruction 30 %). The weather (MeteoSwiss, `~/aimsight-data/weather/`): 4 of the 11 labelled stations have a position in `extraction/stations.txt`, none of BREITHORN, CLARIDEN, FLORIDA, SNCF01, WILDSTRUBEL.
|
|
|
|
**What comes next**, by expected gain:
|
|
1. Obstruction: the pre-trained M2 is a candidate beside the rules, on the stations already labelled; NORDEND's 245 instants to read with the analyst first (wrongly flagged, or Obstruction the labels missed).
|
|
2. Visibility needs new information rather than another model: the weather (the positions of the stations in `stations.txt`), the image quality metrics on the labelled stations.
|
|
3. Labels where the model hesitates or disagrees with the rules (active learning through the analyst and the review in the app), on more stations for Obstruction.
|
|
|
|
## Contracts
|
|
|
|
- Reads the measurements through `scripts/data.py` of `aimsight-tools` (on the path through `detectors/scripts/categories.py`), the reading of the benchmark; the labels from `labeling/auto/`; the data from `data/` (read-only). Paths are passed as arguments.
|
|
- Never changes `aimsight-detectors` (neither `scripts/` nor `bench/`): the benchmark is the measure.
|
|
- Episodes in the format of the detectors' contract (`start`, `end`, `kind`, `label`, `text`, `ignore`, `sensors`), so that a model can later sit behind `detect(measurements)`.
|
|
|
|
## Rules
|
|
|
|
- Every time is in UTC. No rule specific to a station, a target or a date.
|
|
- English for code, identifiers and docs.
|