Steps 5 and 6: pre-trained M2 passes the criterion for Obstruction, not for Visibility; episodes and sensors (decode.py)

m2.py: one thread per process, --test to share the stations left out between processes, --out on the time split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-17 11:59:49 +01:00
1 parent 5337dd0dac
commit 4836929bc7
4 files changed
+212 -42

No files matched your search

+41 -28
View File
@@ -6,61 +6,74 @@ Learned detectors for AIMsight: models trained on the reviewed labels to tell, a
- **Families**: Visibility and Obstruction; the others stay rules for now.
- **Model**: multiple-instance learning, since a label names a station and a range, not its sensors, and each station has its own number of sensors: one small MLP shared by every sensor, attention pooling to a score per family for the station; the attention weights give the sensors.
- **Inputs**: each sensor against its own usual level (median and MAD over the 3 days before), on windows in hours (the step goes from 5 min to 4 h); the station context (share of high and rejected sensors, `sigma0`, references, camera pose, housing temperature, common mode by PCA); the image quality metrics when present, with a mask.
- **Inputs**: each sensor against its own usual level (median and IQR over the 3 days before), on windows in hours (the step goes from 5 min to 4 h); the station context (share of high and rejected sensors, `sigma0`, references, camera pose, housing temperature, common mode as the median over the targets); the image quality metrics when present, with a mask.
- **Training**: pre-training on the outputs of the current detectors over all the data, then fine-tuning on the reviewed labels of `labeling/auto/`.
- **Evaluation**: the measure of `detectors/bench/score.py` (found, correct, per family and per instant), against the rules on the same instants.
- **The criterion** (decided 2026-11-16): on the stations seen, the way the model will run (every station followed keeps being labelled in the app): trained on the start of every station, tested on its end (a cut in time, never at random). The model beats the rules by at least 5 points of found and of correct, in total, for Visibility and for Obstruction, and loses nowhere more than 2 points on a family of a station (the regression of `detectors/CLAUDE.md`). A new station stays under the rules until it has a few weeks of reviewed labels; the score with a station left out is followed at every step, as an indicator, not a gate.
- **The criterion** (decided 2026-11-16): on the stations seen, the way the model will run (every station followed keeps being labelled in the app): trained on the start of every station, tested on its end (a cut in time, never at random). The model beats the rules by at least 5 points of found and of correct, in total, for Visibility and for Obstruction, with no regression as `detectors/bench/compare.py` counts them (a family of a station losing more than 5 points and more than 20 instants; since 2026-11-16, before: 2 points). A new station stays under the rules until it has a few weeks of reviewed labels; the score with a station left out is followed at every step, as an indicator, not a gate.
- **Steps**: dataset, evaluator (reproducing the rules' figures first), gradient boosting baseline, the PyTorch model, pre-training compared with and without, decoding into episodes.
## Use
```bash
uv venv .venv && uv pip install -p .venv -r requirements.txt
.venv/bin/python test_dataset.py ~/aimsight-detectors
.venv/bin/python test_dataset.py ~/aimsight-detectors && .venv/bin/python test_evaluate.py && .venv/bin/python test_decode.py
.venv/bin/python dataset.py --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --out cache
```
`dataset.py` (step 1, ~2 min): one `cache/<STATION>.npz` per labelled station, over its labelled range as the benchmark reads it: `xs` [instant, sensor, feature], `xc` [instant, feature], `y` [instant, family] with the conventions of `bench/score.py` (its labelled instants per family are the benchmark's, exactly). NaN where a value does not exist (a 1 h window at a 4 h step, no temperature, no image quality outside LENZSPITZE).
```bash
.venv/bin/python test_evaluate.py
.venv/bin/python evaluate.py rules --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --cache cache
.venv/bin/python evaluate.py score <predictions folder> --cache cache
.venv/bin/python pretrain.py --detectors ~/aimsight-detectors --data ~/aimsight-data/data --out cache/pre
```
`evaluate.py rules` (step 2, ~20 min, once per change of the detectors): the rules' predictions per instant in `cache/rules/`, and the check that their counts are those of `bench/score.py`, station by station (exit 1 otherwise). `evaluate.py score`: a folder of `<STATION>.npz` with `yhat` [instant, family] against the labels, beside the rules, with the regressions of `detectors/CLAUDE.md` (more than 2 points of found or correct lost on a family of a station).
- `dataset.py` (step 1, ~1 min): one `cache/<STATION>.npz` per labelled station, over its labelled range as the benchmark reads it: `xs` [instant, sensor, feature], `xc` [instant, feature], `y` [instant, family] with the conventions of `bench/score.py` (its labelled instants per family, exactly). NaN where a value does not exist (a 1 h window at a 4 h step, no temperature, no image quality outside LENZSPITZE).
- `evaluate.py rules` (step 2, ~1 min, after every change of the detectors or the labels): the rules' predictions per instant in `cache/rules/`, checked against the counts of `bench/score.py` station by station (exit 1 otherwise). `evaluate.py score <folder>`: predictions per station against the labels, beside the rules, with the regressions of `bench/compare.py`.
- `pretrain.py` (step 5, ~3 min): every station of the data folder, labelled or not, over its whole range, in chunks of two weeks: the features and the families the rules flag (`cache/pre/`).
```bash
.venv/bin/python m1.py --cache cache --out runs/m1 --balance && .venv/bin/python evaluate.py score runs/m1 --cache cache
.venv/bin/python m1.py --cache cache --out - --split-time --balance
.venv/bin/python m2.py --cache cache --pretrain cache/pre --out runs/m2_split_pre
for g in BREITHORN,CLARIDEN,FLORIDA JUNGFRAU,LENZSPITZE,MICHIGAN NEBRASKA,NIEDERHORN,NORDEND SNCF01,WILDSTRUBEL; do
.venv/bin/python m2.py --cache cache --pretrain cache/pre --leave-out --seeds 3 --test $g --out runs/m2_pre & done; wait
.venv/bin/python evaluate.py score runs/m2_pre --cache cache
.venv/bin/python decode.py runs/m2_split_pre --cache cache --labels ~/aimsight-labeling/auto --out runs/episodes
```
`m1.py` (step 3, a few minutes): gradient boosting on the station's features and the aggregates of its sensors', Visibility and Obstruction, leaving one station out; `--split-time`, the diagnosis on the last 30 % of every station, all stations seen in training. First results (2026-11-16, found / correct, total):
- `m1.py` (step 3, ~4 min): gradient boosting on the station's features and the aggregates of its sensors' (median, q90, min, max over the targets and over the references); `--split-time` is the criterion, without it a station left out.
- `m2.py` (steps 4 and 5, PyTorch, CPU, one thread per process): multiple-instance learning, one MLP shared by every sensor and an attention per family over the sensors. One model alone varies by 10 to 20 points with its seed: 5 averaged (3 with a station left out). `--pretrain`: first the six families the rules flag on every station and month (never the instants tested), then the fine-tuning on the reviewed labels. Time split ~40 min; a station left out, ~1 h in four processes (memory: ~0.6 GB each with the pre-training set).
- `decode.py` (step 6): the flagged instants into episodes (a gap of one instant bridged), scored per episode beside the rules decoded the same way; the sensors of the attention against the sensors the labels name; `--out`, the episodes in the detectors' format.
## Results
On 2026-11-17, after the simplification of the detectors (2026-11-16), which raised the rules a lot (on the last 30 %, Visibility found 49 → 58 %, Obstruction 15 → 33 %). Found / correct per instant, in total:
| | Visibility | Obstruction |
|---|---|---|
| rules, all stations | 47 / 77 % | 26 / 28 % |
| M1, station left out | 44 / 62 % | 9 / 7 % |
| M1, station left out, `--balance` | 46 / 70 % | 21 / 23 % |
| rules, last 30 % | 49 / 77 % | 15 / 41 % |
| M1, last 30 % (`--split-time`) | 58 / 87 % | 39 / 75 % |
| M2, last 30 % (5 seeds) | 61 / 81 % | 28 / 78 % |
| M2, station left out (3 seeds) | 58 / 56 % | 26 / 19 % |
| rules, last 30 % of every station | 58 / 78 % | 33 / 41 % |
| M1, last 30 % (`--balance`) | 58 / 85 % | 33 / 75 % |
| M2, last 30 % | 58 / 85 % | 28 / 79 % |
| **M2 pre-trained, last 30 %** | **58 / 84 %** | **56 / 74 %** |
| rules, all stations | 57 / 72 % | 40 / 34 % |
| M2, station left out | 57 / 56 % | 13 / 14 % |
| M2 pre-trained, station left out | 57 / 60 % | 9 / 17 % |
The information is in the features: with every station seen, M1 beats the rules; moving to a new station is what fails (Obstruction is labelled on three stations only). Against the criterion, M1 passes the totals but not the regressions: on the last 30 %, Obstruction found at BREITHORN (90 → 57 %, 21 instants), LENZSPITZE (15 → 9 %, 33) and SNCF01 (100 → 56 %, 9 instants, where the rules flag 272 for 9 right); Visibility at LENZSPITZE (found 64 → 58 %, correct 98 → 81 %), SNCF01 and WILDSTRUBEL (correct 96 → 89 % and 97 → 93 %, with found 10 → 68 % and 86 → 99 %).
**Against the criterion** (stations seen, the last 30 %), the pre-trained M2:
- Obstruction passes the totals by far: found +23 points, correct +33. The pre-training is what does it (28 → 56 % found).
- Visibility does not: found +0, correct +6. No model, no threshold passes both: lowering M1's threshold to 0.3 gives 63 / 80 % (+5, +2), to 0.2, 67 / 77 %. With these inputs, Visibility is at a ceiling.
- Six regressions: NORDEND Obstruction correct 99 → 81 % (245 instants, flagging twice as much: found 33 → 52 %); NORDEND Visibility found 51 → 43 % (94); JUNGFRAU Visibility found 81 → 69 % (26); LENZSPITZE Visibility correct 98 → 79 % (28); WILDSTRUBEL Visibility found 94 → 89 % (21) and correct 97 → 90 % (28).
```bash
.venv/bin/python m2.py --cache cache
.venv/bin/python m2.py --cache cache --leave-out --seeds 3 --out runs/m2 && .venv/bin/python evaluate.py score runs/m2 --cache cache
```
**A new station** stays under the rules, with or without pre-training (Obstruction is labelled on three stations; leaving NORDEND out removes half of its labels): as decided, a new station keeps the rules until it has a few weeks of reviewed labels.
`m2.py` (step 4, PyTorch, CPU): multiple-instance learning, one MLP shared by every sensor and an attention per family over the sensors; trained on the reviewed labels only (no pre-training: step 5). One run alone varies by 10 to 20 points with the seed, hence the average of several (`--seeds`). Last 30 %: ~15 min; station left out: ~1.5 h.
**Episodes** (pre-trained M2, last 30 %, `decode.py`): the model finds more labels than the rules (Visibility 72/86 against 58/86, Obstruction 77/90 against 65/90), but its Visibility episodes are cut up: 64 % correct against 85 %, bounds 0.55 against 0.68 (shared instants over the instants of either). Obstruction: 79 % correct against 77 %, bounds 0.33 against 0.17. Bridging two instants instead of one gives 68 % correct for Visibility.
M2 is not better than M1. On the last 30 %, more Visibility found but Obstruction found falls back (39 → 28 %), and the same regressions as M1 (BREITHORN, LENZSPITZE, SNCF01 Obstruction; LENZSPITZE Visibility correct 98 → 60 %; SNCF01, WILDSTRUBEL). Leaving a station out, the attention does not carry over: Visibility correct falls to 56 % and Obstruction to 19 %, 18 regressions over 9 stations (Visibility correct 100 → 5 % at CLARIDEN, 88 → 17 % at NIEDERHORN; Obstruction found 91 → 5 % at SNCF01, 100 → 0 % at JUNGFRAU). With 11 stations, 3 of them labelled for Obstruction, the reviewed labels are too few for a new station; the station's own behaviour, seen in its first weeks, is what the models use.
**Sensors**: for the labels naming 1 to 3 sensors, a named sensor among the first three of the attention: Visibility 13/16, Obstruction 28/32; first: 9/16 and 11/32. A plain ranking by error_value against its usual level (`ev_z`) does as well for Obstruction (27/32 among three, 17/32 first) and worse for Visibility (10/16, 3/16). The attention does not add much to the plain ranking.
**Tried and dropped**: the rules' flags as inputs of M1 (`--with-rules`): found falls (Visibility 56 %, Obstruction 30 %). The weather (MeteoSwiss, `~/aimsight-data/weather/`): 4 of the 11 labelled stations have a position in `extraction/stations.txt`, none of BREITHORN, CLARIDEN, FLORIDA, SNCF01, WILDSTRUBEL.
**What comes next**, by expected gain:
1. Obstruction: the pre-trained M2 is a candidate beside the rules, on the stations already labelled; NORDEND's 245 instants to read with the analyst first (wrongly flagged, or Obstruction the labels missed).
2. Visibility needs new information rather than another model: the weather (the positions of the stations in `stations.txt`), the image quality metrics on the labelled stations.
3. Labels where the model hesitates or disagrees with the rules (active learning through the analyst and the review in the app), on more stations for Obstruction.
## Contracts
- Reads the measurements through `detectors/scripts/data.py`, the reading of the benchmark; the labels from `labeling/auto/`; the data from `data/` (read-only). Paths are passed as arguments.
- Reads the measurements through `scripts/data.py` of `aimsight-tools` (on the path through `detectors/scripts/categories.py`), the reading of the benchmark; the labels from `labeling/auto/`; the data from `data/` (read-only). Paths are passed as arguments.
- Never changes `aimsight-detectors` (neither `scripts/` nor `bench/`): the benchmark is the measure.
- Episodes in the format of the detectors' contract (`start`, `end`, `kind`, `label`, `text`, `ignore`, `sensors`), so that a model can later sit behind `detect(measurements)`.