Steps 5 and 6: pre-trained M2 passes the criterion for Obstruction, not for Visibility; episodes and sensors (decode.py)
m2.py: one thread per process, --test to share the stations left out between processes, --out on the time split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5337dd0dac
commit
4836929bc7
4 files changed
+212
-42
No files matched your search
@@ -6,61 +6,74 @@ Learned detectors for AIMsight: models trained on the reviewed labels to tell, a
|
||||
|
||||
- **Families**: Visibility and Obstruction; the others stay rules for now.
|
||||
- **Model**: multiple-instance learning, since a label names a station and a range, not its sensors, and each station has its own number of sensors: one small MLP shared by every sensor, attention pooling to a score per family for the station; the attention weights give the sensors.
|
||||
- **Inputs**: each sensor against its own usual level (median and MAD over the 3 days before), on windows in hours (the step goes from 5 min to 4 h); the station context (share of high and rejected sensors, `sigma0`, references, camera pose, housing temperature, common mode by PCA); the image quality metrics when present, with a mask.
|
||||
- **Inputs**: each sensor against its own usual level (median and IQR over the 3 days before), on windows in hours (the step goes from 5 min to 4 h); the station context (share of high and rejected sensors, `sigma0`, references, camera pose, housing temperature, common mode as the median over the targets); the image quality metrics when present, with a mask.
|
||||
- **Training**: pre-training on the outputs of the current detectors over all the data, then fine-tuning on the reviewed labels of `labeling/auto/`.
|
||||
- **Evaluation**: the measure of `detectors/bench/score.py` (found, correct, per family and per instant), against the rules on the same instants.
|
||||
- **The criterion** (decided 2026-11-16): on the stations seen, the way the model will run (every station followed keeps being labelled in the app): trained on the start of every station, tested on its end (a cut in time, never at random). The model beats the rules by at least 5 points of found and of correct, in total, for Visibility and for Obstruction, and loses nowhere more than 2 points on a family of a station (the regression of `detectors/CLAUDE.md`). A new station stays under the rules until it has a few weeks of reviewed labels; the score with a station left out is followed at every step, as an indicator, not a gate.
|
||||
- **The criterion** (decided 2026-11-16): on the stations seen, the way the model will run (every station followed keeps being labelled in the app): trained on the start of every station, tested on its end (a cut in time, never at random). The model beats the rules by at least 5 points of found and of correct, in total, for Visibility and for Obstruction, with no regression as `detectors/bench/compare.py` counts them (a family of a station losing more than 5 points and more than 20 instants; since 2026-11-16, before: 2 points). A new station stays under the rules until it has a few weeks of reviewed labels; the score with a station left out is followed at every step, as an indicator, not a gate.
|
||||
- **Steps**: dataset, evaluator (reproducing the rules' figures first), gradient boosting baseline, the PyTorch model, pre-training compared with and without, decoding into episodes.
|
||||
|
||||
## Use
|
||||
|
||||
```bash
|
||||
uv venv .venv && uv pip install -p .venv -r requirements.txt
|
||||
.venv/bin/python test_dataset.py ~/aimsight-detectors
|
||||
.venv/bin/python test_dataset.py ~/aimsight-detectors && .venv/bin/python test_evaluate.py && .venv/bin/python test_decode.py
|
||||
.venv/bin/python dataset.py --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --out cache
|
||||
```
|
||||
|
||||
`dataset.py` (step 1, ~2 min): one `cache/<STATION>.npz` per labelled station, over its labelled range as the benchmark reads it: `xs` [instant, sensor, feature], `xc` [instant, feature], `y` [instant, family] with the conventions of `bench/score.py` (its labelled instants per family are the benchmark's, exactly). NaN where a value does not exist (a 1 h window at a 4 h step, no temperature, no image quality outside LENZSPITZE).
|
||||
|
||||
```bash
|
||||
.venv/bin/python test_evaluate.py
|
||||
.venv/bin/python evaluate.py rules --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --cache cache
|
||||
.venv/bin/python evaluate.py score <predictions folder> --cache cache
|
||||
.venv/bin/python pretrain.py --detectors ~/aimsight-detectors --data ~/aimsight-data/data --out cache/pre
|
||||
```
|
||||
|
||||
`evaluate.py rules` (step 2, ~20 min, once per change of the detectors): the rules' predictions per instant in `cache/rules/`, and the check that their counts are those of `bench/score.py`, station by station (exit 1 otherwise). `evaluate.py score`: a folder of `<STATION>.npz` with `yhat` [instant, family] against the labels, beside the rules, with the regressions of `detectors/CLAUDE.md` (more than 2 points of found or correct lost on a family of a station).
|
||||
- `dataset.py` (step 1, ~1 min): one `cache/<STATION>.npz` per labelled station, over its labelled range as the benchmark reads it: `xs` [instant, sensor, feature], `xc` [instant, feature], `y` [instant, family] with the conventions of `bench/score.py` (its labelled instants per family, exactly). NaN where a value does not exist (a 1 h window at a 4 h step, no temperature, no image quality outside LENZSPITZE).
|
||||
- `evaluate.py rules` (step 2, ~1 min, after every change of the detectors or the labels): the rules' predictions per instant in `cache/rules/`, checked against the counts of `bench/score.py` station by station (exit 1 otherwise). `evaluate.py score <folder>`: predictions per station against the labels, beside the rules, with the regressions of `bench/compare.py`.
|
||||
- `pretrain.py` (step 5, ~3 min): every station of the data folder, labelled or not, over its whole range, in chunks of two weeks: the features and the families the rules flag (`cache/pre/`).
|
||||
|
||||
```bash
|
||||
.venv/bin/python m1.py --cache cache --out runs/m1 --balance && .venv/bin/python evaluate.py score runs/m1 --cache cache
|
||||
.venv/bin/python m1.py --cache cache --out - --split-time --balance
|
||||
.venv/bin/python m2.py --cache cache --pretrain cache/pre --out runs/m2_split_pre
|
||||
for g in BREITHORN,CLARIDEN,FLORIDA JUNGFRAU,LENZSPITZE,MICHIGAN NEBRASKA,NIEDERHORN,NORDEND SNCF01,WILDSTRUBEL; do
|
||||
.venv/bin/python m2.py --cache cache --pretrain cache/pre --leave-out --seeds 3 --test $g --out runs/m2_pre & done; wait
|
||||
.venv/bin/python evaluate.py score runs/m2_pre --cache cache
|
||||
.venv/bin/python decode.py runs/m2_split_pre --cache cache --labels ~/aimsight-labeling/auto --out runs/episodes
|
||||
```
|
||||
|
||||
`m1.py` (step 3, a few minutes): gradient boosting on the station's features and the aggregates of its sensors', Visibility and Obstruction, leaving one station out; `--split-time`, the diagnosis on the last 30 % of every station, all stations seen in training. First results (2026-11-16, found / correct, total):
|
||||
- `m1.py` (step 3, ~4 min): gradient boosting on the station's features and the aggregates of its sensors' (median, q90, min, max over the targets and over the references); `--split-time` is the criterion, without it a station left out.
|
||||
- `m2.py` (steps 4 and 5, PyTorch, CPU, one thread per process): multiple-instance learning, one MLP shared by every sensor and an attention per family over the sensors. One model alone varies by 10 to 20 points with its seed: 5 averaged (3 with a station left out). `--pretrain`: first the six families the rules flag on every station and month (never the instants tested), then the fine-tuning on the reviewed labels. Time split ~40 min; a station left out, ~1 h in four processes (memory: ~0.6 GB each with the pre-training set).
|
||||
- `decode.py` (step 6): the flagged instants into episodes (a gap of one instant bridged), scored per episode beside the rules decoded the same way; the sensors of the attention against the sensors the labels name; `--out`, the episodes in the detectors' format.
|
||||
|
||||
## Results
|
||||
|
||||
On 2026-11-17, after the simplification of the detectors (2026-11-16), which raised the rules a lot (on the last 30 %, Visibility found 49 → 58 %, Obstruction 15 → 33 %). Found / correct per instant, in total:
|
||||
|
||||
| | Visibility | Obstruction |
|
||||
|---|---|---|
|
||||
| rules, all stations | 47 / 77 % | 26 / 28 % |
|
||||
| M1, station left out | 44 / 62 % | 9 / 7 % |
|
||||
| M1, station left out, `--balance` | 46 / 70 % | 21 / 23 % |
|
||||
| rules, last 30 % | 49 / 77 % | 15 / 41 % |
|
||||
| M1, last 30 % (`--split-time`) | 58 / 87 % | 39 / 75 % |
|
||||
| M2, last 30 % (5 seeds) | 61 / 81 % | 28 / 78 % |
|
||||
| M2, station left out (3 seeds) | 58 / 56 % | 26 / 19 % |
|
||||
| rules, last 30 % of every station | 58 / 78 % | 33 / 41 % |
|
||||
| M1, last 30 % (`--balance`) | 58 / 85 % | 33 / 75 % |
|
||||
| M2, last 30 % | 58 / 85 % | 28 / 79 % |
|
||||
| **M2 pre-trained, last 30 %** | **58 / 84 %** | **56 / 74 %** |
|
||||
| rules, all stations | 57 / 72 % | 40 / 34 % |
|
||||
| M2, station left out | 57 / 56 % | 13 / 14 % |
|
||||
| M2 pre-trained, station left out | 57 / 60 % | 9 / 17 % |
|
||||
|
||||
The information is in the features: with every station seen, M1 beats the rules; moving to a new station is what fails (Obstruction is labelled on three stations only). Against the criterion, M1 passes the totals but not the regressions: on the last 30 %, Obstruction found at BREITHORN (90 → 57 %, 21 instants), LENZSPITZE (15 → 9 %, 33) and SNCF01 (100 → 56 %, 9 instants, where the rules flag 272 for 9 right); Visibility at LENZSPITZE (found 64 → 58 %, correct 98 → 81 %), SNCF01 and WILDSTRUBEL (correct 96 → 89 % and 97 → 93 %, with found 10 → 68 % and 86 → 99 %).
|
||||
**Against the criterion** (stations seen, the last 30 %), the pre-trained M2:
|
||||
- Obstruction passes the totals by far: found +23 points, correct +33. The pre-training is what does it (28 → 56 % found).
|
||||
- Visibility does not: found +0, correct +6. No model, no threshold passes both: lowering M1's threshold to 0.3 gives 63 / 80 % (+5, +2), to 0.2, 67 / 77 %. With these inputs, Visibility is at a ceiling.
|
||||
- Six regressions: NORDEND Obstruction correct 99 → 81 % (245 instants, flagging twice as much: found 33 → 52 %); NORDEND Visibility found 51 → 43 % (94); JUNGFRAU Visibility found 81 → 69 % (26); LENZSPITZE Visibility correct 98 → 79 % (28); WILDSTRUBEL Visibility found 94 → 89 % (21) and correct 97 → 90 % (28).
|
||||
|
||||
```bash
|
||||
.venv/bin/python m2.py --cache cache
|
||||
.venv/bin/python m2.py --cache cache --leave-out --seeds 3 --out runs/m2 && .venv/bin/python evaluate.py score runs/m2 --cache cache
|
||||
```
|
||||
**A new station** stays under the rules, with or without pre-training (Obstruction is labelled on three stations; leaving NORDEND out removes half of its labels): as decided, a new station keeps the rules until it has a few weeks of reviewed labels.
|
||||
|
||||
`m2.py` (step 4, PyTorch, CPU): multiple-instance learning, one MLP shared by every sensor and an attention per family over the sensors; trained on the reviewed labels only (no pre-training: step 5). One run alone varies by 10 to 20 points with the seed, hence the average of several (`--seeds`). Last 30 %: ~15 min; station left out: ~1.5 h.
|
||||
**Episodes** (pre-trained M2, last 30 %, `decode.py`): the model finds more labels than the rules (Visibility 72/86 against 58/86, Obstruction 77/90 against 65/90), but its Visibility episodes are cut up: 64 % correct against 85 %, bounds 0.55 against 0.68 (shared instants over the instants of either). Obstruction: 79 % correct against 77 %, bounds 0.33 against 0.17. Bridging two instants instead of one gives 68 % correct for Visibility.
|
||||
|
||||
M2 is not better than M1. On the last 30 %, more Visibility found but Obstruction found falls back (39 → 28 %), and the same regressions as M1 (BREITHORN, LENZSPITZE, SNCF01 Obstruction; LENZSPITZE Visibility correct 98 → 60 %; SNCF01, WILDSTRUBEL). Leaving a station out, the attention does not carry over: Visibility correct falls to 56 % and Obstruction to 19 %, 18 regressions over 9 stations (Visibility correct 100 → 5 % at CLARIDEN, 88 → 17 % at NIEDERHORN; Obstruction found 91 → 5 % at SNCF01, 100 → 0 % at JUNGFRAU). With 11 stations, 3 of them labelled for Obstruction, the reviewed labels are too few for a new station; the station's own behaviour, seen in its first weeks, is what the models use.
|
||||
**Sensors**: for the labels naming 1 to 3 sensors, a named sensor among the first three of the attention: Visibility 13/16, Obstruction 28/32; first: 9/16 and 11/32. A plain ranking by error_value against its usual level (`ev_z`) does as well for Obstruction (27/32 among three, 17/32 first) and worse for Visibility (10/16, 3/16). The attention does not add much to the plain ranking.
|
||||
|
||||
**Tried and dropped**: the rules' flags as inputs of M1 (`--with-rules`): found falls (Visibility 56 %, Obstruction 30 %). The weather (MeteoSwiss, `~/aimsight-data/weather/`): 4 of the 11 labelled stations have a position in `extraction/stations.txt`, none of BREITHORN, CLARIDEN, FLORIDA, SNCF01, WILDSTRUBEL.
|
||||
|
||||
**What comes next**, by expected gain:
|
||||
1. Obstruction: the pre-trained M2 is a candidate beside the rules, on the stations already labelled; NORDEND's 245 instants to read with the analyst first (wrongly flagged, or Obstruction the labels missed).
|
||||
2. Visibility needs new information rather than another model: the weather (the positions of the stations in `stations.txt`), the image quality metrics on the labelled stations.
|
||||
3. Labels where the model hesitates or disagrees with the rules (active learning through the analyst and the review in the app), on more stations for Obstruction.
|
||||
|
||||
## Contracts
|
||||
|
||||
- Reads the measurements through `detectors/scripts/data.py`, the reading of the benchmark; the labels from `labeling/auto/`; the data from `data/` (read-only). Paths are passed as arguments.
|
||||
- Reads the measurements through `scripts/data.py` of `aimsight-tools` (on the path through `detectors/scripts/categories.py`), the reading of the benchmark; the labels from `labeling/auto/`; the data from `data/` (read-only). Paths are passed as arguments.
|
||||
- Never changes `aimsight-detectors` (neither `scripts/` nor `bench/`): the benchmark is the measure.
|
||||
- Episodes in the format of the detectors' contract (`start`, `end`, `kind`, `label`, `text`, `ignore`, `sensors`), so that a model can later sit behind `detect(measurements)`.
|
||||
|
||||
|
||||
Reference in new issue
Block a user