The criterion: stations seen, cut in time; M1's regressions station by station; no regression when nothing is flagged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-16 17:59:06 +01:00
1 parent f93cdf3cd1
commit 975ebc7c1a
4 files changed
+10 -4

No files matched your search

+3 -2
View File
@@ -8,7 +8,8 @@ Learned detectors for AIMsight: models trained on the reviewed labels to tell, a
- **Model**: multiple-instance learning, since a label names a station and a range, not its sensors, and each station has its own number of sensors: one small MLP shared by every sensor, attention pooling to a score per family for the station; the attention weights give the sensors.
- **Inputs**: each sensor against its own usual level (median and MAD over the 3 days before), on windows in hours (the step goes from 5 min to 4 h); the station context (share of high and rejected sensors, `sigma0`, references, camera pose, housing temperature, common mode by PCA); the image quality metrics when present, with a mask.
- **Training**: pre-training on the outputs of the current detectors over all the data, then fine-tuning on the reviewed labels of `labeling/auto/`.
- **Evaluation**: leave one station out; the measure of `detectors/bench/score.py` (found, correct, per family and per instant), against the rules on the same stations.
- **Evaluation**: the measure of `detectors/bench/score.py` (found, correct, per family and per instant), against the rules on the same instants.
- **The criterion** (decided 2026-11-16): on the stations seen, the way the model will run (every station followed keeps being labelled in the app): trained on the start of every station, tested on its end (a cut in time, never at random). The model beats the rules by at least 5 points of found and of correct, in total, for Visibility and for Obstruction, and loses nowhere more than 2 points on a family of a station (the regression of `detectors/CLAUDE.md`). A new station stays under the rules until it has a few weeks of reviewed labels; the score with a station left out is followed at every step, as an indicator, not a gate.
- **Steps**: dataset, evaluator (reproducing the rules' figures first), gradient boosting baseline, the PyTorch model, pre-training compared with and without, decoding into episodes.
## Use
@@ -44,7 +45,7 @@ uv venv .venv && uv pip install -p .venv -r requirements.txt
| rules, last 30 % | 49 / 77 % | 15 / 41 % |
| M1, last 30 % (`--split-time`) | 58 / 87 % | 39 / 75 % |
The information is in the features: with every station seen, M1 beats the rules; moving to a new station is what fails (Obstruction is labelled on three stations only).
The information is in the features: with every station seen, M1 beats the rules; moving to a new station is what fails (Obstruction is labelled on three stations only). Against the criterion, M1 passes the totals but not the regressions: on the last 30 %, Obstruction found at BREITHORN (90 → 57 %, 21 instants), LENZSPITZE (15 → 9 %, 33) and SNCF01 (100 → 56 %, 9 instants, where the rules flag 272 for 9 right); Visibility at LENZSPITZE (found 64 → 58 %, correct 98 → 81 %), SNCF01 and WILDSTRUBEL (correct 96 → 89 % and 97 → 93 %, with found 10 → 68 % and 86 → 99 %).
## Contracts