71 lines
6.8 KiB
Markdown
71 lines
6.8 KiB
Markdown
# aimsight-models
|
|
|
|
Learned detectors for AIMsight: models trained on the reviewed labels to tell, at each measurement instant of a station, which families are present (`labeling/families.md`), and which sensors carry them. A pilot, not yet in use: the detectors in production are the rules of `aimsight-detectors`.
|
|
|
|
## The pilot
|
|
|
|
- **Families**: Visibility and Obstruction; the others stay rules for now.
|
|
- **Model**: multiple-instance learning, since a label names a station and a range, not its sensors, and each station has its own number of sensors: one small MLP shared by every sensor, attention pooling to a score per family for the station; the attention weights give the sensors.
|
|
- **Inputs**: each sensor against its own usual level (median and MAD over the 3 days before), on windows in hours (the step goes from 5 min to 4 h); the station context (share of high and rejected sensors, `sigma0`, references, camera pose, housing temperature, common mode by PCA); the image quality metrics when present, with a mask.
|
|
- **Training**: pre-training on the outputs of the current detectors over all the data, then fine-tuning on the reviewed labels of `labeling/auto/`.
|
|
- **Evaluation**: the measure of `detectors/bench/score.py` (found, correct, per family and per instant), against the rules on the same instants.
|
|
- **The criterion** (decided 2026-11-16): on the stations seen, the way the model will run (every station followed keeps being labelled in the app): trained on the start of every station, tested on its end (a cut in time, never at random). The model beats the rules by at least 5 points of found and of correct, in total, for Visibility and for Obstruction, and loses nowhere more than 2 points on a family of a station (the regression of `detectors/CLAUDE.md`). A new station stays under the rules until it has a few weeks of reviewed labels; the score with a station left out is followed at every step, as an indicator, not a gate.
|
|
- **Steps**: dataset, evaluator (reproducing the rules' figures first), gradient boosting baseline, the PyTorch model, pre-training compared with and without, decoding into episodes.
|
|
|
|
## Use
|
|
|
|
```bash
|
|
uv venv .venv && uv pip install -p .venv -r requirements.txt
|
|
.venv/bin/python test_dataset.py ~/aimsight-detectors
|
|
.venv/bin/python dataset.py --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --out cache
|
|
```
|
|
|
|
`dataset.py` (step 1, ~2 min): one `cache/<STATION>.npz` per labelled station, over its labelled range as the benchmark reads it: `xs` [instant, sensor, feature], `xc` [instant, feature], `y` [instant, family] with the conventions of `bench/score.py` (its labelled instants per family are the benchmark's, exactly). NaN where a value does not exist (a 1 h window at a 4 h step, no temperature, no image quality outside LENZSPITZE).
|
|
|
|
```bash
|
|
.venv/bin/python test_evaluate.py
|
|
.venv/bin/python evaluate.py rules --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --cache cache
|
|
.venv/bin/python evaluate.py score <predictions folder> --cache cache
|
|
```
|
|
|
|
`evaluate.py rules` (step 2, ~20 min, once per change of the detectors): the rules' predictions per instant in `cache/rules/`, and the check that their counts are those of `bench/score.py`, station by station (exit 1 otherwise). `evaluate.py score`: a folder of `<STATION>.npz` with `yhat` [instant, family] against the labels, beside the rules, with the regressions of `detectors/CLAUDE.md` (more than 2 points of found or correct lost on a family of a station).
|
|
|
|
```bash
|
|
.venv/bin/python m1.py --cache cache --out runs/m1 --balance && .venv/bin/python evaluate.py score runs/m1 --cache cache
|
|
.venv/bin/python m1.py --cache cache --out - --split-time --balance
|
|
```
|
|
|
|
`m1.py` (step 3, a few minutes): gradient boosting on the station's features and the aggregates of its sensors', Visibility and Obstruction, leaving one station out; `--split-time`, the diagnosis on the last 30 % of every station, all stations seen in training. First results (2026-11-16, found / correct, total):
|
|
|
|
| | Visibility | Obstruction |
|
|
|---|---|---|
|
|
| rules, all stations | 47 / 77 % | 26 / 28 % |
|
|
| M1, station left out | 44 / 62 % | 9 / 7 % |
|
|
| M1, station left out, `--balance` | 46 / 70 % | 21 / 23 % |
|
|
| rules, last 30 % | 49 / 77 % | 15 / 41 % |
|
|
| M1, last 30 % (`--split-time`) | 58 / 87 % | 39 / 75 % |
|
|
| M2, last 30 % (5 seeds) | 61 / 81 % | 28 / 78 % |
|
|
| M2, station left out (3 seeds) | 58 / 56 % | 26 / 19 % |
|
|
|
|
The information is in the features: with every station seen, M1 beats the rules; moving to a new station is what fails (Obstruction is labelled on three stations only). Against the criterion, M1 passes the totals but not the regressions: on the last 30 %, Obstruction found at BREITHORN (90 → 57 %, 21 instants), LENZSPITZE (15 → 9 %, 33) and SNCF01 (100 → 56 %, 9 instants, where the rules flag 272 for 9 right); Visibility at LENZSPITZE (found 64 → 58 %, correct 98 → 81 %), SNCF01 and WILDSTRUBEL (correct 96 → 89 % and 97 → 93 %, with found 10 → 68 % and 86 → 99 %).
|
|
|
|
```bash
|
|
.venv/bin/python m2.py --cache cache
|
|
.venv/bin/python m2.py --cache cache --leave-out --seeds 3 --out runs/m2 && .venv/bin/python evaluate.py score runs/m2 --cache cache
|
|
```
|
|
|
|
`m2.py` (step 4, PyTorch, CPU): multiple-instance learning, one MLP shared by every sensor and an attention per family over the sensors; trained on the reviewed labels only (no pre-training: step 5). One run alone varies by 10 to 20 points with the seed, hence the average of several (`--seeds`). Last 30 %: ~15 min; station left out: ~1.5 h.
|
|
|
|
M2 is not better than M1. On the last 30 %, more Visibility found but Obstruction found falls back (39 → 28 %), and the same regressions as M1 (BREITHORN, LENZSPITZE, SNCF01 Obstruction; LENZSPITZE Visibility correct 98 → 60 %; SNCF01, WILDSTRUBEL). Leaving a station out, the attention does not carry over: Visibility correct falls to 56 % and Obstruction to 19 %, 18 regressions over 9 stations (Visibility correct 100 → 5 % at CLARIDEN, 88 → 17 % at NIEDERHORN; Obstruction found 91 → 5 % at SNCF01, 100 → 0 % at JUNGFRAU). With 11 stations, 3 of them labelled for Obstruction, the reviewed labels are too few for a new station; the station's own behaviour, seen in its first weeks, is what the models use.
|
|
|
|
## Contracts
|
|
|
|
- Reads the measurements through `detectors/scripts/data.py`, the reading of the benchmark; the labels from `labeling/auto/`; the data from `data/` (read-only). Paths are passed as arguments.
|
|
- Never changes `aimsight-detectors` (neither `scripts/` nor `bench/`): the benchmark is the measure.
|
|
- Episodes in the format of the detectors' contract (`start`, `end`, `kind`, `label`, `text`, `ignore`, `sensors`), so that a model can later sit behind `detect(measurements)`.
|
|
|
|
## Rules
|
|
|
|
- Every time is in UTC. No rule specific to a station, a target or a date.
|
|
- English for code, identifiers and docs.
|