Files
aimsight-models/README.md
T

4.5 KiB

aimsight-models

Learned detectors for AIMsight: models trained on the reviewed labels to tell, at each measurement instant of a station, which families are present (labeling/families.md), and which sensors carry them. A pilot, not yet in use: the detectors in production are the rules of aimsight-detectors.

The pilot

  • Families: Visibility and Obstruction; the others stay rules for now.
  • Model: multiple-instance learning, since a label names a station and a range, not its sensors, and each station has its own number of sensors: one small MLP shared by every sensor, attention pooling to a score per family for the station; the attention weights give the sensors.
  • Inputs: each sensor against its own usual level (median and MAD over the 3 days before), on windows in hours (the step goes from 5 min to 4 h); the station context (share of high and rejected sensors, sigma0, references, camera pose, housing temperature, common mode by PCA); the image quality metrics when present, with a mask.
  • Training: pre-training on the outputs of the current detectors over all the data, then fine-tuning on the reviewed labels of labeling/auto/.
  • Evaluation: leave one station out; the measure of detectors/bench/score.py (found, correct, per family and per instant), against the rules on the same stations.
  • Steps: dataset, evaluator (reproducing the rules' figures first), gradient boosting baseline, the PyTorch model, pre-training compared with and without, decoding into episodes.

Use

uv venv .venv && uv pip install -p .venv -r requirements.txt
.venv/bin/python test_dataset.py ~/aimsight-detectors
.venv/bin/python dataset.py --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --out cache

dataset.py (step 1, ~2 min): one cache/<STATION>.npz per labelled station, over its labelled range as the benchmark reads it: xs [instant, sensor, feature], xc [instant, feature], y [instant, family] with the conventions of bench/score.py (its labelled instants per family are the benchmark's, exactly). NaN where a value does not exist (a 1 h window at a 4 h step, no temperature, no image quality outside LENZSPITZE).

.venv/bin/python test_evaluate.py
.venv/bin/python evaluate.py rules --detectors ~/aimsight-detectors --data ~/aimsight-data/data --labels ~/aimsight-labeling/auto --cache cache
.venv/bin/python evaluate.py score <predictions folder> --cache cache

evaluate.py rules (step 2, ~20 min, once per change of the detectors): the rules' predictions per instant in cache/rules/, and the check that their counts are those of bench/score.py, station by station (exit 1 otherwise). evaluate.py score: a folder of <STATION>.npz with yhat [instant, family] against the labels, beside the rules, with the regressions of detectors/CLAUDE.md (more than 2 points of found or correct lost on a family of a station).

.venv/bin/python m1.py --cache cache --out runs/m1 --balance && .venv/bin/python evaluate.py score runs/m1 --cache cache
.venv/bin/python m1.py --cache cache --out - --split-time --balance

m1.py (step 3, a few minutes): gradient boosting on the station's features and the aggregates of its sensors', Visibility and Obstruction, leaving one station out; --split-time, the diagnosis on the last 30 % of every station, all stations seen in training. First results (2026-11-16, found / correct, total):

Visibility Obstruction
rules, all stations 47 / 77 % 26 / 28 %
M1, station left out 44 / 62 % 9 / 7 %
M1, station left out, --balance 46 / 70 % 21 / 23 %
rules, last 30 % 49 / 77 % 15 / 41 %
M1, last 30 % (--split-time) 58 / 87 % 39 / 75 %

The information is in the features: with every station seen, M1 beats the rules; moving to a new station is what fails (Obstruction is labelled on three stations only).

Contracts

  • Reads the measurements through detectors/scripts/data.py, the reading of the benchmark; the labels from labeling/auto/; the data from data/ (read-only). Paths are passed as arguments.
  • Never changes aimsight-detectors (neither scripts/ nor bench/): the benchmark is the measure.
  • Episodes in the format of the detectors' contract (start, end, kind, label, text, ignore, sensors), so that a model can later sit behind detect(measurements).

Rules

  • Every time is in UTC. No rule specific to a station, a target or a date.
  • English for code, identifiers and docs.