28 lines
5.6 KiB
Markdown
28 lines
5.6 KiB
Markdown
# detectors
|
|
|
|
AIMsight anomaly detectors (Python 3, standard library only) and their benchmark (`bench/`); nothing else. They are improved by Optimizer (`optimize` skill) with the upgrader and validator of the agents repository. Everything is in English: code, identifiers, docs, outputs. Layout and usage: `README.md`.
|
|
|
|
## Contract (do not break)
|
|
|
|
- `scripts/categories.py` runs the detectors of `DETECTORS` (modules of `scripts/`: `visibility`, `target_issue`, `thermal_displacement`, `no_data`, `displacement`) and makes their episodes events. A module returns every family it finds: `visibility` also gives Brief obstruction and `target_issue` Target obscured (Obstruction). `episodes.py` holds the helpers they share.
|
|
- `detect(measurements)` (`no_data`: also `end`, the end of the window read) returns episodes with the keys `start`, `end`, `kind` (the family), `label` (the variety), `text` (the sentence shown), `ignore`, `sensors`; a Poor visibility also has `level`.
|
|
- The measurements are read by `scripts/data.py` of the tools repository, the sibling `~/aimsight-tools` (`categories.py` puts it on the path); a reading fix is made there.
|
|
- Label files (JSONL): `id`, `start`, `end`, `family`, `description` (and `review`, `analysis`); the project uses `labeling/auto` only.
|
|
- Two repositories use this one through a `detectors -> ~/aimsight-detectors` link (one copy on the workstation: a commit here is live for both at once); a rename here needs the matching change there:
|
|
- aimsight-app: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the event fields (`start`, `end`, `category`, `text`, `sensors`, `ignore`, `episodes`), the `--from/--to` CLIs, `bench/score.py` (the command shown in the page);
|
|
- Optimizer (`optimize` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `compare.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix.
|
|
|
|
## Rules
|
|
|
|
- A change to improve the detectors touches `scripts/*.py` and `thresholds.md` only; never `bench/`, which is the measure. Anything is possible there: add, remove or change a constant, a rule or a whole detector, as long as the six families remain.
|
|
- **Simple first**: a rule earns its place by its score on the benchmark, not by one case; prefer removing or widening a rule to adding a special case. The benchmark ends with the complexity of the detectors (code lines and constants of `scripts/`), `bench/compare.py` shows it before → after. A fix that adds a constant or more than 20 code lines earns them with at least 1 point on the `Total` of a family ("found" or "correct"); otherwise it fits one range: simplify it or drop it. A change that removes code without a regression is a gain in itself.
|
|
- **Pruning**: after a fix, each rule of the detector it touched (a condition, a branch, a constant a plainer value could replace) is tried without it, on a copy in the working folder (the copy's `bench/regression.py`, then `bench/compare.py`); a rule whose removal makes no regression is removed, one commit per rule, with its line of `thresholds.md`.
|
|
- **Everything is detected**: what is shown or not to a client is not the detectors' business. A setting chosen not to show something may be changed, saying so.
|
|
- A constant added or changed updates its line of `thresholds.md` ("Tuned on": the run that changed it) and the detector's docstring in the same change; a constant or rule removed removes its line; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
|
|
- A detector that fails is skipped: a `skipped` line in the output of `categories.py` or of the benchmark means the change is broken.
|
|
- No rule specific to a station, a sensor or a date.
|
|
- A change to a detector is checked with the benchmark: `python3 bench/regression.py --labels <labels folder> --data <data folder>` (aimsight-app and Optimizer: `labeling/auto`, `data`, links to the shared `~/aimsight-labeling` and `~/aimsight-data/data`; about 1 min). A refactoring must leave every figure unchanged.
|
|
- **The benchmark's scores**, per family of each station: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family. **A regression**, computed by `python3 bench/compare.py <before.txt> <after.txt>` (its constants): a family `Total` whose "found" or "correct" loses more than 1 point; a family of a station whose "found" or "correct" loses more than 5 points and more than 20 instants (labelled instants no longer flagged, or wrongly flagged instants added), even if the `Total` rises; below, it is the noise of a few instants (a station at one measurement every 4 h, a family with few labels). A rise is a plus, never a requirement. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain: labels contradicting each other from one station to another (the same state in two families), the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score (per instant, without the sensors) counted as found by an event on another sensor. A drop counted as not a regression is said, with the labels (id). Labels have their ambiguities: the goal is detectors that do their best, not perfect, and not overtrained on the labels.
|
|
- History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in aimsight-app/Support-Agent; keep them short when editing a rule.
|
|
- The bench tools take their paths as arguments; the architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`.
|