Regression computed by bench/compare.py (Total > 1 point; station > 5 points and > 20 instants), complexity in the benchmark, pruning after a fix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-16 17:53:09 +01:00
1 parent 63d954a126
commit a008ea8fcc
4 files changed
+77 -5

No files matched your search

+4 -3
View File
@@ -10,17 +10,18 @@ AIMsight anomaly detectors (Python 3, standard library only) and their benchmark
- Label files (JSONL): `id`, `start`, `end`, `family`, `description` (and `review`, `analysis`); the project uses `labeling/auto` only.
- Two repositories use this one through a `detectors -> ~/aimsight-detectors` link (one copy on the workstation: a commit here is live for both at once); a rename here needs the matching change there:
- aimsight-app: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the event fields (`start`, `end`, `category`, `text`, `sensors`, `ignore`, `episodes`), the `--from/--to` CLIs, `bench/score.py` (the command shown in the page);
- Optimizer (`optimize` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix.
- Optimizer (`optimize` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `compare.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix.
## Rules
- A change to improve the detectors touches `scripts/*.py` and `thresholds.md` only; never `bench/`, which is the measure. Anything is possible there: add, remove or change a constant, a rule or a whole detector, as long as the six families remain.
- **Simple first**: a rule earns its place by its score on the benchmark, not by one case; prefer removing or widening a rule to adding a special case.
- **Simple first**: a rule earns its place by its score on the benchmark, not by one case; prefer removing or widening a rule to adding a special case. The benchmark ends with the complexity of the detectors (code lines and constants of `scripts/`), `bench/compare.py` shows it before → after. A fix that adds a constant or more than 20 code lines earns them with at least 1 point on the `Total` of a family ("found" or "correct"); otherwise it fits one range: simplify it or drop it. A change that removes code without a regression is a gain in itself.
- **Pruning**: after a fix, each rule of the detector it touched (a condition, a branch, a constant a plainer value could replace) is tried without it, on a copy in the working folder (the copy's `bench/regression.py`, then `bench/compare.py`); a rule whose removal makes no regression is removed, one commit per rule, with its line of `thresholds.md`.
- **Everything is detected**: what is shown or not to a client is not the detectors' business. A setting chosen not to show something may be changed, saying so.
- A constant added or changed updates its line of `thresholds.md` ("Tuned on": the run that changed it) and the detector's docstring in the same change; a constant or rule removed removes its line; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
- A detector that fails is skipped: a `skipped` line in the output of `categories.py` or of the benchmark means the change is broken.
- No rule specific to a station, a sensor or a date.
- A change to a detector is checked with the benchmark: `python3 bench/regression.py --labels <labels folder> --data <data folder>` (aimsight-app and Optimizer: `labeling/auto`, `data`, links to the shared `~/aimsight-labeling` and `~/aimsight-data/data`; about 1 min). A refactoring must leave every figure unchanged.
- **The benchmark's scores**, per family of each station: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family. **A regression**, on a family of a station: "found" or "correct" loses more than 2 points; it is a regression even if the `Total` rises; under 2 points it is not one; a rise is a plus, never a requirement. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain: labels contradicting each other from one station to another (the same state in two families), the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score (per instant, without the sensors) counted as found by an event on another sensor. Labels have their ambiguities: the goal is detectors that do their best, not perfect, and not overtrained on the labels.
- **The benchmark's scores**, per family of each station: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family. **A regression**, computed by `python3 bench/compare.py <before.txt> <after.txt>` (its constants): a family `Total` whose "found" or "correct" loses more than 1 point; a family of a station whose "found" or "correct" loses more than 5 points and more than 20 instants (labelled instants no longer flagged, or wrongly flagged instants added), even if the `Total` rises; below, it is the noise of a few instants (a station at one measurement every 4 h, a family with few labels). A rise is a plus, never a requirement. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain: labels contradicting each other from one station to another (the same state in two families), the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score (per instant, without the sensors) counted as found by an event on another sensor. A drop counted as not a regression is said, with the labels (id). Labels have their ambiguities: the goal is detectors that do their best, not perfect, and not overtrained on the labels.
- History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in aimsight-app/Support-Agent; keep them short when editing a rule.
- The bench tools take their paths as arguments; the architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`.
+1
View File
@@ -29,6 +29,7 @@ The detectors against labels (JSONL, one per line, `{"id", "start", "end", "fami
```bash
python3 bench/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD # the score of a range
python3 bench/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> # the benchmark; --from/--to: one range for all
python3 bench/compare.py <before.txt> <after.txt> # two benchmarks: the regressions, the complexity
python3 bench/sensors.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD # the sensors of each event
```
+61
View File
@@ -0,0 +1,61 @@
"""Two benchmark outputs (regression.py), before and after: what changed, and the regressions by the definition of
CLAUDE.md, computed.
- A family Total: "found" or "correct" loses more than TOTAL_POINTS.
- A family of a station: "found" or "correct" loses more than POINTS and more than INSTANTS instants ("found": labelled
instants no longer flagged; "correct": wrongly flagged instants added). Below, the loss is noise of a few instants.
Then the complexity before → after. The verdict is the reader's: a drop the labels explain (CLAUDE.md, "Not a
regression") is said, not hidden.
Usage: python3 compare.py <before.txt> <after.txt>
"""
import re
import sys
TOTAL_POINTS = 1
POINTS = 5
INSTANTS = 20
CELL = r"(\d+)/(\d+) \(\d+ %\)|—"
def parse(path):
"""Benchmark output -> ({(station, family): (both, labelled, flagged)}, complexity line or '')."""
out, station, cx = {}, None, ""
for line in open(path, encoding="utf-8"):
if line.startswith("Complexity:"):
cx = line.split(":", 1)[1].strip()
elif line.strip() and not line.startswith(" "):
station = line.split(":")[0].strip()
elif m := re.match(rf" (\S.*?)\s{{2,}}({CELL})\s+({CELL})\s*$", line):
g = m.groups() # family, found (both, labelled), correct (both, flagged)
out[(station, g[0])] = (int(g[2] or g[5] or 0), int(g[3] or 0), int(g[6] or 0))
return out, cx
def pct(a, b):
return 100 * a / b if b else None
def changes(before, after):
"""-> [(station, family, measure, before %, after %, lost instants, regression)] for every measure that moved."""
rows = []
for key in sorted(set(before) | set(after)):
(b0, l0, f0), (b1, l1, f1) = before.get(key, (0, 0, 0)), after.get(key, (0, 0, 0))
for measure, p0, p1, lost in (("found", pct(b0, l0), pct(b1, l1), b0 - b1),
("correct", pct(b0, f0), pct(b1, f1), (f1 - b1) - (f0 - b0))):
if p0 is None or p1 is None or round(p0) == round(p1):
continue
drop = p0 - p1
bad = drop > TOTAL_POINTS if key[0] == "Total" else drop > POINTS and lost > INSTANTS
rows.append((*key, measure, p0, p1, lost, bad))
return rows
if __name__ == "__main__":
if len(sys.argv) != 3:
sys.exit(__doc__)
(before, cx0), (after, cx1) = parse(sys.argv[1]), parse(sys.argv[2])
rows = changes(before, after)
for station, family, measure, p0, p1, lost, bad in rows:
print(f"{'REGRESSION' if bad else '':<11}{station:<13}{family:<22}{measure:<8}{p0:4.0f} → {p1:3.0f} %"
+ (f" ({lost} instants lost)" if p1 < p0 else ""))
n = sum(r[-1] for r in rows)
print(f"\n{n} regression(s); complexity: {cx0 or '?'} → {cx1 or '?'}")
+11 -2
View File
@@ -3,12 +3,13 @@
For each <STATION>.jsonl of --labels that has its folder <STATION>/ in --data, the score per family and per instant of
score.py, over the range of its labels: from the day of the first to the midnight after the last; with --from/--to, over
that range for every station, a station without a label on it included (several stations labelled the same days). Then the total,
family by family, which says whether the detectors do better overall. The block of a station is its score; the
others, the non-regression: a family that goes back on a station is a regression, even if the total rises.
family by family, which says whether the detectors do better overall, and the complexity of the detectors (code lines
and constants of scripts/). Two outputs are compared by compare.py (the regressions, CLAUDE.md).
Usage: python3 regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> [--stations A,B]
[--from YYYY-MM-DD --to YYYY-MM-DD]
"""
import argparse
import re
from datetime import timedelta
from pathlib import Path
@@ -17,6 +18,13 @@ import score # next to it
from data import iso, parse_time # on the path through score
def complexity(scripts=Path(__file__).resolve().parents[1] / "scripts"):
"""-> (code lines, constants) of the detectors: non-blank lines that are not a comment; NAME = lines at module level."""
lines = [x for f in sorted(scripts.glob("*.py")) for x in f.read_text().splitlines()]
code = [x for x in lines if x.strip() and not x.strip().startswith("#")]
return len(code), sum(bool(re.match(r"[A-Z][A-Z_0-9]* = ", x)) for x in lines)
def label_range(path):
"""Labels JSONL -> (start, end) iso, in whole UTC days, end excluded."""
labels = score.read_labels(path)
@@ -53,3 +61,4 @@ if __name__ == "__main__":
for k, v in n.items():
total[k] = [x + y for x, y in zip(total.get(k, [0, 0, 0]), v)]
score.show("Total", total)
print("\nComplexity: %d code lines, %d constants" % complexity())