Regression computed by bench/compare.py (Total > 1 point; station > 5 points and > 20 instants), complexity in the benchmark, pruning after a fix
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
63d954a126
commit
a008ea8fcc
4 files changed
+77
-5
No files matched your search
@@ -10,17 +10,18 @@ AIMsight anomaly detectors (Python 3, standard library only) and their benchmark
|
||||
- Label files (JSONL): `id`, `start`, `end`, `family`, `description` (and `review`, `analysis`); the project uses `labeling/auto` only.
|
||||
- Two repositories use this one through a `detectors -> ~/aimsight-detectors` link (one copy on the workstation: a commit here is live for both at once); a rename here needs the matching change there:
|
||||
- aimsight-app: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the event fields (`start`, `end`, `category`, `text`, `sensors`, `ignore`, `episodes`), the `--from/--to` CLIs, `bench/score.py` (the command shown in the page);
|
||||
- Optimizer (`optimize` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix.
|
||||
- Optimizer (`optimize` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `compare.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix.
|
||||
|
||||
## Rules
|
||||
|
||||
- A change to improve the detectors touches `scripts/*.py` and `thresholds.md` only; never `bench/`, which is the measure. Anything is possible there: add, remove or change a constant, a rule or a whole detector, as long as the six families remain.
|
||||
- **Simple first**: a rule earns its place by its score on the benchmark, not by one case; prefer removing or widening a rule to adding a special case.
|
||||
- **Simple first**: a rule earns its place by its score on the benchmark, not by one case; prefer removing or widening a rule to adding a special case. The benchmark ends with the complexity of the detectors (code lines and constants of `scripts/`), `bench/compare.py` shows it before → after. A fix that adds a constant or more than 20 code lines earns them with at least 1 point on the `Total` of a family ("found" or "correct"); otherwise it fits one range: simplify it or drop it. A change that removes code without a regression is a gain in itself.
|
||||
- **Pruning**: after a fix, each rule of the detector it touched (a condition, a branch, a constant a plainer value could replace) is tried without it, on a copy in the working folder (the copy's `bench/regression.py`, then `bench/compare.py`); a rule whose removal makes no regression is removed, one commit per rule, with its line of `thresholds.md`.
|
||||
- **Everything is detected**: what is shown or not to a client is not the detectors' business. A setting chosen not to show something may be changed, saying so.
|
||||
- A constant added or changed updates its line of `thresholds.md` ("Tuned on": the run that changed it) and the detector's docstring in the same change; a constant or rule removed removes its line; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
|
||||
- A detector that fails is skipped: a `skipped` line in the output of `categories.py` or of the benchmark means the change is broken.
|
||||
- No rule specific to a station, a sensor or a date.
|
||||
- A change to a detector is checked with the benchmark: `python3 bench/regression.py --labels <labels folder> --data <data folder>` (aimsight-app and Optimizer: `labeling/auto`, `data`, links to the shared `~/aimsight-labeling` and `~/aimsight-data/data`; about 1 min). A refactoring must leave every figure unchanged.
|
||||
- **The benchmark's scores**, per family of each station: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family. **A regression**, on a family of a station: "found" or "correct" loses more than 2 points; it is a regression even if the `Total` rises; under 2 points it is not one; a rise is a plus, never a requirement. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain: labels contradicting each other from one station to another (the same state in two families), the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score (per instant, without the sensors) counted as found by an event on another sensor. Labels have their ambiguities: the goal is detectors that do their best, not perfect, and not overtrained on the labels.
|
||||
- **The benchmark's scores**, per family of each station: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family. **A regression**, computed by `python3 bench/compare.py <before.txt> <after.txt>` (its constants): a family `Total` whose "found" or "correct" loses more than 1 point; a family of a station whose "found" or "correct" loses more than 5 points and more than 20 instants (labelled instants no longer flagged, or wrongly flagged instants added), even if the `Total` rises; below, it is the noise of a few instants (a station at one measurement every 4 h, a family with few labels). A rise is a plus, never a requirement. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain: labels contradicting each other from one station to another (the same state in two families), the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score (per instant, without the sensors) counted as found by an event on another sensor. A drop counted as not a regression is said, with the labels (id). Labels have their ambiguities: the goal is detectors that do their best, not perfect, and not overtrained on the labels.
|
||||
- History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in aimsight-app/Support-Agent; keep them short when editing a rule.
|
||||
- The bench tools take their paths as arguments; the architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`.
|
||||
@@ -29,6 +29,7 @@ The detectors against labels (JSONL, one per line, `{"id", "start", "end", "fami
|
||||
```bash
|
||||
python3 bench/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD # the score of a range
|
||||
python3 bench/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> # the benchmark; --from/--to: one range for all
|
||||
python3 bench/compare.py <before.txt> <after.txt> # two benchmarks: the regressions, the complexity
|
||||
python3 bench/sensors.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD # the sensors of each event
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Two benchmark outputs (regression.py), before and after: what changed, and the regressions by the definition of
|
||||
CLAUDE.md, computed.
|
||||
- A family Total: "found" or "correct" loses more than TOTAL_POINTS.
|
||||
- A family of a station: "found" or "correct" loses more than POINTS and more than INSTANTS instants ("found": labelled
|
||||
instants no longer flagged; "correct": wrongly flagged instants added). Below, the loss is noise of a few instants.
|
||||
Then the complexity before → after. The verdict is the reader's: a drop the labels explain (CLAUDE.md, "Not a
|
||||
regression") is said, not hidden.
|
||||
Usage: python3 compare.py <before.txt> <after.txt>
|
||||
"""
|
||||
import re
|
||||
import sys
|
||||
|
||||
TOTAL_POINTS = 1
|
||||
POINTS = 5
|
||||
INSTANTS = 20
|
||||
CELL = r"(\d+)/(\d+) \(\d+ %\)|—"
|
||||
|
||||
|
||||
def parse(path):
|
||||
"""Benchmark output -> ({(station, family): (both, labelled, flagged)}, complexity line or '')."""
|
||||
out, station, cx = {}, None, ""
|
||||
for line in open(path, encoding="utf-8"):
|
||||
if line.startswith("Complexity:"):
|
||||
cx = line.split(":", 1)[1].strip()
|
||||
elif line.strip() and not line.startswith(" "):
|
||||
station = line.split(":")[0].strip()
|
||||
elif m := re.match(rf" (\S.*?)\s{{2,}}({CELL})\s+({CELL})\s*$", line):
|
||||
g = m.groups() # family, found (both, labelled), correct (both, flagged)
|
||||
out[(station, g[0])] = (int(g[2] or g[5] or 0), int(g[3] or 0), int(g[6] or 0))
|
||||
return out, cx
|
||||
|
||||
|
||||
def pct(a, b):
|
||||
return 100 * a / b if b else None
|
||||
|
||||
|
||||
def changes(before, after):
|
||||
"""-> [(station, family, measure, before %, after %, lost instants, regression)] for every measure that moved."""
|
||||
rows = []
|
||||
for key in sorted(set(before) | set(after)):
|
||||
(b0, l0, f0), (b1, l1, f1) = before.get(key, (0, 0, 0)), after.get(key, (0, 0, 0))
|
||||
for measure, p0, p1, lost in (("found", pct(b0, l0), pct(b1, l1), b0 - b1),
|
||||
("correct", pct(b0, f0), pct(b1, f1), (f1 - b1) - (f0 - b0))):
|
||||
if p0 is None or p1 is None or round(p0) == round(p1):
|
||||
continue
|
||||
drop = p0 - p1
|
||||
bad = drop > TOTAL_POINTS if key[0] == "Total" else drop > POINTS and lost > INSTANTS
|
||||
rows.append((*key, measure, p0, p1, lost, bad))
|
||||
return rows
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
sys.exit(__doc__)
|
||||
(before, cx0), (after, cx1) = parse(sys.argv[1]), parse(sys.argv[2])
|
||||
rows = changes(before, after)
|
||||
for station, family, measure, p0, p1, lost, bad in rows:
|
||||
print(f"{'REGRESSION' if bad else '':<11}{station:<13}{family:<22}{measure:<8}{p0:4.0f} → {p1:3.0f} %"
|
||||
+ (f" ({lost} instants lost)" if p1 < p0 else ""))
|
||||
n = sum(r[-1] for r in rows)
|
||||
print(f"\n{n} regression(s); complexity: {cx0 or '?'} → {cx1 or '?'}")
|
||||
+11
-2
@@ -3,12 +3,13 @@
|
||||
For each <STATION>.jsonl of --labels that has its folder <STATION>/ in --data, the score per family and per instant of
|
||||
score.py, over the range of its labels: from the day of the first to the midnight after the last; with --from/--to, over
|
||||
that range for every station, a station without a label on it included (several stations labelled the same days). Then the total,
|
||||
family by family, which says whether the detectors do better overall. The block of a station is its score; the
|
||||
others, the non-regression: a family that goes back on a station is a regression, even if the total rises.
|
||||
family by family, which says whether the detectors do better overall, and the complexity of the detectors (code lines
|
||||
and constants of scripts/). Two outputs are compared by compare.py (the regressions, CLAUDE.md).
|
||||
Usage: python3 regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> [--stations A,B]
|
||||
[--from YYYY-MM-DD --to YYYY-MM-DD]
|
||||
"""
|
||||
import argparse
|
||||
import re
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
|
||||
@@ -17,6 +18,13 @@ import score # next to it
|
||||
from data import iso, parse_time # on the path through score
|
||||
|
||||
|
||||
def complexity(scripts=Path(__file__).resolve().parents[1] / "scripts"):
|
||||
"""-> (code lines, constants) of the detectors: non-blank lines that are not a comment; NAME = lines at module level."""
|
||||
lines = [x for f in sorted(scripts.glob("*.py")) for x in f.read_text().splitlines()]
|
||||
code = [x for x in lines if x.strip() and not x.strip().startswith("#")]
|
||||
return len(code), sum(bool(re.match(r"[A-Z][A-Z_0-9]* = ", x)) for x in lines)
|
||||
|
||||
|
||||
def label_range(path):
|
||||
"""Labels JSONL -> (start, end) iso, in whole UTC days, end excluded."""
|
||||
labels = score.read_labels(path)
|
||||
@@ -53,3 +61,4 @@ if __name__ == "__main__":
|
||||
for k, v in n.items():
|
||||
total[k] = [x + y for x, y in zip(total.get(k, [0, 0, 0]), v)]
|
||||
score.show("Total", total)
|
||||
print("\nComplexity: %d code lines, %d constants" % complexity())
|
||||
Reference in new issue
Block a user