diff --git a/CLAUDE.md b/CLAUDE.md index bd7efd0..8caa3f4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,17 +10,18 @@ AIMsight anomaly detectors (Python 3, standard library only) and their benchmark - Label files (JSONL): `id`, `start`, `end`, `family`, `description` (and `review`, `analysis`); the project uses `labeling/auto` only. - Two repositories use this one through a `detectors -> ~/aimsight-detectors` link (one copy on the workstation: a commit here is live for both at once); a rename here needs the matching change there: - aimsight-app: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the event fields (`start`, `end`, `category`, `text`, `sensors`, `ignore`, `episodes`), the `--from/--to` CLIs, `bench/score.py` (the command shown in the page); - - Optimizer (`optimize` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix. + - Optimizer (`optimize` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `compare.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix. ## Rules - A change to improve the detectors touches `scripts/*.py` and `thresholds.md` only; never `bench/`, which is the measure. Anything is possible there: add, remove or change a constant, a rule or a whole detector, as long as the six families remain. -- **Simple first**: a rule earns its place by its score on the benchmark, not by one case; prefer removing or widening a rule to adding a special case. +- **Simple first**: a rule earns its place by its score on the benchmark, not by one case; prefer removing or widening a rule to adding a special case. The benchmark ends with the complexity of the detectors (code lines and constants of `scripts/`), `bench/compare.py` shows it before → after. A fix that adds a constant or more than 20 code lines earns them with at least 1 point on the `Total` of a family ("found" or "correct"); otherwise it fits one range: simplify it or drop it. A change that removes code without a regression is a gain in itself. +- **Pruning**: after a fix, each rule of the detector it touched (a condition, a branch, a constant a plainer value could replace) is tried without it, on a copy in the working folder (the copy's `bench/regression.py`, then `bench/compare.py`); a rule whose removal makes no regression is removed, one commit per rule, with its line of `thresholds.md`. - **Everything is detected**: what is shown or not to a client is not the detectors' business. A setting chosen not to show something may be changed, saying so. - A constant added or changed updates its line of `thresholds.md` ("Tuned on": the run that changed it) and the detector's docstring in the same change; a constant or rule removed removes its line; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`. - A detector that fails is skipped: a `skipped` line in the output of `categories.py` or of the benchmark means the change is broken. - No rule specific to a station, a sensor or a date. - A change to a detector is checked with the benchmark: `python3 bench/regression.py --labels --data ` (aimsight-app and Optimizer: `labeling/auto`, `data`, links to the shared `~/aimsight-labeling` and `~/aimsight-data/data`; about 1 min). A refactoring must leave every figure unchanged. -- **The benchmark's scores**, per family of each station: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family. **A regression**, on a family of a station: "found" or "correct" loses more than 2 points; it is a regression even if the `Total` rises; under 2 points it is not one; a rise is a plus, never a requirement. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain: labels contradicting each other from one station to another (the same state in two families), the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score (per instant, without the sensors) counted as found by an event on another sensor. Labels have their ambiguities: the goal is detectors that do their best, not perfect, and not overtrained on the labels. +- **The benchmark's scores**, per family of each station: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family. **A regression**, computed by `python3 bench/compare.py ` (its constants): a family `Total` whose "found" or "correct" loses more than 1 point; a family of a station whose "found" or "correct" loses more than 5 points and more than 20 instants (labelled instants no longer flagged, or wrongly flagged instants added), even if the `Total` rises; below, it is the noise of a few instants (a station at one measurement every 4 h, a family with few labels). A rise is a plus, never a requirement. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain: labels contradicting each other from one station to another (the same state in two families), the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score (per instant, without the sensors) counted as found by an event on another sensor. A drop counted as not a regression is said, with the labels (id). Labels have their ambiguities: the goal is detectors that do their best, not perfect, and not overtrained on the labels. - History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in aimsight-app/Support-Agent; keep them short when editing a rule. - The bench tools take their paths as arguments; the architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`. diff --git a/README.md b/README.md index 746c53b..e5e1eb0 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,7 @@ The detectors against labels (JSONL, one per line, `{"id", "start", "end", "fami ```bash python3 bench/score.py --labels --from YYYY-MM-DD --to YYYY-MM-DD # the score of a range python3 bench/regression.py --labels .jsonl> --data /> # the benchmark; --from/--to: one range for all +python3 bench/compare.py # two benchmarks: the regressions, the complexity python3 bench/sensors.py --from YYYY-MM-DD --to YYYY-MM-DD # the sensors of each event ``` diff --git a/bench/compare.py b/bench/compare.py new file mode 100644 index 0000000..ca942fb --- /dev/null +++ b/bench/compare.py @@ -0,0 +1,61 @@ +"""Two benchmark outputs (regression.py), before and after: what changed, and the regressions by the definition of +CLAUDE.md, computed. +- A family Total: "found" or "correct" loses more than TOTAL_POINTS. +- A family of a station: "found" or "correct" loses more than POINTS and more than INSTANTS instants ("found": labelled + instants no longer flagged; "correct": wrongly flagged instants added). Below, the loss is noise of a few instants. +Then the complexity before → after. The verdict is the reader's: a drop the labels explain (CLAUDE.md, "Not a +regression") is said, not hidden. +Usage: python3 compare.py +""" +import re +import sys + +TOTAL_POINTS = 1 +POINTS = 5 +INSTANTS = 20 +CELL = r"(\d+)/(\d+) \(\d+ %\)|—" + + +def parse(path): + """Benchmark output -> ({(station, family): (both, labelled, flagged)}, complexity line or '').""" + out, station, cx = {}, None, "" + for line in open(path, encoding="utf-8"): + if line.startswith("Complexity:"): + cx = line.split(":", 1)[1].strip() + elif line.strip() and not line.startswith(" "): + station = line.split(":")[0].strip() + elif m := re.match(rf" (\S.*?)\s{{2,}}({CELL})\s+({CELL})\s*$", line): + g = m.groups() # family, found (both, labelled), correct (both, flagged) + out[(station, g[0])] = (int(g[2] or g[5] or 0), int(g[3] or 0), int(g[6] or 0)) + return out, cx + + +def pct(a, b): + return 100 * a / b if b else None + + +def changes(before, after): + """-> [(station, family, measure, before %, after %, lost instants, regression)] for every measure that moved.""" + rows = [] + for key in sorted(set(before) | set(after)): + (b0, l0, f0), (b1, l1, f1) = before.get(key, (0, 0, 0)), after.get(key, (0, 0, 0)) + for measure, p0, p1, lost in (("found", pct(b0, l0), pct(b1, l1), b0 - b1), + ("correct", pct(b0, f0), pct(b1, f1), (f1 - b1) - (f0 - b0))): + if p0 is None or p1 is None or round(p0) == round(p1): + continue + drop = p0 - p1 + bad = drop > TOTAL_POINTS if key[0] == "Total" else drop > POINTS and lost > INSTANTS + rows.append((*key, measure, p0, p1, lost, bad)) + return rows + + +if __name__ == "__main__": + if len(sys.argv) != 3: + sys.exit(__doc__) + (before, cx0), (after, cx1) = parse(sys.argv[1]), parse(sys.argv[2]) + rows = changes(before, after) + for station, family, measure, p0, p1, lost, bad in rows: + print(f"{'REGRESSION' if bad else '':<11}{station:<13}{family:<22}{measure:<8}{p0:4.0f} → {p1:3.0f} %" + + (f" ({lost} instants lost)" if p1 < p0 else "")) + n = sum(r[-1] for r in rows) + print(f"\n{n} regression(s); complexity: {cx0 or '?'} → {cx1 or '?'}") diff --git a/bench/regression.py b/bench/regression.py index ca04447..3c30ebe 100644 --- a/bench/regression.py +++ b/bench/regression.py @@ -3,12 +3,13 @@ For each .jsonl of --labels that has its folder / in --data, the score per family and per instant of score.py, over the range of its labels: from the day of the first to the midnight after the last; with --from/--to, over that range for every station, a station without a label on it included (several stations labelled the same days). Then the total, -family by family, which says whether the detectors do better overall. The block of a station is its score; the -others, the non-regression: a family that goes back on a station is a regression, even if the total rises. +family by family, which says whether the detectors do better overall, and the complexity of the detectors (code lines +and constants of scripts/). Two outputs are compared by compare.py (the regressions, CLAUDE.md). Usage: python3 regression.py --labels .jsonl> --data /> [--stations A,B] [--from YYYY-MM-DD --to YYYY-MM-DD] """ import argparse +import re from datetime import timedelta from pathlib import Path @@ -17,6 +18,13 @@ import score # next to it from data import iso, parse_time # on the path through score +def complexity(scripts=Path(__file__).resolve().parents[1] / "scripts"): + """-> (code lines, constants) of the detectors: non-blank lines that are not a comment; NAME = lines at module level.""" + lines = [x for f in sorted(scripts.glob("*.py")) for x in f.read_text().splitlines()] + code = [x for x in lines if x.strip() and not x.strip().startswith("#")] + return len(code), sum(bool(re.match(r"[A-Z][A-Z_0-9]* = ", x)) for x in lines) + + def label_range(path): """Labels JSONL -> (start, end) iso, in whole UTC days, end excluded.""" labels = score.read_labels(path) @@ -53,3 +61,4 @@ if __name__ == "__main__": for k, v in n.items(): total[k] = [x + y for x, y in zip(total.get(k, [0, 0, 0]), v)] score.show("Total", total) + print("\nComplexity: %d code lines, %d constants" % complexity())