diff --git a/CLAUDE.md b/CLAUDE.md index 366b27c..67cb117 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,14 +4,18 @@ AIMsight anomaly detectors (Python 3, standard library only) and the Claude Code ## Contract (do not break) -- One detector per family of `labeling/familles.md` (D-119), in `scripts/`: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py`. `categories.py` loads every `scripts/*.py` that has `detect`: never put another script with a `detect` function, or one with side effects at import, directly in `scripts/` (tools go in `skills/wf-improve/scripts/`). +- One detector per family, in `scripts/`: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py` (D-119). `categories.py` loads every `scripts/*.py` that has `detect`: never put another script with a `detect` function, or with side effects at import, directly in `scripts/`; tools go in `skills/wf-improve/scripts/`. - `detect(measurements)` returns episodes with the keys `start`, `end`, `kind` (the family), `label` (the variety), `text`, `ignore`, `sensors`. A detector may also declare `temperature`, `camera`, `metrics` (`categories.CONTEXT`) or `end`; `categories.events` passes them by introspection. -- Diagnostic-app reads this repository as it is: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the `--from/--to` CLIs, `/detectors:wf-improve`, `wf-improve//summary.md`. A rename here needs the matching change there. -- Label fields come from Diagnostic-app and stay French: `debut`, `fin`, `categorie`, `description`; likewise `demande.json` (`STATION`, `début`, `fin`). +- Label files are external data and keep their French keys: `debut`, `fin`, `categorie`, `description`. +- Diagnostic-app uses this repository as its `detectors/` submodule: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the `--from/--to` CLIs, `/detectors:wf-improve`. A rename here needs the matching change there. + +## The workflow stays generic + +`skills/wf-improve/SKILL.md` and `agents/` describe only the workflow (range + labels → upgrader → validator → summary). Everything specific to a caller (its paths, its files, its git rules, where the summary goes) is given in the request at launch, never written here. The tools take their paths as arguments. ## Rules - A constant added or changed updates `thresholds.md` in the same change; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`. - No rule specific to a station, a sensor or a date. -- A change to a detector is checked with the benchmark, from a working repository where this repository is mounted (or copied, not symlinked) as `detectors/` next to `data/` and `labeling/`: `python3 detectors/skills/wf-improve/scripts/regression.py`. A refactoring must leave every figure unchanged. -- History references in docstrings ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past sessions and to decisions recorded in Diagnostic-app/Support-Agent; keep them when editing a rule. +- A change to a detector is checked with the benchmark: `python3 skills/wf-improve/scripts/regression.py --labels --data ` (Diagnostic-app: `labeling/auto`, `data`; about 3 min). A refactoring must leave every figure unchanged. +- History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in Diagnostic-app/Support-Agent; keep them when editing a rule. diff --git a/README.md b/README.md index 1a98e3a..ceb6961 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,57 @@ # detectors -The AIMsight anomaly detectors, and their improvement. One detector per anomaly family of the displacements, classified into events by `scripts/categories.py`; `scripts/data.py` reads a station folder. Standard library only (Python 3). +The AIMsight anomaly detectors, and the workflow that improves them. Python 3, standard library only. + +## The detectors + +One detector per anomaly family of the displacements, in `scripts/`: + +| Family | Detector | +|---|---| +| Visibility | `visibility.py` | +| Obstruction | `obstruction.py` | +| Target issue | `target_issue.py` | +| Thermal displacement | `thermal_displacement.py` | +| No data | `no_data.py` | +| Displacement | `displacement.py` | + +`categories.py` runs them all and classifies their episodes into events, one line per event; `data.py` reads a station folder (`displacement/`, `metrics/`, `camera/`). ```bash -python3 scripts/categories.py --from YYYY-MM-DD --to YYYY-MM-DD +python3 scripts/categories.py --from YYYY-MM-DD --to YYYY-MM-DD # --chunk N, --history N +python3 scripts/visibility.py # one detector alone ``` -The station folder: `displacement/`, `metrics/`, `camera/` (the format of `data//` in Diagnostic-app). +`thresholds.md` is the registry of the detectors' thresholds and what each was tuned on. -- `scripts/`: the detectors (`visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py`), `categories.py`, `data.py`. -- `thresholds.md`: the registry of the detectors' thresholds. +## Improving them: `/detectors:wf-improve` -## The wf-improve workflow: a Claude Code plugin +The repository is also a Claude Code plugin (`.claude-plugin/plugin.json`). Its workflow improves the detectors on a range of data where they perform poorly, given labels as the truth on that range: -The repository is also a Claude Code plugin (`.claude-plugin/plugin.json`), to be mounted as `detectors/` in a working repository that has `labeling/` (the labels, `labeling/auto/.jsonl`; the analyses, `labeling/auto/analyses/`), `data/` and `agent-tools/`: Diagnostic-app, where it is a submodule. +1. measures before: the score on the range, and a benchmark on a reference truth (every labelled station); +2. the `upgrader` agent compares the detectors' events with the labels and fixes the detectors with general rules, one commit per fix; +3. the `validator` agent, an LLM-as-judge that does not read the upgrader's diagnosis, keeps the fixes, removes some, or cancels them: no station may go back, no overfitting; +4. a summary. -- `skills/wf-improve/`: `/detectors:wf-improve ` (`SKILL.md`), the session that improves the detectors from an analysis of the Data page, without overfitting; its tools in `scripts/`, run from the working repository: `regression.py` (the benchmark, all labels of `labeling/auto/`), `score.py` (the score of a range), `sensors.py` (the sensors of each event). -- `agents/`: `upgrader` (diagnoses and fixes the detectors) and `validator` (LLM-as-judge: keeps, removes or cancels the fixes), called `detectors:`. -- `wf-improve/`: the sessions, one folder `_/` each, written by the skill. +The result: detectors a little better on the range without going back elsewhere, or unchanged. Nothing is pushed. -From the working repository: +The caller gives the context of each run: the range, the labels, the reference truth, the families document, any context documents, the output folder, and its own rules (git branch, what else to commit). For example: ```bash -claude --plugin-dir detectors # then /detectors:wf-improve -python3 detectors/skills/wf-improve/scripts/regression.py # the benchmark +claude --plugin-dir detectors -p "/detectors:wf-improve DETECTORS: detectors/. STATION: data/NORDEND. \ +FROM: 2026-11-04. TO: 2026-11-08. LABELS: labeling/auto/NORDEND.jsonl. \ +REFERENCE: --labels labeling/auto --data data. FAMILIES: labeling/familles.md. \ +CONTEXT: labeling/auto/analyses/NORDEND_x/rapport.md. OUT: detectors/runs/NORDEND_x/." ``` -The session commits the detectors here, on `main`, without pushing: push `detectors` so that everyone has the updated detectors. +Labels: JSONL, one per line, `{"id", "debut", "fin", "categorie", "description"}` (UTC ISO times, `categorie` = the family). -Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here. +- `skills/wf-improve/SKILL.md`: the workflow; its tools in `skills/wf-improve/scripts/`: `score.py` (the score of a range), `regression.py` (the benchmark), `sensors.py` (the sensors of each event). +- `agents/`: `upgrader`, `validator` (called `detectors:`). + +```bash +python3 skills/wf-improve/scripts/score.py --labels --from YYYY-MM-DD --to YYYY-MM-DD +python3 skills/wf-improve/scripts/regression.py --labels .jsonl> --data /> +``` + +Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here. Used by Diagnostic-app, where it is the `detectors/` submodule. diff --git a/agents/upgrader.md b/agents/upgrader.md index f22f224..d6b6433 100644 --- a/agents/upgrader.md +++ b/agents/upgrader.md @@ -66,7 +66,7 @@ Your working folder is yours alone: `mktemp -d`. To try a variant without touchi The validator can have a single fix removed without touching the others: each has its commit. For each fix, in order: 1. change, then check as above, against the benchmark of the last committed state (the benchmark before, or that of your previous kept fix); before your output, the whole against the benchmark before; keep the output of the benchmark in `/benchmark-S.txt`; -2. kept: `git -C add && git -C commit -m "wf-improve: S" -- `, the message ending with an empty line then `Co-Authored-By: Claude Opus 5.5 `; +2. kept: `git -C add && git -C commit -m "wf-improve: S" -m "Co-Authored-By: Claude Opus 5.5 " -- `; 3. not kept: `git -C checkout -- `, only them, and it goes to "Not fixed". Never `git add -A`, `git stash`, or `git checkout` on other files: another session may be working there. diff --git a/agents/validator.md b/agents/validator.md index 818674d..b32253b 100644 --- a/agents/validator.md +++ b/agents/validator.md @@ -17,7 +17,7 @@ The detectors repository (`DETECTORS`: the detectors `scripts/*.py`, `thresholds - the range score before → after: the gain the modification sought; - in `modification.md`, the parts "What changed" and "Why", nothing else: `sed -n '/^## What changed/,/^## Before/p' `; -- the diff of the detectors since the base: `git -C diff -- 'scripts/*.py' thresholds.md`, and the code it touches; each fix has its commit (`git -C log --oneline ..HEAD`, message `wf-improve: S`), its diff by `git -C show `; +- the diff of the detectors since the base: `git -C diff HEAD -- 'scripts/*.py' thresholds.md`, and the code it touches; each fix has its commit (`git -C log --oneline ..HEAD`, message `wf-improve: S`), its diff by `git -C show `; - the benchmarks (the score per family on each station of the reference truth, then their `Total`), all stations in full; the benchmarks after each fix, in order, say which one makes a station go back; you may rerun the benchmark command; - to understand a regression on a station: the events command on its range (from the header of the station's block in the benchmark), with `2>&1`, at HEAD and, to compare, at the base (`git -C archive | tar -x -C ` in your working folder, then the same command with the copy's `scripts/categories.py`), the station's labels, and the families document. diff --git a/scripts/categories.py b/scripts/categories.py index f447de6..2044655 100644 --- a/scripts/categories.py +++ b/scripts/categories.py @@ -23,7 +23,7 @@ episodes; their label gives the variety (Poor visibility, Heat shimmer, Brief ob Each event gets its sentence, in English (the site): what happened and what to do about it, as the "For the client" column of the families document says; the cause is only given if the variety establishes it, otherwise the images show it. No detect function: the loader does not take this file for a detector. -Usage: python3 detectors/scripts/categories.py (one line per event: family, variety, sensors) +Usage: python3 scripts/categories.py (one line per event: family, variety, sensors) ... [--from YYYY-MM-DD] [--to YYYY-MM-DD] [--chunk 3] [--history N] (batch run: each chunk of N days classified alone, with the days of history before it if they exist; only the events touching the chunk are shown; "nothing" for a quiet chunk) @@ -50,7 +50,7 @@ def _context(folder, key): return CONTEXT[key](folder) -# The families of the families document, in the order of the timeline rows; ignore: greyed zone (None: as its episodes +# The families, in the order of the timeline rows; ignore: greyed zone (None: as its episodes # say: a total loss of visibility, strong heat shimmer); process: Images button, nothing, or Images and a link to # Diagnosis. CATEGORIES = { diff --git a/scripts/no_data.py b/scripts/no_data.py index e5d3c71..628d672 100644 --- a/scripts/no_data.py +++ b/scripts/no_data.py @@ -7,7 +7,7 @@ End of the window read (end, when the caller gives it: the chunked run of catego measurement and end, the episode running up to end (CLARIDEN, batch 01: 44 h without a measurement up to the end of the batch, not reported). No measurement in the export does not mean the station stopped: the text only says that the export has nothing. -Usage: python3 detectors/no_data.py (prints the episodes) +Usage: python3 scripts/no_data.py (prints the episodes) """ import os import statistics diff --git a/scripts/obstruction.py b/scripts/obstruction.py index 690c096..2ce9711 100644 --- a/scripts/obstruction.py +++ b/scripts/obstruction.py @@ -7,7 +7,7 @@ A sensor already in an overlapping Target issue episode (Degraded target, Target its curve is already to be ignored, the family is Target issue (boundary "Target issue or an episode of another family" of the families document; wf-improve session Obstruction, mod. 1: S1, problem targets of FLORIDA, NORDEND, SNCF01, NIEDERHORN, JUNGFRAU, CLARIDEN cut into Target obscured; S3, their isolated rejections as Brief obstruction). -Usage: python3 detectors/obstruction.py (prints the episodes) +Usage: python3 scripts/obstruction.py (prints the episodes) """ import os import sys diff --git a/scripts/target_issue.py b/scripts/target_issue.py index 0580f48..41789a6 100644 --- a/scripts/target_issue.py +++ b/scripts/target_issue.py @@ -34,7 +34,7 @@ Stuck target (E-011, FLORIDA: a target defined on the prism of a reference): per corr_y (median of the gaps from one measurement to the next) under STUCK times the median noise of the other targets, on both axes; one day is enough (a day without noise does not happen by accident); label "Target stuck". On E-011: C2 at 7e-5 px and C3 at 2e-4 against 0.013 for the others. -Usage: python3 detectors/target_issue.py (prints the episodes) +Usage: python3 scripts/target_issue.py (prints the episodes) """ import os import statistics diff --git a/skills/wf-improve/SKILL.md b/skills/wf-improve/SKILL.md index 71c5de7..c839909 100644 --- a/skills/wf-improve/SKILL.md +++ b/skills/wf-improve/SKILL.md @@ -48,13 +48,13 @@ Each agent is called with the Agent tool, its `subagent_type` (`detectors:upgrad 4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: . Families: . Modification: /modification.md. Range score: /range-before.txt → /range-after.txt. Benchmark: /benchmark-before.txt → /benchmark-after.txt, after each fix: /benchmark-S*.txt; command: <…>. Events command for any station: python3 /scripts/categories.py --from --to . Base: . Write /validation.md.` - `keep`: go to the summary. - `remove`: for each fix of the `**Remove**:` line, its commits (`git -C log --format=%h --grep ': S$' ..HEAD`), from the most recent: `git -C revert --no-edit `, then `git -C commit --amend -m "wf-improve: S removed" -m "Co-Authored-By: Claude Opus 5.5 "`. A conflict: `git -C revert --abort`, and do as for `cancel`. - - `cancel`, or anything else: `git -C checkout -- 'scripts/*.py' thresholds.md` (without the files already modified), commit of these paths, message `wf-improve: modification cancelled`. + - `cancel`, or anything else: `git -C revert --no-commit ..HEAD && git -C commit -m "wf-improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 "` (only what the fix commits changed; files already modified stay as they are). An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to the summary. Do not rerun an agent. ## Summary -Measures, step `end` (unless nothing changed: then `end` = `before`). Then `/summary.md`: +Measures, step `end`, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed). Then `/summary.md`: ```markdown # wf-improve: , → diff --git a/skills/wf-improve/scripts/score.py b/skills/wf-improve/scripts/score.py index 3de4848..a5052fb 100644 --- a/skills/wf-improve/scripts/score.py +++ b/skills/wf-improve/scripts/score.py @@ -25,7 +25,8 @@ HISTORY = timedelta(days=3) # as the batch command of categories.py def events(measurements, start, end, folder=None): - """Detectors' events touching the range, classified with its days of history (as the batch command).""" + """Detectors' events touching the range, classified with its days of history. Unlike the batch command, end is not + passed: no_data does not see a hole touching the end of the range.""" h, b = parse_time(start) - HISTORY, parse_time(end) chunk = {s: k for s, ms in measurements.items() if (k := [m for m in ms if h <= m["t"] < b])} return [v for v in (categories.events(chunk, folder=folder) if chunk else []) if v["end"] >= start] @@ -88,6 +89,8 @@ if __name__ == "__main__": p.add_argument("--from", dest="start", required=True) p.add_argument("--to", dest="end", required=True) a = p.parse_args() + if not os.path.exists(a.labels): + sys.exit(f"No labels file: {a.labels}") station, start, end = os.path.basename(os.path.normpath(a.folder)), iso(parse_time(a.start)), iso(parse_time(a.end)) n, labels = family_scores(a.folder, a.labels, start, end) if not labels: