diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json deleted file mode 100644 index 14de2a8..0000000 --- a/.claude-plugin/plugin.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "name": "detectors", - "description": "AIMsight anomaly detectors and their improvement on a range of labelled data (/detectors:wf-improve)." -} diff --git a/CLAUDE.md b/CLAUDE.md index 97abecd..e8b1424 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,23 +1,20 @@ # detectors -AIMsight anomaly detectors (Python 3, standard library only) and the Claude Code plugin that improves them (`/detectors:wf-improve`). Everything is in English: code, identifiers, docs, outputs. Layout and usage: `README.md`. +AIMsight anomaly detectors (Python 3, standard library only) and their benchmark (`bench/`); nothing else. They are improved by Optimizer (`improve` skill) with the upgrader and validator of the agents repository. Everything is in English: code, identifiers, docs, outputs. Layout and usage: `README.md`. ## Contract (do not break) -- One detector per family, in `scripts/`: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py` (D-119). `categories.py` loads every `scripts/*.py` that has `detect`: never put another script with a `detect` function, or with side effects at import, directly in `scripts/`; tools go in `skills/wf-improve/scripts/`. +- One detector per family, in `scripts/`: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py` (D-119). `categories.py` loads every `scripts/*.py` that has `detect`: never put another script with a `detect` function, or with side effects at import, directly in `scripts/`; tools go in `bench/`. - `detect(measurements)` returns episodes with the keys `start`, `end`, `kind` (the family), `label` (the variety), `text`, `ignore`, `sensors`. A detector may also declare `temperature`, `camera`, `metrics` (`categories.CONTEXT`) or `end`; `categories.events` passes them by introspection. - Label files are external data and keep their French keys: `debut`, `fin`, `categorie`, `description`. - Two repositories use this one through a `detectors -> ~/detectors` link (one copy on the workstation: a commit here is live for both at once); a rename here needs the matching change there: - - Diagnostic-app: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the `--from/--to` CLIs, `/detectors:wf-improve`; - - Optimizer (`batch.py`): `categories.events`, `categories.CATEGORIES`, and from `scripts/data.py` `load`, `load_temperature`, `is_reference`, `rejected`, `parse_time`, `iso`. - -## The workflow stays generic - -`skills/wf-improve/SKILL.md` and `agents/` describe only the workflow (range + labels → upgrader → validator → summary). Everything specific to a caller (its paths, its files, its git rules, where the summary goes) is given in the request at launch, never written here. The tools take their paths as arguments. + - Diagnostic-app: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the `--from/--to` CLIs, `bench/score.py` (the command shown in the page); + - Optimizer (`improve` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix. ## Rules - A constant added or changed updates `thresholds.md` in the same change; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`. - No rule specific to a station, a sensor or a date. -- A change to a detector is checked with the benchmark: `python3 skills/wf-improve/scripts/regression.py --labels --data ` (Diagnostic-app and Optimizer: `labeling/auto`, `data`, links to the shared `~/labeling` and `~/aimsight-data/data`; about 3 min). A refactoring must leave every figure unchanged. +- A change to a detector is checked with the benchmark: `python3 bench/regression.py --labels --data ` (Diagnostic-app and Optimizer: `labeling/auto`, `data`, links to the shared `~/labeling` and `~/aimsight-data/data`; about 3 min). A refactoring must leave every figure unchanged. - History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in Diagnostic-app/Support-Agent; keep them when editing a rule. +- The bench tools take their paths as arguments; the architecture of the repositories: `~/Diagnostic-App/docs/architecture.drawio`. diff --git a/README.md b/README.md index 449cf69..6d823c5 100644 --- a/README.md +++ b/README.md @@ -24,39 +24,20 @@ python3 scripts/visibility.py `thresholds.md` is the registry of the detectors' thresholds and what each was tuned on. -## Improving them: `/detectors:wf-improve` +## Measuring them: `bench/` -The repository is also a Claude Code plugin (`.claude-plugin/plugin.json`). Its workflow improves the detectors on a range of data where they perform poorly, given labels as the truth on that range: - -1. measures before: the score on the range, and a benchmark on a reference truth (every labelled station); -2. the `upgrader` agent compares the detectors' events with the labels and fixes the detectors with general rules, one commit per fix; -3. the `validator` agent, an LLM-as-judge that does not read the upgrader's diagnosis, keeps the fixes, removes some, or cancels them: no station may go back, no overfitting; -4. a summary. - -The result: detectors a little better on the range without going back elsewhere, or unchanged. Nothing is pushed. - -The caller gives the context of each run: the range, the labels, the reference truth, the families document, any context documents, the output folder, and its own rules (git branch, what else to commit). For example: +The detectors against labels (JSONL, one per line, `{"id", "debut", "fin", "categorie", "description"}`, UTC ISO times, `categorie` = the family): ```bash -claude --plugin-dir detectors -p "/detectors:wf-improve DETECTORS: detectors/. STATION: data/NORDEND. \ -FROM: 2026-11-04. TO: 2026-11-08. LABELS: labeling/auto/NORDEND.jsonl. \ -REFERENCE: --labels labeling/auto --data data. FAMILIES: labeling/familles.md. \ -CONTEXT: labeling/auto/analyses/NORDEND_x/rapport.md. OUT: detectors/runs/NORDEND_x/." +python3 bench/score.py --labels --from YYYY-MM-DD --to YYYY-MM-DD # the score of a range +python3 bench/regression.py --labels .jsonl> --data /> # the benchmark; --from/--to: one range for all +python3 bench/sensors.py --from YYYY-MM-DD --to YYYY-MM-DD # the sensors of each event ``` -Several stations labelled on the same range (Optimizer's daily batch): `DATA: data. STATIONS: A,B. LABELS: .jsonl>` in place of `STATION` and a labels file; a single session for all of them. +A change to a detector is checked with the benchmark; a refactoring leaves every figure unchanged. -Labels: JSONL, one per line, `{"id", "debut", "fin", "categorie", "description"}` (UTC ISO times, `categorie` = the family). +## Improving them -The agents keep the scripts they write in `/tools/` and list them in their outputs; the summary copies the list. +The detectors are improved by Optimizer (its `improve` skill), with the `upgrader` and `validator` agents of the agents repository: on the range of an analysis, its labels as the truth, the upgrader fixes the detectors (one commit per fix here) and the validator keeps, removes or cancels the fixes against overfitting, judged by this benchmark. Nothing is pushed. -- `skills/wf-improve/SKILL.md`: the workflow; its tools in `skills/wf-improve/scripts/`: `score.py` (the score of a range), `regression.py` (the benchmark), `sensors.py` (the sensors of each event). -- `agents/`: `upgrader`, `validator` (called `detectors:`). - -```bash -python3 skills/wf-improve/scripts/score.py --labels --from YYYY-MM-DD --to YYYY-MM-DD -python3 skills/wf-improve/scripts/regression.py --labels .jsonl> --data /> # --from/--to: one range for all -``` - -Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here. Used by Diagnostic-app and Optimizer, through a `detectors -> ~/detectors` link (one copy on the workstation); their labels come from the `labeling` repository, their data from `aimsight-data`. +Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here. Used by Diagnostic-app and Optimizer, through a `detectors -> ~/detectors` link (one copy on the workstation); their labels come from the `labeling` repository, their data from `aimsight-data`. The architecture of the repositories: `~/Diagnostic-App/docs/architecture.drawio`. diff --git a/agents/upgrader.md b/agents/upgrader.md deleted file mode 100644 index 42532be..0000000 --- a/agents/upgrader.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -name: upgrader -description: Upgrader of the wf-improve workflow — on a range of data where the detectors perform poorly, compares what they find, all families, with the labels of the range (the truth), then fixes the detectors (adds, removes, changes) so that they do better there without overfitting, one commit per fix. Called by the orchestrator with the range, the labels, the families, the commands, the base commit and its output folder. -tools: Read, Edit, Write, Bash -model: inherit ---- - -You are the upgrader of the wf-improve workflow: you make the detectors better on a range of data of a station where they perform poorly. The labels of the range are the truth. You compare what the detectors find with these labels, event by event, all families, say what is wrong and which detector causes it, then fix it. Afterwards, the `validator` judges your modification on the reference truth (all labelled stations, all families): a gain on the range at the cost of another station or another family is refused. - -**Avoid overfitting**: a small step. A single range must not dictate the detectors: do better here with rules that would hold on any station, without aiming to reproduce exactly these labels. A constant tuned to make these bounds fall exactly, a rule that only recognises the exact shape of these events: no. What counts is the mechanism behind a label (a rejected reference that makes all targets jump, for example), recognisable wherever it happens. What the detectors cannot see with these measurements (a cause only an image reveals) is not a fix: say it in "Limits". - -## Your request - -The detectors repository (`DETECTORS`: the detectors `scripts/*.py`, their threshold registry `thresholds.md`), the station's data folder and the range (UTC), the labels file (or several stations on the same range, a labels file each, in one folder: the events and sensors commands run per station, the range score covers them all; a mechanism that comes back from one station to another comes before a single case), the families document, the context documents if any, the events, sensors, range score and benchmark commands with their outputs before, the base commit, the files already modified, the output folder and the path of your output `modification.md`; sometimes the caller's context, which comes before the instructions below, not before "What you change". - -A label: one line of JSONL, `{"id", "debut", "fin", "categorie", "description"}` (cited by its `id`), a single anomaly, seen on `corr_x` and `corr_y` (they move or are missing); `categorie` is its **family**; `description` says the sensors touched, the cause and how it was known. Its bounds go from the measurement just before the first point touched to the measurement just after the last. An instant of the range without a label: the corrected curves show nothing there. - -## 1. Diagnosis - -Read: -- the labels touching the range (`debut` < `TO` and `fin` ≥ `FROM`), and the context documents: the mechanism of each event, the sensors touched and those that do not move, the level of proof; a document may say that a label is wrong: a limit to report, not a target; -- the families document **in full**: the families and their boundaries; the category of an event is its family (one detector per family); -- the outputs of the events command (run with `2>&1`) and of the sensors command (the cadence, then the sensors of each event); the range score; the benchmark before; -- the code of the detectors, `scripts/categories.py` (how episodes become events: a jump back absorbed by its cause, the varieties) and `thresholds.md`. - -Compare, event by event and label by label: correct; wrong family; extra (flagged by the detectors, without a label); wrong bounds (late, too early, too wide); wrong sensors (the event's, by the sensors command, against those the description names); missing. A few measurements of gap at the bounds are not a defect. Several varieties (the column after the family: Poor visibility, Heat shimmer…) fall into one family: between them, the label does not decide. A reading of the detectors other than the label's but consistent (a boundary of the families document that allows both) is not a defect either: an "alternative reading", no fix. - -Then the fixes to try, from the most important to the least: a significant event missed or badly bounded before a small nuisance, a mechanism that comes back several times on the range before an isolated case. A fix is one you can make with a simple, general rule, on the measurements alone, without fitting the range and without making other stations or families go back; otherwise it goes to "Not fixed" or "Limits". You may read the station's data to understand why a rule fails. - -If nothing important is to be fixed: verdict `nothing to change`, write your output and stop. - -## 2. What you change - -Only the detectors (`/scripts/*.py`) and `/thresholds.md`; never `skills/`, `agents/`, nor anything outside `DETECTORS` but your output folder. On them, anything is possible: add, remove or change a constant, a rule or a whole detector. You never commit a file the request says was already modified. - -**Everything is detected**: what is shown or not to a client is not the detectors' business. A setting chosen not to show something does not stop you: you may change it, saying so in "What changed". - -**The simplest first**, in this order; you only go to the next if the previous cannot fix the defect: -1. remove or simplify: a rule, a constant or a detector that causes the defect, or that another rule makes useless; -2. change a constant; -3. adjust an existing rule; -4. add a new, short rule; -5. as a last resort, a new variety (a `label`) in the family's detector. - -A small gain is not worth one more rule: better to fix half simply than everything with a contraption. - -**Forbidden.** -- One detector per family: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py`, no more, no less; the `kind` of an episode is its family. A fix that needs a new family goes to "Not fixed". -- No rule specific to a station, a sensor or a date. -- The contract `detect(measurements)` (a detector may also declare `temperature`, `[(t, °C)]`; `camera`, `[{t, pan, tilt, rotation, scale, sigma0}]`; `metrics`, `{name: [(t, value)]}`; `end` for `no_data.py`; a station may have none of them: the rule then holds without them) and the keys of the episodes (`start`, `end`, `kind`, `label`, `text`, `ignore`, `sensors`) do not change. -- Standard library only. - -**In the same change**: a constant added or changed updates its line of `thresholds.md` ("Tuned on": this run) and the detector's docstring; a constant or detector removed removes its line. Check: `grep -nE "^[A-Z_0-9]+ = " /scripts/*.py`. - -## 3. Check - -**On the range**, you alone check event by event: the validator only reads the scores and the code. Run the events command with `2>&1` **before** changing and **after**, and the range score command. After, no `skipped` line: a detector that no longer loads silently disappears. - -Then the **benchmark** against the benchmark before: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family. -- **The range score**: what you make rise, without losing a whole event. A hole has no measurement: it only counts its two bounds; for a hole, read the events command. Do not aim for 100 %. -- **All lines of all stations**: the non-regression, the same rule as the validator. **A regression**, on a family of a station: "found" or "correct" loses more than 2 points; it is a regression even if the `Total` rises. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain (labels contradicting each other from one station to another, the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score counted as found by an event on another sensor). A regression: rework your change before committing it, or drop the fix. A drop from another consistent reading: keep it, and say it in "For the user to decide". - -Your working folder is yours alone: `mktemp -d`. A script you write to read the data or to check a rule (anything longer than a one-line command) is a file in `/tools/`, its first line saying what it does, and runs from there, never inline (heredoc, `python3 -c`): the caller keeps these scripts to reuse them; your output lists them (`## Tools`). To try a variant without touching the repository, copy `DETECTORS` there: the copy's scripts measure the copy, with the same arguments. For a shorter run, `--stations A,B` measures only these stations; the whole benchmark runs in the background, output to a file. - -## 4. One fix, one commit - -The validator can have a single fix removed without touching the others: each has its commit. For each fix, in order: -1. change, then check as above, against the benchmark of the last committed state (the benchmark before, or that of your previous kept fix); before your output, the whole against the benchmark before; keep the output of the benchmark in `/benchmark-S.txt`; -2. kept: `git -C add && git -C commit -m "wf-improve: S" -m "Co-Authored-By: Claude Opus 5.5 " -- `; -3. not kept: `git -C checkout -- `, only them, and it goes to "Not fixed". - -Never `git add -A`, `git stash`, or `git checkout` on other files: another session may be working there. - -## Your output, `modification.md` - -```markdown -# Modification - -**Verdict**: changed | nothing to change - -## Diagnosis -; | Family | labelled | found | flagged | wrongly | - - -## What changed - and its commit: file, constant, rule or detector added, removed or changed; old and new value> - -## Why - - -## Before / after on the range - - -## Benchmark - - -## For the user to decide -none | - - -## Not fixed -none | - -## Limits -none | - - -## Tools -none | - <`/tools/