Review fixes; README and CLAUDE.md finalised
- cancel reverts only the run's commits (base..HEAD), never the caller's edits - validator diffs base..HEAD, not the working tree - fix commits carry their Co-Authored-By line; no redundant end benchmark - score.py: clear error on a missing labels file; docstring says end is not passed - usage lines; README and CLAUDE.md for the generic workflow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0de006bcb2
commit
388619685a
10 files changed
+63
-30
No files matched your search
@@ -4,14 +4,18 @@ AIMsight anomaly detectors (Python 3, standard library only) and the Claude Code
|
||||
|
||||
## Contract (do not break)
|
||||
|
||||
- One detector per family of `labeling/familles.md` (D-119), in `scripts/`: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py`. `categories.py` loads every `scripts/*.py` that has `detect`: never put another script with a `detect` function, or one with side effects at import, directly in `scripts/` (tools go in `skills/wf-improve/scripts/`).
|
||||
- One detector per family, in `scripts/`: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py` (D-119). `categories.py` loads every `scripts/*.py` that has `detect`: never put another script with a `detect` function, or with side effects at import, directly in `scripts/`; tools go in `skills/wf-improve/scripts/`.
|
||||
- `detect(measurements)` returns episodes with the keys `start`, `end`, `kind` (the family), `label` (the variety), `text`, `ignore`, `sensors`. A detector may also declare `temperature`, `camera`, `metrics` (`categories.CONTEXT`) or `end`; `categories.events` passes them by introspection.
|
||||
- Diagnostic-app reads this repository as it is: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the `--from/--to` CLIs, `/detectors:wf-improve`, `wf-improve/<session>/summary.md`. A rename here needs the matching change there.
|
||||
- Label fields come from Diagnostic-app and stay French: `debut`, `fin`, `categorie`, `description`; likewise `demande.json` (`STATION`, `début`, `fin`).
|
||||
- Label files are external data and keep their French keys: `debut`, `fin`, `categorie`, `description`.
|
||||
- Diagnostic-app uses this repository as its `detectors/` submodule: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the `--from/--to` CLIs, `/detectors:wf-improve`. A rename here needs the matching change there.
|
||||
|
||||
## The workflow stays generic
|
||||
|
||||
`skills/wf-improve/SKILL.md` and `agents/` describe only the workflow (range + labels → upgrader → validator → summary). Everything specific to a caller (its paths, its files, its git rules, where the summary goes) is given in the request at launch, never written here. The tools take their paths as arguments.
|
||||
|
||||
## Rules
|
||||
|
||||
- A constant added or changed updates `thresholds.md` in the same change; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
|
||||
- No rule specific to a station, a sensor or a date.
|
||||
- A change to a detector is checked with the benchmark, from a working repository where this repository is mounted (or copied, not symlinked) as `detectors/` next to `data/` and `labeling/`: `python3 detectors/skills/wf-improve/scripts/regression.py`. A refactoring must leave every figure unchanged.
|
||||
- History references in docstrings ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past sessions and to decisions recorded in Diagnostic-app/Support-Agent; keep them when editing a rule.
|
||||
- A change to a detector is checked with the benchmark: `python3 skills/wf-improve/scripts/regression.py --labels <labels folder> --data <data folder>` (Diagnostic-app: `labeling/auto`, `data`; about 3 min). A refactoring must leave every figure unchanged.
|
||||
- History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in Diagnostic-app/Support-Agent; keep them when editing a rule.
|
||||
@@ -1,31 +1,57 @@
|
||||
# detectors
|
||||
|
||||
The AIMsight anomaly detectors, and their improvement. One detector per anomaly family of the displacements, classified into events by `scripts/categories.py`; `scripts/data.py` reads a station folder. Standard library only (Python 3).
|
||||
The AIMsight anomaly detectors, and the workflow that improves them. Python 3, standard library only.
|
||||
|
||||
## The detectors
|
||||
|
||||
One detector per anomaly family of the displacements, in `scripts/`:
|
||||
|
||||
| Family | Detector |
|
||||
|---|---|
|
||||
| Visibility | `visibility.py` |
|
||||
| Obstruction | `obstruction.py` |
|
||||
| Target issue | `target_issue.py` |
|
||||
| Thermal displacement | `thermal_displacement.py` |
|
||||
| No data | `no_data.py` |
|
||||
| Displacement | `displacement.py` |
|
||||
|
||||
`categories.py` runs them all and classifies their episodes into events, one line per event; `data.py` reads a station folder (`displacement/`, `metrics/`, `camera/`).
|
||||
|
||||
```bash
|
||||
python3 scripts/categories.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD
|
||||
python3 scripts/categories.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD # --chunk N, --history N
|
||||
python3 scripts/visibility.py <station folder> # one detector alone
|
||||
```
|
||||
|
||||
The station folder: `displacement/`, `metrics/`, `camera/` (the format of `data/<STATION>/` in Diagnostic-app).
|
||||
`thresholds.md` is the registry of the detectors' thresholds and what each was tuned on.
|
||||
|
||||
- `scripts/`: the detectors (`visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py`), `categories.py`, `data.py`.
|
||||
- `thresholds.md`: the registry of the detectors' thresholds.
|
||||
## Improving them: `/detectors:wf-improve`
|
||||
|
||||
## The wf-improve workflow: a Claude Code plugin
|
||||
The repository is also a Claude Code plugin (`.claude-plugin/plugin.json`). Its workflow improves the detectors on a range of data where they perform poorly, given labels as the truth on that range:
|
||||
|
||||
The repository is also a Claude Code plugin (`.claude-plugin/plugin.json`), to be mounted as `detectors/` in a working repository that has `labeling/` (the labels, `labeling/auto/<STATION>.jsonl`; the analyses, `labeling/auto/analyses/`), `data/` and `agent-tools/`: Diagnostic-app, where it is a submodule.
|
||||
1. measures before: the score on the range, and a benchmark on a reference truth (every labelled station);
|
||||
2. the `upgrader` agent compares the detectors' events with the labels and fixes the detectors with general rules, one commit per fix;
|
||||
3. the `validator` agent, an LLM-as-judge that does not read the upgrader's diagnosis, keeps the fixes, removes some, or cancels them: no station may go back, no overfitting;
|
||||
4. a summary.
|
||||
|
||||
- `skills/wf-improve/`: `/detectors:wf-improve <analysis>` (`SKILL.md`), the session that improves the detectors from an analysis of the Data page, without overfitting; its tools in `scripts/`, run from the working repository: `regression.py` (the benchmark, all labels of `labeling/auto/`), `score.py` (the score of a range), `sensors.py` (the sensors of each event).
|
||||
- `agents/`: `upgrader` (diagnoses and fixes the detectors) and `validator` (LLM-as-judge: keeps, removes or cancels the fixes), called `detectors:<name>`.
|
||||
- `wf-improve/`: the sessions, one folder `<session>_<analysis>/` each, written by the skill.
|
||||
The result: detectors a little better on the range without going back elsewhere, or unchanged. Nothing is pushed.
|
||||
|
||||
From the working repository:
|
||||
The caller gives the context of each run: the range, the labels, the reference truth, the families document, any context documents, the output folder, and its own rules (git branch, what else to commit). For example:
|
||||
|
||||
```bash
|
||||
claude --plugin-dir detectors # then /detectors:wf-improve <analysis>
|
||||
python3 detectors/skills/wf-improve/scripts/regression.py # the benchmark
|
||||
claude --plugin-dir detectors -p "/detectors:wf-improve DETECTORS: detectors/. STATION: data/NORDEND. \
|
||||
FROM: 2026-11-04. TO: 2026-11-08. LABELS: labeling/auto/NORDEND.jsonl. \
|
||||
REFERENCE: --labels labeling/auto --data data. FAMILIES: labeling/familles.md. \
|
||||
CONTEXT: labeling/auto/analyses/NORDEND_x/rapport.md. OUT: detectors/runs/NORDEND_x/."
|
||||
```
|
||||
|
||||
The session commits the detectors here, on `main`, without pushing: push `detectors` so that everyone has the updated detectors.
|
||||
Labels: JSONL, one per line, `{"id", "debut", "fin", "categorie", "description"}` (UTC ISO times, `categorie` = the family).
|
||||
|
||||
Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here.
|
||||
- `skills/wf-improve/SKILL.md`: the workflow; its tools in `skills/wf-improve/scripts/`: `score.py` (the score of a range), `regression.py` (the benchmark), `sensors.py` (the sensors of each event).
|
||||
- `agents/`: `upgrader`, `validator` (called `detectors:<name>`).
|
||||
|
||||
```bash
|
||||
python3 skills/wf-improve/scripts/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD
|
||||
python3 skills/wf-improve/scripts/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/>
|
||||
```
|
||||
|
||||
Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here. Used by Diagnostic-app, where it is the `detectors/` submodule.
|
||||
+1
-1
@@ -66,7 +66,7 @@ Your working folder is yours alone: `mktemp -d`. To try a variant without touchi
|
||||
|
||||
The validator can have a single fix removed without touching the others: each has its commit. For each fix, in order:
|
||||
1. change, then check as above, against the benchmark of the last committed state (the benchmark before, or that of your previous kept fix); before your output, the whole against the benchmark before; keep the output of the benchmark in `<output folder>/benchmark-S<n>.txt`;
|
||||
2. kept: `git -C <DETECTORS> add <the changed files> && git -C <DETECTORS> commit -m "wf-improve: S<n>" -- <the same>`, the message ending with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`;
|
||||
2. kept: `git -C <DETECTORS> add <the changed files> && git -C <DETECTORS> commit -m "wf-improve: S<n>" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" -- <the same>`;
|
||||
3. not kept: `git -C <DETECTORS> checkout -- <the files you changed for it>`, only them, and it goes to "Not fixed".
|
||||
|
||||
Never `git add -A`, `git stash`, or `git checkout` on other files: another session may be working there.
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ The detectors repository (`DETECTORS`: the detectors `scripts/*.py`, `thresholds
|
||||
|
||||
- the range score before → after: the gain the modification sought;
|
||||
- in `modification.md`, the parts "What changed" and "Why", nothing else: `sed -n '/^## What changed/,/^## Before/p' <modification>`;
|
||||
- the diff of the detectors since the base: `git -C <DETECTORS> diff <base> -- 'scripts/*.py' thresholds.md`, and the code it touches; each fix has its commit (`git -C <DETECTORS> log --oneline <base>..HEAD`, message `wf-improve: S<n>`), its diff by `git -C <DETECTORS> show <commit>`;
|
||||
- the diff of the detectors since the base: `git -C <DETECTORS> diff <base> HEAD -- 'scripts/*.py' thresholds.md`, and the code it touches; each fix has its commit (`git -C <DETECTORS> log --oneline <base>..HEAD`, message `wf-improve: S<n>`), its diff by `git -C <DETECTORS> show <commit>`;
|
||||
- the benchmarks (the score per family on each station of the reference truth, then their `Total`), all stations in full; the benchmarks after each fix, in order, say which one makes a station go back; you may rerun the benchmark command;
|
||||
- to understand a regression on a station: the events command on its range (from the header of the station's block in the benchmark), with `2>&1`, at HEAD and, to compare, at the base (`git -C <DETECTORS> archive <base> | tar -x -C <folder>` in your working folder, then the same command with the copy's `scripts/categories.py`), the station's labels, and the families document.
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ episodes; their label gives the variety (Poor visibility, Heat shimmer, Brief ob
|
||||
Each event gets its sentence, in English (the site): what happened and what to do about it, as the "For the client"
|
||||
column of the families document says; the cause is only given if the variety establishes it, otherwise the images show it.
|
||||
No detect function: the loader does not take this file for a detector.
|
||||
Usage: python3 detectors/scripts/categories.py <station folder> (one line per event: family, variety, sensors)
|
||||
Usage: python3 scripts/categories.py <station folder> (one line per event: family, variety, sensors)
|
||||
... <folder> [--from YYYY-MM-DD] [--to YYYY-MM-DD] [--chunk 3] [--history N] (batch run: each chunk of N days
|
||||
classified alone, with the days of history before it if they exist; only the events touching the chunk are
|
||||
shown; "nothing" for a quiet chunk)
|
||||
@@ -50,7 +50,7 @@ def _context(folder, key):
|
||||
return CONTEXT[key](folder)
|
||||
|
||||
|
||||
# The families of the families document, in the order of the timeline rows; ignore: greyed zone (None: as its episodes
|
||||
# The families, in the order of the timeline rows; ignore: greyed zone (None: as its episodes
|
||||
# say: a total loss of visibility, strong heat shimmer); process: Images button, nothing, or Images and a link to
|
||||
# Diagnosis.
|
||||
CATEGORIES = {
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ End of the window read (end, when the caller gives it: the chunked run of catego
|
||||
measurement and end, the episode running up to end (CLARIDEN, batch 01: 44 h without a measurement up to the end of
|
||||
the batch, not reported). No measurement in the export does not mean the station stopped: the text only says that the
|
||||
export has nothing.
|
||||
Usage: python3 detectors/no_data.py <station folder> (prints the episodes)
|
||||
Usage: python3 scripts/no_data.py <station folder> (prints the episodes)
|
||||
"""
|
||||
import os
|
||||
import statistics
|
||||
|
||||
@@ -7,7 +7,7 @@ A sensor already in an overlapping Target issue episode (Degraded target, Target
|
||||
its curve is already to be ignored, the family is Target issue (boundary "Target issue or an episode of another family"
|
||||
of the families document; wf-improve session Obstruction, mod. 1: S1, problem targets of FLORIDA, NORDEND, SNCF01,
|
||||
NIEDERHORN, JUNGFRAU, CLARIDEN cut into Target obscured; S3, their isolated rejections as Brief obstruction).
|
||||
Usage: python3 detectors/obstruction.py <station folder> (prints the episodes)
|
||||
Usage: python3 scripts/obstruction.py <station folder> (prints the episodes)
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
@@ -34,7 +34,7 @@ Stuck target (E-011, FLORIDA: a target defined on the prism of a reference): per
|
||||
corr_y (median of the gaps from one measurement to the next) under STUCK times the median noise of the other targets, on
|
||||
both axes; one day is enough (a day without noise does not happen by accident); label "Target stuck". On E-011: C2 at
|
||||
7e-5 px and C3 at 2e-4 against 0.013 for the others.
|
||||
Usage: python3 detectors/target_issue.py <station folder> (prints the episodes)
|
||||
Usage: python3 scripts/target_issue.py <station folder> (prints the episodes)
|
||||
"""
|
||||
import os
|
||||
import statistics
|
||||
|
||||
@@ -48,13 +48,13 @@ Each agent is called with the Agent tool, its `subagent_type` (`detectors:upgrad
|
||||
4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: <DETECTORS>. Families: <FAMILIES>. Modification: <OUT>/modification.md. Range score: <OUT>/range-before.txt → <OUT>/range-after.txt. Benchmark: <OUT>/benchmark-before.txt → <OUT>/benchmark-after.txt, after each fix: <OUT>/benchmark-S*.txt; command: <…>. Events command for any station: python3 <DETECTORS>/scripts/categories.py <station folder> --from <from> --to <to>. Base: <base>. Write <OUT>/validation.md.`
|
||||
- `keep`: go to the summary.
|
||||
- `remove`: for each fix of the `**Remove**:` line, its commits (`git -C <DETECTORS> log --format=%h --grep ': S<n>$' <base>..HEAD`), from the most recent: `git -C <DETECTORS> revert --no-edit <commit>`, then `git -C <DETECTORS> commit --amend -m "wf-improve: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`. A conflict: `git -C <DETECTORS> revert --abort`, and do as for `cancel`.
|
||||
- `cancel`, or anything else: `git -C <DETECTORS> checkout <base> -- 'scripts/*.py' thresholds.md` (without the files already modified), commit of these paths, message `wf-improve: modification cancelled`.
|
||||
- `cancel`, or anything else: `git -C <DETECTORS> revert --no-commit <base>..HEAD && git -C <DETECTORS> commit -m "wf-improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"` (only what the fix commits changed; files already modified stay as they are).
|
||||
|
||||
An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to the summary. Do not rerun an agent.
|
||||
|
||||
## Summary
|
||||
|
||||
Measures, step `end` (unless nothing changed: then `end` = `before`). Then `<OUT>/summary.md`:
|
||||
Measures, step `end`, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed). Then `<OUT>/summary.md`:
|
||||
|
||||
```markdown
|
||||
# wf-improve: <station>, <FROM> → <TO>
|
||||
|
||||
@@ -25,7 +25,8 @@ HISTORY = timedelta(days=3) # as the batch command of categories.py
|
||||
|
||||
|
||||
def events(measurements, start, end, folder=None):
|
||||
"""Detectors' events touching the range, classified with its days of history (as the batch command)."""
|
||||
"""Detectors' events touching the range, classified with its days of history. Unlike the batch command, end is not
|
||||
passed: no_data does not see a hole touching the end of the range."""
|
||||
h, b = parse_time(start) - HISTORY, parse_time(end)
|
||||
chunk = {s: k for s, ms in measurements.items() if (k := [m for m in ms if h <= m["t"] < b])}
|
||||
return [v for v in (categories.events(chunk, folder=folder) if chunk else []) if v["end"] >= start]
|
||||
@@ -88,6 +89,8 @@ if __name__ == "__main__":
|
||||
p.add_argument("--from", dest="start", required=True)
|
||||
p.add_argument("--to", dest="end", required=True)
|
||||
a = p.parse_args()
|
||||
if not os.path.exists(a.labels):
|
||||
sys.exit(f"No labels file: {a.labels}")
|
||||
station, start, end = os.path.basename(os.path.normpath(a.folder)), iso(parse_time(a.start)), iso(parse_time(a.end))
|
||||
n, labels = family_scores(a.folder, a.labels, start, end)
|
||||
if not labels:
|
||||
|
||||
Reference in new issue
Block a user