Le benchmark dans bench/ ; plus de plugin : l'amélioration vit dans Optimizer, ses agents dans le dépôt agents
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5438b0b469
commit
e7c03e7304
9 files changed
+17
-316
No files matched your search
@@ -24,39 +24,20 @@ python3 scripts/visibility.py <station folder>
|
||||
|
||||
`thresholds.md` is the registry of the detectors' thresholds and what each was tuned on.
|
||||
|
||||
## Improving them: `/detectors:wf-improve`
|
||||
## Measuring them: `bench/`
|
||||
|
||||
The repository is also a Claude Code plugin (`.claude-plugin/plugin.json`). Its workflow improves the detectors on a range of data where they perform poorly, given labels as the truth on that range:
|
||||
|
||||
1. measures before: the score on the range, and a benchmark on a reference truth (every labelled station);
|
||||
2. the `upgrader` agent compares the detectors' events with the labels and fixes the detectors with general rules, one commit per fix;
|
||||
3. the `validator` agent, an LLM-as-judge that does not read the upgrader's diagnosis, keeps the fixes, removes some, or cancels them: no station may go back, no overfitting;
|
||||
4. a summary.
|
||||
|
||||
The result: detectors a little better on the range without going back elsewhere, or unchanged. Nothing is pushed.
|
||||
|
||||
The caller gives the context of each run: the range, the labels, the reference truth, the families document, any context documents, the output folder, and its own rules (git branch, what else to commit). For example:
|
||||
The detectors against labels (JSONL, one per line, `{"id", "debut", "fin", "categorie", "description"}`, UTC ISO times, `categorie` = the family):
|
||||
|
||||
```bash
|
||||
claude --plugin-dir detectors -p "/detectors:wf-improve DETECTORS: detectors/. STATION: data/NORDEND. \
|
||||
FROM: 2026-11-04. TO: 2026-11-08. LABELS: labeling/auto/NORDEND.jsonl. \
|
||||
REFERENCE: --labels labeling/auto --data data. FAMILIES: labeling/familles.md. \
|
||||
CONTEXT: labeling/auto/analyses/NORDEND_x/rapport.md. OUT: detectors/runs/NORDEND_x/."
|
||||
python3 bench/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD # the score of a range
|
||||
python3 bench/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> # the benchmark; --from/--to: one range for all
|
||||
python3 bench/sensors.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD # the sensors of each event
|
||||
```
|
||||
|
||||
Several stations labelled on the same range (Optimizer's daily batch): `DATA: data. STATIONS: A,B. LABELS: <folder of
|
||||
<STATION>.jsonl>` in place of `STATION` and a labels file; a single session for all of them.
|
||||
A change to a detector is checked with the benchmark; a refactoring leaves every figure unchanged.
|
||||
|
||||
Labels: JSONL, one per line, `{"id", "debut", "fin", "categorie", "description"}` (UTC ISO times, `categorie` = the family).
|
||||
## Improving them
|
||||
|
||||
The agents keep the scripts they write in `<OUT>/tools/` and list them in their outputs; the summary copies the list.
|
||||
The detectors are improved by Optimizer (its `improve` skill), with the `upgrader` and `validator` agents of the agents repository: on the range of an analysis, its labels as the truth, the upgrader fixes the detectors (one commit per fix here) and the validator keeps, removes or cancels the fixes against overfitting, judged by this benchmark. Nothing is pushed.
|
||||
|
||||
- `skills/wf-improve/SKILL.md`: the workflow; its tools in `skills/wf-improve/scripts/`: `score.py` (the score of a range), `regression.py` (the benchmark), `sensors.py` (the sensors of each event).
|
||||
- `agents/`: `upgrader`, `validator` (called `detectors:<name>`).
|
||||
|
||||
```bash
|
||||
python3 skills/wf-improve/scripts/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD
|
||||
python3 skills/wf-improve/scripts/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> # --from/--to: one range for all
|
||||
```
|
||||
|
||||
Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here. Used by Diagnostic-app and Optimizer, through a `detectors -> ~/detectors` link (one copy on the workstation); their labels come from the `labeling` repository, their data from `aimsight-data`.
|
||||
Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here. Used by Diagnostic-app and Optimizer, through a `detectors -> ~/detectors` link (one copy on the workstation); their labels come from the `labeling` repository, their data from `aimsight-data`. The architecture of the repositories: `~/Diagnostic-App/docs/architecture.drawio`.
|
||||
Reference in new issue
Block a user