From 68768dfd39f3c84d60958b69f1146f02f639f182 Mon Sep 17 00:00:00 2001 From: maxime Date: Mon, 16 Nov 2026 14:13:38 +0100 Subject: [PATCH] Optimize: /optimize, the detectors and the tools from one analysis (ex /improve) Co-Authored-By: Claude Opus 5.5 --- .claude/skills/improve/SKILL.md | 89 --------------- .claude/skills/optimize/SKILL.md | 104 ++++++++++++++++++ .gitignore | 1 + CLAUDE.md | 8 +- README.md | 26 ++--- docs/superpowers/plans/2026-11-16-optimize.md | 88 +++++++++++++++ .../specs/2026-11-16-optimize-design.md | 90 +++++++++++++++ 7 files changed, 299 insertions(+), 107 deletions(-) delete mode 100644 .claude/skills/improve/SKILL.md create mode 100644 .claude/skills/optimize/SKILL.md create mode 100644 docs/superpowers/plans/2026-11-16-optimize.md create mode 100644 docs/superpowers/specs/2026-11-16-optimize-design.md diff --git a/.claude/skills/improve/SKILL.md b/.claude/skills/improve/SKILL.md deleted file mode 100644 index 9988168..0000000 --- a/.claude/skills/improve/SKILL.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -name: improve -description: Workflow that improves the detectors on a range of data where they perform poorly, given labels as the truth on that range — the upgrader fixes, the validator judges; the result, detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged. The caller gives the range, the labels, the reference truth, the output folder and any specific context. Run from Optimizer's root with the agents plugin (--plugin-dir ). /improve . -argument-hint: "" ---- - -# improve: improve the detectors on a range - -You are the orchestrator. The detectors (the detectors repository, `DETECTORS`: `scripts/`, `thresholds.md`) perform poorly on a range of data of a station, or of several stations on the same range; labels give the truth on this range. You run the upgrader, which fixes the detectors, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end. - -## Inputs - -Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope. - -- `DETECTORS`: the path of the detectors repository (default: `detectors/`, a link in Optimizer). -- `STATION`: the station's data folder (read by `DETECTORS/scripts/data.py`); or, several stations on the same range: `DATA`, the folder of the station folders, and `STATIONS`, `A,B`, the stations to improve. -- `FROM`, `TO`: the range, UTC days (YYYY-MM-DD, `TO` excluded). -- `LABELS`: the station's labels file (JSONL: `id`, `start`, `end`, `family`, `description`), the truth on the range; with `STATIONS`, a folder of `.jsonl`, one per station. -- `REFERENCE`: the reference truth for the non-regression, `--labels .jsonl> --data />`. -- `FAMILIES`: the document defining the families and their boundaries (one detector per family). -- `CONTEXT` (optional): documents explaining the range's labels (an analyst's report, for example). -- `OUT`: the output folder of this run (default: `sessions/_/` in Optimizer). - -A required input missing, or `DETECTORS` not a git repository: say so and stop, changing nothing. - -## Commands - -- events: `python3 /scripts/categories.py --from --to ` -- sensors: `python3 /bench/sensors.py --from --to ` -- range score (the goal): `python3 /bench/score.py --labels --from --to `; it exits with an error if no label touches the range: note it in the summary and stop without touching the detectors. -- with `STATIONS`: the events and sensors commands once per station, `/` for each `S` of `STATIONS`; the range score, all of them at once: `python3 /bench/regression.py --labels --data --stations --from --to `. -- benchmark (the non-regression, about 3 min): `python3 /bench/regression.py ` - -The measures of a step: ` > /range-.txt 2>&1` and ` > /benchmark-.txt 2>&1`. - -## Git - -The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C rev-parse HEAD`. Files already modified (`git -C status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. Always commit by path (`git -C commit -m "" -- `), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 `. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's. - -## Steps - -Each agent is called with the Agent tool, its `subagent_type` (`agents:upgrader`, `agents:validator`, from the agents plugin loaded with `--plugin-dir`; not available: say so and stop, changing nothing) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' `. - -1. **Measures before**: step `before`. -2. **Upgrader**, a single pass: `Detectors: . Station: / for each S of STATIONS>, range → (UTC). Labels: . Families: . Context: . Events command: <…>. Sensors command: <…>. Range score command: <…> (before: /range-before.txt). Benchmark command: <…> (before: /benchmark-before.txt). Base: . Files already modified: . Output folder: . Write /modification.md.` - It commits each fix it keeps, message `improve: S`, with its benchmark `/benchmark-S.txt`. - Verdict `nothing to change`, or no fix commit (`git -C log --format=%h --grep ': S[0-9]*$' ..HEAD` empty): go to the summary. -3. **Measures after**: step `after`. -4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: . Families: . Modification: /modification.md. Range score: /range-before.txt → /range-after.txt. Benchmark: /benchmark-before.txt → /benchmark-after.txt, after each fix: /benchmark-S*.txt; command: <…>. Events command for any station: python3 /scripts/categories.py --from --to . Base: . Write /validation.md.` - - `keep`: go to the summary. - - `remove`: for each fix of the `**Remove**:` line, its commits (`git -C log --format=%h --grep ': S$' ..HEAD`), from the most recent: `git -C revert --no-edit `, then `git -C commit --amend -m "improve: S removed" -m "Co-Authored-By: Claude Opus 5.5 "`. A conflict: `git -C revert --abort`, and do as for `cancel`. - - `cancel`, or anything else: `git -C revert --no-commit ..HEAD && git -C commit -m "improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 "` (only what the fix commits changed; files already modified stay as they are). - -An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to the summary. Do not rerun an agent. - -## Summary - -Measures, step `end`, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed). Then `/summary.md`: - -```markdown -# improve: , → - -**Result**: modification kept | kept without S… | cancelled | none | error - -## The range -the range score, before → end, one line per family (found, correct) - -## Elsewhere -the `Total` line of each family, before → end; then the other changed lines (station, family), before → end - -## What changed -; the upgrader's verdict, the validator's - -## For the user to decide - - -## Not fixed -<"Not fixed" of the modification; none> - -## Tools -/tools/`); none> - -## Errors - log --oneline ..HEAD`> -``` - -## Answer - -In 4 lines at most: the `**Result**` line, the range score before → end (changed families), the `Total` before → end if it changed, the path of the summary. diff --git a/.claude/skills/optimize/SKILL.md b/.claude/skills/optimize/SKILL.md new file mode 100644 index 0000000..66e059a --- /dev/null +++ b/.claude/skills/optimize/SKILL.md @@ -0,0 +1,104 @@ +--- +name: optimize +description: Workflow that, from one analysis (its labels as the truth on its range, its scripts and what it missed), optimises the detectors — the upgrader fixes, the validator judges; detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged — then the reading tools — the tool-builder brings the analyst's useful scripts into them. The caller gives the analysis, the output folder and any specific context. Run from Optimizer's root with the agents plugin (--plugin-dir ). /optimize . +argument-hint: "ANALYSIS: . OUT: . [context]" +--- + +# optimize: the detectors and the tools, from one analysis + +You are the orchestrator. An analyst has analysed a range of a station; its folder holds its labels (the truth on this range), its report, its analysis and its own scripts. Two parts follow, in this order. **Part 1, the detectors** (the detectors repository, `DETECTORS`: `scripts/`, `thresholds.md`): you run the upgrader, which fixes them, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **Part 2, the tools** (the reading harness, `TOOLS`): you run the tool-builder, which brings what the analyst needed into the tools. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end. + +## Inputs + +Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope. + +- `ANALYSIS` (required): the analysis folder: `request.json` (`station`, `start`, `end`, UTC), `labels.jsonl`, `report.md`, `analysis.md`, `scripts/`. +- `OUT` (required): the output folder of this run (`sessions/_/` in Optimizer). +- Defaults, relative to Optimizer's root (its links), which the request may override: `DETECTORS: detectors/`, `TOOLS: tools/`, `DATA: data`, `LABELS: labeling/auto` (the folder of the `.jsonl` label files, JSONL: `id`, `start`, `end`, `family`, `description`), `FAMILIES: labeling/families.md` (the families and their boundaries, one detector per family). + +Derived from `/request.json`: + +- `STATION`: `/` (read by `DETECTORS/scripts/data.py`); +- `FROM`: the day of `start`; `TO`: the day after the day of `end` (UTC days, YYYY-MM-DD, `TO` excluded); +- the labels of the range: `/.jsonl` (the analysis's labels are submitted there); +- `REFERENCE`, the reference truth for the non-regression: `--labels --data `; +- `CONTEXT`: `/report.md`. + +`ANALYSIS` or `OUT` missing, no `/request.json`, `DETECTORS` or `TOOLS` not a git repository: say so and stop, changing nothing. + +## Commands + +- events: `python3 /scripts/categories.py --from --to ` +- sensors: `python3 /bench/sensors.py --from --to ` +- range score (the goal): `python3 /bench/score.py --labels /.jsonl --from --to `; it exits with an error if no label touches the range: note it in the summary, touch nothing in the detectors and go to part 2. +- benchmark (the non-regression, about 3 min): `python3 /bench/regression.py ` + +The measures of a step: ` > /range-.txt 2>&1` and ` > /benchmark-.txt 2>&1`. + +## Git + +The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C rev-parse HEAD`. Files already modified (`git -C status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. Always commit by path (`git -C commit -m "" -- `), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 `. The tool-builder commits in `TOOLS` itself. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's. + +## Steps + +Each agent is called with the Agent tool, its `subagent_type` (`agents:upgrader`, `agents:validator`, `agents:tool-builder`, from the agents plugin loaded with `--plugin-dir`; not available: say so and stop, changing nothing) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' `. + +### Part 1: the detectors + +1. **Measures before**: step `before`. +2. **Upgrader**, a single pass: `Detectors: . Station: , range → (UTC). Labels: /.jsonl. Families: . Context: . Events command: <…>. Sensors command: <…>. Range score command: <…> (before: /range-before.txt). Benchmark command: <…> (before: /benchmark-before.txt). Base: . Files already modified: . Output folder: . Write /modification.md.` + It commits each fix it keeps, message `improve: S`, with its benchmark `/benchmark-S.txt`. + Verdict `nothing to change`, or no fix commit (`git -C log --format=%h --grep ': S[0-9]*$' ..HEAD` empty): go to part 2. +3. **Measures after**: step `after`. +4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: . Families: . Modification: /modification.md. Range score: /range-before.txt → /range-after.txt. Benchmark: /benchmark-before.txt → /benchmark-after.txt, after each fix: /benchmark-S*.txt; command: <…>. Events command for any station: python3 /scripts/categories.py --from --to . Base: . Write /validation.md.` + - `keep`: go to part 2. + - `remove`: for each fix of the `**Remove**:` line, its commits (`git -C log --format=%h --grep ': S$' ..HEAD`), from the most recent: `git -C revert --no-edit `, then `git -C commit --amend -m "improve: S removed" -m "Co-Authored-By: Claude Opus 5.5 "`. A conflict: `git -C revert --abort`, and do as for `cancel`. + - `cancel`, or anything else: `git -C revert --no-commit ..HEAD && git -C commit -m "improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 "` (only what the fix commits changed; files already modified stay as they are). +5. **Measures end**, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed). + +An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to part 2. Do not rerun an agent. + +### Part 2: the tools + +Always, whatever part 1 gave (a stop in part 1 does not stop part 2). + +6. **Tool-builder**: `Analysis: /. Tools: /. Write /tools.md.` It reads the analysis, writes only in `TOOLS` and in ``, and commits in `TOOLS`. Its output missing: note it. Do not rerun it. + +## Summary + +`/summary.md`: + +```markdown +# optimize: , → + +**Result**: modification kept | kept without S… | cancelled | none | error +**Tools changed**: /tools.md> | error + +## The range +the range score, before → end, one line per family (found, correct) + +## Elsewhere +the `Total` line of each family, before → end; then the other changed lines (station, family), before → end + +## What changed +; the upgrader's verdict, the validator's + +## For the user to decide + + +## Not fixed +<"Not fixed" of the modification; none> + +## Reading tools +/tools.md, and its "data.py" line if changed; none> + +## Scripts of the agents +/tools/`); none> + +## Errors + log --oneline ..HEAD`; `git -C log --oneline -1` if the tools changed> +``` + +## Answer + +In 5 lines at most: the `**Result**` line, the range score before → end (changed families), the `Total` before → end if it changed, the `**Tools changed**` line, the path of the summary. diff --git a/.gitignore b/.gitignore index 26eec4a..876c16c 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ data labeling detectors agents +tools .venv/ __pycache__/ archive/batches/*.tmp/ diff --git a/CLAUDE.md b/CLAUDE.md index e7ec06c..faf9879 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,11 +1,11 @@ # CLAUDE.md -Optimizer: the optimisation of the AIMsight detectors on the labels of `labeling/auto`, and nothing else (`README.md`). The workflow: `.claude/skills/improve/SKILL.md` (`/improve`), launched from the root of Optimizer with the agents plugin (`--plugin-dir agents`); its agents `agents:upgrader` and `agents:validator`; the benchmark `detectors/bench/`. +Optimizer: the optimisation of the AIMsight detectors and of the reading tools, from one analysis of `labeling/auto`, and nothing else (`README.md`). The workflow: `.claude/skills/optimize/SKILL.md` (`/optimize`), launched from the root of Optimizer with the agents plugin (`--plugin-dir agents`); its agents `agents:upgrader`, `agents:validator` (the detectors) and `agents:tool-builder` (the tools); the benchmark `detectors/bench/`. -- **The links**, out of git: `detectors/` → `~/detectors` (the session commits there); `data/` → `~/aimsight-data/data` and `labeling/` → `~/labeling` (read-only); `agents/` → `~/agents` (the plugin, loaded read-only). +- **The links**, out of git: `detectors/` → `~/detectors` and `tools/` → `~/tools` (the session commits there); `data/` → `~/aimsight-data/data` and `labeling/` → `~/labeling` (read-only); `agents/` → `~/agents` (the plugin, loaded read-only). - **The sessions**: `sessions/_/`; Diagnostic-App reads their summary (`summary.md`). -- **The workflow stays generic**: the skill describes only the workflow (range + labels → upgrader → validator → summary); what is specific to a caller (its paths, its git rules, `Caller's context`) arrives in the request. -- **Check a change of the skill**: `env -u ANTHROPIC_API_KEY claude -p "/improve" --plugin-dir agents --model haiku` must say the inputs are missing and stop without touching `detectors/`; then a real session through the app's button. +- **The workflow stays generic**: the skill describes only the workflow (analysis → detectors: upgrader → validator; tools: tool-builder → summary); what is specific to a caller (its paths, its git rules, `Caller's context`) arrives in the request. +- **Check a change of the skill**: `env -u ANTHROPIC_API_KEY claude -p "/optimize" --plugin-dir agents --model haiku` must say the inputs are missing and stop without touching `detectors/` or `tools/`; then a real session through the app's button. - `archive/`: the former batch and its scanner, stopped; do not run them again. Rules: diff --git a/README.md b/README.md index 4a1d7f4..5beeea8 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,20 @@ # Optimizer -The optimisation of the AIMsight detectors, on the labels of `labeling/auto`, and nothing else. On the range of an in-depth analysis (made in the Diagnostic-App app), its labels as the truth, the `improve` skill: +The optimisation of the AIMsight detectors and of the reading tools, from one analysis, and nothing else. An analysis (made in the Diagnostic-App app by the analyst) leaves in its folder the two things needed: its labels, the truth on its range, and its own scripts with what it missed. On that analysis, the `optimize` skill: -1. measures the detectors' score on the range, and the benchmark on every label of `labeling/auto`; -2. has the detectors fixed by the `upgrader`, one fix per commit in `~/detectors`; -3. has them judged by the `validator`, which keeps, has some removed, or cancels: no station goes back, no overfitting; -4. writes the summary of the session. +1. **the detectors**: measures their score on the range, and the benchmark on every label of `labeling/auto`; has them fixed by the `upgrader`, one fix per commit in `~/detectors`; has them judged by the `validator`, which keeps, has some removed, or cancels: no station goes back, no overfitting; +2. **the tools**: has the `tool-builder` bring the analyst's useful scripts into the reading harness, commits in `~/tools`; always, whatever the detectors gave; +3. writes the summary of the session. -The `upgrader` and the `validator` come from the `agents` repository; the benchmark (`bench/`) from the `detectors` repository. Nothing is pushed. +The `upgrader`, the `validator` and the `tool-builder` come from the `agents` repository; the benchmark (`bench/`) from the `detectors` repository. Nothing is pushed. ## Install ```bash ln -s ~/detectors detectors # the detectors, shared with Diagnostic-App +ln -s ~/tools tools # the reading tools, shared with Diagnostic-App ln -s ~/aimsight-data/data data # the pumped data, out of git -ln -s ~/labeling labeling # the labels: the truth and the families +ln -s ~/labeling labeling # the labels, the analyses and the families ln -s ~/agents agents # the generic agents (Claude Code plugin) ``` @@ -22,21 +22,19 @@ ln -s ~/agents agents # the generic agents (Claude Code p ## Run -From the "Optimize the detectors" button of Diagnostic-App, on a finished analysis; or by hand, from the root of Optimizer: +From the "Optimize" button of Diagnostic-App, on a finished analysis; or by hand, from the root of Optimizer: ```bash -env -u ANTHROPIC_API_KEY claude -p "/improve DETECTORS: detectors/. STATION: data/NORDEND. FROM: 2026-11-04. TO: 2026-11-08. \ -LABELS: labeling/auto/NORDEND.jsonl. REFERENCE: --labels labeling/auto --data data. FAMILIES: labeling/families.md. \ -CONTEXT: labeling/auto/analyses//report.md. OUT: sessions/_/." \ - --plugin-dir agents --add-dir ~/detectors --add-dir ~/labeling --add-dir ~/aimsight-data/data \ +env -u ANTHROPIC_API_KEY claude -p "/optimize ANALYSIS: labeling/auto/analyses//. OUT: sessions/_/." \ + --plugin-dir agents --add-dir ~/detectors --add-dir ~/tools --add-dir ~/labeling --add-dir ~/aimsight-data/data \ --setting-sources project,local --strict-mcp-config \ --allowedTools Read,Write,Edit,Bash,Agent --permission-mode acceptEdits ``` -`TO` is excluded. Several stations on the same range: `DATA: data. STATIONS: A,B. LABELS: .jsonl>` instead of `STATION` and a labels file. +The station and the range come from the analysis's `request.json` (the range's last day included); the labels from `labeling/auto/.jsonl`; the context from its `report.md`. `DETECTORS`, `TOOLS`, `DATA`, `LABELS`, `FAMILIES` can be overridden in the request. ## What a session leaves -`sessions/_