Optimize: /optimize, the detectors and the tools from one analysis (ex /improve)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
63bebbc415
commit
68768dfd39
7 files changed
+299
-107
No files matched your search
@@ -1,89 +0,0 @@
|
||||
---
|
||||
name: improve
|
||||
description: Workflow that improves the detectors on a range of data where they perform poorly, given labels as the truth on that range — the upgrader fixes, the validator judges; the result, detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged. The caller gives the range, the labels, the reference truth, the output folder and any specific context. Run from Optimizer's root with the agents plugin (--plugin-dir <agents>). /improve <request>.
|
||||
argument-hint: "<range, labels, reference, output folder, context>"
|
||||
---
|
||||
|
||||
# improve: improve the detectors on a range
|
||||
|
||||
You are the orchestrator. The detectors (the detectors repository, `DETECTORS`: `scripts/`, `thresholds.md`) perform poorly on a range of data of a station, or of several stations on the same range; labels give the truth on this range. You run the upgrader, which fixes the detectors, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||
|
||||
## Inputs
|
||||
|
||||
Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope.
|
||||
|
||||
- `DETECTORS`: the path of the detectors repository (default: `detectors/`, a link in Optimizer).
|
||||
- `STATION`: the station's data folder (read by `DETECTORS/scripts/data.py`); or, several stations on the same range: `DATA`, the folder of the station folders, and `STATIONS`, `A,B`, the stations to improve.
|
||||
- `FROM`, `TO`: the range, UTC days (YYYY-MM-DD, `TO` excluded).
|
||||
- `LABELS`: the station's labels file (JSONL: `id`, `start`, `end`, `family`, `description`), the truth on the range; with `STATIONS`, a folder of `<STATION>.jsonl`, one per station.
|
||||
- `REFERENCE`: the reference truth for the non-regression, `--labels <folder of <STATION>.jsonl> --data <folder of <STATION>/>`.
|
||||
- `FAMILIES`: the document defining the families and their boundaries (one detector per family).
|
||||
- `CONTEXT` (optional): documents explaining the range's labels (an analyst's report, for example).
|
||||
- `OUT`: the output folder of this run (default: `sessions/<analysis>_<YYYYMMDDTHHMMZ>/` in Optimizer).
|
||||
|
||||
A required input missing, or `DETECTORS` not a git repository: say so and stop, changing nothing.
|
||||
|
||||
## Commands
|
||||
|
||||
- events: `python3 <DETECTORS>/scripts/categories.py <STATION> --from <FROM> --to <TO>`
|
||||
- sensors: `python3 <DETECTORS>/bench/sensors.py <STATION> --from <FROM> --to <TO>`
|
||||
- range score (the goal): `python3 <DETECTORS>/bench/score.py <STATION> --labels <LABELS> --from <FROM> --to <TO>`; it exits with an error if no label touches the range: note it in the summary and stop without touching the detectors.
|
||||
- with `STATIONS`: the events and sensors commands once per station, `<DATA>/<S>` for each `S` of `STATIONS`; the range score, all of them at once: `python3 <DETECTORS>/bench/regression.py --labels <LABELS> --data <DATA> --stations <STATIONS> --from <FROM> --to <TO>`.
|
||||
- benchmark (the non-regression, about 3 min): `python3 <DETECTORS>/bench/regression.py <REFERENCE>`
|
||||
|
||||
The measures of a step: `<range score> > <OUT>/range-<step>.txt 2>&1` and `<benchmark> > <OUT>/benchmark-<step>.txt 2>&1`.
|
||||
|
||||
## Git
|
||||
|
||||
The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C <DETECTORS> rev-parse HEAD`. Files already modified (`git -C <DETECTORS> status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. Always commit by path (`git -C <DETECTORS> commit -m "<message>" -- <paths>`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's.
|
||||
|
||||
## Steps
|
||||
|
||||
Each agent is called with the Agent tool, its `subagent_type` (`agents:upgrader`, `agents:validator`, from the agents plugin loaded with `--plugin-dir`; not available: say so and stop, changing nothing) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
|
||||
|
||||
1. **Measures before**: step `before`.
|
||||
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION | <DATA>/<S> for each S of STATIONS>, range <FROM> → <TO> (UTC). Labels: <LABELS>. Families: <FAMILIES>. Context: <CONTEXT | none>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
|
||||
It commits each fix it keeps, message `improve: S<n>`, with its benchmark `<OUT>/benchmark-S<n>.txt`.
|
||||
Verdict `nothing to change`, or no fix commit (`git -C <DETECTORS> log --format=%h --grep ': S[0-9]*$' <base>..HEAD` empty): go to the summary.
|
||||
3. **Measures after**: step `after`.
|
||||
4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: <DETECTORS>. Families: <FAMILIES>. Modification: <OUT>/modification.md. Range score: <OUT>/range-before.txt → <OUT>/range-after.txt. Benchmark: <OUT>/benchmark-before.txt → <OUT>/benchmark-after.txt, after each fix: <OUT>/benchmark-S*.txt; command: <…>. Events command for any station: python3 <DETECTORS>/scripts/categories.py <station folder> --from <from> --to <to>. Base: <base>. Write <OUT>/validation.md.`
|
||||
- `keep`: go to the summary.
|
||||
- `remove`: for each fix of the `**Remove**:` line, its commits (`git -C <DETECTORS> log --format=%h --grep ': S<n>$' <base>..HEAD`), from the most recent: `git -C <DETECTORS> revert --no-edit <commit>`, then `git -C <DETECTORS> commit --amend -m "improve: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`. A conflict: `git -C <DETECTORS> revert --abort`, and do as for `cancel`.
|
||||
- `cancel`, or anything else: `git -C <DETECTORS> revert --no-commit <base>..HEAD && git -C <DETECTORS> commit -m "improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"` (only what the fix commits changed; files already modified stay as they are).
|
||||
|
||||
An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to the summary. Do not rerun an agent.
|
||||
|
||||
## Summary
|
||||
|
||||
Measures, step `end`, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed). Then `<OUT>/summary.md`:
|
||||
|
||||
```markdown
|
||||
# improve: <station | STATIONS>, <FROM> → <TO>
|
||||
|
||||
**Result**: modification kept | kept without S<n>… | cancelled | none | error
|
||||
|
||||
## The range
|
||||
the range score, before → end, one line per family (found, correct)
|
||||
|
||||
## Elsewhere
|
||||
the `Total` line of each family, before → end; then the other changed lines (station, family), before → end
|
||||
|
||||
## What changed
|
||||
<per fix kept, one sentence from "What changed" of the modification>; the upgrader's verdict, the validator's
|
||||
|
||||
## For the user to decide
|
||||
<copied as they are: "Limits" and "For the user to decide" of the modification, "Reservations" of the validator; none>
|
||||
|
||||
## Not fixed
|
||||
<"Not fixed" of the modification; none>
|
||||
|
||||
## Tools
|
||||
<copied as they are: "Tools" of the modification and of the validation (the scripts the agents kept in `<OUT>/tools/`); none>
|
||||
|
||||
## Errors
|
||||
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`>
|
||||
```
|
||||
|
||||
## Answer
|
||||
|
||||
In 4 lines at most: the `**Result**` line, the range score before → end (changed families), the `Total` before → end if it changed, the path of the summary.
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
name: optimize
|
||||
description: Workflow that, from one analysis (its labels as the truth on its range, its scripts and what it missed), optimises the detectors — the upgrader fixes, the validator judges; detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged — then the reading tools — the tool-builder brings the analyst's useful scripts into them. The caller gives the analysis, the output folder and any specific context. Run from Optimizer's root with the agents plugin (--plugin-dir <agents>). /optimize <request>.
|
||||
argument-hint: "ANALYSIS: <analysis folder>. OUT: <output folder>. [context]"
|
||||
---
|
||||
|
||||
# optimize: the detectors and the tools, from one analysis
|
||||
|
||||
You are the orchestrator. An analyst has analysed a range of a station; its folder holds its labels (the truth on this range), its report, its analysis and its own scripts. Two parts follow, in this order. **Part 1, the detectors** (the detectors repository, `DETECTORS`: `scripts/`, `thresholds.md`): you run the upgrader, which fixes them, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **Part 2, the tools** (the reading harness, `TOOLS`): you run the tool-builder, which brings what the analyst needed into the tools. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||
|
||||
## Inputs
|
||||
|
||||
Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope.
|
||||
|
||||
- `ANALYSIS` (required): the analysis folder: `request.json` (`station`, `start`, `end`, UTC), `labels.jsonl`, `report.md`, `analysis.md`, `scripts/`.
|
||||
- `OUT` (required): the output folder of this run (`sessions/<analysis>_<YYYYMMDDTHHMMZ>/` in Optimizer).
|
||||
- Defaults, relative to Optimizer's root (its links), which the request may override: `DETECTORS: detectors/`, `TOOLS: tools/`, `DATA: data`, `LABELS: labeling/auto` (the folder of the `<STATION>.jsonl` label files, JSONL: `id`, `start`, `end`, `family`, `description`), `FAMILIES: labeling/families.md` (the families and their boundaries, one detector per family).
|
||||
|
||||
Derived from `<ANALYSIS>/request.json`:
|
||||
|
||||
- `STATION`: `<DATA>/<station>` (read by `DETECTORS/scripts/data.py`);
|
||||
- `FROM`: the day of `start`; `TO`: the day after the day of `end` (UTC days, YYYY-MM-DD, `TO` excluded);
|
||||
- the labels of the range: `<LABELS>/<station>.jsonl` (the analysis's labels are submitted there);
|
||||
- `REFERENCE`, the reference truth for the non-regression: `--labels <LABELS> --data <DATA>`;
|
||||
- `CONTEXT`: `<ANALYSIS>/report.md`.
|
||||
|
||||
`ANALYSIS` or `OUT` missing, no `<ANALYSIS>/request.json`, `DETECTORS` or `TOOLS` not a git repository: say so and stop, changing nothing.
|
||||
|
||||
## Commands
|
||||
|
||||
- events: `python3 <DETECTORS>/scripts/categories.py <STATION> --from <FROM> --to <TO>`
|
||||
- sensors: `python3 <DETECTORS>/bench/sensors.py <STATION> --from <FROM> --to <TO>`
|
||||
- range score (the goal): `python3 <DETECTORS>/bench/score.py <STATION> --labels <LABELS>/<station>.jsonl --from <FROM> --to <TO>`; it exits with an error if no label touches the range: note it in the summary, touch nothing in the detectors and go to part 2.
|
||||
- benchmark (the non-regression, about 3 min): `python3 <DETECTORS>/bench/regression.py <REFERENCE>`
|
||||
|
||||
The measures of a step: `<range score> > <OUT>/range-<step>.txt 2>&1` and `<benchmark> > <OUT>/benchmark-<step>.txt 2>&1`.
|
||||
|
||||
## Git
|
||||
|
||||
The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C <DETECTORS> rev-parse HEAD`. Files already modified (`git -C <DETECTORS> status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. Always commit by path (`git -C <DETECTORS> commit -m "<message>" -- <paths>`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. The tool-builder commits in `TOOLS` itself. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's.
|
||||
|
||||
## Steps
|
||||
|
||||
Each agent is called with the Agent tool, its `subagent_type` (`agents:upgrader`, `agents:validator`, `agents:tool-builder`, from the agents plugin loaded with `--plugin-dir`; not available: say so and stop, changing nothing) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
|
||||
|
||||
### Part 1: the detectors
|
||||
|
||||
1. **Measures before**: step `before`.
|
||||
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION>, range <FROM> → <TO> (UTC). Labels: <LABELS>/<station>.jsonl. Families: <FAMILIES>. Context: <CONTEXT>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
|
||||
It commits each fix it keeps, message `improve: S<n>`, with its benchmark `<OUT>/benchmark-S<n>.txt`.
|
||||
Verdict `nothing to change`, or no fix commit (`git -C <DETECTORS> log --format=%h --grep ': S[0-9]*$' <base>..HEAD` empty): go to part 2.
|
||||
3. **Measures after**: step `after`.
|
||||
4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: <DETECTORS>. Families: <FAMILIES>. Modification: <OUT>/modification.md. Range score: <OUT>/range-before.txt → <OUT>/range-after.txt. Benchmark: <OUT>/benchmark-before.txt → <OUT>/benchmark-after.txt, after each fix: <OUT>/benchmark-S*.txt; command: <…>. Events command for any station: python3 <DETECTORS>/scripts/categories.py <station folder> --from <from> --to <to>. Base: <base>. Write <OUT>/validation.md.`
|
||||
- `keep`: go to part 2.
|
||||
- `remove`: for each fix of the `**Remove**:` line, its commits (`git -C <DETECTORS> log --format=%h --grep ': S<n>$' <base>..HEAD`), from the most recent: `git -C <DETECTORS> revert --no-edit <commit>`, then `git -C <DETECTORS> commit --amend -m "improve: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`. A conflict: `git -C <DETECTORS> revert --abort`, and do as for `cancel`.
|
||||
- `cancel`, or anything else: `git -C <DETECTORS> revert --no-commit <base>..HEAD && git -C <DETECTORS> commit -m "improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"` (only what the fix commits changed; files already modified stay as they are).
|
||||
5. **Measures end**, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed).
|
||||
|
||||
An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to part 2. Do not rerun an agent.
|
||||
|
||||
### Part 2: the tools
|
||||
|
||||
Always, whatever part 1 gave (a stop in part 1 does not stop part 2).
|
||||
|
||||
6. **Tool-builder**: `Analysis: <ANALYSIS>/. Tools: <TOOLS>/. Write <OUT>/tools.md.` It reads the analysis, writes only in `TOOLS` and in `<OUT>`, and commits in `TOOLS`. Its output missing: note it. Do not rerun it.
|
||||
|
||||
## Summary
|
||||
|
||||
`<OUT>/summary.md`:
|
||||
|
||||
```markdown
|
||||
# optimize: <station>, <FROM> → <TO>
|
||||
|
||||
**Result**: modification kept | kept without S<n>… | cancelled | none | error
|
||||
**Tools changed**: <the "Tools changed" line of <OUT>/tools.md> | error
|
||||
|
||||
## The range
|
||||
the range score, before → end, one line per family (found, correct)
|
||||
|
||||
## Elsewhere
|
||||
the `Total` line of each family, before → end; then the other changed lines (station, family), before → end
|
||||
|
||||
## What changed
|
||||
<per fix kept, one sentence from "What changed" of the modification>; the upgrader's verdict, the validator's
|
||||
|
||||
## For the user to decide
|
||||
<copied as they are: "Limits" and "For the user to decide" of the modification, "Reservations" of the validator; none>
|
||||
|
||||
## Not fixed
|
||||
<"Not fixed" of the modification; none>
|
||||
|
||||
## Reading tools
|
||||
<copied as they are: "What I decided" and "What changed" of <OUT>/tools.md, and its "data.py" line if changed; none>
|
||||
|
||||
## Scripts of the agents
|
||||
<copied as they are: "Tools" of the modification and of the validation (the scripts the agents kept in `<OUT>/tools/`); none>
|
||||
|
||||
## Errors
|
||||
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`; `git -C <TOOLS> log --oneline -1` if the tools changed>
|
||||
```
|
||||
|
||||
## Answer
|
||||
|
||||
In 5 lines at most: the `**Result**` line, the range score before → end (changed families), the `Total` before → end if it changed, the `**Tools changed**` line, the path of the summary.
|
||||
@@ -2,6 +2,7 @@ data
|
||||
labeling
|
||||
detectors
|
||||
agents
|
||||
tools
|
||||
.venv/
|
||||
__pycache__/
|
||||
archive/batches/*.tmp/
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# CLAUDE.md
|
||||
|
||||
Optimizer: the optimisation of the AIMsight detectors on the labels of `labeling/auto`, and nothing else (`README.md`). The workflow: `.claude/skills/improve/SKILL.md` (`/improve`), launched from the root of Optimizer with the agents plugin (`--plugin-dir agents`); its agents `agents:upgrader` and `agents:validator`; the benchmark `detectors/bench/`.
|
||||
Optimizer: the optimisation of the AIMsight detectors and of the reading tools, from one analysis of `labeling/auto`, and nothing else (`README.md`). The workflow: `.claude/skills/optimize/SKILL.md` (`/optimize`), launched from the root of Optimizer with the agents plugin (`--plugin-dir agents`); its agents `agents:upgrader`, `agents:validator` (the detectors) and `agents:tool-builder` (the tools); the benchmark `detectors/bench/`.
|
||||
|
||||
- **The links**, out of git: `detectors/` → `~/detectors` (the session commits there); `data/` → `~/aimsight-data/data` and `labeling/` → `~/labeling` (read-only); `agents/` → `~/agents` (the plugin, loaded read-only).
|
||||
- **The links**, out of git: `detectors/` → `~/detectors` and `tools/` → `~/tools` (the session commits there); `data/` → `~/aimsight-data/data` and `labeling/` → `~/labeling` (read-only); `agents/` → `~/agents` (the plugin, loaded read-only).
|
||||
- **The sessions**: `sessions/<analysis>_<YYYYMMDDTHHMMZ>/`; Diagnostic-App reads their summary (`summary.md`).
|
||||
- **The workflow stays generic**: the skill describes only the workflow (range + labels → upgrader → validator → summary); what is specific to a caller (its paths, its git rules, `Caller's context`) arrives in the request.
|
||||
- **Check a change of the skill**: `env -u ANTHROPIC_API_KEY claude -p "/improve" --plugin-dir agents --model haiku` must say the inputs are missing and stop without touching `detectors/`; then a real session through the app's button.
|
||||
- **The workflow stays generic**: the skill describes only the workflow (analysis → detectors: upgrader → validator; tools: tool-builder → summary); what is specific to a caller (its paths, its git rules, `Caller's context`) arrives in the request.
|
||||
- **Check a change of the skill**: `env -u ANTHROPIC_API_KEY claude -p "/optimize" --plugin-dir agents --model haiku` must say the inputs are missing and stop without touching `detectors/` or `tools/`; then a real session through the app's button.
|
||||
- `archive/`: the former batch and its scanner, stopped; do not run them again.
|
||||
|
||||
Rules:
|
||||
|
||||
@@ -1,20 +1,20 @@
|
||||
# Optimizer
|
||||
|
||||
The optimisation of the AIMsight detectors, on the labels of `labeling/auto`, and nothing else. On the range of an in-depth analysis (made in the Diagnostic-App app), its labels as the truth, the `improve` skill:
|
||||
The optimisation of the AIMsight detectors and of the reading tools, from one analysis, and nothing else. An analysis (made in the Diagnostic-App app by the analyst) leaves in its folder the two things needed: its labels, the truth on its range, and its own scripts with what it missed. On that analysis, the `optimize` skill:
|
||||
|
||||
1. measures the detectors' score on the range, and the benchmark on every label of `labeling/auto`;
|
||||
2. has the detectors fixed by the `upgrader`, one fix per commit in `~/detectors`;
|
||||
3. has them judged by the `validator`, which keeps, has some removed, or cancels: no station goes back, no overfitting;
|
||||
4. writes the summary of the session.
|
||||
1. **the detectors**: measures their score on the range, and the benchmark on every label of `labeling/auto`; has them fixed by the `upgrader`, one fix per commit in `~/detectors`; has them judged by the `validator`, which keeps, has some removed, or cancels: no station goes back, no overfitting;
|
||||
2. **the tools**: has the `tool-builder` bring the analyst's useful scripts into the reading harness, commits in `~/tools`; always, whatever the detectors gave;
|
||||
3. writes the summary of the session.
|
||||
|
||||
The `upgrader` and the `validator` come from the `agents` repository; the benchmark (`bench/`) from the `detectors` repository. Nothing is pushed.
|
||||
The `upgrader`, the `validator` and the `tool-builder` come from the `agents` repository; the benchmark (`bench/`) from the `detectors` repository. Nothing is pushed.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
ln -s ~/detectors detectors # the detectors, shared with Diagnostic-App
|
||||
ln -s ~/tools tools # the reading tools, shared with Diagnostic-App
|
||||
ln -s ~/aimsight-data/data data # the pumped data, out of git
|
||||
ln -s ~/labeling labeling # the labels: the truth and the families
|
||||
ln -s ~/labeling labeling # the labels, the analyses and the families
|
||||
ln -s ~/agents agents # the generic agents (Claude Code plugin)
|
||||
```
|
||||
|
||||
@@ -22,21 +22,19 @@ ln -s ~/agents agents # the generic agents (Claude Code p
|
||||
|
||||
## Run
|
||||
|
||||
From the "Optimize the detectors" button of Diagnostic-App, on a finished analysis; or by hand, from the root of Optimizer:
|
||||
From the "Optimize" button of Diagnostic-App, on a finished analysis; or by hand, from the root of Optimizer:
|
||||
|
||||
```bash
|
||||
env -u ANTHROPIC_API_KEY claude -p "/improve DETECTORS: detectors/. STATION: data/NORDEND. FROM: 2026-11-04. TO: 2026-11-08. \
|
||||
LABELS: labeling/auto/NORDEND.jsonl. REFERENCE: --labels labeling/auto --data data. FAMILIES: labeling/families.md. \
|
||||
CONTEXT: labeling/auto/analyses/<analysis>/report.md. OUT: sessions/<analysis>_<YYYYMMDDTHHMMZ>/." \
|
||||
--plugin-dir agents --add-dir ~/detectors --add-dir ~/labeling --add-dir ~/aimsight-data/data \
|
||||
env -u ANTHROPIC_API_KEY claude -p "/optimize ANALYSIS: labeling/auto/analyses/<analysis>/. OUT: sessions/<analysis>_<YYYYMMDDTHHMMZ>/." \
|
||||
--plugin-dir agents --add-dir ~/detectors --add-dir ~/tools --add-dir ~/labeling --add-dir ~/aimsight-data/data \
|
||||
--setting-sources project,local --strict-mcp-config \
|
||||
--allowedTools Read,Write,Edit,Bash,Agent --permission-mode acceptEdits
|
||||
```
|
||||
|
||||
`TO` is excluded. Several stations on the same range: `DATA: data. STATIONS: A,B. LABELS: <folder of <STATION>.jsonl>` instead of `STATION` and a labels file.
|
||||
The station and the range come from the analysis's `request.json` (the range's last day included); the labels from `labeling/auto/<STATION>.jsonl`; the context from its `report.md`. `DETECTORS`, `TOOLS`, `DATA`, `LABELS`, `FAMILIES` can be overridden in the request.
|
||||
|
||||
## What a session leaves
|
||||
|
||||
`sessions/<analysis>_<time>/`: the measures before and after (`range-*.txt`, `benchmark-*.txt`), `modification.md` (the upgrader), `validation.md` (the validator), `summary.md` (the summary, shown by the app under the analysis report), `tools/` (the scripts the agents wrote). The fixes: `improve: S<n>` commits in `~/detectors`, on its current branch.
|
||||
`sessions/<analysis>_<time>/`: the measures before and after (`range-*.txt`, `benchmark-*.txt`), `modification.md` (the upgrader), `validation.md` (the validator), `tools.md` (the tool-builder) and its trials (`work/`), `summary.md` (the summary, shown by the app under the analysis report), `tools/` (the scripts the upgrader and validator wrote). The fixes: `improve: S<n>` commits in `~/detectors`, the tools a `Tools: <analysis>` commit in `~/tools`, each on its current branch.
|
||||
|
||||
`archive/`: the former daily batch and its scanner (auto-labeling by the detectors), stopped on 2026-11-16. The architecture of the repositories: `~/Diagnostic-App/docs/architecture.drawio`.
|
||||
@@ -0,0 +1,88 @@
|
||||
# `/optimize` Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Optimizer's `/optimize ANALYSIS: <folder>` optimises the detectors and the reading tools from one analysis; Diagnostic-App calls it once.
|
||||
|
||||
**Architecture:** `/improve` is moved to `/optimize`, derives its inputs from the analysis folder, keeps its detectors steps, then runs `agents:tool-builder` as a subagent and writes one summary. The tool-builder writes only in the session folder. Diagnostic-App's optimisation becomes a single `claude -p` turn in Optimizer.
|
||||
|
||||
**Tech Stack:** Claude Code skills and agents (Markdown), Python 3 (Diagnostic-App server and its `check.py`).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-11-16-optimize-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- English everywhere; every time UTC.
|
||||
- Commit by path, never `git add -A`; each repository on its current branch (Optimizer `master`, the others `main`); nothing pushed.
|
||||
- Commit messages end with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
|
||||
- Optimizer is read-only on `data/` and `labeling/`; the agents never run on an API key.
|
||||
- Detector fix commits stay `improve: S<n>`.
|
||||
|
||||
## Review Focus
|
||||
|
||||
- `/optimize` without inputs, or with an analysis folder lacking `request.json`: says what is missing, stops, touches neither `detectors/` nor `tools/` (Task 2, Step 4).
|
||||
- An analysis whose range carries no label (`score.py` exits with an error): the detectors part stops, the tools part still runs (Task 2 skill text, Step 4 review).
|
||||
- An analysis that ends at midnight: `TO` = the day after `end`, as Diagnostic-App does today (Task 2 skill text).
|
||||
- The tool-builder never writes in the analysis folder (Task 1).
|
||||
- Diagnostic-App with Optimizer lacking the `tools` link: a clear refusal (Task 3, Step 1).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: the tool-builder writes in its report's folder (agents)
|
||||
|
||||
**Files:**
|
||||
- Modify: `~/agents/agents/tool-builder.md` (lines 3, 44-45, 65)
|
||||
- Modify: `~/agents/agents/upgrader.md:3,8`, `~/agents/agents/validator.md:3,8`, `~/agents/README.md:8-10`
|
||||
|
||||
- [ ] **Step 1:** In `tool-builder.md`, Check step 1: `--out <the folder of your report>/work/tools/` instead of `<analysis>/work/tools/`; Check step 2: rerun the report's commands with their `--out` replaced by `<the folder of your report>/work/`; add "You write nothing in the analysis folder." to "What you touch"; report template "figures in <the folder of your report>/work/tools/". Description: launched by Optimizer's `/optimize` (Agent tool, `agents:tool-builder`).
|
||||
- [ ] **Step 2:** "improve workflow" → "optimize workflow" in `upgrader.md`, `validator.md`; README table: requests "that of the `optimize` skill", the tool-builder's caller "Optimizer (`/optimize`)".
|
||||
- [ ] **Step 3:** `grep -n "work/tools\|improve workflow\|improve\` skill" ~/agents -r` returns only `<the folder of your report>` lines.
|
||||
- [ ] **Step 4:** Commit in `~/agents`: `Optimize: the tool-builder writes in its report's folder; run by Optimizer's /optimize`.
|
||||
|
||||
### Task 2: the `/optimize` skill (Optimizer)
|
||||
|
||||
**Files:**
|
||||
- Move: `.claude/skills/improve/SKILL.md` → `.claude/skills/optimize/SKILL.md` (`git mv`)
|
||||
- Create: link `tools -> ~/tools` (ignored or untracked, like the others)
|
||||
- Modify: `README.md`, `CLAUDE.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: request `/optimize ANALYSIS: <analysis folder>/. OUT: sessions/<analysis>_<YYYYMMDDTHHMMZ>/. Caller's context: …`; optional overrides `DETECTORS`, `TOOLS`, `DATA`, `LABELS`, `FAMILIES`; output `<OUT>/summary.md` (first line `# optimize: <station>, <FROM> → <TO>`, lines `**Result**:` and `**Tools changed**:`), `<OUT>/tools.md`.
|
||||
|
||||
- [ ] **Step 1:** Skill frontmatter `name: optimize`, description: the workflow that, from one analysis, improves the detectors (upgrader, validator) and the reading tools (tool-builder).
|
||||
- [ ] **Step 2:** Inputs section per the spec: `ANALYSIS`, `OUT` required; defaults `DETECTORS: detectors/`, `TOOLS: tools/`, `DATA: data`, `LABELS: labeling/auto`, `FAMILIES: labeling/families.md`; derived `STATION`, `FROM`, `TO` (day after `end`), labels file, reference, `CONTEXT`. Missing `ANALYSIS`/`OUT`, no `request.json`, `DETECTORS` or `TOOLS` not a git repository: say so, stop, change nothing. Remove the `DATA`+`STATIONS` mode and its commands.
|
||||
- [ ] **Step 3:** Steps: "Part 1, detectors" = the current steps (unchanged; a stop in part 1 — no label on the range, an agent missing — goes to part 2, not to the end); "Part 2, tools", always: Agent tool `agents:tool-builder`, request `Analysis: <ANALYSIS>/. Tools: <TOOLS>/. Write <OUT>/tools.md.`, plus the caller's context; its output missing: note it. Summary: title `# optimize: …`, the detectors part as now, `**Tools changed**:` under `**Result**:`, section `## Reading tools` (copied: "Tools changed", "What I decided" of `<OUT>/tools.md`), `## Tools` renamed `## Scripts of the agents`. Answer: add the `**Tools changed**` line.
|
||||
- [ ] **Step 4:** Run `env -u ANTHROPIC_API_KEY claude -p "/optimize" --plugin-dir agents --model haiku`, then the same with `"/optimize ANALYSIS: labeling/auto/analyses/nothing/. OUT: /tmp/x/"`. Expected: both say what is missing and stop; `git -C detectors status --short` and `git -C tools status --short` empty.
|
||||
- [ ] **Step 5:** `ln -s ~/tools tools`; check `git status` does not show `tools` as a new file to commit (add it to `.gitignore` if the other links are there).
|
||||
- [ ] **Step 6:** `README.md`: scope "the detectors and the tools, from an analysis"; the steps; the `tools` link; the by-hand command `/optimize ANALYSIS: labeling/auto/analyses/<analysis>/. OUT: sessions/<analysis>_<YYYYMMDDTHHMMZ>/.` with `--add-dir ~/tools`; what a session leaves (`tools.md`, `work/`). `CLAUDE.md`: the same scope, link and check command.
|
||||
- [ ] **Step 7:** Commit in Optimizer by path: the moved skill, `README.md`, `CLAUDE.md`, `.gitignore` if changed, the spec and this plan: `Optimize: /optimize, the detectors and the tools from one analysis (ex /improve)`.
|
||||
|
||||
### Task 3: Diagnostic-App calls `/optimize` once
|
||||
|
||||
**Files:**
|
||||
- Modify: `~/Diagnostic-App/app/server/analysis.py` (docstring 1-11, `IMPROVE` 29-33, `optimize` 262-268, `optimize_thread` 275-297)
|
||||
- Test: `~/Diagnostic-App/app/server/check.py` (`check_optimization`, 135-166)
|
||||
- Modify: `~/Diagnostic-App/CLAUDE.md:5,8`, `README.md:6,25`, `app/README.md:24,41`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Task 2's request.
|
||||
|
||||
- [ ] **Step 1: the failing check.** In `check_optimization`: create `skills/optimize/SKILL.md` instead of `improve`; the links loop `("data", "labeling", "detectors", "tools")` with a refusal asserted before `tools` exists; after the run `assert len(launched) == 1`; `request.startswith("/optimize ANALYSIS: labeling/auto/analyses/NORDEND_a/. OUT: sessions/NORDEND_a_")`, `k["cwd"] == analysis.OPTIMIZER.resolve()`, `"--plugin-dir" in args`; remove the `launched[1]` tool-builder assertions; keep the old `<analysis>/tools.md` summary assertion.
|
||||
- [ ] **Step 2:** `cd ~/Diagnostic-App && .venv/bin/python app/server/check.py` → FAIL in `check_optimization`.
|
||||
- [ ] **Step 3:** `IMPROVE` → `OPTIMIZE = "/optimize ANALYSIS: labeling/auto/analyses/{name}/. OUT: sessions/{name}_{time}/. Caller's context: Diagnostic-app, through Optimizer; read-only on data/ and labeling/; commit only in detectors/ and tools/, on their current branch; do not commit OUT."`; `optimize()` checks `skills/optimize/SKILL.md` and the `tools` link; `optimize_thread` one turn in `OPTIMIZER.resolve()` with `--plugin-dir AGENTS.resolve()`, `--add-dir` detectors (station.links() already adds tools), `--allowedTools Read,Write,Edit,Bash,Agent`; the `request.json` read and `TO` computation removed (Optimizer derives them); docstrings say `/optimize`.
|
||||
- [ ] **Step 4:** `.venv/bin/python app/server/check.py` → passes.
|
||||
- [ ] **Step 5:** Docs: "Optimize the detectors" launches `/optimize` in `optimizer/` on the analysis (detectors committed in `detectors/`, tools in `tools/`); contract "the request of `/optimize`"; `app/README.md` the single launch; summary path.
|
||||
- [ ] **Step 6:** Commit in Diagnostic-App by path: `Optimize: a single /optimize call in Optimizer, the detectors and the tools`.
|
||||
|
||||
### Task 4: wording in detectors and tools
|
||||
|
||||
**Files:** `~/detectors/CLAUDE.md:3,12`, `~/detectors/README.md` ("Improving them"), `~/tools/README.md` ("Who changes it")
|
||||
|
||||
- [ ] **Step 1:** `improve` skill → `optimize` skill (detectors); the tool-builder "run by Optimizer's `/optimize`, after each analysis optimised" (tools). History references (`wf-improve`, `improve: S<n>`) unchanged.
|
||||
- [ ] **Step 2:** Commit each repository by path: `Optimize: the skill is /optimize`.
|
||||
|
||||
### Task 5: final check
|
||||
|
||||
- [ ] `grep -rn "/improve\|improve\` skill\|skills/improve" ~/Optimizer --include=*.md -l | grep -v "archive/\|sessions/\|docs/superpowers"` and the same on Diagnostic-App (outside `docs/superpowers`, `node_modules`), agents, detectors, tools: nothing left.
|
||||
- [ ] Diagnostic-App `check.py` passes; the Task 2 haiku runs stop cleanly.
|
||||
- [ ] Left to the user: a real session through the app's button (it commits in `detectors` and `tools`).
|
||||
@@ -0,0 +1,90 @@
|
||||
# `/optimize`: one analysis in, detectors and tools optimised
|
||||
|
||||
Date: 2026-11-16. Size: a small change, across five repositories.
|
||||
|
||||
## Intent
|
||||
|
||||
Everything that optimises lives in Optimizer. Today Optimizer optimises the detectors (`/improve`), while the
|
||||
optimisation of the reading tools is a bare `claude -p --agent agents:tool-builder` call written in Diagnostic-App's
|
||||
server. An analysis run by the analyst already leaves, in its folder, the two things both optimisations need: its labels
|
||||
(`labels.jsonl`, the truth for the detectors) and its own scripts plus what it missed (`scripts/`, `analysis.md`, the
|
||||
material for the tools). So the workflow takes **one analysis** and optimises **both**.
|
||||
|
||||
## Goals
|
||||
|
||||
- One skill, `/optimize`, in Optimizer: input an analysis folder, output one session folder and one summary.
|
||||
- Diagnostic-App's "Optimize" button makes a single call, to `/optimize`.
|
||||
- Optimizer's scope becomes "the detectors and the tools, from an analysis".
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No change to the analyst: it already writes its scripts in `<output folder>/scripts/` and says what it missed.
|
||||
- No validator for the tools; the tool-builder stays as it is in its decisions.
|
||||
- No toolbox for the scripts of the upgrader and validator (`sessions/*/tools/`): later.
|
||||
- No change to the benchmark, the agents' judgement, or the labels.
|
||||
|
||||
## Anti-goals
|
||||
|
||||
- Optimizer writing in `labeling/` (read-only): the tool-builder's outputs move to the session folder.
|
||||
- One step's failure blocking the other.
|
||||
|
||||
## Approaches
|
||||
|
||||
1. **Extend `/improve` into `/optimize ANALYSIS: <folder>`** (picked). The skill derives its inputs from the analysis,
|
||||
runs the current detectors steps, then the tool-builder, then writes one summary. One file, one session, one line in
|
||||
Diagnostic-App. `/improve` has a single caller, so nothing needs it alone.
|
||||
2. Keep `/improve`, add `/optimize` calling it then the tool-builder. Nothing existing moves, but two layers and two
|
||||
summaries to merge, for no caller that wants the detectors alone.
|
||||
3. Two skills called in turn by Diagnostic-App. Smallest diff, but the process (the order, the session) stays in
|
||||
Diagnostic-App, against the intent.
|
||||
|
||||
## Design
|
||||
|
||||
**Skill** `.claude/skills/optimize/SKILL.md` (moved from `improve/`). Inputs:
|
||||
|
||||
- `ANALYSIS` (required): the analysis folder (`request.json`, `labels.jsonl`, `report.md`, `analysis.md`, `scripts/`).
|
||||
- `OUT` (required): the session folder, `sessions/<analysis>_<YYYYMMDDTHHMMZ>/`.
|
||||
- Defaults, relative to Optimizer's links, overridable in the request: `DETECTORS: detectors/`, `TOOLS: tools/`,
|
||||
`DATA: data`, `LABELS: labeling/auto` (the folder of `<STATION>.jsonl`), `FAMILIES: labeling/families.md`.
|
||||
- Derived: `STATION` = `<DATA>/<request.json station>`; `FROM` = the day of `start`; `TO` = the day after `end`
|
||||
(excluded); labels = `<LABELS>/<station>.jsonl`; reference = `--labels <LABELS> --data <DATA>`;
|
||||
context = `<ANALYSIS>/report.md`.
|
||||
- `Caller's context`, passed on as today.
|
||||
|
||||
The multi-station mode (`DATA` + `STATIONS`) is dropped: an analysis is one station and no caller uses it (my call).
|
||||
|
||||
**Steps.**
|
||||
1. Detectors: the current steps of `/improve`, unchanged (measures, upgrader, measures, validator, remove/cancel).
|
||||
Commits stay `improve: S<n>` in `DETECTORS` (the agents grep it).
|
||||
2. Tools, always, whatever step 1 gave: the Agent tool, `subagent_type: agents:tool-builder`, request
|
||||
`Analysis: <ANALYSIS>/. Tools: <TOOLS>/. Write <OUT>/tools.md.` It commits in `TOOLS` itself.
|
||||
3. Summary `<OUT>/summary.md`: the current detectors part, then a `## Reading tools` part (the tool-builder's
|
||||
"Tools changed" line and its "What I decided", copied). The current `## Tools` section (the agents' scripts) is renamed
|
||||
`## Scripts of the agents`.
|
||||
|
||||
**agents repository.** `tool-builder.md`: its trial figures and the reruns of the report's commands write in
|
||||
`<the folder of its report>/work/`, never in the analysis folder. The "improve workflow" wording of `upgrader.md`,
|
||||
`validator.md` and `README.md` becomes "optimize workflow"; the tool-builder's caller becomes Optimizer.
|
||||
|
||||
**Optimizer.** Link `tools -> ~/tools`; the launch adds `--add-dir ~/tools`. `README.md` and `CLAUDE.md`: scope "the
|
||||
detectors and the tools", the `/optimize` request, the check command.
|
||||
|
||||
**Diagnostic-App.** `optimize_thread`: one turn, `/optimize ANALYSIS: labeling/auto/analyses/<name>/. OUT: sessions/
|
||||
<name>_<time>/. Caller's context: …`, from Optimizer, with `--add-dir` on detectors and tools. `optimize()` also requires
|
||||
the `tools` link and `optimize/SKILL.md`. `summary()` unchanged: it shows the session's `summary.md`, and the old
|
||||
`<analysis>/tools.md` if one exists. `check.py`: one launch, `/optimize` in Optimizer. Docs (`CLAUDE.md`, `README.md`,
|
||||
`app/README.md`) follow.
|
||||
|
||||
**detectors, tools.** The wording `improve` skill → `optimize` skill in `detectors/CLAUDE.md`, `detectors/README.md`;
|
||||
`tools/README.md` "Who changes it": the tool-builder, run by Optimizer's `/optimize`.
|
||||
|
||||
## Check
|
||||
|
||||
- `env -u ANTHROPIC_API_KEY claude -p "/optimize" --plugin-dir agents --model haiku`: says the inputs are missing, stops,
|
||||
touches neither `detectors/` nor `tools/`.
|
||||
- Diagnostic-App's `check.py` passes.
|
||||
- A real session through the app's button, on an analysis with scripts.
|
||||
|
||||
## Left to the builder
|
||||
|
||||
The exact wording of the skill and of the summary's tools part; the order of the doc edits.
|
||||
Reference in new issue
Block a user