Compare commits
6
Commits
63bebbc415
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e52479adf8 | ||
|
|
1200996254 | ||
|
|
c44b9286c9 | ||
|
|
6596b68721 | ||
|
|
459d505705 | ||
|
|
68768dfd39 |
No files matched your search
@@ -1,89 +0,0 @@
|
|||||||
---
|
|
||||||
name: improve
|
|
||||||
description: Workflow that improves the detectors on a range of data where they perform poorly, given labels as the truth on that range — the upgrader fixes, the validator judges; the result, detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged. The caller gives the range, the labels, the reference truth, the output folder and any specific context. Run from Optimizer's root with the agents plugin (--plugin-dir <agents>). /improve <request>.
|
|
||||||
argument-hint: "<range, labels, reference, output folder, context>"
|
|
||||||
---
|
|
||||||
|
|
||||||
# improve: improve the detectors on a range
|
|
||||||
|
|
||||||
You are the orchestrator. The detectors (the detectors repository, `DETECTORS`: `scripts/`, `thresholds.md`) perform poorly on a range of data of a station, or of several stations on the same range; labels give the truth on this range. You run the upgrader, which fixes the detectors, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
|
||||||
|
|
||||||
## Inputs
|
|
||||||
|
|
||||||
Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope.
|
|
||||||
|
|
||||||
- `DETECTORS`: the path of the detectors repository (default: `detectors/`, a link in Optimizer).
|
|
||||||
- `STATION`: the station's data folder (read by `DETECTORS/scripts/data.py`); or, several stations on the same range: `DATA`, the folder of the station folders, and `STATIONS`, `A,B`, the stations to improve.
|
|
||||||
- `FROM`, `TO`: the range, UTC days (YYYY-MM-DD, `TO` excluded).
|
|
||||||
- `LABELS`: the station's labels file (JSONL: `id`, `start`, `end`, `family`, `description`), the truth on the range; with `STATIONS`, a folder of `<STATION>.jsonl`, one per station.
|
|
||||||
- `REFERENCE`: the reference truth for the non-regression, `--labels <folder of <STATION>.jsonl> --data <folder of <STATION>/>`.
|
|
||||||
- `FAMILIES`: the document defining the families and their boundaries (one detector per family).
|
|
||||||
- `CONTEXT` (optional): documents explaining the range's labels (an analyst's report, for example).
|
|
||||||
- `OUT`: the output folder of this run (default: `sessions/<analysis>_<YYYYMMDDTHHMMZ>/` in Optimizer).
|
|
||||||
|
|
||||||
A required input missing, or `DETECTORS` not a git repository: say so and stop, changing nothing.
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
|
|
||||||
- events: `python3 <DETECTORS>/scripts/categories.py <STATION> --from <FROM> --to <TO>`
|
|
||||||
- sensors: `python3 <DETECTORS>/bench/sensors.py <STATION> --from <FROM> --to <TO>`
|
|
||||||
- range score (the goal): `python3 <DETECTORS>/bench/score.py <STATION> --labels <LABELS> --from <FROM> --to <TO>`; it exits with an error if no label touches the range: note it in the summary and stop without touching the detectors.
|
|
||||||
- with `STATIONS`: the events and sensors commands once per station, `<DATA>/<S>` for each `S` of `STATIONS`; the range score, all of them at once: `python3 <DETECTORS>/bench/regression.py --labels <LABELS> --data <DATA> --stations <STATIONS> --from <FROM> --to <TO>`.
|
|
||||||
- benchmark (the non-regression, about 3 min): `python3 <DETECTORS>/bench/regression.py <REFERENCE>`
|
|
||||||
|
|
||||||
The measures of a step: `<range score> > <OUT>/range-<step>.txt 2>&1` and `<benchmark> > <OUT>/benchmark-<step>.txt 2>&1`.
|
|
||||||
|
|
||||||
## Git
|
|
||||||
|
|
||||||
The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C <DETECTORS> rev-parse HEAD`. Files already modified (`git -C <DETECTORS> status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. Always commit by path (`git -C <DETECTORS> commit -m "<message>" -- <paths>`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's.
|
|
||||||
|
|
||||||
## Steps
|
|
||||||
|
|
||||||
Each agent is called with the Agent tool, its `subagent_type` (`agents:upgrader`, `agents:validator`, from the agents plugin loaded with `--plugin-dir`; not available: say so and stop, changing nothing) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
|
|
||||||
|
|
||||||
1. **Measures before**: step `before`.
|
|
||||||
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION | <DATA>/<S> for each S of STATIONS>, range <FROM> → <TO> (UTC). Labels: <LABELS>. Families: <FAMILIES>. Context: <CONTEXT | none>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
|
|
||||||
It commits each fix it keeps, message `improve: S<n>`, with its benchmark `<OUT>/benchmark-S<n>.txt`.
|
|
||||||
Verdict `nothing to change`, or no fix commit (`git -C <DETECTORS> log --format=%h --grep ': S[0-9]*$' <base>..HEAD` empty): go to the summary.
|
|
||||||
3. **Measures after**: step `after`.
|
|
||||||
4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: <DETECTORS>. Families: <FAMILIES>. Modification: <OUT>/modification.md. Range score: <OUT>/range-before.txt → <OUT>/range-after.txt. Benchmark: <OUT>/benchmark-before.txt → <OUT>/benchmark-after.txt, after each fix: <OUT>/benchmark-S*.txt; command: <…>. Events command for any station: python3 <DETECTORS>/scripts/categories.py <station folder> --from <from> --to <to>. Base: <base>. Write <OUT>/validation.md.`
|
|
||||||
- `keep`: go to the summary.
|
|
||||||
- `remove`: for each fix of the `**Remove**:` line, its commits (`git -C <DETECTORS> log --format=%h --grep ': S<n>$' <base>..HEAD`), from the most recent: `git -C <DETECTORS> revert --no-edit <commit>`, then `git -C <DETECTORS> commit --amend -m "improve: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`. A conflict: `git -C <DETECTORS> revert --abort`, and do as for `cancel`.
|
|
||||||
- `cancel`, or anything else: `git -C <DETECTORS> revert --no-commit <base>..HEAD && git -C <DETECTORS> commit -m "improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"` (only what the fix commits changed; files already modified stay as they are).
|
|
||||||
|
|
||||||
An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to the summary. Do not rerun an agent.
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Measures, step `end`, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed). Then `<OUT>/summary.md`:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# improve: <station | STATIONS>, <FROM> → <TO>
|
|
||||||
|
|
||||||
**Result**: modification kept | kept without S<n>… | cancelled | none | error
|
|
||||||
|
|
||||||
## The range
|
|
||||||
the range score, before → end, one line per family (found, correct)
|
|
||||||
|
|
||||||
## Elsewhere
|
|
||||||
the `Total` line of each family, before → end; then the other changed lines (station, family), before → end
|
|
||||||
|
|
||||||
## What changed
|
|
||||||
<per fix kept, one sentence from "What changed" of the modification>; the upgrader's verdict, the validator's
|
|
||||||
|
|
||||||
## For the user to decide
|
|
||||||
<copied as they are: "Limits" and "For the user to decide" of the modification, "Reservations" of the validator; none>
|
|
||||||
|
|
||||||
## Not fixed
|
|
||||||
<"Not fixed" of the modification; none>
|
|
||||||
|
|
||||||
## Tools
|
|
||||||
<copied as they are: "Tools" of the modification and of the validation (the scripts the agents kept in `<OUT>/tools/`); none>
|
|
||||||
|
|
||||||
## Errors
|
|
||||||
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`>
|
|
||||||
```
|
|
||||||
|
|
||||||
## Answer
|
|
||||||
|
|
||||||
In 4 lines at most: the `**Result**` line, the range score before → end (changed families), the `Total` before → end if it changed, the path of the summary.
|
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
---
|
||||||
|
name: optimize
|
||||||
|
description: Workflow that, from one analysis (its labels as the truth on its range, its scripts and what it missed), optimises the detectors — the upgrader fixes, the validator judges; detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged — then the reading tools — the upgrader brings the analyst's useful scripts into them. Each agent gets its instructions from `instructions/`. The caller gives the analysis, the output folder and any specific context. Run from Optimizer's root with the agents plugin (--plugin-dir <agents>). /optimize <request>.
|
||||||
|
argument-hint: "ANALYSIS: <analysis folder>. OUT: <output folder>. [context]"
|
||||||
|
---
|
||||||
|
|
||||||
|
# optimize: the detectors and the tools, from one analysis
|
||||||
|
|
||||||
|
You are the orchestrator. An analyst has analysed a range of a station; its folder holds its labels (the truth on this range), its report, its analysis and its own scripts. Two parts follow, in this order. **Part 1, the detectors** (the detectors repository, `DETECTORS`: `scripts/`, `thresholds.md`): you run the upgrader, which fixes them, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **Part 2, the tools** (the reading harness, `TOOLS`): you run the upgrader on them, which brings what the analyst needed into the tools. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope.
|
||||||
|
|
||||||
|
- `ANALYSIS` (required): the analysis folder: `request.json` (`station`, `start`, `end`, UTC), `labels.jsonl`, `report.md`, `analysis.md`, `scripts/`.
|
||||||
|
- `OUT` (required): the output folder of this run (`sessions/<analysis>_<YYYYMMDDTHHMMZ>/` in Optimizer).
|
||||||
|
- Defaults, relative to Optimizer's root (its links), which the request may override: `DETECTORS: detectors/`, `TOOLS: tools/`, `DATA: data`, `LABELS: labeling/auto` (the folder of the `<STATION>.jsonl` label files, JSONL: `id`, `start`, `end`, `family`, `description`), `FAMILIES: labeling/families.md` (the families and their boundaries, one detector per family).
|
||||||
|
|
||||||
|
Derived from `<ANALYSIS>/request.json`:
|
||||||
|
|
||||||
|
- `STATION`: `<DATA>/<station>` (read by `DETECTORS/scripts/data.py`);
|
||||||
|
- `FROM`: the day of `start`; `TO`: the day after the day of `end` (UTC days, YYYY-MM-DD, `TO` excluded);
|
||||||
|
- the labels of the range: `<LABELS>/<station>.jsonl` (the analysis's labels are submitted there);
|
||||||
|
- `REFERENCE`, the reference truth for the non-regression: `--labels <LABELS> --data <DATA>`;
|
||||||
|
- `CONTEXT`: `<ANALYSIS>/report.md`.
|
||||||
|
|
||||||
|
`ANALYSIS` or `OUT` missing, no `<ANALYSIS>/request.json`, `DETECTORS` or `TOOLS` not a git repository: say so and stop, changing nothing.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
- events: `python3 <DETECTORS>/scripts/categories.py <STATION> --from <FROM> --to <TO>`
|
||||||
|
- sensors: `python3 <DETECTORS>/bench/sensors.py <STATION> --from <FROM> --to <TO>`
|
||||||
|
- range score (the goal): `python3 <DETECTORS>/bench/score.py <STATION> --labels <LABELS>/<station>.jsonl --from <FROM> --to <TO>`; it exits with an error if no label touches the range: note it in the summary, touch nothing in the detectors and go to part 2.
|
||||||
|
- benchmark (the non-regression, about 3 min): `python3 <DETECTORS>/bench/regression.py <REFERENCE>`
|
||||||
|
|
||||||
|
The measures of a step: `<range score> > <OUT>/range-<step>.txt 2>&1` and `<benchmark> > <OUT>/benchmark-<step>.txt 2>&1`.
|
||||||
|
|
||||||
|
## Git
|
||||||
|
|
||||||
|
The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C <DETECTORS> rev-parse HEAD`. Files already modified (`git -C <DETECTORS> status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. The same for `TOOLS` before part 2: its **tools base** (`git -C <TOOLS> rev-parse HEAD`) and its files already modified. Always commit by path (`git -C <DETECTORS> commit -m "<message>" -- <paths>`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. The upgrader commits in `TOOLS` itself in part 2, one commit per fix. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
Each agent is called with the Agent tool, its `subagent_type` (`agents:upgrader`, `agents:validator`, from the agents plugin loaded with `--plugin-dir`; not available: say so and stop, changing nothing) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
|
||||||
|
|
||||||
|
### Part 1: the detectors
|
||||||
|
|
||||||
|
1. **Measures before**: step `before`.
|
||||||
|
2. **Upgrader**, a single pass: `Target: <DETECTORS>. Evidence: station <STATION>, range <FROM> → <TO> (UTC); labels <LABELS>/<station>.jsonl; families <FAMILIES>; context <CONTEXT>. Checks: events: <events> (before: run it); sensors: <sensors> (before: run it); range: <range score> (before: <OUT>/range-before.txt); benchmark: <benchmark> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md. INSTRUCTIONS: instructions/upgrader-detectors.md.`
|
||||||
|
It commits each fix it keeps, message `improve: S<n>`, with its benchmark `<OUT>/benchmark-S<n>.txt`.
|
||||||
|
Verdict `nothing to change`, or no fix commit (`git -C <DETECTORS> log --format=%h --grep ': S[0-9]*$' <base>..HEAD` empty): go to part 2.
|
||||||
|
3. **Measures after**: step `after`.
|
||||||
|
4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Target: <DETECTORS>. Modification: <OUT>/modification.md. Base: <base>. Checks: range: <range score>; before: <OUT>/range-before.txt; after: <OUT>/range-after.txt; benchmark: <benchmark>; before: <OUT>/benchmark-before.txt; after: <OUT>/benchmark-after.txt; after each fix: <OUT>/benchmark-S*.txt; events, for any station: python3 <DETECTORS>/scripts/categories.py <station folder> --from <from> --to <to>. Families: <FAMILIES>. Output: <OUT>/validation.md. INSTRUCTIONS: instructions/validator-detectors.md.`
|
||||||
|
- `keep`: go to part 2.
|
||||||
|
- `remove`: for each fix of the `**Remove**:` line, its commits (`git -C <DETECTORS> log --format=%h --grep ': S<n>$' <base>..HEAD`), from the most recent: `git -C <DETECTORS> revert --no-edit <commit>`, then `git -C <DETECTORS> commit --amend -m "improve: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`. A conflict: `git -C <DETECTORS> revert --abort`, and do as for `cancel`.
|
||||||
|
- `cancel`, or anything else: `git -C <DETECTORS> revert --no-commit <base>..HEAD && git -C <DETECTORS> commit -m "improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"` (only what the fix commits changed; files already modified stay as they are).
|
||||||
|
5. **Measures end**, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed).
|
||||||
|
|
||||||
|
An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to part 2. Do not rerun an agent.
|
||||||
|
|
||||||
|
### Part 2: the tools
|
||||||
|
|
||||||
|
Always, whatever part 1 gave (a stop in part 1 does not stop part 2).
|
||||||
|
|
||||||
|
6. **Upgrader on the tools**: `Target: <TOOLS>. Evidence: <ANALYSIS>/. Checks: none. Base: <tools base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/tools.md. INSTRUCTIONS: instructions/upgrader-tools.md.` It reads the analysis, writes only in `TOOLS` and in `<OUT>`, and commits in `TOOLS`, one commit per fix. Its output missing: note it. Do not rerun it.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
`<OUT>/summary.md`:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# optimize: <station>, <FROM> → <TO>
|
||||||
|
|
||||||
|
**Result**: modification kept | kept without S<n>… | cancelled | none | error
|
||||||
|
**Tools changed**: <the value of the `**Changed**` line of <OUT>/tools.md> | error
|
||||||
|
|
||||||
|
## The range
|
||||||
|
the range score, before → end, one line per family (found, correct)
|
||||||
|
|
||||||
|
## Elsewhere
|
||||||
|
the `Total` line of each family, before → end; then the other changed lines (station, family), before → end
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
<per fix kept, one sentence from "What changed" of the modification>; the upgrader's verdict, the validator's
|
||||||
|
|
||||||
|
## For the user to decide
|
||||||
|
<copied as they are: "Limits" and "For the user to decide" of the modification, "Reservations" of the validator; none>
|
||||||
|
|
||||||
|
## Not fixed
|
||||||
|
<"Not fixed" of the modification; none>
|
||||||
|
|
||||||
|
## Reading tools
|
||||||
|
<copied as they are: "What I decided" and "What changed" of <OUT>/tools.md, and its "data.py" line if changed; none>
|
||||||
|
|
||||||
|
## Scripts of the agents
|
||||||
|
<copied as they are: "Tools" of the modification and of the validation (the scripts the agents kept in `<OUT>/tools/`); none>
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`; `git -C <TOOLS> log --oneline <tools base>..HEAD` if the tools changed>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Answer
|
||||||
|
|
||||||
|
In 5 lines at most: the `**Result**` line, the range score before → end (changed families), the `Total` before → end if it changed, the `**Tools changed**` line, the path of the summary.
|
||||||
@@ -2,6 +2,8 @@ data
|
|||||||
labeling
|
labeling
|
||||||
detectors
|
detectors
|
||||||
agents
|
agents
|
||||||
|
/tools
|
||||||
|
sessions/*/work/
|
||||||
.venv/
|
.venv/
|
||||||
__pycache__/
|
__pycache__/
|
||||||
archive/batches/*.tmp/
|
archive/batches/*.tmp/
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
# CLAUDE.md
|
# CLAUDE.md
|
||||||
|
|
||||||
Optimizer: the optimisation of the AIMsight detectors on the labels of `labeling/auto`, and nothing else (`README.md`). The workflow: `.claude/skills/improve/SKILL.md` (`/improve`), launched from the root of Optimizer with the agents plugin (`--plugin-dir agents`); its agents `agents:upgrader` and `agents:validator`; the benchmark `detectors/bench/`.
|
Optimizer: the optimisation of the AIMsight detectors and of the reading tools, from one analysis of `labeling/auto`, and nothing else (`README.md`). The workflow: `.claude/skills/optimize/SKILL.md` (`/optimize`), launched from the root of Optimizer with the agents plugin (`--plugin-dir agents`); its agents `agents:upgrader`, `agents:validator` (the detectors) and `agents:upgrader` again (the tools), each with its file of `instructions/`; the benchmark `detectors/bench/`.
|
||||||
|
|
||||||
- **The links**, out of git: `detectors/` → `~/detectors` (the session commits there); `data/` → `~/aimsight-data/data` and `labeling/` → `~/labeling` (read-only); `agents/` → `~/agents` (the plugin, loaded read-only).
|
- **The links**, out of git: `detectors/` → `~/aimsight-detectors` and `tools/` → `~/aimsight-tools` (the session commits there); `data/` → `~/aimsight-data/data` and `labeling/` → `~/aimsight-labeling` (read-only); `agents/` → `~/aimsight-agents` (the plugin, loaded read-only).
|
||||||
- **The sessions**: `sessions/<analysis>_<YYYYMMDDTHHMMZ>/`; Diagnostic-App reads their summary (`summary.md`).
|
- **The sessions**: `sessions/<analysis>_<YYYYMMDDTHHMMZ>/`; aimsight-app reads their summary (`summary.md`).
|
||||||
- **The workflow stays generic**: the skill describes only the workflow (range + labels → upgrader → validator → summary); what is specific to a caller (its paths, its git rules, `Caller's context`) arrives in the request.
|
- **The workflow stays generic**: the skill describes only the workflow (analysis → detectors: upgrader → validator; tools: upgrader → summary); what is specific to a caller (its paths, its git rules, `Caller's context`) arrives in the request.
|
||||||
- **Check a change of the skill**: `env -u ANTHROPIC_API_KEY claude -p "/improve" --plugin-dir agents --model haiku` must say the inputs are missing and stop without touching `detectors/`; then a real session through the app's button.
|
- **Check a change of the skill**: `env -u ANTHROPIC_API_KEY claude -p "/optimize" --plugin-dir agents --model haiku` must say the inputs are missing and stop without touching `detectors/` or `tools/`; then a real session through the app's button.
|
||||||
- `archive/`: the former batch and its scanner, stopped; do not run them again.
|
- `archive/`: the former batch and its scanner, stopped; do not run them again.
|
||||||
|
|
||||||
Rules:
|
Rules:
|
||||||
|
|||||||
@@ -1,42 +1,40 @@
|
|||||||
# Optimizer
|
# Optimizer
|
||||||
|
|
||||||
The optimisation of the AIMsight detectors, on the labels of `labeling/auto`, and nothing else. On the range of an in-depth analysis (made in the Diagnostic-App app), its labels as the truth, the `improve` skill:
|
The optimisation of the AIMsight detectors and of the reading tools, from one analysis, and nothing else. An analysis (made in the aimsight-app app by the analyst) leaves in its folder the two things needed: its labels, the truth on its range, and its own scripts with what it missed. On that analysis, the `optimize` skill:
|
||||||
|
|
||||||
1. measures the detectors' score on the range, and the benchmark on every label of `labeling/auto`;
|
1. **the detectors**: measures their score on the range, and the benchmark on every label of `labeling/auto`; has them fixed by the `upgrader`, one fix per commit in `~/aimsight-detectors`; has them judged by the `validator`, which keeps, has some removed, or cancels: no station goes back, no overfitting;
|
||||||
2. has the detectors fixed by the `upgrader`, one fix per commit in `~/detectors`;
|
2. **the tools**: has the `upgrader` bring the analyst's useful scripts into the reading harness, one fix per commit in `~/aimsight-tools`; always, whatever the detectors gave;
|
||||||
3. has them judged by the `validator`, which keeps, has some removed, or cancels: no station goes back, no overfitting;
|
3. writes the summary of the session.
|
||||||
4. writes the summary of the session.
|
|
||||||
|
|
||||||
The `upgrader` and the `validator` come from the `agents` repository; the benchmark (`bench/`) from the `detectors` repository. Nothing is pushed.
|
The `upgrader` and the `validator` come from the `agents` repository, each with its instructions from `instructions/` (`upgrader-detectors.md`, `validator-detectors.md`, `upgrader-tools.md`); the rules of each target are its own `CLAUDE.md`; the benchmark (`bench/`) from the `detectors` repository. Nothing is pushed.
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ln -s ~/detectors detectors # the detectors, shared with Diagnostic-App
|
ln -s ~/aimsight-detectors detectors # the detectors, shared with aimsight-app
|
||||||
|
ln -s ~/aimsight-tools tools # the reading tools, shared with aimsight-app
|
||||||
ln -s ~/aimsight-data/data data # the pumped data, out of git
|
ln -s ~/aimsight-data/data data # the pumped data, out of git
|
||||||
ln -s ~/labeling labeling # the labels: the truth and the families
|
ln -s ~/aimsight-labeling labeling # the labels, the analyses and the families
|
||||||
ln -s ~/agents agents # the generic agents (Claude Code plugin)
|
ln -s ~/aimsight-agents agents # the generic agents (Claude Code plugin)
|
||||||
```
|
```
|
||||||
|
|
||||||
`claude` (Claude Code) must be on the `PATH`, logged in to the Claude subscription (`/login`), never to an API key.
|
`claude` (Claude Code) must be on the `PATH`, logged in to the Claude subscription (`/login`), never to an API key.
|
||||||
|
|
||||||
## Run
|
## Run
|
||||||
|
|
||||||
From the "Optimize the detectors" button of Diagnostic-App, on a finished analysis; or by hand, from the root of Optimizer:
|
From the "Optimize" button of aimsight-app, on a finished analysis; or by hand, from the root of Optimizer:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
env -u ANTHROPIC_API_KEY claude -p "/improve DETECTORS: detectors/. STATION: data/NORDEND. FROM: 2026-11-04. TO: 2026-11-08. \
|
env -u ANTHROPIC_API_KEY claude -p "/optimize ANALYSIS: labeling/auto/analyses/<analysis>/. OUT: sessions/<analysis>_<YYYYMMDDTHHMMZ>/." \
|
||||||
LABELS: labeling/auto/NORDEND.jsonl. REFERENCE: --labels labeling/auto --data data. FAMILIES: labeling/families.md. \
|
--plugin-dir agents --add-dir ~/aimsight-detectors --add-dir ~/aimsight-tools --add-dir ~/aimsight-labeling --add-dir ~/aimsight-data/data \
|
||||||
CONTEXT: labeling/auto/analyses/<analysis>/report.md. OUT: sessions/<analysis>_<YYYYMMDDTHHMMZ>/." \
|
|
||||||
--plugin-dir agents --add-dir ~/detectors --add-dir ~/labeling --add-dir ~/aimsight-data/data \
|
|
||||||
--setting-sources project,local --strict-mcp-config \
|
--setting-sources project,local --strict-mcp-config \
|
||||||
--allowedTools Read,Write,Edit,Bash,Agent --permission-mode acceptEdits
|
--allowedTools Read,Write,Edit,Bash,Agent --permission-mode acceptEdits
|
||||||
```
|
```
|
||||||
|
|
||||||
`TO` is excluded. Several stations on the same range: `DATA: data. STATIONS: A,B. LABELS: <folder of <STATION>.jsonl>` instead of `STATION` and a labels file.
|
The station and the range come from the analysis's `request.json` (the range's last day included); the labels from `labeling/auto/<STATION>.jsonl`; the context from its `report.md`. `DETECTORS`, `TOOLS`, `DATA`, `LABELS`, `FAMILIES` can be overridden in the request.
|
||||||
|
|
||||||
## What a session leaves
|
## What a session leaves
|
||||||
|
|
||||||
`sessions/<analysis>_<time>/`: the measures before and after (`range-*.txt`, `benchmark-*.txt`), `modification.md` (the upgrader), `validation.md` (the validator), `summary.md` (the summary, shown by the app under the analysis report), `tools/` (the scripts the agents wrote). The fixes: `improve: S<n>` commits in `~/detectors`, on its current branch.
|
`sessions/<analysis>_<time>/`: the measures before and after (`range-*.txt`, `benchmark-*.txt`), `modification.md` (the upgrader), `validation.md` (the validator), `tools.md` (the upgrader on the tools) and its trials (`work/`), `summary.md` (the summary, shown by the app under the analysis report), `tools/` (the scripts the upgrader and validator wrote). The fixes: `improve: S<n>` commits in `~/aimsight-detectors`, the tools `improve: S<n>` commits in `~/aimsight-tools`, each on its current branch.
|
||||||
|
|
||||||
`archive/`: the former daily batch and its scanner (auto-labeling by the detectors), stopped on 2026-11-16. The architecture of the repositories: `~/Diagnostic-App/docs/architecture.drawio`.
|
`archive/`: the former daily batch and its scanner (auto-labeling by the detectors), stopped on 2026-11-16. The architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`.
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# `/optimize` Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Optimizer's `/optimize ANALYSIS: <folder>` optimises the detectors and the reading tools from one analysis; Diagnostic-App calls it once.
|
||||||
|
|
||||||
|
**Architecture:** `/improve` is moved to `/optimize`, derives its inputs from the analysis folder, keeps its detectors steps, then runs `agents:tool-builder` as a subagent and writes one summary. The tool-builder writes only in the session folder. Diagnostic-App's optimisation becomes a single `claude -p` turn in Optimizer.
|
||||||
|
|
||||||
|
**Tech Stack:** Claude Code skills and agents (Markdown), Python 3 (Diagnostic-App server and its `check.py`).
|
||||||
|
|
||||||
|
**Spec:** `docs/superpowers/specs/2026-11-16-optimize-design.md`
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- English everywhere; every time UTC.
|
||||||
|
- Commit by path, never `git add -A`; each repository on its current branch (Optimizer `master`, the others `main`); nothing pushed.
|
||||||
|
- Commit messages end with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
|
||||||
|
- Optimizer is read-only on `data/` and `labeling/`; the agents never run on an API key.
|
||||||
|
- Detector fix commits stay `improve: S<n>`.
|
||||||
|
|
||||||
|
## Review Focus
|
||||||
|
|
||||||
|
- `/optimize` without inputs, or with an analysis folder lacking `request.json`: says what is missing, stops, touches neither `detectors/` nor `tools/` (Task 2, Step 4).
|
||||||
|
- An analysis whose range carries no label (`score.py` exits with an error): the detectors part stops, the tools part still runs (Task 2 skill text, Step 4 review).
|
||||||
|
- An analysis that ends at midnight: `TO` = the day after `end`, as Diagnostic-App does today (Task 2 skill text).
|
||||||
|
- The tool-builder never writes in the analysis folder (Task 1).
|
||||||
|
- Diagnostic-App with Optimizer lacking the `tools` link: a clear refusal (Task 3, Step 1).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: the tool-builder writes in its report's folder (agents)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `~/agents/agents/tool-builder.md` (lines 3, 44-45, 65)
|
||||||
|
- Modify: `~/agents/agents/upgrader.md:3,8`, `~/agents/agents/validator.md:3,8`, `~/agents/README.md:8-10`
|
||||||
|
|
||||||
|
- [ ] **Step 1:** In `tool-builder.md`, Check step 1: `--out <the folder of your report>/work/tools/` instead of `<analysis>/work/tools/`; Check step 2: rerun the report's commands with their `--out` replaced by `<the folder of your report>/work/`; add "You write nothing in the analysis folder." to "What you touch"; report template "figures in <the folder of your report>/work/tools/". Description: launched by Optimizer's `/optimize` (Agent tool, `agents:tool-builder`).
|
||||||
|
- [ ] **Step 2:** "improve workflow" → "optimize workflow" in `upgrader.md`, `validator.md`; README table: requests "that of the `optimize` skill", the tool-builder's caller "Optimizer (`/optimize`)".
|
||||||
|
- [ ] **Step 3:** `grep -n "work/tools\|improve workflow\|improve\` skill" ~/agents -r` returns only `<the folder of your report>` lines.
|
||||||
|
- [ ] **Step 4:** Commit in `~/agents`: `Optimize: the tool-builder writes in its report's folder; run by Optimizer's /optimize`.
|
||||||
|
|
||||||
|
### Task 2: the `/optimize` skill (Optimizer)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Move: `.claude/skills/improve/SKILL.md` → `.claude/skills/optimize/SKILL.md` (`git mv`)
|
||||||
|
- Create: link `tools -> ~/tools` (ignored or untracked, like the others)
|
||||||
|
- Modify: `README.md`, `CLAUDE.md`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Produces: request `/optimize ANALYSIS: <analysis folder>/. OUT: sessions/<analysis>_<YYYYMMDDTHHMMZ>/. Caller's context: …`; optional overrides `DETECTORS`, `TOOLS`, `DATA`, `LABELS`, `FAMILIES`; output `<OUT>/summary.md` (first line `# optimize: <station>, <FROM> → <TO>`, lines `**Result**:` and `**Tools changed**:`), `<OUT>/tools.md`.
|
||||||
|
|
||||||
|
- [ ] **Step 1:** Skill frontmatter `name: optimize`, description: the workflow that, from one analysis, improves the detectors (upgrader, validator) and the reading tools (tool-builder).
|
||||||
|
- [ ] **Step 2:** Inputs section per the spec: `ANALYSIS`, `OUT` required; defaults `DETECTORS: detectors/`, `TOOLS: tools/`, `DATA: data`, `LABELS: labeling/auto`, `FAMILIES: labeling/families.md`; derived `STATION`, `FROM`, `TO` (day after `end`), labels file, reference, `CONTEXT`. Missing `ANALYSIS`/`OUT`, no `request.json`, `DETECTORS` or `TOOLS` not a git repository: say so, stop, change nothing. Remove the `DATA`+`STATIONS` mode and its commands.
|
||||||
|
- [ ] **Step 3:** Steps: "Part 1, detectors" = the current steps (unchanged; a stop in part 1 — no label on the range, an agent missing — goes to part 2, not to the end); "Part 2, tools", always: Agent tool `agents:tool-builder`, request `Analysis: <ANALYSIS>/. Tools: <TOOLS>/. Write <OUT>/tools.md.`, plus the caller's context; its output missing: note it. Summary: title `# optimize: …`, the detectors part as now, `**Tools changed**:` under `**Result**:`, section `## Reading tools` (copied: "Tools changed", "What I decided" of `<OUT>/tools.md`), `## Tools` renamed `## Scripts of the agents`. Answer: add the `**Tools changed**` line.
|
||||||
|
- [ ] **Step 4:** Run `env -u ANTHROPIC_API_KEY claude -p "/optimize" --plugin-dir agents --model haiku`, then the same with `"/optimize ANALYSIS: labeling/auto/analyses/nothing/. OUT: /tmp/x/"`. Expected: both say what is missing and stop; `git -C detectors status --short` and `git -C tools status --short` empty.
|
||||||
|
- [ ] **Step 5:** `ln -s ~/tools tools`; check `git status` does not show `tools` as a new file to commit (add it to `.gitignore` if the other links are there).
|
||||||
|
- [ ] **Step 6:** `README.md`: scope "the detectors and the tools, from an analysis"; the steps; the `tools` link; the by-hand command `/optimize ANALYSIS: labeling/auto/analyses/<analysis>/. OUT: sessions/<analysis>_<YYYYMMDDTHHMMZ>/.` with `--add-dir ~/tools`; what a session leaves (`tools.md`, `work/`). `CLAUDE.md`: the same scope, link and check command.
|
||||||
|
- [ ] **Step 7:** Commit in Optimizer by path: the moved skill, `README.md`, `CLAUDE.md`, `.gitignore` if changed, the spec and this plan: `Optimize: /optimize, the detectors and the tools from one analysis (ex /improve)`.
|
||||||
|
|
||||||
|
### Task 3: Diagnostic-App calls `/optimize` once
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `~/Diagnostic-App/app/server/analysis.py` (docstring 1-11, `IMPROVE` 29-33, `optimize` 262-268, `optimize_thread` 275-297)
|
||||||
|
- Test: `~/Diagnostic-App/app/server/check.py` (`check_optimization`, 135-166)
|
||||||
|
- Modify: `~/Diagnostic-App/CLAUDE.md:5,8`, `README.md:6,25`, `app/README.md:24,41`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: Task 2's request.
|
||||||
|
|
||||||
|
- [ ] **Step 1: the failing check.** In `check_optimization`: create `skills/optimize/SKILL.md` instead of `improve`; the links loop `("data", "labeling", "detectors", "tools")` with a refusal asserted before `tools` exists; after the run `assert len(launched) == 1`; `request.startswith("/optimize ANALYSIS: labeling/auto/analyses/NORDEND_a/. OUT: sessions/NORDEND_a_")`, `k["cwd"] == analysis.OPTIMIZER.resolve()`, `"--plugin-dir" in args`; remove the `launched[1]` tool-builder assertions; keep the old `<analysis>/tools.md` summary assertion.
|
||||||
|
- [ ] **Step 2:** `cd ~/Diagnostic-App && .venv/bin/python app/server/check.py` → FAIL in `check_optimization`.
|
||||||
|
- [ ] **Step 3:** `IMPROVE` → `OPTIMIZE = "/optimize ANALYSIS: labeling/auto/analyses/{name}/. OUT: sessions/{name}_{time}/. Caller's context: Diagnostic-app, through Optimizer; read-only on data/ and labeling/; commit only in detectors/ and tools/, on their current branch; do not commit OUT."`; `optimize()` checks `skills/optimize/SKILL.md` and the `tools` link; `optimize_thread` one turn in `OPTIMIZER.resolve()` with `--plugin-dir AGENTS.resolve()`, `--add-dir` detectors (station.links() already adds tools), `--allowedTools Read,Write,Edit,Bash,Agent`; the `request.json` read and `TO` computation removed (Optimizer derives them); docstrings say `/optimize`.
|
||||||
|
- [ ] **Step 4:** `.venv/bin/python app/server/check.py` → passes.
|
||||||
|
- [ ] **Step 5:** Docs: "Optimize the detectors" launches `/optimize` in `optimizer/` on the analysis (detectors committed in `detectors/`, tools in `tools/`); contract "the request of `/optimize`"; `app/README.md` the single launch; summary path.
|
||||||
|
- [ ] **Step 6:** Commit in Diagnostic-App by path: `Optimize: a single /optimize call in Optimizer, the detectors and the tools`.
|
||||||
|
|
||||||
|
### Task 4: wording in detectors and tools
|
||||||
|
|
||||||
|
**Files:** `~/detectors/CLAUDE.md:3,12`, `~/detectors/README.md` ("Improving them"), `~/tools/README.md` ("Who changes it")
|
||||||
|
|
||||||
|
- [ ] **Step 1:** `improve` skill → `optimize` skill (detectors); the tool-builder "run by Optimizer's `/optimize`, after each analysis optimised" (tools). History references (`wf-improve`, `improve: S<n>`) unchanged.
|
||||||
|
- [ ] **Step 2:** Commit each repository by path: `Optimize: the skill is /optimize`.
|
||||||
|
|
||||||
|
### Task 5: final check
|
||||||
|
|
||||||
|
- [ ] `grep -rn "/improve\|improve\` skill\|skills/improve" ~/Optimizer --include=*.md -l | grep -v "archive/\|sessions/\|docs/superpowers"` and the same on Diagnostic-App (outside `docs/superpowers`, `node_modules`), agents, detectors, tools: nothing left.
|
||||||
|
- [ ] Diagnostic-App `check.py` passes; the Task 2 haiku runs stop cleanly.
|
||||||
|
- [ ] Left to the user: a real session through the app's button (it commits in `detectors` and `tools`).
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
# `/optimize`: one analysis in, detectors and tools optimised
|
||||||
|
|
||||||
|
Date: 2026-11-16. Size: a small change, across five repositories.
|
||||||
|
|
||||||
|
## Intent
|
||||||
|
|
||||||
|
Everything that optimises lives in Optimizer. Today Optimizer optimises the detectors (`/improve`), while the
|
||||||
|
optimisation of the reading tools is a bare `claude -p --agent agents:tool-builder` call written in Diagnostic-App's
|
||||||
|
server. An analysis run by the analyst already leaves, in its folder, the two things both optimisations need: its labels
|
||||||
|
(`labels.jsonl`, the truth for the detectors) and its own scripts plus what it missed (`scripts/`, `analysis.md`, the
|
||||||
|
material for the tools). So the workflow takes **one analysis** and optimises **both**.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- One skill, `/optimize`, in Optimizer: input an analysis folder, output one session folder and one summary.
|
||||||
|
- Diagnostic-App's "Optimize" button makes a single call, to `/optimize`.
|
||||||
|
- Optimizer's scope becomes "the detectors and the tools, from an analysis".
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- No change to the analyst: it already writes its scripts in `<output folder>/scripts/` and says what it missed.
|
||||||
|
- No validator for the tools; the tool-builder stays as it is in its decisions.
|
||||||
|
- No toolbox for the scripts of the upgrader and validator (`sessions/*/tools/`): later.
|
||||||
|
- No change to the benchmark, the agents' judgement, or the labels.
|
||||||
|
|
||||||
|
## Anti-goals
|
||||||
|
|
||||||
|
- Optimizer writing in `labeling/` (read-only): the tool-builder's outputs move to the session folder.
|
||||||
|
- One step's failure blocking the other.
|
||||||
|
|
||||||
|
## Approaches
|
||||||
|
|
||||||
|
1. **Extend `/improve` into `/optimize ANALYSIS: <folder>`** (picked). The skill derives its inputs from the analysis,
|
||||||
|
runs the current detectors steps, then the tool-builder, then writes one summary. One file, one session, one line in
|
||||||
|
Diagnostic-App. `/improve` has a single caller, so nothing needs it alone.
|
||||||
|
2. Keep `/improve`, add `/optimize` calling it then the tool-builder. Nothing existing moves, but two layers and two
|
||||||
|
summaries to merge, for no caller that wants the detectors alone.
|
||||||
|
3. Two skills called in turn by Diagnostic-App. Smallest diff, but the process (the order, the session) stays in
|
||||||
|
Diagnostic-App, against the intent.
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
**Skill** `.claude/skills/optimize/SKILL.md` (moved from `improve/`). Inputs:
|
||||||
|
|
||||||
|
- `ANALYSIS` (required): the analysis folder (`request.json`, `labels.jsonl`, `report.md`, `analysis.md`, `scripts/`).
|
||||||
|
- `OUT` (required): the session folder, `sessions/<analysis>_<YYYYMMDDTHHMMZ>/`.
|
||||||
|
- Defaults, relative to Optimizer's links, overridable in the request: `DETECTORS: detectors/`, `TOOLS: tools/`,
|
||||||
|
`DATA: data`, `LABELS: labeling/auto` (the folder of `<STATION>.jsonl`), `FAMILIES: labeling/families.md`.
|
||||||
|
- Derived: `STATION` = `<DATA>/<request.json station>`; `FROM` = the day of `start`; `TO` = the day after `end`
|
||||||
|
(excluded); labels = `<LABELS>/<station>.jsonl`; reference = `--labels <LABELS> --data <DATA>`;
|
||||||
|
context = `<ANALYSIS>/report.md`.
|
||||||
|
- `Caller's context`, passed on as today.
|
||||||
|
|
||||||
|
The multi-station mode (`DATA` + `STATIONS`) is dropped: an analysis is one station and no caller uses it (my call).
|
||||||
|
|
||||||
|
**Steps.**
|
||||||
|
1. Detectors: the current steps of `/improve`, unchanged (measures, upgrader, measures, validator, remove/cancel).
|
||||||
|
Commits stay `improve: S<n>` in `DETECTORS` (the agents grep it).
|
||||||
|
2. Tools, always, whatever step 1 gave: the Agent tool, `subagent_type: agents:tool-builder`, request
|
||||||
|
`Analysis: <ANALYSIS>/. Tools: <TOOLS>/. Write <OUT>/tools.md.` It commits in `TOOLS` itself.
|
||||||
|
3. Summary `<OUT>/summary.md`: the current detectors part, then a `## Reading tools` part (the tool-builder's
|
||||||
|
"Tools changed" line and its "What I decided", copied). The current `## Tools` section (the agents' scripts) is renamed
|
||||||
|
`## Scripts of the agents`.
|
||||||
|
|
||||||
|
**agents repository.** `tool-builder.md`: its trial figures and the reruns of the report's commands write in
|
||||||
|
`<the folder of its report>/work/`, never in the analysis folder. The "improve workflow" wording of `upgrader.md`,
|
||||||
|
`validator.md` and `README.md` becomes "optimize workflow"; the tool-builder's caller becomes Optimizer.
|
||||||
|
|
||||||
|
**Optimizer.** Link `tools -> ~/tools`; the launch adds `--add-dir ~/tools`. `README.md` and `CLAUDE.md`: scope "the
|
||||||
|
detectors and the tools", the `/optimize` request, the check command.
|
||||||
|
|
||||||
|
**Diagnostic-App.** `optimize_thread`: one turn, `/optimize ANALYSIS: labeling/auto/analyses/<name>/. OUT: sessions/
|
||||||
|
<name>_<time>/. Caller's context: …`, from Optimizer, with `--add-dir` on detectors and tools. `optimize()` also requires
|
||||||
|
the `tools` link and `optimize/SKILL.md`. `summary()` unchanged: it shows the session's `summary.md`, and the old
|
||||||
|
`<analysis>/tools.md` if one exists. `check.py`: one launch, `/optimize` in Optimizer. Docs (`CLAUDE.md`, `README.md`,
|
||||||
|
`app/README.md`) follow.
|
||||||
|
|
||||||
|
**detectors, tools.** The wording `improve` skill → `optimize` skill in `detectors/CLAUDE.md`, `detectors/README.md`;
|
||||||
|
`tools/README.md` "Who changes it": the tool-builder, run by Optimizer's `/optimize`.
|
||||||
|
|
||||||
|
## Check
|
||||||
|
|
||||||
|
- `env -u ANTHROPIC_API_KEY claude -p "/optimize" --plugin-dir agents --model haiku`: says the inputs are missing, stops,
|
||||||
|
touches neither `detectors/` nor `tools/`.
|
||||||
|
- Diagnostic-App's `check.py` passes.
|
||||||
|
- A real session through the app's button, on an analysis with scripts.
|
||||||
|
|
||||||
|
## Left to the builder
|
||||||
|
|
||||||
|
The exact wording of the skill and of the summary's tools part; the order of the doc edits.
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# Optimizer's instructions to the upgrader: the detectors
|
||||||
|
|
||||||
|
What `/optimize` asks of the upgrader (agents repository, `agents:upgrader`) in part 1, on top of its trade. The target is the detectors repository; its rules (what may change, the regression, everything is detected) are in `<Target>/CLAUDE.md`, read in full.
|
||||||
|
|
||||||
|
## Why this range
|
||||||
|
|
||||||
|
An analyst has analysed a range of a station where the detectors perform poorly. The labels of the range are the truth. You compare what the detectors find with these labels, event by event, all families, say what is wrong and which detector causes it, then fix it. Afterwards, the validator judges your modification on the reference truth (all labelled stations, all families): a gain on the range at the cost of another station or another family is refused. What counts is the mechanism behind a label (a rejected reference that makes all targets jump, for example), recognisable wherever it happens; a constant tuned to make these bounds fall exactly, a rule that only recognises the exact shape of these events: no. What the detectors cannot see with these measurements (a cause only an image reveals) goes to "Limits".
|
||||||
|
|
||||||
|
## The evidence
|
||||||
|
|
||||||
|
- The labels file of the station (or several stations on the same range, a labels file each, in one folder: the events and sensors checks run per station, the range score covers them all; a mechanism that comes back from one station to another comes before a single case). A label: one line of JSONL, `{"id", "start", "end", "family", "description"}` (cited by its `id`), a single anomaly, seen on `corr_x` and `corr_y` (they move or are missing); `description` says the sensors touched, the cause and how it was known. Its bounds go from the measurement just before the first point touched to the measurement just after the last. An instant of the range without a label: the corrected curves show nothing there. Read the labels touching the range (`start` < `TO` and `end` ≥ `FROM`).
|
||||||
|
- The context documents (the analysis report): the mechanism of each event, the sensors touched and those that do not move, the level of proof; a document may say that a label is wrong: a limit to report, not a target.
|
||||||
|
- The families document **in full**: the families and their boundaries; the category of an event is its family.
|
||||||
|
- The checks' outputs before: events (run it with `2>&1`), sensors (the cadence, then the sensors of each event), range score, benchmark.
|
||||||
|
- The code of the detectors, `scripts/categories.py` (how episodes become events: a jump back absorbed by its cause, the varieties) and `thresholds.md`. You may read the station's data to understand why a rule fails.
|
||||||
|
|
||||||
|
## The diagnosis
|
||||||
|
|
||||||
|
Compare, event by event and label by label: correct; wrong family; extra (flagged by the detectors, without a label); wrong bounds (late, too early, too wide); wrong sensors (the event's, by the sensors check, against those the description names); missing. A few measurements of gap at the bounds are not a defect. Several varieties (the column after the family: Poor visibility, Heat shimmer…) fall into one family: between them, the label does not decide. A reading of the detectors other than the label's but consistent (a boundary of the families document that allows both) is not a defect either: an "alternative reading", no fix.
|
||||||
|
|
||||||
|
Fixes to try first: a significant event missed or badly bounded before a small nuisance. The simplest first, for the detectors, ends with: as a last resort, a new variety (a `label`) in the family's detector.
|
||||||
|
|
||||||
|
## The checks
|
||||||
|
|
||||||
|
- **events**: before and after each fix, with `2>&1`.
|
||||||
|
- **range**: the range score, what you make rise, without losing a whole event. A hole has no measurement: it only counts its two bounds; for a hole, read the events check. Do not aim for 100 %.
|
||||||
|
- **benchmark**: all lines of all stations, against the benchmark of the last committed state (the benchmark before, or that of your previous kept fix); before your output, the whole against the benchmark before. A regression (`<Target>/CLAUDE.md`): rework your change before committing it, or drop the fix. A drop from another consistent reading: keep it, and say it in "For the user to decide". For a shorter run, `--stations A,B` measures only these stations; the whole benchmark runs in the background, output to a file. To try a variant without touching the repository, copy the target in your working folder: the copy's scripts measure the copy, with the same arguments.
|
||||||
|
|
||||||
|
## Your output
|
||||||
|
|
||||||
|
On top of the skeleton of your trade:
|
||||||
|
|
||||||
|
- `## Diagnosis` opens with `<station, range, cadence>` and the table `| Family | labelled | found | flagged | wrongly |`, then one line per event or label: its judgement and why; same judgement and cause grouped.
|
||||||
|
- After `## Checks`: `## Before / after on the range` (the changed lines of the two outputs of the events check; the range score before → after, per family) and `## Benchmark` (the lines that changed, station and family, before → after, then the changed lines of the `Total` block).
|
||||||
|
- "For the user to decide": a reading of the detectors other than the labels', kept: the labels (id, station, moment, family), the detectors' reading, what it changes in the benchmark. "Limits": the label (id, bounds, family), what it would take to see it (image…); a label the context says is wrong (id).
|
||||||
|
|
||||||
|
Times in UTC. The orchestrator reads the `**Verdict**` line as it is.
|
||||||
|
|
||||||
|
## Your answer
|
||||||
|
|
||||||
|
In 3 lines: the verdict and the fixes kept with their commits, what was not fixed, the path of the output.
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# Optimizer's instructions to the upgrader: the tools
|
||||||
|
|
||||||
|
What `/optimize` asks of the upgrader (agents repository, `agents:upgrader`) in part 2, on top of its trade. The target is the tools repository (the reading harness); its rules (remove nothing, the simplest first, no judgement in a tool's output, `data.py`) are in `<Target>/CLAUDE.md`, read in full.
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
The analyst has just analysed a range as an expert, with the tools of the target, their guide (`GUIDE.md`) and, where they were not enough, its own scripts. It is up to you to decide what deserves to enter the tools, so that the next analysis, on any station, reads better. The analyst follows the guide; the callers read the data through `scripts/data.py`.
|
||||||
|
|
||||||
|
## The evidence
|
||||||
|
|
||||||
|
The analysis folder: `report.md` first, if it exists (the commands behind each fact, hence the tools the analyst used); then `analysis.md` (its "The documentation" section and its "Difficulties" say what it missed or what got in its way); its scripts, in `scripts/`. Then `<Target>/GUIDE.md` and the tools of `<Target>/scripts/`; the station folder, to try things. The station and the range are in the heading of `analysis.md`. Nothing else: neither the detectors (`detectors/`) nor `CLAUDE.md` of the root you are called from.
|
||||||
|
|
||||||
|
## What you decide
|
||||||
|
|
||||||
|
- **Optimise**: a tool that got in the analyst's way (output too long, unreadable figure, option to guess) is tuned: a better default, an option, a sentence of the guide.
|
||||||
|
- **Add**: a script of the analyst that showed what the tools did not show, and that would help on other stations, enters `<Target>/scripts/`, made general; or an option of an existing tool does the same, which is better. A script that only served this range stays where it is.
|
||||||
|
- **Nothing to do** is a decision: if the tools were enough and nothing got in the way, touch nothing and say so.
|
||||||
|
|
||||||
|
You write nothing in the analysis folder.
|
||||||
|
|
||||||
|
## The checks
|
||||||
|
|
||||||
|
There is no check command (`Checks: none`). For each fix, before its commit:
|
||||||
|
|
||||||
|
1. Run each changed or added tool on the range, from the root you are called from (`python3 <Target>/scripts/<tool>.py data/<STATION> …`), with `--start` and `--end` of the range and `--out <output folder>/work/tools/`; look at each PNG with Read.
|
||||||
|
2. Rerun, on the range, the commands of the analysis report that use a changed tool, with `--out <output folder>/work/` (replacing theirs, or added if absent): they must still work.
|
||||||
|
|
||||||
|
## Your output
|
||||||
|
|
||||||
|
On top of the skeleton of your trade:
|
||||||
|
|
||||||
|
- after the `**Changed**` line: `**data.py**: unchanged | changed: <the reading error fixed, in one sentence>`;
|
||||||
|
- `## What the analyst used` (before `## Diagnosis`): the tools and the scripts it used, and what it missed or what got in its way, summarised;
|
||||||
|
- `## What I decided` (after `## Diagnosis`): `- <tool or script>: <optimised, added or left>, and why`;
|
||||||
|
- under `## Checks`: the trial commands and what they return, figures in `<output folder>/work/tools/`; then `Report commands rerun: <number>, and "no failure" or the failures`.
|
||||||
|
|
||||||
|
## Your answer
|
||||||
|
|
||||||
|
In 2 lines: what changed (or "no tool changed"), the path of your output.
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Optimizer's instructions to the validator: the detectors
|
||||||
|
|
||||||
|
What `/optimize` asks of the validator (agents repository, `agents:validator`) in part 1, on top of its trade. The target is the detectors repository; its rules (what may change, what loads, the regression) are in `<Target>/CLAUDE.md`, read in full.
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
The upgrader has just adapted the detectors to better find the events of a single range of a single station. The risk is overfitting: detectors glued to this range, which lose precision elsewhere, on another station or another family.
|
||||||
|
|
||||||
|
## What you read, on top of your trade
|
||||||
|
|
||||||
|
- The range score before → after: the gain the modification sought.
|
||||||
|
- The benchmarks (the score per family on each station of the reference truth, then their `Total`), all stations in full; the benchmarks after each fix, in order, say which fix makes a station go back.
|
||||||
|
- The families document.
|
||||||
|
- To understand a regression on a station: the events command on its range (from the header of the station's block in the benchmark), with `2>&1`, at HEAD and, to compare, at the base (`git -C <Target> archive <base> | tar -x -C <folder>` in your working folder, then the same command with the copy's `scripts/categories.py`), and the station's labels.
|
||||||
|
|
||||||
|
## What you judge, for the detectors
|
||||||
|
|
||||||
|
- Regressions: station by station and family by family, the range's station included, by the definition of `<Target>/CLAUDE.md`. A drop you do not count as a regression goes to "Reservations".
|
||||||
|
- A range gain of nothing or almost, at the price of one more rule, means cancel (or remove for the fix in question); a rule removed is better than a rule added.
|
||||||
|
|
||||||
|
## Your output
|
||||||
|
|
||||||
|
On top of the skeleton of your trade: `## The range` (the range score before → after, per family), placed before `## Checks`; under `## Checks`, the non-regression: the lines that changed, station and family, before → after, then the changed lines of the `Total` block. "Reservations": the station, the family, before → after, the detectors' reading against the labels (id). Times in UTC.
|
||||||
|
|
||||||
|
## Your answer
|
||||||
|
|
||||||
|
In 2 lines: the verdict, the path of the output.
|
||||||
Reference in new issue
Block a user