Optimize: /optimize, the detectors and the tools from one analysis (ex /improve)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
63bebbc415
commit
68768dfd39
7 files changed
+299
-107
No files matched your search
@@ -1,89 +0,0 @@
|
||||
---
|
||||
name: improve
|
||||
description: Workflow that improves the detectors on a range of data where they perform poorly, given labels as the truth on that range — the upgrader fixes, the validator judges; the result, detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged. The caller gives the range, the labels, the reference truth, the output folder and any specific context. Run from Optimizer's root with the agents plugin (--plugin-dir <agents>). /improve <request>.
|
||||
argument-hint: "<range, labels, reference, output folder, context>"
|
||||
---
|
||||
|
||||
# improve: improve the detectors on a range
|
||||
|
||||
You are the orchestrator. The detectors (the detectors repository, `DETECTORS`: `scripts/`, `thresholds.md`) perform poorly on a range of data of a station, or of several stations on the same range; labels give the truth on this range. You run the upgrader, which fixes the detectors, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||
|
||||
## Inputs
|
||||
|
||||
Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope.
|
||||
|
||||
- `DETECTORS`: the path of the detectors repository (default: `detectors/`, a link in Optimizer).
|
||||
- `STATION`: the station's data folder (read by `DETECTORS/scripts/data.py`); or, several stations on the same range: `DATA`, the folder of the station folders, and `STATIONS`, `A,B`, the stations to improve.
|
||||
- `FROM`, `TO`: the range, UTC days (YYYY-MM-DD, `TO` excluded).
|
||||
- `LABELS`: the station's labels file (JSONL: `id`, `start`, `end`, `family`, `description`), the truth on the range; with `STATIONS`, a folder of `<STATION>.jsonl`, one per station.
|
||||
- `REFERENCE`: the reference truth for the non-regression, `--labels <folder of <STATION>.jsonl> --data <folder of <STATION>/>`.
|
||||
- `FAMILIES`: the document defining the families and their boundaries (one detector per family).
|
||||
- `CONTEXT` (optional): documents explaining the range's labels (an analyst's report, for example).
|
||||
- `OUT`: the output folder of this run (default: `sessions/<analysis>_<YYYYMMDDTHHMMZ>/` in Optimizer).
|
||||
|
||||
A required input missing, or `DETECTORS` not a git repository: say so and stop, changing nothing.
|
||||
|
||||
## Commands
|
||||
|
||||
- events: `python3 <DETECTORS>/scripts/categories.py <STATION> --from <FROM> --to <TO>`
|
||||
- sensors: `python3 <DETECTORS>/bench/sensors.py <STATION> --from <FROM> --to <TO>`
|
||||
- range score (the goal): `python3 <DETECTORS>/bench/score.py <STATION> --labels <LABELS> --from <FROM> --to <TO>`; it exits with an error if no label touches the range: note it in the summary and stop without touching the detectors.
|
||||
- with `STATIONS`: the events and sensors commands once per station, `<DATA>/<S>` for each `S` of `STATIONS`; the range score, all of them at once: `python3 <DETECTORS>/bench/regression.py --labels <LABELS> --data <DATA> --stations <STATIONS> --from <FROM> --to <TO>`.
|
||||
- benchmark (the non-regression, about 3 min): `python3 <DETECTORS>/bench/regression.py <REFERENCE>`
|
||||
|
||||
The measures of a step: `<range score> > <OUT>/range-<step>.txt 2>&1` and `<benchmark> > <OUT>/benchmark-<step>.txt 2>&1`.
|
||||
|
||||
## Git
|
||||
|
||||
The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C <DETECTORS> rev-parse HEAD`. Files already modified (`git -C <DETECTORS> status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. Always commit by path (`git -C <DETECTORS> commit -m "<message>" -- <paths>`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's.
|
||||
|
||||
## Steps
|
||||
|
||||
Each agent is called with the Agent tool, its `subagent_type` (`agents:upgrader`, `agents:validator`, from the agents plugin loaded with `--plugin-dir`; not available: say so and stop, changing nothing) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
|
||||
|
||||
1. **Measures before**: step `before`.
|
||||
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION | <DATA>/<S> for each S of STATIONS>, range <FROM> → <TO> (UTC). Labels: <LABELS>. Families: <FAMILIES>. Context: <CONTEXT | none>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
|
||||
It commits each fix it keeps, message `improve: S<n>`, with its benchmark `<OUT>/benchmark-S<n>.txt`.
|
||||
Verdict `nothing to change`, or no fix commit (`git -C <DETECTORS> log --format=%h --grep ': S[0-9]*$' <base>..HEAD` empty): go to the summary.
|
||||
3. **Measures after**: step `after`.
|
||||
4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: <DETECTORS>. Families: <FAMILIES>. Modification: <OUT>/modification.md. Range score: <OUT>/range-before.txt → <OUT>/range-after.txt. Benchmark: <OUT>/benchmark-before.txt → <OUT>/benchmark-after.txt, after each fix: <OUT>/benchmark-S*.txt; command: <…>. Events command for any station: python3 <DETECTORS>/scripts/categories.py <station folder> --from <from> --to <to>. Base: <base>. Write <OUT>/validation.md.`
|
||||
- `keep`: go to the summary.
|
||||
- `remove`: for each fix of the `**Remove**:` line, its commits (`git -C <DETECTORS> log --format=%h --grep ': S<n>$' <base>..HEAD`), from the most recent: `git -C <DETECTORS> revert --no-edit <commit>`, then `git -C <DETECTORS> commit --amend -m "improve: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`. A conflict: `git -C <DETECTORS> revert --abort`, and do as for `cancel`.
|
||||
- `cancel`, or anything else: `git -C <DETECTORS> revert --no-commit <base>..HEAD && git -C <DETECTORS> commit -m "improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"` (only what the fix commits changed; files already modified stay as they are).
|
||||
|
||||
An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to the summary. Do not rerun an agent.
|
||||
|
||||
## Summary
|
||||
|
||||
Measures, step `end`, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed). Then `<OUT>/summary.md`:
|
||||
|
||||
```markdown
|
||||
# improve: <station | STATIONS>, <FROM> → <TO>
|
||||
|
||||
**Result**: modification kept | kept without S<n>… | cancelled | none | error
|
||||
|
||||
## The range
|
||||
the range score, before → end, one line per family (found, correct)
|
||||
|
||||
## Elsewhere
|
||||
the `Total` line of each family, before → end; then the other changed lines (station, family), before → end
|
||||
|
||||
## What changed
|
||||
<per fix kept, one sentence from "What changed" of the modification>; the upgrader's verdict, the validator's
|
||||
|
||||
## For the user to decide
|
||||
<copied as they are: "Limits" and "For the user to decide" of the modification, "Reservations" of the validator; none>
|
||||
|
||||
## Not fixed
|
||||
<"Not fixed" of the modification; none>
|
||||
|
||||
## Tools
|
||||
<copied as they are: "Tools" of the modification and of the validation (the scripts the agents kept in `<OUT>/tools/`); none>
|
||||
|
||||
## Errors
|
||||
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`>
|
||||
```
|
||||
|
||||
## Answer
|
||||
|
||||
In 4 lines at most: the `**Result**` line, the range score before → end (changed families), the `Total` before → end if it changed, the path of the summary.
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
name: optimize
|
||||
description: Workflow that, from one analysis (its labels as the truth on its range, its scripts and what it missed), optimises the detectors — the upgrader fixes, the validator judges; detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged — then the reading tools — the tool-builder brings the analyst's useful scripts into them. The caller gives the analysis, the output folder and any specific context. Run from Optimizer's root with the agents plugin (--plugin-dir <agents>). /optimize <request>.
|
||||
argument-hint: "ANALYSIS: <analysis folder>. OUT: <output folder>. [context]"
|
||||
---
|
||||
|
||||
# optimize: the detectors and the tools, from one analysis
|
||||
|
||||
You are the orchestrator. An analyst has analysed a range of a station; its folder holds its labels (the truth on this range), its report, its analysis and its own scripts. Two parts follow, in this order. **Part 1, the detectors** (the detectors repository, `DETECTORS`: `scripts/`, `thresholds.md`): you run the upgrader, which fixes them, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **Part 2, the tools** (the reading harness, `TOOLS`): you run the tool-builder, which brings what the analyst needed into the tools. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||
|
||||
## Inputs
|
||||
|
||||
Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope.
|
||||
|
||||
- `ANALYSIS` (required): the analysis folder: `request.json` (`station`, `start`, `end`, UTC), `labels.jsonl`, `report.md`, `analysis.md`, `scripts/`.
|
||||
- `OUT` (required): the output folder of this run (`sessions/<analysis>_<YYYYMMDDTHHMMZ>/` in Optimizer).
|
||||
- Defaults, relative to Optimizer's root (its links), which the request may override: `DETECTORS: detectors/`, `TOOLS: tools/`, `DATA: data`, `LABELS: labeling/auto` (the folder of the `<STATION>.jsonl` label files, JSONL: `id`, `start`, `end`, `family`, `description`), `FAMILIES: labeling/families.md` (the families and their boundaries, one detector per family).
|
||||
|
||||
Derived from `<ANALYSIS>/request.json`:
|
||||
|
||||
- `STATION`: `<DATA>/<station>` (read by `DETECTORS/scripts/data.py`);
|
||||
- `FROM`: the day of `start`; `TO`: the day after the day of `end` (UTC days, YYYY-MM-DD, `TO` excluded);
|
||||
- the labels of the range: `<LABELS>/<station>.jsonl` (the analysis's labels are submitted there);
|
||||
- `REFERENCE`, the reference truth for the non-regression: `--labels <LABELS> --data <DATA>`;
|
||||
- `CONTEXT`: `<ANALYSIS>/report.md`.
|
||||
|
||||
`ANALYSIS` or `OUT` missing, no `<ANALYSIS>/request.json`, `DETECTORS` or `TOOLS` not a git repository: say so and stop, changing nothing.
|
||||
|
||||
## Commands
|
||||
|
||||
- events: `python3 <DETECTORS>/scripts/categories.py <STATION> --from <FROM> --to <TO>`
|
||||
- sensors: `python3 <DETECTORS>/bench/sensors.py <STATION> --from <FROM> --to <TO>`
|
||||
- range score (the goal): `python3 <DETECTORS>/bench/score.py <STATION> --labels <LABELS>/<station>.jsonl --from <FROM> --to <TO>`; it exits with an error if no label touches the range: note it in the summary, touch nothing in the detectors and go to part 2.
|
||||
- benchmark (the non-regression, about 3 min): `python3 <DETECTORS>/bench/regression.py <REFERENCE>`
|
||||
|
||||
The measures of a step: `<range score> > <OUT>/range-<step>.txt 2>&1` and `<benchmark> > <OUT>/benchmark-<step>.txt 2>&1`.
|
||||
|
||||
## Git
|
||||
|
||||
The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C <DETECTORS> rev-parse HEAD`. Files already modified (`git -C <DETECTORS> status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. Always commit by path (`git -C <DETECTORS> commit -m "<message>" -- <paths>`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. The tool-builder commits in `TOOLS` itself. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's.
|
||||
|
||||
## Steps
|
||||
|
||||
Each agent is called with the Agent tool, its `subagent_type` (`agents:upgrader`, `agents:validator`, `agents:tool-builder`, from the agents plugin loaded with `--plugin-dir`; not available: say so and stop, changing nothing) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
|
||||
|
||||
### Part 1: the detectors
|
||||
|
||||
1. **Measures before**: step `before`.
|
||||
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION>, range <FROM> → <TO> (UTC). Labels: <LABELS>/<station>.jsonl. Families: <FAMILIES>. Context: <CONTEXT>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
|
||||
It commits each fix it keeps, message `improve: S<n>`, with its benchmark `<OUT>/benchmark-S<n>.txt`.
|
||||
Verdict `nothing to change`, or no fix commit (`git -C <DETECTORS> log --format=%h --grep ': S[0-9]*$' <base>..HEAD` empty): go to part 2.
|
||||
3. **Measures after**: step `after`.
|
||||
4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: <DETECTORS>. Families: <FAMILIES>. Modification: <OUT>/modification.md. Range score: <OUT>/range-before.txt → <OUT>/range-after.txt. Benchmark: <OUT>/benchmark-before.txt → <OUT>/benchmark-after.txt, after each fix: <OUT>/benchmark-S*.txt; command: <…>. Events command for any station: python3 <DETECTORS>/scripts/categories.py <station folder> --from <from> --to <to>. Base: <base>. Write <OUT>/validation.md.`
|
||||
- `keep`: go to part 2.
|
||||
- `remove`: for each fix of the `**Remove**:` line, its commits (`git -C <DETECTORS> log --format=%h --grep ': S<n>$' <base>..HEAD`), from the most recent: `git -C <DETECTORS> revert --no-edit <commit>`, then `git -C <DETECTORS> commit --amend -m "improve: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`. A conflict: `git -C <DETECTORS> revert --abort`, and do as for `cancel`.
|
||||
- `cancel`, or anything else: `git -C <DETECTORS> revert --no-commit <base>..HEAD && git -C <DETECTORS> commit -m "improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"` (only what the fix commits changed; files already modified stay as they are).
|
||||
5. **Measures end**, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed).
|
||||
|
||||
An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to part 2. Do not rerun an agent.
|
||||
|
||||
### Part 2: the tools
|
||||
|
||||
Always, whatever part 1 gave (a stop in part 1 does not stop part 2).
|
||||
|
||||
6. **Tool-builder**: `Analysis: <ANALYSIS>/. Tools: <TOOLS>/. Write <OUT>/tools.md.` It reads the analysis, writes only in `TOOLS` and in `<OUT>`, and commits in `TOOLS`. Its output missing: note it. Do not rerun it.
|
||||
|
||||
## Summary
|
||||
|
||||
`<OUT>/summary.md`:
|
||||
|
||||
```markdown
|
||||
# optimize: <station>, <FROM> → <TO>
|
||||
|
||||
**Result**: modification kept | kept without S<n>… | cancelled | none | error
|
||||
**Tools changed**: <the "Tools changed" line of <OUT>/tools.md> | error
|
||||
|
||||
## The range
|
||||
the range score, before → end, one line per family (found, correct)
|
||||
|
||||
## Elsewhere
|
||||
the `Total` line of each family, before → end; then the other changed lines (station, family), before → end
|
||||
|
||||
## What changed
|
||||
<per fix kept, one sentence from "What changed" of the modification>; the upgrader's verdict, the validator's
|
||||
|
||||
## For the user to decide
|
||||
<copied as they are: "Limits" and "For the user to decide" of the modification, "Reservations" of the validator; none>
|
||||
|
||||
## Not fixed
|
||||
<"Not fixed" of the modification; none>
|
||||
|
||||
## Reading tools
|
||||
<copied as they are: "What I decided" and "What changed" of <OUT>/tools.md, and its "data.py" line if changed; none>
|
||||
|
||||
## Scripts of the agents
|
||||
<copied as they are: "Tools" of the modification and of the validation (the scripts the agents kept in `<OUT>/tools/`); none>
|
||||
|
||||
## Errors
|
||||
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`; `git -C <TOOLS> log --oneline -1` if the tools changed>
|
||||
```
|
||||
|
||||
## Answer
|
||||
|
||||
In 5 lines at most: the `**Result**` line, the range score before → end (changed families), the `Total` before → end if it changed, the `**Tools changed**` line, the path of the summary.
|
||||
Reference in new issue
Block a user