optimize: generic agents with instructions/; the tools by the upgrader
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6596b68721
commit
c44b9286c9
3 files changed
+14
-14
No files matched your search
@@ -1,12 +1,12 @@
|
||||
---
|
||||
name: optimize
|
||||
description: Workflow that, from one analysis (its labels as the truth on its range, its scripts and what it missed), optimises the detectors — the upgrader fixes, the validator judges; detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged — then the reading tools — the tool-builder brings the analyst's useful scripts into them. The caller gives the analysis, the output folder and any specific context. Run from Optimizer's root with the agents plugin (--plugin-dir <agents>). /optimize <request>.
|
||||
description: Workflow that, from one analysis (its labels as the truth on its range, its scripts and what it missed), optimises the detectors — the upgrader fixes, the validator judges; detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged — then the reading tools — the upgrader brings the analyst's useful scripts into them. Each agent gets its instructions from `instructions/`. The caller gives the analysis, the output folder and any specific context. Run from Optimizer's root with the agents plugin (--plugin-dir <agents>). /optimize <request>.
|
||||
argument-hint: "ANALYSIS: <analysis folder>. OUT: <output folder>. [context]"
|
||||
---
|
||||
|
||||
# optimize: the detectors and the tools, from one analysis
|
||||
|
||||
You are the orchestrator. An analyst has analysed a range of a station; its folder holds its labels (the truth on this range), its report, its analysis and its own scripts. Two parts follow, in this order. **Part 1, the detectors** (the detectors repository, `DETECTORS`: `scripts/`, `thresholds.md`): you run the upgrader, which fixes them, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **Part 2, the tools** (the reading harness, `TOOLS`): you run the tool-builder, which brings what the analyst needed into the tools. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||
You are the orchestrator. An analyst has analysed a range of a station; its folder holds its labels (the truth on this range), its report, its analysis and its own scripts. Two parts follow, in this order. **Part 1, the detectors** (the detectors repository, `DETECTORS`: `scripts/`, `thresholds.md`): you run the upgrader, which fixes them, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **Part 2, the tools** (the reading harness, `TOOLS`): you run the upgrader on them, which brings what the analyst needed into the tools. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||
|
||||
## Inputs
|
||||
|
||||
@@ -37,20 +37,20 @@ The measures of a step: `<range score> > <OUT>/range-<step>.txt 2>&1` and `<benc
|
||||
|
||||
## Git
|
||||
|
||||
The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C <DETECTORS> rev-parse HEAD`. Files already modified (`git -C <DETECTORS> status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. Always commit by path (`git -C <DETECTORS> commit -m "<message>" -- <paths>`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. The tool-builder commits in `TOOLS` itself. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's.
|
||||
The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C <DETECTORS> rev-parse HEAD`. Files already modified (`git -C <DETECTORS> status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. The same for `TOOLS` before part 2: its **tools base** (`git -C <TOOLS> rev-parse HEAD`) and its files already modified. Always commit by path (`git -C <DETECTORS> commit -m "<message>" -- <paths>`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. The upgrader commits in `TOOLS` itself in part 2, one commit per fix. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's.
|
||||
|
||||
## Steps
|
||||
|
||||
Each agent is called with the Agent tool, its `subagent_type` (`agents:upgrader`, `agents:validator`, `agents:tool-builder`, from the agents plugin loaded with `--plugin-dir`; not available: say so and stop, changing nothing) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
|
||||
Each agent is called with the Agent tool, its `subagent_type` (`agents:upgrader`, `agents:validator`, from the agents plugin loaded with `--plugin-dir`; not available: say so and stop, changing nothing) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
|
||||
|
||||
### Part 1: the detectors
|
||||
|
||||
1. **Measures before**: step `before`.
|
||||
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION>, range <FROM> → <TO> (UTC). Labels: <LABELS>/<station>.jsonl. Families: <FAMILIES>. Context: <CONTEXT>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
|
||||
2. **Upgrader**, a single pass: `Target: <DETECTORS>. Evidence: station <STATION>, range <FROM> → <TO> (UTC); labels <LABELS>/<station>.jsonl; families <FAMILIES>; context <CONTEXT>. Checks: events: <events> (before: run it); sensors: <sensors> (before: run it); range: <range score> (before: <OUT>/range-before.txt); benchmark: <benchmark> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md. INSTRUCTIONS: instructions/upgrader-detectors.md.`
|
||||
It commits each fix it keeps, message `improve: S<n>`, with its benchmark `<OUT>/benchmark-S<n>.txt`.
|
||||
Verdict `nothing to change`, or no fix commit (`git -C <DETECTORS> log --format=%h --grep ': S[0-9]*$' <base>..HEAD` empty): go to part 2.
|
||||
3. **Measures after**: step `after`.
|
||||
4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: <DETECTORS>. Families: <FAMILIES>. Modification: <OUT>/modification.md. Range score: <OUT>/range-before.txt → <OUT>/range-after.txt. Benchmark: <OUT>/benchmark-before.txt → <OUT>/benchmark-after.txt, after each fix: <OUT>/benchmark-S*.txt; command: <…>. Events command for any station: python3 <DETECTORS>/scripts/categories.py <station folder> --from <from> --to <to>. Base: <base>. Write <OUT>/validation.md.`
|
||||
4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Target: <DETECTORS>. Modification: <OUT>/modification.md. Base: <base>. Checks: range: <range score>; before: <OUT>/range-before.txt; after: <OUT>/range-after.txt; benchmark: <benchmark>; before: <OUT>/benchmark-before.txt; after: <OUT>/benchmark-after.txt; after each fix: <OUT>/benchmark-S*.txt; events, for any station: python3 <DETECTORS>/scripts/categories.py <station folder> --from <from> --to <to>. Families: <FAMILIES>. Output: <OUT>/validation.md. INSTRUCTIONS: instructions/validator-detectors.md.`
|
||||
- `keep`: go to part 2.
|
||||
- `remove`: for each fix of the `**Remove**:` line, its commits (`git -C <DETECTORS> log --format=%h --grep ': S<n>$' <base>..HEAD`), from the most recent: `git -C <DETECTORS> revert --no-edit <commit>`, then `git -C <DETECTORS> commit --amend -m "improve: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`. A conflict: `git -C <DETECTORS> revert --abort`, and do as for `cancel`.
|
||||
- `cancel`, or anything else: `git -C <DETECTORS> revert --no-commit <base>..HEAD && git -C <DETECTORS> commit -m "improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"` (only what the fix commits changed; files already modified stay as they are).
|
||||
@@ -62,7 +62,7 @@ An output missing or without a readable verdict: note it, put the detectors back
|
||||
|
||||
Always, whatever part 1 gave (a stop in part 1 does not stop part 2).
|
||||
|
||||
6. **Tool-builder**: `Analysis: <ANALYSIS>/. Tools: <TOOLS>/. Write <OUT>/tools.md.` It reads the analysis, writes only in `TOOLS` and in `<OUT>`, and commits in `TOOLS`. Its output missing: note it. Do not rerun it.
|
||||
6. **Upgrader on the tools**: `Target: <TOOLS>. Evidence: <ANALYSIS>/. Checks: none. Base: <tools base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/tools.md. INSTRUCTIONS: instructions/upgrader-tools.md.` It reads the analysis, writes only in `TOOLS` and in `<OUT>`, and commits in `TOOLS`, one commit per fix. Its output missing: note it. Do not rerun it.
|
||||
|
||||
## Summary
|
||||
|
||||
@@ -72,7 +72,7 @@ Always, whatever part 1 gave (a stop in part 1 does not stop part 2).
|
||||
# optimize: <station>, <FROM> → <TO>
|
||||
|
||||
**Result**: modification kept | kept without S<n>… | cancelled | none | error
|
||||
**Tools changed**: <the "Tools changed" line of <OUT>/tools.md> | error
|
||||
**Tools changed**: <the value of the `**Changed**` line of <OUT>/tools.md> | error
|
||||
|
||||
## The range
|
||||
the range score, before → end, one line per family (found, correct)
|
||||
@@ -96,7 +96,7 @@ the `Total` line of each family, before → end; then the other changed lines (s
|
||||
<copied as they are: "Tools" of the modification and of the validation (the scripts the agents kept in `<OUT>/tools/`); none>
|
||||
|
||||
## Errors
|
||||
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`; `git -C <TOOLS> log --oneline -1` if the tools changed>
|
||||
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`; `git -C <TOOLS> log --oneline <tools base>..HEAD` if the tools changed>
|
||||
```
|
||||
|
||||
## Answer
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# CLAUDE.md
|
||||
|
||||
Optimizer: the optimisation of the AIMsight detectors and of the reading tools, from one analysis of `labeling/auto`, and nothing else (`README.md`). The workflow: `.claude/skills/optimize/SKILL.md` (`/optimize`), launched from the root of Optimizer with the agents plugin (`--plugin-dir agents`); its agents `agents:upgrader`, `agents:validator` (the detectors) and `agents:tool-builder` (the tools); the benchmark `detectors/bench/`.
|
||||
Optimizer: the optimisation of the AIMsight detectors and of the reading tools, from one analysis of `labeling/auto`, and nothing else (`README.md`). The workflow: `.claude/skills/optimize/SKILL.md` (`/optimize`), launched from the root of Optimizer with the agents plugin (`--plugin-dir agents`); its agents `agents:upgrader`, `agents:validator` (the detectors) and `agents:upgrader` again (the tools), each with its file of `instructions/`; the benchmark `detectors/bench/`.
|
||||
|
||||
- **The links**, out of git: `detectors/` → `~/detectors` and `tools/` → `~/tools` (the session commits there); `data/` → `~/aimsight-data/data` and `labeling/` → `~/labeling` (read-only); `agents/` → `~/agents` (the plugin, loaded read-only).
|
||||
- **The sessions**: `sessions/<analysis>_<YYYYMMDDTHHMMZ>/`; Diagnostic-App reads their summary (`summary.md`).
|
||||
- **The workflow stays generic**: the skill describes only the workflow (analysis → detectors: upgrader → validator; tools: tool-builder → summary); what is specific to a caller (its paths, its git rules, `Caller's context`) arrives in the request.
|
||||
- **The workflow stays generic**: the skill describes only the workflow (analysis → detectors: upgrader → validator; tools: upgrader → summary); what is specific to a caller (its paths, its git rules, `Caller's context`) arrives in the request.
|
||||
- **Check a change of the skill**: `env -u ANTHROPIC_API_KEY claude -p "/optimize" --plugin-dir agents --model haiku` must say the inputs are missing and stop without touching `detectors/` or `tools/`; then a real session through the app's button.
|
||||
- `archive/`: the former batch and its scanner, stopped; do not run them again.
|
||||
|
||||
|
||||
@@ -3,10 +3,10 @@
|
||||
The optimisation of the AIMsight detectors and of the reading tools, from one analysis, and nothing else. An analysis (made in the Diagnostic-App app by the analyst) leaves in its folder the two things needed: its labels, the truth on its range, and its own scripts with what it missed. On that analysis, the `optimize` skill:
|
||||
|
||||
1. **the detectors**: measures their score on the range, and the benchmark on every label of `labeling/auto`; has them fixed by the `upgrader`, one fix per commit in `~/detectors`; has them judged by the `validator`, which keeps, has some removed, or cancels: no station goes back, no overfitting;
|
||||
2. **the tools**: has the `tool-builder` bring the analyst's useful scripts into the reading harness, commits in `~/tools`; always, whatever the detectors gave;
|
||||
2. **the tools**: has the `upgrader` bring the analyst's useful scripts into the reading harness, one fix per commit in `~/tools`; always, whatever the detectors gave;
|
||||
3. writes the summary of the session.
|
||||
|
||||
The `upgrader`, the `validator` and the `tool-builder` come from the `agents` repository; the benchmark (`bench/`) from the `detectors` repository. Nothing is pushed.
|
||||
The `upgrader` and the `validator` come from the `agents` repository, each with its instructions from `instructions/` (`upgrader-detectors.md`, `validator-detectors.md`, `upgrader-tools.md`); the rules of each target are its own `CLAUDE.md`; the benchmark (`bench/`) from the `detectors` repository. Nothing is pushed.
|
||||
|
||||
## Install
|
||||
|
||||
@@ -35,6 +35,6 @@ The station and the range come from the analysis's `request.json` (the range's l
|
||||
|
||||
## What a session leaves
|
||||
|
||||
`sessions/<analysis>_<time>/`: the measures before and after (`range-*.txt`, `benchmark-*.txt`), `modification.md` (the upgrader), `validation.md` (the validator), `tools.md` (the tool-builder) and its trials (`work/`), `summary.md` (the summary, shown by the app under the analysis report), `tools/` (the scripts the upgrader and validator wrote). The fixes: `improve: S<n>` commits in `~/detectors`, the tools a `Tools: <analysis>` commit in `~/tools`, each on its current branch.
|
||||
`sessions/<analysis>_<time>/`: the measures before and after (`range-*.txt`, `benchmark-*.txt`), `modification.md` (the upgrader), `validation.md` (the validator), `tools.md` (the upgrader on the tools) and its trials (`work/`), `summary.md` (the summary, shown by the app under the analysis report), `tools/` (the scripts the upgrader and validator wrote). The fixes: `improve: S<n>` commits in `~/detectors`, the tools `improve: S<n>` commits in `~/tools`, each on its current branch.
|
||||
|
||||
`archive/`: the former daily batch and its scanner (auto-labeling by the detectors), stopped on 2026-11-16. The architecture of the repositories: `~/Diagnostic-App/docs/architecture.drawio`.
|
||||
Reference in new issue
Block a user