Design: generic upgrader and validator, target rules in the target's CLAUDE.md, tool-builder folded into the upgrader
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2225ff5e86
commit
791ac80125
1 file changed
+128
@@ -0,0 +1,128 @@
|
||||
# Generic agents: design
|
||||
|
||||
Date: 2026-11-16. Status: approved by the user.
|
||||
|
||||
## Intent
|
||||
|
||||
`agents` is the core of the AIMsight processes: one copy of each agent, used by every process of the gitea repositories. An agent knows its trade and the AIMsight domain (stations, `labeling/`, `tools/`, the families); it knows nothing of the process that launches it. What is specific to a process reaches it through the request and an `INSTRUCTIONS` file, as the analyst already does with Diagnostic-App's `instructions/analyst.md`.
|
||||
|
||||
Today only the analyst works this way. The `upgrader` and the `validator` are generic in name only: they hard-code the detectors (the six detector files, `thresholds.md`, the benchmark, the 2-point regression rule, the `optimize` workflow). The `tool-builder` is an upgrader whose target is `tools/`, written separately.
|
||||
|
||||
## Goals
|
||||
|
||||
- Three agents, one trade each:
|
||||
- `analyst`: reads a time range of a station like the expert and labels it (unchanged);
|
||||
- `upgrader`: improves a **target** (a git repository) from evidence, without overfitting, one commit per fix;
|
||||
- `validator`: judges an upgrader's change, independently, by the checks' outputs (scripts) and by reading the code (LLM-as-judge); keep, remove some fixes, or cancel.
|
||||
- The rules of a target are written once, in the target's own `CLAUDE.md`, read by the upgrader, the validator and the user alike.
|
||||
- What is specific to a process lives in the caller's `INSTRUCTIONS` file.
|
||||
- `tool-builder` disappears: Optimizer's part 2 becomes the upgrader on `tools/` with its own instructions.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Support-AI (runs its agents through the API, out of scope by the user's decision).
|
||||
- Support-Agent and its agents (user's decision).
|
||||
- A `writer` agent (reports, mails, documents): the report stays asked of the analyst by Diagnostic-App's instructions until a second process needs a writer. *(Claude's call.)*
|
||||
- A validator on Optimizer's part 2 (the tools): none today, none added. *(Claude's call.)*
|
||||
- A validator judging something other than a git change (a labels file, a document): added when a process needs it.
|
||||
|
||||
## Anti-goals
|
||||
|
||||
- An agent that names a caller, a caller's path, its report format or its git rules.
|
||||
- A target rule written in two places (agent file and instructions, or upgrader and validator instructions).
|
||||
- A change of the detectors' or tools' behaviour: this is a move of text, the processes must behave as before.
|
||||
|
||||
## Constraints
|
||||
|
||||
From `CLAUDE.md` of this repository and the users of the agents: every time UTC; no rule specific to a station, a target or a date; never read `.env` nor copy `SIM_CCID`, `ip*`; `labeling/auto` only; commit by path, never `git add -A`; English; the agents run on the Claude subscription (`ANTHROPIC_API_KEY` removed); the plugin is loaded read-only, nothing is written in `agents/` by an agent. One copy of `agents/` on the workstation: a change here is live for Optimizer and Diagnostic-App at once.
|
||||
|
||||
## Approaches
|
||||
|
||||
1. **The rules in the target's repository** (picked by the user). `detectors/CLAUDE.md` and `tools/CLAUDE.md` carry what may and may not change in them; the caller's instructions carry only the process. Each rule is written once; a new target must first write its rules.
|
||||
2. **Everything in the caller's instructions.** One file per agent and process; the detectors' rules would be duplicated between the upgrader's and the validator's instructions, and again for a second caller.
|
||||
3. **Target profiles in `agents/`.** Rejected: `agents/` would know a specific repository, against this repository's `CLAUDE.md`.
|
||||
|
||||
The target's rules file is its existing `CLAUDE.md` rather than a new `RULES.md`: both already hold most of the rules. *(Claude's call.)*
|
||||
|
||||
## Precedence
|
||||
|
||||
For the upgrader and the validator, from strongest to weakest:
|
||||
1. the target's `CLAUDE.md`: what is forbidden in the target; instructions may narrow it (fewer files to touch), never loosen it;
|
||||
2. the agent's core: overfitting, one fix one commit, git safety (upgrader); independence, single pass (validator);
|
||||
3. `INSTRUCTIONS`: everything else (why, scope inside the target, extra sections, how to answer); it prevails over the rest of the agent file.
|
||||
|
||||
## The upgrader
|
||||
|
||||
**Request**: `Target: <repository>. Evidence: <files or folder>. Checks: <command: what it measures; output before: <file>>, … Base: <commit>. Files already modified: <list | none>. Output folder: <folder>. Write <folder>/<name>.md. INSTRUCTIONS: <file>.`
|
||||
|
||||
**Trade** (taken from today's upgrader and tool-builder, stripped of the detectors):
|
||||
- read `INSTRUCTIONS` in full first, then `<Target>/CLAUDE.md` in full, then the evidence;
|
||||
- diagnosis: what the target does wrong on the evidence, and which part of it causes it; most important first; a mechanism that comes back before an isolated case;
|
||||
- overfitting: the evidence is one sample; fix the mechanism, never fit the sample; what the target cannot see with its inputs goes to "Limits";
|
||||
- the simplest first: remove or simplify, change a constant or a default, adjust, add a short rule, add a new part as a last resort; a small gain is not worth a contraption; nothing to do is a decision;
|
||||
- checks: run each `Checks` command before and after each fix; output per fix in `<Output>/check-S<n>.txt`;
|
||||
- one fix one commit: `git -C <Target> commit -m "improve: S<n>" -- <paths>`; a fix not kept is reverted by `git checkout -- <its files>` only; never `git add -A`, `stash`, or touching a file listed as already modified;
|
||||
- its scripts in `<Output>/tools/`, never inline.
|
||||
|
||||
**Output**: the skeleton below; the instructions may add sections or lines.
|
||||
|
||||
```markdown
|
||||
# Modification
|
||||
|
||||
**Verdict**: changed | nothing to change
|
||||
**Changed**: none | <files, comma-separated>
|
||||
|
||||
## Diagnosis
|
||||
## What changed (per fix S<n> and its commit; stands alone, no data of the evidence)
|
||||
## Why (stands alone)
|
||||
## Checks (before → after, the lines that changed)
|
||||
## For the user to decide
|
||||
## Not fixed
|
||||
## Limits
|
||||
## Tools
|
||||
## Difficulties
|
||||
```
|
||||
|
||||
## The validator
|
||||
|
||||
**Request**: `Target: <repository>. Modification: <file>. Base: <commit>. Checks: <command; outputs before, after, after each fix>. Output: <file>. INSTRUCTIONS: <file>.`
|
||||
|
||||
**Trade** (today's validator, stripped of the detectors):
|
||||
- reads only: `INSTRUCTIONS`, `<Target>/CLAUDE.md`, "What changed" and "Why" of the modification, `git diff <base> HEAD` and the code it touches, the per-fix commits, the checks' outputs (may rerun them), and what the instructions add; never the diagnosis;
|
||||
- judges: the target still loads/runs (as its `CLAUDE.md` or the instructions define it); no regression (the definition comes from the target's `CLAUDE.md`); the gain justifies the code; no overfitting; the target's rules respected;
|
||||
- single pass, no sending back; do not cancel for perfection.
|
||||
|
||||
**Output**: `**Verdict**: keep | remove | cancel`, `**Remove**: S<n>, …` (with `remove`), then `## Answers`, `## Checks`, `## Reservations`, `## Tools`. The instructions may add sections.
|
||||
|
||||
## The analyst
|
||||
|
||||
Unchanged. Its README line stays.
|
||||
|
||||
## What moves where
|
||||
|
||||
| Today in | Moves to |
|
||||
|---|---|
|
||||
| `upgrader.md`: six detector files, `detect()` contract, episode keys, `thresholds.md` update, "everything is detected", stdlib | `detectors/CLAUDE.md` (most already there; add "everything is detected" and the regression definition) |
|
||||
| `upgrader.md` / `validator.md`: regression = "found" or "correct" loses > 2 points on a family of a station, and what is not a regression | `detectors/CLAUDE.md`, under the benchmark rule |
|
||||
| `upgrader.md`: labels format, range score, benchmark, `--stations`, section "Benchmark", "Before / after on the range" | `Optimizer/instructions/upgrader-detectors.md` |
|
||||
| `validator.md`: range score, events command to explain a regression, base archive trick | `Optimizer/instructions/validator-detectors.md` |
|
||||
| `tool-builder.md`: never remove a tool, option, output or guide sentence; a new tool enters the guide | `tools/CLAUDE.md` (partly there) |
|
||||
| `tool-builder.md`: read the analysis (`report.md`, `analysis.md`, `scripts/`), trial on the range, rerun the report's commands with `--out`, the `data.py` line, one commit `Tools: <analysis>` | `Optimizer/instructions/upgrader-tools.md` (one commit per fix replaces the single commit) |
|
||||
| `optimize` skill: request of each agent | the generic requests above, with `INSTRUCTIONS: instructions/<…>.md`; part 2 calls `agents:upgrader`; the summary reads `**Changed**` instead of `**Tools changed**` |
|
||||
|
||||
Then `agents/tool-builder.md` is deleted; README table, `plugin.json` description and the references in Optimizer and Diagnostic-App READMEs updated in the same change.
|
||||
|
||||
## Order
|
||||
|
||||
The plugin is live for every caller: the Optimizer skill and its instructions land with the new agents, in one session, before anyone runs `/optimize`. `detectors/` and `tools/` rules first (additive), then Optimizer's instructions, then the agents and the skill together.
|
||||
|
||||
## Checks
|
||||
|
||||
- Each agent loads: `claude -p "<without reading or running anything, list the fields of your request and the order of precedence>" --plugin-dir ~/agents --agent agents:<name> --model haiku`.
|
||||
- Optimizer: `env -u ANTHROPIC_API_KEY claude -p "/optimize" --plugin-dir agents --model haiku` stops on missing inputs, touching nothing.
|
||||
- A real `/optimize` on an existing analysis: `modification.md`, `validation.md`, `tools.md` each have a readable verdict line, `summary.md` is filled, commits in `detectors/` and `tools/` follow `improve: S<n>`.
|
||||
- `grep -nE "visibility.py|thresholds.md|benchmark|regression.py" agents/agents/*.md` returns only the analyst's mentions, if any.
|
||||
|
||||
## Left to the builder
|
||||
|
||||
The wording of each agent file; how the summary maps the new sections; the exact names of the instruction files.
|
||||
Reference in new issue
Block a user