diff --git a/docs/superpowers/specs/2026-11-16-generic-agents-design.md b/docs/superpowers/specs/2026-11-16-generic-agents-design.md new file mode 100644 index 0000000..a3858fb --- /dev/null +++ b/docs/superpowers/specs/2026-11-16-generic-agents-design.md @@ -0,0 +1,128 @@ +# Generic agents: design + +Date: 2026-11-16. Status: approved by the user. + +## Intent + +`agents` is the core of the AIMsight processes: one copy of each agent, used by every process of the gitea repositories. An agent knows its trade and the AIMsight domain (stations, `labeling/`, `tools/`, the families); it knows nothing of the process that launches it. What is specific to a process reaches it through the request and an `INSTRUCTIONS` file, as the analyst already does with Diagnostic-App's `instructions/analyst.md`. + +Today only the analyst works this way. The `upgrader` and the `validator` are generic in name only: they hard-code the detectors (the six detector files, `thresholds.md`, the benchmark, the 2-point regression rule, the `optimize` workflow). The `tool-builder` is an upgrader whose target is `tools/`, written separately. + +## Goals + +- Three agents, one trade each: + - `analyst`: reads a time range of a station like the expert and labels it (unchanged); + - `upgrader`: improves a **target** (a git repository) from evidence, without overfitting, one commit per fix; + - `validator`: judges an upgrader's change, independently, by the checks' outputs (scripts) and by reading the code (LLM-as-judge); keep, remove some fixes, or cancel. +- The rules of a target are written once, in the target's own `CLAUDE.md`, read by the upgrader, the validator and the user alike. +- What is specific to a process lives in the caller's `INSTRUCTIONS` file. +- `tool-builder` disappears: Optimizer's part 2 becomes the upgrader on `tools/` with its own instructions. + +## Non-goals + +- Support-AI (runs its agents through the API, out of scope by the user's decision). +- Support-Agent and its agents (user's decision). +- A `writer` agent (reports, mails, documents): the report stays asked of the analyst by Diagnostic-App's instructions until a second process needs a writer. *(Claude's call.)* +- A validator on Optimizer's part 2 (the tools): none today, none added. *(Claude's call.)* +- A validator judging something other than a git change (a labels file, a document): added when a process needs it. + +## Anti-goals + +- An agent that names a caller, a caller's path, its report format or its git rules. +- A target rule written in two places (agent file and instructions, or upgrader and validator instructions). +- A change of the detectors' or tools' behaviour: this is a move of text, the processes must behave as before. + +## Constraints + +From `CLAUDE.md` of this repository and the users of the agents: every time UTC; no rule specific to a station, a target or a date; never read `.env` nor copy `SIM_CCID`, `ip*`; `labeling/auto` only; commit by path, never `git add -A`; English; the agents run on the Claude subscription (`ANTHROPIC_API_KEY` removed); the plugin is loaded read-only, nothing is written in `agents/` by an agent. One copy of `agents/` on the workstation: a change here is live for Optimizer and Diagnostic-App at once. + +## Approaches + +1. **The rules in the target's repository** (picked by the user). `detectors/CLAUDE.md` and `tools/CLAUDE.md` carry what may and may not change in them; the caller's instructions carry only the process. Each rule is written once; a new target must first write its rules. +2. **Everything in the caller's instructions.** One file per agent and process; the detectors' rules would be duplicated between the upgrader's and the validator's instructions, and again for a second caller. +3. **Target profiles in `agents/`.** Rejected: `agents/` would know a specific repository, against this repository's `CLAUDE.md`. + +The target's rules file is its existing `CLAUDE.md` rather than a new `RULES.md`: both already hold most of the rules. *(Claude's call.)* + +## Precedence + +For the upgrader and the validator, from strongest to weakest: +1. the target's `CLAUDE.md`: what is forbidden in the target; instructions may narrow it (fewer files to touch), never loosen it; +2. the agent's core: overfitting, one fix one commit, git safety (upgrader); independence, single pass (validator); +3. `INSTRUCTIONS`: everything else (why, scope inside the target, extra sections, how to answer); it prevails over the rest of the agent file. + +## The upgrader + +**Request**: `Target: . Evidence: . Checks: >, … Base: . Files already modified: . Output folder: . Write /.md. INSTRUCTIONS: .` + +**Trade** (taken from today's upgrader and tool-builder, stripped of the detectors): +- read `INSTRUCTIONS` in full first, then `/CLAUDE.md` in full, then the evidence; +- diagnosis: what the target does wrong on the evidence, and which part of it causes it; most important first; a mechanism that comes back before an isolated case; +- overfitting: the evidence is one sample; fix the mechanism, never fit the sample; what the target cannot see with its inputs goes to "Limits"; +- the simplest first: remove or simplify, change a constant or a default, adjust, add a short rule, add a new part as a last resort; a small gain is not worth a contraption; nothing to do is a decision; +- checks: run each `Checks` command before and after each fix; output per fix in `/check-S.txt`; +- one fix one commit: `git -C commit -m "improve: S" -- `; a fix not kept is reverted by `git checkout -- ` only; never `git add -A`, `stash`, or touching a file listed as already modified; +- its scripts in `/tools/`, never inline. + +**Output**: the skeleton below; the instructions may add sections or lines. + +```markdown +# Modification + +**Verdict**: changed | nothing to change +**Changed**: none | + +## Diagnosis +## What changed (per fix S and its commit; stands alone, no data of the evidence) +## Why (stands alone) +## Checks (before → after, the lines that changed) +## For the user to decide +## Not fixed +## Limits +## Tools +## Difficulties +``` + +## The validator + +**Request**: `Target: . Modification: . Base: . Checks: . Output: . INSTRUCTIONS: .` + +**Trade** (today's validator, stripped of the detectors): +- reads only: `INSTRUCTIONS`, `/CLAUDE.md`, "What changed" and "Why" of the modification, `git diff HEAD` and the code it touches, the per-fix commits, the checks' outputs (may rerun them), and what the instructions add; never the diagnosis; +- judges: the target still loads/runs (as its `CLAUDE.md` or the instructions define it); no regression (the definition comes from the target's `CLAUDE.md`); the gain justifies the code; no overfitting; the target's rules respected; +- single pass, no sending back; do not cancel for perfection. + +**Output**: `**Verdict**: keep | remove | cancel`, `**Remove**: S, …` (with `remove`), then `## Answers`, `## Checks`, `## Reservations`, `## Tools`. The instructions may add sections. + +## The analyst + +Unchanged. Its README line stays. + +## What moves where + +| Today in | Moves to | +|---|---| +| `upgrader.md`: six detector files, `detect()` contract, episode keys, `thresholds.md` update, "everything is detected", stdlib | `detectors/CLAUDE.md` (most already there; add "everything is detected" and the regression definition) | +| `upgrader.md` / `validator.md`: regression = "found" or "correct" loses > 2 points on a family of a station, and what is not a regression | `detectors/CLAUDE.md`, under the benchmark rule | +| `upgrader.md`: labels format, range score, benchmark, `--stations`, section "Benchmark", "Before / after on the range" | `Optimizer/instructions/upgrader-detectors.md` | +| `validator.md`: range score, events command to explain a regression, base archive trick | `Optimizer/instructions/validator-detectors.md` | +| `tool-builder.md`: never remove a tool, option, output or guide sentence; a new tool enters the guide | `tools/CLAUDE.md` (partly there) | +| `tool-builder.md`: read the analysis (`report.md`, `analysis.md`, `scripts/`), trial on the range, rerun the report's commands with `--out`, the `data.py` line, one commit `Tools: ` | `Optimizer/instructions/upgrader-tools.md` (one commit per fix replaces the single commit) | +| `optimize` skill: request of each agent | the generic requests above, with `INSTRUCTIONS: instructions/<…>.md`; part 2 calls `agents:upgrader`; the summary reads `**Changed**` instead of `**Tools changed**` | + +Then `agents/tool-builder.md` is deleted; README table, `plugin.json` description and the references in Optimizer and Diagnostic-App READMEs updated in the same change. + +## Order + +The plugin is live for every caller: the Optimizer skill and its instructions land with the new agents, in one session, before anyone runs `/optimize`. `detectors/` and `tools/` rules first (additive), then Optimizer's instructions, then the agents and the skill together. + +## Checks + +- Each agent loads: `claude -p "" --plugin-dir ~/agents --agent agents: --model haiku`. +- Optimizer: `env -u ANTHROPIC_API_KEY claude -p "/optimize" --plugin-dir agents --model haiku` stops on missing inputs, touching nothing. +- A real `/optimize` on an existing analysis: `modification.md`, `validation.md`, `tools.md` each have a readable verdict line, `summary.md` is filled, commits in `detectors/` and `tools/` follow `improve: S`. +- `grep -nE "visibility.py|thresholds.md|benchmark|regression.py" agents/agents/*.md` returns only the analyst's mentions, if any. + +## Left to the builder + +The wording of each agent file; how the summary maps the new sections; the exact names of the instruction files.