Optimize: a single /optimize call in Optimizer, the detectors and the tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-16 14:15:55 +01:00
1 parent 91cd39525e
commit beaa248687
6 files changed
+47 -55

No files matched your search

+2 -2
View File
@@ -2,10 +2,10 @@
Diagnostic-app: the control station of the AIMsight diagnostic (`README.md`): the web app, and the launching of processes whose code lives elsewhere. A camera follows targets on a structure; the detectors read the displacement series of the stations and flag the moments when the curves must not be trusted (rain, fog, spider web, turbulence…) or a real movement. The architecture of the repositories: `docs/architecture.drawio`.
- **The processes**: the Analysis tab launches `agents:analyst` (plugin `agents/`) on a range, with `instructions/analyst.md` and `DETECTORS: no` (labels and report in `labeling/auto/analyses/<STATION>_<start>_<end>/`, submitted to `labeling/auto/<STATION>.jsonl`); "Optimize the detectors" launches `/improve` in `optimizer/` (its request in `app/server/analysis.py`, its summary in `optimizer/sessions/<analysis>_<time>/summary.md`), which commits in `detectors/`, then `agents:tool-builder`, which commits in `tools/`.
- **The processes**: the Analysis tab launches `agents:analyst` (plugin `agents/`) on a range, with `instructions/analyst.md` and `DETECTORS: no` (labels and report in `labeling/auto/analyses/<STATION>_<start>_<end>/`, submitted to `labeling/auto/<STATION>.jsonl`); "Optimize the detectors" launches `/optimize` in `optimizer/` on the analysis (its request in `app/server/analysis.py`, its summary in `optimizer/sessions/<analysis>_<time>/summary.md`), which commits the detectors in `detectors/` and the reading tools in `tools/`.
- **The links** (ignored by git, one copy of each repository on the workstation): `data/`, `labeling/`, `detectors/`, `tools/`, `agents/`, `optimizer/`; each thing is committed in its own repository, never here. This repository carries no agent, no reading tool, no workflow.
- **The truth**: the labels of `labeling/auto/`; the labeling knowledge is in `labeling/`.
- **The contracts**: with the detectors, `detectors/CLAUDE.md` (`categories.events`, `bench/score.py`); with the agents, `agents/README.md` (the requests of the analyst and of the tool-builder); with Optimizer, `optimizer/README.md` (the request of `/improve`). A renaming over there requires the change here. The Data page passes to the detectors the measurements read by `tools/scripts/data.py`, whereas the benchmark and Optimizer read through `detectors/scripts/data.py`: a reading fix in one is carried over to the other.
- **The contracts**: with the detectors, `detectors/CLAUDE.md` (`categories.events`, `bench/score.py`); with the agents, `agents/README.md` (the request of the analyst); with Optimizer, `optimizer/README.md` (the request of `/optimize`). A renaming over there requires the change here. The Data page passes to the detectors the measurements read by `tools/scripts/data.py`, whereas the benchmark and Optimizer read through `detectors/scripts/data.py`: a reading fix in one is carried over to the other.
- **The benchmark**: `python3 detectors/bench/regression.py --labels labeling/auto --data data` (~3 min); the score of a range: `python3 detectors/bench/score.py data/<STATION> --labels labeling/auto/<STATION>.jsonl --from YYYY-MM-DD --to YYYY-MM-DD` (`--to` excluded).
- **The pumping**: in `~/aimsight-data` (`extraction/pump.sh`, VS Code tasks **pump**, **pump a range**), read-only on the platform; user guide `~/aimsight-data/extraction/README.md`. Claude may launch it on small ranges; a long range or a history, outside busy hours.
- **Check a change of the app**: `.venv/bin/python app/server/check.py`, then `cd app/web && npm run build` and launch the server. The labels: `python3 labeling/validate.py`.
+3 -3
View File
@@ -3,7 +3,7 @@
The control station of the AIMsight diagnostic: see, analyze, launch. The Data page shows the curves, the camera images, the labels and the detectors' events; its Analysis tab launches, by `claude -p`, processes whose code lives in the other repositories:
- **"Analyze the range"**: the `analyst` of the agents repository, on a chosen range, with the app's instructions (`instructions/analyst.md`) and without the detectors (`DETECTORS: no`): labels, submitted to `labeling/auto/`, and a diagnostic report;
- **"Optimize the detectors"**, on a finished analysis: `/improve` in Optimizer improves the detectors on the range of the analysis, its labels as truth, without overfitting; then the `tool-builder` of the agents repository brings the analyst's scripts into the reading harness (tools repository).
- **"Optimize the detectors"**, on a finished analysis: `/optimize` in Optimizer, on the analysis: it improves the detectors on the range of the analysis, its labels as truth, without overfitting; then it brings the analyst's scripts into the reading harness (tools repository).
The architecture of the repositories, one responsibility each: `docs/architecture.drawio`.
@@ -22,9 +22,9 @@ And six links, ignored by git, to the neighbouring repositories (a single copy o
| `detectors/` | `~/detectors` | reads the detectors' events (`scripts/categories.py`) |
| `tools/` | `~/tools` | reads the data through `tools/scripts/data.py`; the analyst uses it, the tool-builder writes in it |
| `agents/` | `~/agents` | the plugin of the agents (`--plugin-dir agents`, `--agent agents:analyst`) |
| `optimizer/` | `~/Optimizer` | launches `/improve` there; reads the summaries in `optimizer/sessions/` |
| `optimizer/` | `~/Optimizer` | launches `/optimize` there; reads the summaries in `optimizer/sessions/` |
Each thing is committed in its own repository: the labels in `~/labeling`, the detectors in `~/detectors` (the optimization session does it, without pushing), the tools in `~/tools` (the tool-builder does it).
Each thing is committed in its own repository: the labels in `~/labeling`, the detectors in `~/detectors` (the optimization session does it, without pushing), the tools in `~/tools` (the same session, through the tool-builder).
## Install and launch
+2 -2
View File
@@ -21,7 +21,7 @@ The address keeps the station and the range (UTC, `end` excluded): `#/data?stati
- **Analyze the range**: a range selected, optional hints, then `server/analysis.py` launches the analyst of the agents repository by `claude -p` (non-interactive), with its instructions (`instructions/analyst.md`) and without the detectors (`DETECTORS: no`). It writes in `labeling/auto/analyses/<STATION>_<start>_<end>/` its labels and a diagnostic report (`report.md` and its figures; its work and claude's log in `work/`, outside git). The page follows its progress, then shows the report; the server submits its labels to `labeling/auto/<STATION>.jsonl` (field `analysis`). "View the report" on such a label opens it again.
- **The sidebar** sorts the analyses of `labeling/auto/analyses/` into folders, one per station (open by default if it has one running), and inside one analysis per range, the most recent first, with its state; a click opens Data on it.
- **The messages**: after the report, the user writes to the analyst, the same session resumes (`--resume`). An analysis stops with the server ("interrupted"); "continue" in the message field resumes it.
- **Optimize the detectors**, on a finished analysis: `/improve` in Optimizer (`optimizer/`, with the plugin `agents/`) on the range of the analysis, then the `tool-builder` of the agents repository; the detectors committed in `detectors/`, the reading tools in `tools/`, each on its current branch. The page follows the thread of the session (`work/optimization.jsonl`), then shows its summary under the report (`optimizer/sessions/<analysis>_<time>/summary.md`, then `<analysis>/tools.md`). One optimization at a time; meanwhile, no analysis and no message to the analyst; an optimization cut by the server stopping is rerun with the button. The detectors' events are kept in memory until the end of an optimization: after an `/improve` run outside the app, restart the server.
- **Optimize the detectors**, on a finished analysis: `/optimize` in Optimizer (`optimizer/`, with the plugin `agents/`) on the analysis: the detectors, then the reading tools; the detectors committed in `detectors/`, the reading tools in `tools/`, each on its current branch. The page follows the thread of the session (`work/optimization.jsonl`), then shows its summary under the report (`optimizer/sessions/<analysis>_<time>/summary.md`, and the `<analysis>/tools.md` of an older session). One optimization at a time; meanwhile, no analysis and no message to the analyst; an optimization cut by the server stopping is rerun with the button. The detectors' events are kept in memory until the end of an optimization: after an `/optimize` run outside the app, restart the server.
## Launch
@@ -38,7 +38,7 @@ The page opens on `http://127.0.0.1:8770`. Settings by variables: `AIMSIGHT_DATA
- `server/server.py`: FastAPI; serves the built page and the API under `/api` (stations, images, curves, detectors, labels, analyses); answers only the pages of the workstation (local Host, declared JSON).
- `server/station.py`: the data of the Data page; read-only on `data/` (through `tools/scripts/data.py`), writes only in `labeling/{auto,manual}/`.
- `server/analysis.py`: the Analysis tab; launches `claude` with the agent `agents:analyst` (Read, Write, Bash allowed without asking); and the optimization, `/improve` in Optimizer (Read, Write, Edit, Bash, Agent) then the agent `agents:tool-builder` (Read, Write, Edit, Bash).
- `server/analysis.py`: the Analysis tab; launches `claude` with the agent `agents:analyst` (Read, Write, Bash allowed without asking); and the optimization, `/optimize` in Optimizer (Read, Write, Edit, Bash, Agent).
- `server/check.py`: checks, without calling claude: `.venv/bin/python app/server/check.py`.
- `web/`: React (Vite) and Mantine; `src/App.jsx` (header, sidebar, address), `src/pages/Data.jsx` (the page), `src/pages/Images.jsx` (the images, in Data), `src/common.jsx` (requests, selector), `src/main.jsx` (theme). Plotly, dressed in Mantine's colours, loads only with Data. `dist/` and `node_modules/` outside git.
+29 -36
View File
@@ -5,17 +5,17 @@ labeling/auto/analyses/<STATION>_<start>_<end>/ its labels (labels.jsonl) and a
next to it); at the end, the server submits its labels to labeling/auto/<STATION>.jsonl, marked with their analysis (field
`analysis`). Then the user can write to it: the same session resumes (`claude -p --resume`), and its labels, if they
change, replace those it had submitted.
On a finished analysis, "Optimize" launches `/improve` in Optimizer (optimizer/, the skill that improves the detectors on the
range of the analysis, its labels as truth), then the tool-builder of the agents repository, which brings the analyst's
scripts into the reading tools (tools/); one optimization at a time, no analysis or message meanwhile; the summary of the
session (optimizer/sessions/<analysis>_<time>/summary.md) and that of the tools (<analysis>/tools.md) show under the report.
On a finished analysis, "Optimize" launches `/optimize` in Optimizer (optimizer/, the skill that, from the analysis, improves
the detectors on its range, its labels as truth, then brings the analyst's scripts into the reading tools, tools/); one
optimization at a time, no analysis or message meanwhile; the summary of the session
(optimizer/sessions/<analysis>_<time>/summary.md, and the <analysis>/tools.md of an older session) shows under the report.
"""
import html
import json
import os
import subprocess
import threading
from datetime import date, datetime, timedelta, timezone
from datetime import datetime, timezone
import markdown
@@ -24,13 +24,13 @@ import station
FOLDER = station.LABELS / "auto" / "analyses"
DETECTORS = station.ROOT / "detectors"
AGENTS = station.ROOT / "agents" # the plugin of the generic agents (~/agents)
OPTIMIZER = station.ROOT / "optimizer" # the Optimizer repository (~/Optimizer), where /improve runs
OPTIMIZER = station.ROOT / "optimizer" # the Optimizer repository (~/Optimizer), where /optimize runs
SESSIONS = OPTIMIZER / "sessions"
# the request of /improve (optimizer/README.md), paths relative to the root of Optimizer (its links data, labeling, detectors)
IMPROVE = """/improve DETECTORS: detectors/. STATION: data/{station}. FROM: {start}. TO: {end}. \
LABELS: labeling/auto/{station}.jsonl. REFERENCE: --labels labeling/auto --data data. FAMILIES: labeling/families.md. \
CONTEXT: labeling/auto/analyses/{name}/report.md. OUT: sessions/{name}_{time}/. Caller's context: Diagnostic-app, through \
Optimizer; read-only on data/ and labeling/; commit only in detectors/, on its current branch; do not commit OUT."""
# the request of /optimize (optimizer/README.md), paths relative to the root of Optimizer (its links); Optimizer reads the
# station and the range in the analysis
OPTIMIZE = """/optimize ANALYSIS: labeling/auto/analyses/{name}/. OUT: sessions/{name}_{time}/. Caller's context: \
Diagnostic-app, through Optimizer; read-only on data/ and labeling/; commit only in detectors/ and tools/, on their current \
branch; do not commit OUT."""
REQUEST = """Station: data/{station}/. Range: {start} → {end} (UTC). Hints: {hints}. \
Output folder: {out}/. DETECTORS: no. INSTRUCTIONS: instructions/analyst.md."""
@@ -229,7 +229,7 @@ def report(name, dark):
return _page(file(name, "report.md").read_text(encoding="utf-8"), name, dark)
# --- "Optimize": /improve in Optimizer on the analysis, then the tool-builder, by claude -p
# --- "Optimize": /optimize in Optimizer on the analysis, by claude -p
def _summary_file(name):
"""The summary of the last session of this analysis, or None. The time pattern rules out the sessions of another
@@ -258,8 +258,8 @@ def optimize(name):
"""The "Optimize" button on a finished analysis: the session in a thread; refused if an optimization or an analysis runs."""
if state(name)["state"] != "done":
raise station.Refusal("only a finished analysis can be optimized")
if not ((OPTIMIZER / ".claude" / "skills" / "improve" / "SKILL.md").exists()
and all((OPTIMIZER / link).exists() for link in ("data", "labeling", "detectors"))
if not ((OPTIMIZER / ".claude" / "skills" / "optimize" / "SKILL.md").exists()
and all((OPTIMIZER / link).exists() for link in ("data", "labeling", "detectors", "tools"))
and (AGENTS / ".claude-plugin" / "plugin.json").exists()):
raise station.Refusal("no Optimizer in optimizer/ or no agents in agents/ (the links: README.md)")
with lock:
@@ -273,32 +273,25 @@ def optimize(name):
def optimize_thread(name):
"""The two turns of the optimization, one after the other: the detectors (/improve, in Optimizer), then the reading
tools (the tool-builder, even if the detectors did not change)."""
"""The optimization: /optimize in Optimizer, on the analysis (the detectors, then the reading tools)."""
o, out = optimizations[name], FOLDER / name
rel = os.path.relpath(out, station.ROOT)
(out / "work").mkdir(exist_ok=True)
with open(out / "work" / "optimization.jsonl", "a", encoding="utf-8") as stream:
try:
d = json.loads((out / "request.json").read_text(encoding="utf-8"))
request = IMPROVE.format( # TO excluded: the day after the end of the range, as in the page
station=d["station"], start=d["start"][:10], end=date.fromisoformat(d["end"][:10]) + timedelta(days=1),
name=name, time=f"{datetime.now(timezone.utc):%Y%m%dT%H%MZ}")
request = OPTIMIZE.format(name=name, time=f"{datetime.now(timezone.utc):%Y%m%dT%H%MZ}")
# ponytail: Optimizer reads data/ and labeling/ through its own links; AIMSIGHT_DATA / AIMSIGHT_LABELS do not follow
turns = [(OPTIMIZER.resolve(), [request, "--plugin-dir", str(AGENTS.resolve()),
"--add-dir", str(DETECTORS.resolve()), # the target of the link, where the session writes
"--allowedTools", "Read,Write,Edit,Bash,Agent"]),
(station.ROOT, [f"Analysis: {rel}/. Tools: tools/. Write {rel}/tools.md.", "--plugin-dir", str(AGENTS),
"--agent", "agents:tool-builder", "--allowedTools", "Read,Write,Edit,Bash"])]
for cwd, (prompt, *options) in turns:
p = subprocess.Popen(["claude", "-p", prompt, *ISOLATED, "--output-format", "stream-json", "--verbose",
"--permission-mode", "acceptEdits", *station.links(), *options],
cwd=cwd, env=_env(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
for line in p.stdout:
stream.write(line)
o["lines"] += digest(line)
if p.wait():
raise RuntimeError(f"claude ended with code {p.returncode}")
p = subprocess.Popen(["claude", "-p", request, *ISOLATED, "--output-format", "stream-json", "--verbose",
"--permission-mode", "acceptEdits", *station.links(), # tools/ among them, where it writes
"--plugin-dir", str(AGENTS.resolve()),
"--add-dir", str(DETECTORS.resolve()), # the target of the link, where the session writes
"--allowedTools", "Read,Write,Edit,Bash,Agent"],
cwd=OPTIMIZER.resolve(), env=_env(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
text=True)
for line in p.stdout:
stream.write(line)
o["lines"] += digest(line)
if p.wait():
raise RuntimeError(f"claude ended with code {p.returncode}")
o["state"] = "done"
except Exception as e: # noqa: BLE001 — the error shows in the page, the server goes on
o["lines"].append(f"Error: {e}")
@@ -309,7 +302,7 @@ def optimize_thread(name):
def summary(name, dark):
"""The summary of the detectors' session, then that of the tools (<analysis>/tools.md), in one page."""
"""The summary of the session (the detectors and the tools), then the <analysis>/tools.md of an older session."""
s, tools = _summary_file(name), FOLDER / name / "tools.md"
if s is None and not tools.exists():
raise station.Refusal("no summary for this analysis")
+10 -11
View File
@@ -134,14 +134,16 @@ def check_optimization():
assert analysis.state("NORDEND_a")["optimization"] is None
# without Optimizer, then without agents: a clear refusal, not an error trace
assert "optimizer/" in refuses(analysis.optimize, "NORDEND_a")
(analysis.OPTIMIZER / ".claude" / "skills" / "improve").mkdir(parents=True)
(analysis.OPTIMIZER / ".claude" / "skills" / "improve" / "SKILL.md").write_text("improve")
(analysis.OPTIMIZER / ".claude" / "skills" / "optimize").mkdir(parents=True)
(analysis.OPTIMIZER / ".claude" / "skills" / "optimize" / "SKILL.md").write_text("optimize")
assert "agents/" in refuses(analysis.optimize, "NORDEND_a")
(analysis.AGENTS / ".claude-plugin").mkdir(parents=True)
(analysis.AGENTS / ".claude-plugin" / "plugin.json").write_text("{}")
assert "optimizer/" in refuses(analysis.optimize, "NORDEND_a") # without the data, labeling, detectors links of Optimizer
assert "optimizer/" in refuses(analysis.optimize, "NORDEND_a") # without the links of Optimizer
for link in ("data", "labeling", "detectors"):
(analysis.OPTIMIZER / link).mkdir()
assert "optimizer/" in refuses(analysis.optimize, "NORDEND_a") # without its tools link
(analysis.OPTIMIZER / "tools").mkdir()
assert analysis.optimize("NORDEND_a") == {}
assert analysis.state("NORDEND_a")["optimization"]["state"] == "running"
refuses(analysis.optimize, "NORDEND_a")
@@ -152,18 +154,15 @@ def check_optimization():
while analysis.state("NORDEND_a")["optimization"]["state"] == "running":
time.sleep(0.05)
assert analysis.state("NORDEND_a")["optimization"]["state"] == "done"
args, k = launched[0] # /improve in Optimizer, on the range of the analysis, TO excluded
assert len(launched) == 1, launched # a single call: /optimize in Optimizer, on the analysis
args, k = launched[0]
request = args[2]
assert request.startswith("/improve ") and "STATION: data/NORDEND. FROM: 2026-11-04. TO: 2026-11-08." in request \
and "OUT: sessions/NORDEND_a_" in request and "FAMILIES: labeling/families.md." in request \
and "CONTEXT: labeling/auto/analyses/NORDEND_a/report.md." in request, request
assert request.startswith("/optimize ANALYSIS: labeling/auto/analyses/NORDEND_a/. OUT: sessions/NORDEND_a_") \
and "Caller's context:" in request, request
assert k["cwd"] == analysis.OPTIMIZER.resolve() and "--plugin-dir" in args, (args, k)
assert all("ANTHROPIC_API_KEY" not in kw["env"] for _, kw in launched), "an API key reaches claude"
# isolated from the settings of the workstation (the user's plugins, hooks, MCP servers)
assert all(a[a.index("--setting-sources") + 1] == "project,local" and "--strict-mcp-config" in a for a, _ in launched)
args, k = launched[1] # then the tool-builder of the agents repository, on the tools repository
assert args[args.index("--agent") + 1] == "agents:tool-builder" and "Tools: tools/" in args[2] \
and args[2].endswith("/tools.md."), args
analysis.analyses["NORDEND_b"]["state"] = "running" # a running analysis blocks the optimization
refuses(analysis.optimize, "NORDEND_a")
# after a restart: the stream ends with {"type": "end"} -> done; without that line -> interrupted; the summary of
@@ -177,7 +176,7 @@ def check_optimization():
analysis.optimizations.clear()
assert analysis.state("NORDEND_a")["optimization"]["state"] == "interrupted"
(analysis.SESSIONS / "NORDEND_a_20261113T1000Z").mkdir(parents=True)
(analysis.SESSIONS / "NORDEND_a_20261113T1000Z" / "summary.md").write_text("# improve\n\n**Result**: none")
(analysis.SESSIONS / "NORDEND_a_20261113T1000Z" / "summary.md").write_text("# optimize\n\n**Result**: none")
# a new analysis of the same range (NORDEND_a_143000): its session is not the summary of NORDEND_a
(analysis.SESSIONS / "NORDEND_a_143000_20261113T1100Z").mkdir(parents=True)
(analysis.SESSIONS / "NORDEND_a_143000_20261113T1100Z" / "summary.md").write_text("# other analysis")
+1 -1
View File
@@ -187,7 +187,7 @@ def curves(station, start, end):
HISTORY = timedelta(days=3) # days before given to the detectors, as categories.py does (rule of 2026-10-24)
# ponytail: kept until the server restarts (a minute or more per range); restart it after an /improve run outside the app
# ponytail: kept until the server restarts (a minute or more per range); restart it after an /optimize run outside the app
@functools.lru_cache(maxsize=32)
def detectors(station, start, end, targets=""):
"""-> the detectors' events over the range, in the label format (start, end, family, description); plus kind,