wf-improve tools in skills/wf-improve/scripts/

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-13 12:31:22 +01:00
1 parent 6df274f580
commit 80e00a4898
8 files changed
+21 -22

No files matched your search

+8 -8
View File
@@ -6,7 +6,7 @@ argument-hint: "<analysis>"
# wf-improve: the detectors from an analysis
You are the orchestrator. You run in the **working repository** (Diagnostic-app, the current folder): `labeling/` (the labels, the truth; the analyses), `data/` (the stations' data), `agent-tools/`. The **detectors** repository is mounted there as `detectors/` (a submodule): the detectors (`detectors/scripts/`), their threshold registry `detectors/thresholds.md`, the session tools (`detectors/scripts/wf-improve/`) and the sessions (`detectors/wf-improve/`). Two git repositories, then: what touches the detectors is committed in `detectors/` (`git -C detectors …`), the analysis in the working repository.
You are the orchestrator. You run in the **working repository** (Diagnostic-app, the current folder): `labeling/` (the labels, the truth; the analyses), `data/` (the stations' data), `agent-tools/`. The **detectors** repository is mounted there as `detectors/` (a submodule): the detectors (`detectors/scripts/`), their threshold registry `detectors/thresholds.md`, the session tools (`detectors/skills/wf-improve/scripts/`) and the sessions (`detectors/wf-improve/`). Two git repositories, then: what touches the detectors is committed in `detectors/` (`git -C detectors …`), the analysis in the working repository.
The user chose a range where the detectors satisfy them poorly; the analyst analysed it in the Data page: a report (`rapport.md`) and labels, added to `labeling/auto/<STATION>.jsonl`. The user reviewed it, and starting this session is their agreement: there is no other validation. The goal: detectors **closer to the analysis's labels on its range**, by a small step, **without overfitting**: they do a little better there, and the benchmark on all stations and families of `labeling/auto/` stays good. You call the agents in order, pass on their outputs and commit. **You judge nothing yourself**: you only read the verdict lines. **You ask the user no question**: the session runs alone to the end.
@@ -17,13 +17,13 @@ The user chose a range where the detectors satisfy them poorly; the analyst anal
- `SESSION` = `date -u +%Y-%m-%dT%H%MZ`; session folder `detectors/wf-improve/<SESSION>_<ANALYSIS>/`, below `<session>`.
- **Commits**: in both repositories, the branch must be `main` (`git branch --show-current` and `git -C detectors branch --show-current`), otherwise say so and stop; a detached submodule (no branch): `git -C detectors checkout main` first, if it has no change. You always commit by path (`git add <paths> && git commit -m "<message>" -- <paths>`, or with `git -C detectors`, paths relative to `detectors/`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working in these repositories. Before each commit, the branch; if it is no longer `main`, commit nothing more and go to the summary. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. You push nothing.
- **Step 0, the analysis in git**: the app does not commit; if `git status --porcelain -- <A> labeling/auto/<STATION>.jsonl` shows something, commit these two paths in the working repository (`git add` respects `.gitignore`: `travail/` stays out), message `wf-improve <ANALYSIS>: labels of the analysis`. Then note the **session base**: `git -C detectors rev-parse HEAD`.
- `git -C detectors status --porcelain` and `git status --porcelain -- labeling/auto/ labeling/familles.md` (`familles.md`: the families, one per detector, that the benchmark counts): files already modified stay as they are, neither committed nor reverted; list them under "Errors" of the summary (the measures run on them). Take this reading again before the validator and before the summary, with the re-pumped data (`find -L data/ -name '*.csv' -newer <session>/benchmark-before.txt | cut -d/ -f2 | sort -u`): a file changed during the session by another session (a label touched up in the Data page, for example) goes under "Errors"; before the validator, then rerun the benchmark before at the session base (in a `mktemp -d` folder: `git -C detectors archive <base> --prefix=detectors/ | tar -x -C <folder>`, links to `data/`, `labeling/` and `agent-tools/` of the working repository, then `python3 <folder>/detectors/scripts/wf-improve/regression.py`) into `<session>/benchmark-before-rerun.txt`, and add to the validator's request: `Changed during the session: <paths or stations>; benchmark before rerun: <session>/benchmark-before-rerun.txt.`
- `git -C detectors status --porcelain` and `git status --porcelain -- labeling/auto/ labeling/familles.md` (`familles.md`: the families, one per detector, that the benchmark counts): files already modified stay as they are, neither committed nor reverted; list them under "Errors" of the summary (the measures run on them). Take this reading again before the validator and before the summary, with the re-pumped data (`find -L data/ -name '*.csv' -newer <session>/benchmark-before.txt | cut -d/ -f2 | sort -u`): a file changed during the session by another session (a label touched up in the Data page, for example) goes under "Errors"; before the validator, then rerun the benchmark before at the session base (in a `mktemp -d` folder: `git -C detectors archive <base> --prefix=detectors/ | tar -x -C <folder>`, links to `data/`, `labeling/` and `agent-tools/` of the working repository, then `python3 <folder>/detectors/skills/wf-improve/scripts/regression.py`) into `<session>/benchmark-before-rerun.txt`, and add to the validator's request: `Changed during the session: <paths or stations>; benchmark before rerun: <session>/benchmark-before-rerun.txt.`
**The two measures** of the session, at each step:
- **the benchmark**, the non-regression: `python3 detectors/scripts/wf-improve/regression.py > <session>/benchmark-<step>.txt 2>&1` (about 3 min). The score per family and per instant on each station of `labeling/auto/`, then their `Total`;
- **the range score**, the goal: `python3 detectors/scripts/wf-improve/score.py data/<STATION> --from <FROM> --to <TO> > <session>/range-<step>.txt 2>&1`. The labels of `labeling/auto/<STATION>.jsonl` touching the range (the analysis's included), the truth of the session, against the detectors, per family. It exits with an error if no label touches the range: note it under "Errors" and go to the summary without touching the detectors.
- **the benchmark**, the non-regression: `python3 detectors/skills/wf-improve/scripts/regression.py > <session>/benchmark-<step>.txt 2>&1` (about 3 min). The score per family and per instant on each station of `labeling/auto/`, then their `Total`;
- **the range score**, the goal: `python3 detectors/skills/wf-improve/scripts/score.py data/<STATION> --from <FROM> --to <TO> > <session>/range-<step>.txt 2>&1`. The labels of `labeling/auto/<STATION>.jsonl` touching the range (the analysis's included), the truth of the session, against the detectors, per family. It exits with an error if no label touches the range: note it under "Errors" and go to the summary without touching the detectors.
**Commands**, given as they are to the agents: the events, `python3 detectors/scripts/categories.py data/<STATION> --from <FROM> --to <TO>`; the sensors, `python3 detectors/scripts/wf-improve/sensors.py data/<STATION> --from <FROM> --to <TO>`. For the validator only, which looks at all stations, the events command with `<STATION>`, `<from>` and `<to>` to fill from the header of the station's block in the benchmark: `python3 detectors/scripts/categories.py data/<STATION> --from <from> --to <to>`.
**Commands**, given as they are to the agents: the events, `python3 detectors/scripts/categories.py data/<STATION> --from <FROM> --to <TO>`; the sensors, `python3 detectors/skills/wf-improve/scripts/sensors.py data/<STATION> --from <FROM> --to <TO>`. For the validator only, which looks at all stations, the events command with `<STATION>`, `<from>` and `<to>` to fill from the header of the station's block in the benchmark: `python3 detectors/scripts/categories.py data/<STATION> --from <from> --to <to>`.
## 2. In order
@@ -34,16 +34,16 @@ Session commits (the session folder, the detectors, `thresholds.md`) are made in
1. **Measures before**: step `before`, so `<session>/benchmark-before.txt` and `<session>/range-before.txt`.
Commit `wf-improve/<SESSION>_<ANALYSIS>/`, message `wf-improve <ANALYSIS>: benchmark`.
2. **Upgrader**, a single pass:
`Analysis: <A>/. Station: <STATION>, range <début> → <fin> (UTC). Events command: <command>. Sensors command: <sensors command>. Range score: python3 detectors/scripts/wf-improve/score.py data/<STATION> --from <FROM> --to <TO> (before: <session>/range-before.txt). Benchmark before: <session>/benchmark-before.txt, to rerun with python3 detectors/scripts/wf-improve/regression.py. Session folder: <session>. Files already modified at the start of the session: <list | none>. Write <session>/modification.md.`
`Analysis: <A>/. Station: <STATION>, range <début> → <fin> (UTC). Events command: <command>. Sensors command: <sensors command>. Range score: python3 detectors/skills/wf-improve/scripts/score.py data/<STATION> --from <FROM> --to <TO> (before: <session>/range-before.txt). Benchmark before: <session>/benchmark-before.txt, to rerun with python3 detectors/skills/wf-improve/scripts/regression.py. Session folder: <session>. Files already modified at the start of the session: <list | none>. Write <session>/modification.md.`
The upgrader commits each fix it keeps itself, in `detectors/`, with its benchmark `<session>/benchmark-S<n>.txt`, message `wf-improve <ANALYSIS>: S<n>`: the validator can thus have a single one removed.
Then the measures, step `after`, in `<session>/benchmark-after.txt` and `<session>/range-after.txt`.
Commit of the session folder, message `wf-improve <ANALYSIS>: modification (<verdict>)`. No fix kept (verdict `nothing to change`, or no `, S<n>` commit: `git -C detectors log --format=%h --grep ': S[0-9]*$' <base>..HEAD`): go to the summary.
3. **Validator**, which reads neither the diagnosis nor the detail of the modification:
`Analysis: <ANALYSIS>. Modification: <session>/modification.md. Benchmark: <session>/benchmark-before.txt → <session>/benchmark-after.txt, to rerun with python3 detectors/scripts/wf-improve/regression.py; after each fix: <session>/benchmark-S*.txt. Range score: <session>/range-before.txt → <session>/range-after.txt. Events command: <validator's command>. Base (in detectors/): <session base>. Write <session>/validation.md.`
`Analysis: <ANALYSIS>. Modification: <session>/modification.md. Benchmark: <session>/benchmark-before.txt → <session>/benchmark-after.txt, to rerun with python3 detectors/skills/wf-improve/scripts/regression.py; after each fix: <session>/benchmark-S*.txt. Range score: <session>/range-before.txt → <session>/range-after.txt. Events command: <validator's command>. Base (in detectors/): <session base>. Write <session>/validation.md.`
Commit of the session folder, message `wf-improve <ANALYSIS>: validator (<verdict>)`.
- `keep`: go to the summary.
- `remove`: for each fix of the `**Remove**:` line, its session commits (`git -C detectors log --format=%h --grep ': S<n>$' <session base>..HEAD`), from the most recent to the oldest: `git -C detectors revert --no-edit <commit>`, then amend its message so that it ends with the `Co-Authored-By` line (`git -C detectors commit --amend -m "wf-improve <ANALYSIS>: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`). A conflict: `git -C detectors revert --abort`, and do as for `cancel`. Then go to the summary.
- `cancel`, or any other word: `git -C detectors checkout <session base> -- 'scripts/*.py' thresholds.md` (without the files already modified noted in §1; `scripts/wf-improve/` is not touched), commit of these paths, message `wf-improve <ANALYSIS>: modification cancelled`.
- `cancel`, or any other word: `git -C detectors checkout <session base> -- 'scripts/*.py' thresholds.md` (without the files already modified noted in §1; `skills/wf-improve/` is not touched), commit of these paths, message `wf-improve <ANALYSIS>: modification cancelled`.
An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to the summary. Do not rerun an agent more than once.
+49
View File
@@ -0,0 +1,49 @@
"""Benchmark of the wf-improve workflow (D-116): the detectors against all the labeller's labels (labeling/auto/).
For each station of labeling/auto/ that has its folder in data/, the score per family and per instant of score.py
(D-112), over the range of its labels: from the day of the first to the midnight after the last. Then the total, family
by family, which says whether the detectors do better overall from one session to the next. The block of the session's
station is its score; the other blocks, the non-regression: a family that goes back on a station is a regression, even
if the total rises.
Usage: python3 detectors/skills/wf-improve/scripts/regression.py [--stations A,B]
"""
import argparse
import json
from datetime import timedelta
import score # next to it
from data import iso, parse_time # on the path through score
def label_range(path):
"""labeling/auto/<STATION>.jsonl -> (start, end) iso, in whole UTC days, end excluded."""
with open(path, encoding="utf-8") as f:
labels = [json.loads(l) for l in f if l.strip()]
start = min(e["debut"] for e in labels)[:10]
# ponytail: the end of the batch is not written; midnight closest to the last bound (a label bound overflows by one
# measurement after midnight); the real end of the batch in labeling/auto/ if the gap matters
end = iso(parse_time(max(e["fin"] for e in labels)) + timedelta(hours=12))[:10]
return iso(parse_time(start)), iso(parse_time(end))
if __name__ == "__main__":
p = argparse.ArgumentParser(usage=__doc__)
p.add_argument("--stations", help="A,B: these stations only (a shorter run)")
a = p.parse_args()
total = {}
for path in sorted((score.WORK / "labeling" / "auto").glob("*.jsonl")):
station = path.stem
if a.stations and station not in a.stations.split(","):
continue
folder = score.WORK / "data" / station
if not folder.is_dir():
print(f"{station}: no data/{station}/, skipped\n")
continue
start, end = label_range(path)
n, labels = score.family_scores(str(folder), start, end)
score.show(f"{station}: {start} → {end}, {len(labels)} labels", n)
print()
for k, v in n.items():
total[k] = [x + y for x, y in zip(total.get(k, [0, 0, 0]), v)]
score.show("Total", total)
+97
View File
@@ -0,0 +1,97 @@
"""Score of the detectors on a range: the detectors' events against the labels of labeling/auto/<STATION>.jsonl that
touch the range, per family (labeling/familles.md) and per instant (D-112). An instant: an hour of the range where at
least one sensor has a measurement; it receives, on one side, the families of the labels covering it, bounds excluded
(except for a hole, without a measurement inside), on the other, those of the detectors' events (categories.py, the
range classified with its 3 days of history). The category of an event is its family.
found: labelled instants that the detectors also flagged (share of the labelled ones);
correct: flagged instants that a label also puts in the family (share of the flagged ones).
The score is a fact, not a verdict: the labels can be wrong too. No label on the range: exits with an error.
Label fields (written by Diagnostic-app): debut, fin, categorie (the family), description.
Run from the working repository, where detectors/ is mounted.
Usage: python3 detectors/skills/wf-improve/scripts/score.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD
"""
import argparse
import json
import os
import sys
from datetime import timedelta
from pathlib import Path
REPO = Path(os.path.abspath(__file__)).parents[3] # the detectors repository
WORK = REPO.parent # the working repository (labeling/, data/)
sys.path.insert(0, str(REPO / "scripts"))
import categories # noqa: E402
from data import iso, load, parse_time # noqa: E402
HISTORY = timedelta(days=3) # as the batch command (D-056)
def events(measurements, start, end, folder=None):
"""Detectors' events touching the range, classified with its days of history (as the batch command)."""
h, b = parse_time(start) - HISTORY, parse_time(end)
chunk = {s: k for s, ms in measurements.items() if (k := [m for m in ms if h <= m["t"] < b])}
return [v for v in (categories.events(chunk, folder=folder) if chunk else []) if v["end"] >= start]
def count(pairs):
"""[(labelled keys, flagged keys)], one pair per instant -> {key: [labelled, flagged, both]}."""
n = {}
for a, d in pairs:
for k in a | d:
x = n.setdefault(k, [0, 0, 0])
x[0], x[1], x[2] = x[0] + (k in a), x[1] + (k in d), x[2] + (k in a and k in d)
return n
def auto_labels(station, start, end):
"""labeling/auto/<STATION>.jsonl -> the labeller's labels touching [start, end)."""
path = WORK / "labeling" / "auto" / f"{station}.jsonl"
if not path.exists():
return []
with open(path, encoding="utf-8") as f:
return [e for e in map(json.loads, filter(str.strip, f)) if e["debut"] < end and e["fin"] >= start]
def family_scores(folder, start, end):
"""Score of a range per family and per instant -> ({family: [labelled, flagged, both]}, labels)."""
# ponytail: per instant and not per sensor, the labels only give their sensors in the text; a sensors field in the
# label if this precision is missing
measurements = load(folder)
labels = auto_labels(os.path.basename(os.path.normpath(folder)), start, end)
evts = events(measurements, start, end, folder)
instants = sorted({iso(m["t"]) for ms in measurements.values() for m in ms if start <= iso(m["t"]) < end})
# bounds excluded: they are the healthy measurement before and after (D-111); a hole, without a measurement
# inside, keeps them
empty = {i for i, e in enumerate(labels) if not any(e["debut"] < t < e["fin"] for t in instants)}
pairs = []
for t in instants:
a = {e.get("categorie", "no family") for i, e in enumerate(labels)
if e["debut"] < t < e["fin"] or (i in empty and e["debut"] <= t <= e["fin"])} - {"None"}
d = {v["category"] for v in evts if v["start"] <= t <= v["end"]}
pairs.append((a, d))
return count(pairs), labels
def pct(a, b):
return f"{a}/{b} ({round(100 * a / b)} %)" if b else "—"
def show(title, n):
print(title)
print(f" {'':<22}{'found':>20}{'correct':>20}")
for k in sorted(n):
labelled, flagged, both = n[k]
print(f" {k:<22}{pct(both, labelled):>20}{pct(both, flagged):>20}")
if __name__ == "__main__":
p = argparse.ArgumentParser(usage=__doc__)
p.add_argument("folder")
p.add_argument("--from", dest="start", required=True)
p.add_argument("--to", dest="end", required=True)
a = p.parse_args()
station, start, end = os.path.basename(os.path.normpath(a.folder)), iso(parse_time(a.start)), iso(parse_time(a.end))
n, labels = family_scores(a.folder, start, end)
if not labels:
sys.exit(f"No label of {station} on {start} → {end} in labeling/auto/{station}.jsonl: analyse the range first.")
show(f"{station}: {start} → {end}, {len(labels)} labels of labeling/auto/{station}.jsonl, per family and per instant", n)
+31
View File
@@ -0,0 +1,31 @@
"""For the upgrader: the cadence of the measurements of the range, the number of targets and references, then each event
of the range with its sensors (categories.py only gives their number). Same computation as categories.py without
--chunk (3 days of history).
Usage: python3 detectors/skills/wf-improve/scripts/sensors.py <folder> --from YYYY-MM-DD --to YYYY-MM-DD
"""
import argparse
import statistics
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts"))
import categories # noqa: E402
from data import iso, load # noqa: E402
p = argparse.ArgumentParser(usage=__doc__)
p.add_argument("folder")
p.add_argument("--from", dest="start", required=True)
p.add_argument("--to", dest="end", required=True)
a = p.parse_args()
day = lambda d: datetime.strptime(d[:10], "%Y-%m-%d").replace(tzinfo=timezone.utc) # noqa: E731
start, end = day(a.start), day(a.end)
batch = {s: k for s, ms in load(a.folder).items() if (k := [m for m in ms if start - timedelta(days=3) <= m["t"] < end])}
ts = sorted({m["t"] for ms in batch.values() for m in ms if m["t"] >= start})
steps = [(y - x).total_seconds() / 60 for x, y in zip(ts, ts[1:])]
print(f"cadence: {statistics.median(steps):.0f} min (median of the steps between measurements)" if steps else "cadence: no measurement")
refs = sum(ms[0]["pattern_type"] == "global_ref" for ms in batch.values())
print(f"sensors: {len(batch) - refs} targets, {refs} references")
for v in (v for v in categories.events(batch, end, a.folder) if v["end"] >= iso(start)) if batch else []:
print(v["start"], "→", v["end"], f"{v['category']:<20}", "|", v["episodes"][0]["label"], "|",
", ".join(sorted(v["sensors"])) or "all sensors")