wf-improve: several stations on the same range (DATA, STATIONS, a labels folder); regression.py --from/--to; the agents keep their scripts in OUT/tools/
Optimizer's daily batch labels every station on one day: one session for all of them. The single-station request (Diagnostic-app) is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c7e34b49f7
commit
887b313ba2
5 files changed
+32
-11
No files matched your search
@@ -6,16 +6,16 @@ argument-hint: "<range, labels, reference, output folder, context>"
|
||||
|
||||
# wf-improve: improve the detectors on a range
|
||||
|
||||
You are the orchestrator. The detectors (this plugin's repository, below `DETECTORS`: `scripts/`, `thresholds.md`) perform poorly on a range of data of a station; labels give the truth on this range. You run the upgrader, which fixes the detectors, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||
You are the orchestrator. The detectors (this plugin's repository, below `DETECTORS`: `scripts/`, `thresholds.md`) perform poorly on a range of data of a station, or of several stations on the same range; labels give the truth on this range. You run the upgrader, which fixes the detectors, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||
|
||||
## Inputs
|
||||
|
||||
Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope.
|
||||
|
||||
- `DETECTORS`: the path of this repository (default: `detectors/`).
|
||||
- `STATION`: the station's data folder (read by `DETECTORS/scripts/data.py`).
|
||||
- `STATION`: the station's data folder (read by `DETECTORS/scripts/data.py`); or, several stations on the same range: `DATA`, the folder of the station folders, and `STATIONS`, `A,B`, the stations to improve.
|
||||
- `FROM`, `TO`: the range, UTC days (YYYY-MM-DD, `TO` excluded).
|
||||
- `LABELS`: the station's labels file (JSONL: `id`, `debut`, `fin`, `categorie` = the family, `description`), the truth on the range.
|
||||
- `LABELS`: the station's labels file (JSONL: `id`, `debut`, `fin`, `categorie` = the family, `description`), the truth on the range; with `STATIONS`, a folder of `<STATION>.jsonl`, one per station.
|
||||
- `REFERENCE`: the reference truth for the non-regression, `--labels <folder of <STATION>.jsonl> --data <folder of <STATION>/>`.
|
||||
- `FAMILIES`: the document defining the families and their boundaries (one detector per family).
|
||||
- `CONTEXT` (optional): documents explaining the range's labels (an analyst's report, for example).
|
||||
@@ -28,6 +28,7 @@ A required input missing, or `DETECTORS` not a git repository: say so and stop,
|
||||
- events: `python3 <DETECTORS>/scripts/categories.py <STATION> --from <FROM> --to <TO>`
|
||||
- sensors: `python3 <DETECTORS>/skills/wf-improve/scripts/sensors.py <STATION> --from <FROM> --to <TO>`
|
||||
- range score (the goal): `python3 <DETECTORS>/skills/wf-improve/scripts/score.py <STATION> --labels <LABELS> --from <FROM> --to <TO>`; it exits with an error if no label touches the range: note it in the summary and stop without touching the detectors.
|
||||
- with `STATIONS`: the events and sensors commands once per station, `<DATA>/<S>` for each `S` of `STATIONS`; the range score, all of them at once: `python3 <DETECTORS>/skills/wf-improve/scripts/regression.py --labels <LABELS> --data <DATA> --stations <STATIONS> --from <FROM> --to <TO>`.
|
||||
- benchmark (the non-regression, about 3 min): `python3 <DETECTORS>/skills/wf-improve/scripts/regression.py <REFERENCE>`
|
||||
|
||||
The measures of a step: `<range score> > <OUT>/range-<step>.txt 2>&1` and `<benchmark> > <OUT>/benchmark-<step>.txt 2>&1`.
|
||||
@@ -41,7 +42,7 @@ The fixes are commits in `DETECTORS`, one per fix, so that the validator can hav
|
||||
Each agent is called with the Agent tool, its `subagent_type` (`detectors:upgrader`, `detectors:validator`) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
|
||||
|
||||
1. **Measures before**: step `before`.
|
||||
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION>, range <FROM> → <TO> (UTC). Labels: <LABELS>. Families: <FAMILIES>. Context: <CONTEXT | none>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
|
||||
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION | <DATA>/<S> for each S of STATIONS>, range <FROM> → <TO> (UTC). Labels: <LABELS>. Families: <FAMILIES>. Context: <CONTEXT | none>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
|
||||
It commits each fix it keeps, message `wf-improve: S<n>`, with its benchmark `<OUT>/benchmark-S<n>.txt`.
|
||||
Verdict `nothing to change`, or no fix commit (`git -C <DETECTORS> log --format=%h --grep ': S[0-9]*$' <base>..HEAD` empty): go to the summary.
|
||||
3. **Measures after**: step `after`.
|
||||
@@ -57,7 +58,7 @@ An output missing or without a readable verdict: note it, put the detectors back
|
||||
Measures, step `end`, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed). Then `<OUT>/summary.md`:
|
||||
|
||||
```markdown
|
||||
# wf-improve: <station>, <FROM> → <TO>
|
||||
# wf-improve: <station | STATIONS>, <FROM> → <TO>
|
||||
|
||||
**Result**: modification kept | kept without S<n>… | cancelled | none | error
|
||||
|
||||
@@ -76,6 +77,9 @@ the `Total` line of each family, before → end; then the other changed lines (s
|
||||
## Not fixed
|
||||
<"Not fixed" of the modification; none>
|
||||
|
||||
## Tools
|
||||
<copied as they are: "Tools" of the modification and of the validation (the scripts the agents kept in `<OUT>/tools/`); none>
|
||||
|
||||
## Errors
|
||||
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`>
|
||||
```
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
"""Benchmark of the detectors against a reference truth: every labelled station.
|
||||
|
||||
For each <STATION>.jsonl of --labels that has its folder <STATION>/ in --data, the score per family and per instant of
|
||||
score.py, over the range of its labels: from the day of the first to the midnight after the last. Then the total,
|
||||
score.py, over the range of its labels: from the day of the first to the midnight after the last; with --from/--to, over
|
||||
that range for every station, a station without a label on it included (several stations labelled the same days). Then the total,
|
||||
family by family, which says whether the detectors do better overall. The block of a station is its score; the
|
||||
others, the non-regression: a family that goes back on a station is a regression, even if the total rises.
|
||||
Usage: python3 regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> [--stations A,B]
|
||||
[--from YYYY-MM-DD --to YYYY-MM-DD]
|
||||
"""
|
||||
import argparse
|
||||
from datetime import timedelta
|
||||
@@ -30,7 +32,11 @@ if __name__ == "__main__":
|
||||
p.add_argument("--labels", required=True, help="folder of the labels, one <STATION>.jsonl per station")
|
||||
p.add_argument("--data", required=True, help="folder of the station folders, <STATION>/")
|
||||
p.add_argument("--stations", help="A,B: these stations only (a shorter run)")
|
||||
p.add_argument("--from", dest="start", help="YYYY-MM-DD: this range for every station, not the range of its labels")
|
||||
p.add_argument("--to", dest="end", help="YYYY-MM-DD, excluded (with --from)")
|
||||
a = p.parse_args()
|
||||
if bool(a.start) != bool(a.end):
|
||||
p.error("--from and --to go together")
|
||||
total = {}
|
||||
for path in sorted(Path(a.labels).glob("*.jsonl")):
|
||||
station = path.stem
|
||||
@@ -40,7 +46,7 @@ if __name__ == "__main__":
|
||||
if not folder.is_dir():
|
||||
print(f"{station}: no {folder}/, skipped\n")
|
||||
continue
|
||||
start, end = label_range(path)
|
||||
start, end = (iso(parse_time(a.start)), iso(parse_time(a.end))) if a.start else label_range(path)
|
||||
n, labels = score.family_scores(str(folder), str(path), start, end)
|
||||
score.show(f"{station}: {start} → {end}, {len(labels)} labels", n)
|
||||
print()
|
||||
|
||||
Reference in new issue
Block a user