wf-improve: several stations on the same range (DATA, STATIONS, a labels folder); regression.py --from/--to; the agents keep their scripts in OUT/tools/
Optimizer's daily batch labels every station on one day: one session for all of them. The single-station request (Diagnostic-app) is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c7e34b49f7
commit
887b313ba2
5 files changed
+32
-11
No files matched your search
@@ -44,14 +44,19 @@ REFERENCE: --labels labeling/auto --data data. FAMILIES: labeling/familles.md. \
|
||||
CONTEXT: labeling/auto/analyses/NORDEND_x/rapport.md. OUT: detectors/runs/NORDEND_x/."
|
||||
```
|
||||
|
||||
Several stations labelled on the same range (Optimizer's daily batch): `DATA: data. STATIONS: A,B. LABELS: <folder of
|
||||
<STATION>.jsonl>` in place of `STATION` and a labels file; a single session for all of them.
|
||||
|
||||
Labels: JSONL, one per line, `{"id", "debut", "fin", "categorie", "description"}` (UTC ISO times, `categorie` = the family).
|
||||
|
||||
The agents keep the scripts they write in `<OUT>/tools/` and list them in their outputs; the summary copies the list.
|
||||
|
||||
- `skills/wf-improve/SKILL.md`: the workflow; its tools in `skills/wf-improve/scripts/`: `score.py` (the score of a range), `regression.py` (the benchmark), `sensors.py` (the sensors of each event).
|
||||
- `agents/`: `upgrader`, `validator` (called `detectors:<name>`).
|
||||
|
||||
```bash
|
||||
python3 skills/wf-improve/scripts/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD
|
||||
python3 skills/wf-improve/scripts/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/>
|
||||
python3 skills/wf-improve/scripts/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> # --from/--to: one range for all
|
||||
```
|
||||
|
||||
Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here. Used by Diagnostic-app and Optimizer, through a `detectors -> ~/detectors` link (one copy on the workstation); their labels come from the `labeling` repository, their data from `aimsight-data`.
|
||||
+5
-2
@@ -11,7 +11,7 @@ You are the upgrader of the wf-improve workflow: you make the detectors better o
|
||||
|
||||
## Your request
|
||||
|
||||
The detectors repository (`DETECTORS`: the detectors `scripts/*.py`, their threshold registry `thresholds.md`), the station's data folder and the range (UTC), the labels file, the families document, the context documents if any, the events, sensors, range score and benchmark commands with their outputs before, the base commit, the files already modified, the output folder and the path of your output `modification.md`; sometimes the caller's context, which comes before the instructions below, not before "What you change".
|
||||
The detectors repository (`DETECTORS`: the detectors `scripts/*.py`, their threshold registry `thresholds.md`), the station's data folder and the range (UTC), the labels file (or several stations on the same range, a labels file each, in one folder: the events and sensors commands run per station, the range score covers them all; a mechanism that comes back from one station to another comes before a single case), the families document, the context documents if any, the events, sensors, range score and benchmark commands with their outputs before, the base commit, the files already modified, the output folder and the path of your output `modification.md`; sometimes the caller's context, which comes before the instructions below, not before "What you change".
|
||||
|
||||
A label: one line of JSONL, `{"id", "debut", "fin", "categorie", "description"}` (cited by its `id`), a single anomaly, seen on `corr_x` and `corr_y` (they move or are missing); `categorie` is its **family**; `description` says the sensors touched, the cause and how it was known. Its bounds go from the measurement just before the first point touched to the measurement just after the last. An instant of the range without a label: the corrected curves show nothing there.
|
||||
|
||||
@@ -60,7 +60,7 @@ Then the **benchmark** against the benchmark before: "found", the share of the l
|
||||
- **The range score**: what you make rise, without losing a whole event. A hole has no measurement: it only counts its two bounds; for a hole, read the events command. Do not aim for 100 %.
|
||||
- **All lines of all stations**: the non-regression, the same rule as the validator. **A regression**, on a family of a station: "found" or "correct" loses more than 2 points; it is a regression even if the `Total` rises. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain (labels contradicting each other from one station to another, the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score counted as found by an event on another sensor). A regression: rework your change before committing it, or drop the fix. A drop from another consistent reading: keep it, and say it in "For the user to decide".
|
||||
|
||||
Your working folder is yours alone: `mktemp -d`. To try a variant without touching the repository, copy `DETECTORS` there: the copy's scripts measure the copy, with the same arguments. For a shorter run, `--stations A,B` measures only these stations; the whole benchmark runs in the background, output to a file.
|
||||
Your working folder is yours alone: `mktemp -d`. A script you write to read the data or to check a rule (anything longer than a one-line command) is a file in `<output folder>/tools/`, its first line saying what it does, and runs from there, never inline (heredoc, `python3 -c`): the caller keeps these scripts to reuse them; your output lists them (`## Tools`). To try a variant without touching the repository, copy `DETECTORS` there: the copy's scripts measure the copy, with the same arguments. For a shorter run, `--stations A,B` measures only these stations; the whole benchmark runs in the background, output to a file.
|
||||
|
||||
## 4. One fix, one commit
|
||||
|
||||
@@ -102,6 +102,9 @@ none | <what remains, and why>
|
||||
|
||||
## Limits
|
||||
none | - <what the detectors cannot see with the measurements alone: the label (id, bounds, family), what it would take to see it (image…); a label the context says is wrong (id)>
|
||||
|
||||
## Tools
|
||||
none | - <`<output folder>/tools/<script>`: what it does, its arguments; worth keeping as a workflow tool or not>
|
||||
```
|
||||
|
||||
The Verdict line carries one of these two words; the orchestrator reads it as it is. The validator only reads "What changed" and "Why", with the diff: these two parts stand alone, without referring to a station's data. Times in UTC.
|
||||
|
||||
+4
-1
@@ -33,7 +33,7 @@ Nothing else: neither the diagnosis nor earlier runs.
|
||||
3. **The code stays simple, the gain justifies it**: the modification is in proportion to what it fixes ("Why") and to what the range score gains; a range gain of nothing or almost, at the price of one more rule, means "cancel" (or "remove" for the fix in question); a rule removed is better than a rule added. **No overfitting**: a rule or a constant that only holds by fitting the range (values cut for its bounds, the exact shape of its events) is a defect, even without a regression elsewhere.
|
||||
4. **The forbidden**: one detector per family (`visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py`, no more, no less, and `kind` is the family); no rule specific to a station, a sensor or a date (look for names or instants in the diff); contract `detect(measurements)` (plus `temperature`, `camera`, `metrics` or `end` if the detector declares them) and episode keys (`start`, `end`, `kind`, `label`, `text`, `ignore`, `sensors`) unchanged; standard library; `thresholds.md` up to date (`grep -nE "^[A-Z_0-9]+ = " <DETECTORS>/scripts/*.py`, against the registry).
|
||||
|
||||
You change no file but your output. Your working folder is yours alone: `mktemp -d`.
|
||||
You change no file but your output and your scripts. Your working folder is yours alone: `mktemp -d`. A script you write to read the data or to check a rule (anything longer than a one-line command) is a file in `<the folder of your output>/tools/`, its first line saying what it does, and runs from there, never inline (heredoc, `python3 -c`): the caller keeps these scripts to reuse them; your output lists them (`## Tools`).
|
||||
|
||||
## Your decision
|
||||
|
||||
@@ -58,6 +58,9 @@ A single pass: no sending back to the upgrader. The upgrader must be able to pro
|
||||
|
||||
## Reservations
|
||||
none | - <what you keep without counting it as a regression, for the user: the station, the family, before → after, the detectors' reading against the labels (id)>
|
||||
|
||||
## Tools
|
||||
none | - <`tools/<script>`: what it does, its arguments; worth keeping as a workflow tool or not>
|
||||
```
|
||||
|
||||
The Verdict line carries a single one of these three words, the Remove line only the fixes to remove; the orchestrator reads them as they are.
|
||||
|
||||
@@ -6,16 +6,16 @@ argument-hint: "<range, labels, reference, output folder, context>"
|
||||
|
||||
# wf-improve: improve the detectors on a range
|
||||
|
||||
You are the orchestrator. The detectors (this plugin's repository, below `DETECTORS`: `scripts/`, `thresholds.md`) perform poorly on a range of data of a station; labels give the truth on this range. You run the upgrader, which fixes the detectors, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||
You are the orchestrator. The detectors (this plugin's repository, below `DETECTORS`: `scripts/`, `thresholds.md`) perform poorly on a range of data of a station, or of several stations on the same range; labels give the truth on this range. You run the upgrader, which fixes the detectors, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
|
||||
|
||||
## Inputs
|
||||
|
||||
Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope.
|
||||
|
||||
- `DETECTORS`: the path of this repository (default: `detectors/`).
|
||||
- `STATION`: the station's data folder (read by `DETECTORS/scripts/data.py`).
|
||||
- `STATION`: the station's data folder (read by `DETECTORS/scripts/data.py`); or, several stations on the same range: `DATA`, the folder of the station folders, and `STATIONS`, `A,B`, the stations to improve.
|
||||
- `FROM`, `TO`: the range, UTC days (YYYY-MM-DD, `TO` excluded).
|
||||
- `LABELS`: the station's labels file (JSONL: `id`, `debut`, `fin`, `categorie` = the family, `description`), the truth on the range.
|
||||
- `LABELS`: the station's labels file (JSONL: `id`, `debut`, `fin`, `categorie` = the family, `description`), the truth on the range; with `STATIONS`, a folder of `<STATION>.jsonl`, one per station.
|
||||
- `REFERENCE`: the reference truth for the non-regression, `--labels <folder of <STATION>.jsonl> --data <folder of <STATION>/>`.
|
||||
- `FAMILIES`: the document defining the families and their boundaries (one detector per family).
|
||||
- `CONTEXT` (optional): documents explaining the range's labels (an analyst's report, for example).
|
||||
@@ -28,6 +28,7 @@ A required input missing, or `DETECTORS` not a git repository: say so and stop,
|
||||
- events: `python3 <DETECTORS>/scripts/categories.py <STATION> --from <FROM> --to <TO>`
|
||||
- sensors: `python3 <DETECTORS>/skills/wf-improve/scripts/sensors.py <STATION> --from <FROM> --to <TO>`
|
||||
- range score (the goal): `python3 <DETECTORS>/skills/wf-improve/scripts/score.py <STATION> --labels <LABELS> --from <FROM> --to <TO>`; it exits with an error if no label touches the range: note it in the summary and stop without touching the detectors.
|
||||
- with `STATIONS`: the events and sensors commands once per station, `<DATA>/<S>` for each `S` of `STATIONS`; the range score, all of them at once: `python3 <DETECTORS>/skills/wf-improve/scripts/regression.py --labels <LABELS> --data <DATA> --stations <STATIONS> --from <FROM> --to <TO>`.
|
||||
- benchmark (the non-regression, about 3 min): `python3 <DETECTORS>/skills/wf-improve/scripts/regression.py <REFERENCE>`
|
||||
|
||||
The measures of a step: `<range score> > <OUT>/range-<step>.txt 2>&1` and `<benchmark> > <OUT>/benchmark-<step>.txt 2>&1`.
|
||||
@@ -41,7 +42,7 @@ The fixes are commits in `DETECTORS`, one per fix, so that the validator can hav
|
||||
Each agent is called with the Agent tool, its `subagent_type` (`detectors:upgrader`, `detectors:validator`) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
|
||||
|
||||
1. **Measures before**: step `before`.
|
||||
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION>, range <FROM> → <TO> (UTC). Labels: <LABELS>. Families: <FAMILIES>. Context: <CONTEXT | none>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
|
||||
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION | <DATA>/<S> for each S of STATIONS>, range <FROM> → <TO> (UTC). Labels: <LABELS>. Families: <FAMILIES>. Context: <CONTEXT | none>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
|
||||
It commits each fix it keeps, message `wf-improve: S<n>`, with its benchmark `<OUT>/benchmark-S<n>.txt`.
|
||||
Verdict `nothing to change`, or no fix commit (`git -C <DETECTORS> log --format=%h --grep ': S[0-9]*$' <base>..HEAD` empty): go to the summary.
|
||||
3. **Measures after**: step `after`.
|
||||
@@ -57,7 +58,7 @@ An output missing or without a readable verdict: note it, put the detectors back
|
||||
Measures, step `end`, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed). Then `<OUT>/summary.md`:
|
||||
|
||||
```markdown
|
||||
# wf-improve: <station>, <FROM> → <TO>
|
||||
# wf-improve: <station | STATIONS>, <FROM> → <TO>
|
||||
|
||||
**Result**: modification kept | kept without S<n>… | cancelled | none | error
|
||||
|
||||
@@ -76,6 +77,9 @@ the `Total` line of each family, before → end; then the other changed lines (s
|
||||
## Not fixed
|
||||
<"Not fixed" of the modification; none>
|
||||
|
||||
## Tools
|
||||
<copied as they are: "Tools" of the modification and of the validation (the scripts the agents kept in `<OUT>/tools/`); none>
|
||||
|
||||
## Errors
|
||||
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`>
|
||||
```
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
"""Benchmark of the detectors against a reference truth: every labelled station.
|
||||
|
||||
For each <STATION>.jsonl of --labels that has its folder <STATION>/ in --data, the score per family and per instant of
|
||||
score.py, over the range of its labels: from the day of the first to the midnight after the last. Then the total,
|
||||
score.py, over the range of its labels: from the day of the first to the midnight after the last; with --from/--to, over
|
||||
that range for every station, a station without a label on it included (several stations labelled the same days). Then the total,
|
||||
family by family, which says whether the detectors do better overall. The block of a station is its score; the
|
||||
others, the non-regression: a family that goes back on a station is a regression, even if the total rises.
|
||||
Usage: python3 regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> [--stations A,B]
|
||||
[--from YYYY-MM-DD --to YYYY-MM-DD]
|
||||
"""
|
||||
import argparse
|
||||
from datetime import timedelta
|
||||
@@ -30,7 +32,11 @@ if __name__ == "__main__":
|
||||
p.add_argument("--labels", required=True, help="folder of the labels, one <STATION>.jsonl per station")
|
||||
p.add_argument("--data", required=True, help="folder of the station folders, <STATION>/")
|
||||
p.add_argument("--stations", help="A,B: these stations only (a shorter run)")
|
||||
p.add_argument("--from", dest="start", help="YYYY-MM-DD: this range for every station, not the range of its labels")
|
||||
p.add_argument("--to", dest="end", help="YYYY-MM-DD, excluded (with --from)")
|
||||
a = p.parse_args()
|
||||
if bool(a.start) != bool(a.end):
|
||||
p.error("--from and --to go together")
|
||||
total = {}
|
||||
for path in sorted(Path(a.labels).glob("*.jsonl")):
|
||||
station = path.stem
|
||||
@@ -40,7 +46,7 @@ if __name__ == "__main__":
|
||||
if not folder.is_dir():
|
||||
print(f"{station}: no {folder}/, skipped\n")
|
||||
continue
|
||||
start, end = label_range(path)
|
||||
start, end = (iso(parse_time(a.start)), iso(parse_time(a.end))) if a.start else label_range(path)
|
||||
n, labels = score.family_scores(str(folder), str(path), start, end)
|
||||
score.show(f"{station}: {start} → {end}, {len(labels)} labels", n)
|
||||
print()
|
||||
|
||||
Reference in new issue
Block a user