wf-improve tools in skills/wf-improve/scripts/
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6df274f580
commit
80e00a4898
8 files changed
+21
-22
No files matched your search
@@ -6,7 +6,7 @@ argument-hint: "<analysis>"
|
||||
|
||||
# wf-improve: the detectors from an analysis
|
||||
|
||||
You are the orchestrator. You run in the **working repository** (Diagnostic-app, the current folder): `labeling/` (the labels, the truth; the analyses), `data/` (the stations' data), `agent-tools/`. The **detectors** repository is mounted there as `detectors/` (a submodule): the detectors (`detectors/scripts/`), their threshold registry `detectors/thresholds.md`, the session tools (`detectors/scripts/wf-improve/`) and the sessions (`detectors/wf-improve/`). Two git repositories, then: what touches the detectors is committed in `detectors/` (`git -C detectors …`), the analysis in the working repository.
|
||||
You are the orchestrator. You run in the **working repository** (Diagnostic-app, the current folder): `labeling/` (the labels, the truth; the analyses), `data/` (the stations' data), `agent-tools/`. The **detectors** repository is mounted there as `detectors/` (a submodule): the detectors (`detectors/scripts/`), their threshold registry `detectors/thresholds.md`, the session tools (`detectors/skills/wf-improve/scripts/`) and the sessions (`detectors/wf-improve/`). Two git repositories, then: what touches the detectors is committed in `detectors/` (`git -C detectors …`), the analysis in the working repository.
|
||||
|
||||
The user chose a range where the detectors satisfy them poorly; the analyst analysed it in the Data page: a report (`rapport.md`) and labels, added to `labeling/auto/<STATION>.jsonl`. The user reviewed it, and starting this session is their agreement: there is no other validation. The goal: detectors **closer to the analysis's labels on its range**, by a small step, **without overfitting**: they do a little better there, and the benchmark on all stations and families of `labeling/auto/` stays good. You call the agents in order, pass on their outputs and commit. **You judge nothing yourself**: you only read the verdict lines. **You ask the user no question**: the session runs alone to the end.
|
||||
|
||||
@@ -17,13 +17,13 @@ The user chose a range where the detectors satisfy them poorly; the analyst anal
|
||||
- `SESSION` = `date -u +%Y-%m-%dT%H%MZ`; session folder `detectors/wf-improve/<SESSION>_<ANALYSIS>/`, below `<session>`.
|
||||
- **Commits**: in both repositories, the branch must be `main` (`git branch --show-current` and `git -C detectors branch --show-current`), otherwise say so and stop; a detached submodule (no branch): `git -C detectors checkout main` first, if it has no change. You always commit by path (`git add <paths> && git commit -m "<message>" -- <paths>`, or with `git -C detectors`, paths relative to `detectors/`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working in these repositories. Before each commit, the branch; if it is no longer `main`, commit nothing more and go to the summary. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. You push nothing.
|
||||
- **Step 0, the analysis in git**: the app does not commit; if `git status --porcelain -- <A> labeling/auto/<STATION>.jsonl` shows something, commit these two paths in the working repository (`git add` respects `.gitignore`: `travail/` stays out), message `wf-improve <ANALYSIS>: labels of the analysis`. Then note the **session base**: `git -C detectors rev-parse HEAD`.
|
||||
- `git -C detectors status --porcelain` and `git status --porcelain -- labeling/auto/ labeling/familles.md` (`familles.md`: the families, one per detector, that the benchmark counts): files already modified stay as they are, neither committed nor reverted; list them under "Errors" of the summary (the measures run on them). Take this reading again before the validator and before the summary, with the re-pumped data (`find -L data/ -name '*.csv' -newer <session>/benchmark-before.txt | cut -d/ -f2 | sort -u`): a file changed during the session by another session (a label touched up in the Data page, for example) goes under "Errors"; before the validator, then rerun the benchmark before at the session base (in a `mktemp -d` folder: `git -C detectors archive <base> --prefix=detectors/ | tar -x -C <folder>`, links to `data/`, `labeling/` and `agent-tools/` of the working repository, then `python3 <folder>/detectors/scripts/wf-improve/regression.py`) into `<session>/benchmark-before-rerun.txt`, and add to the validator's request: `Changed during the session: <paths or stations>; benchmark before rerun: <session>/benchmark-before-rerun.txt.`
|
||||
- `git -C detectors status --porcelain` and `git status --porcelain -- labeling/auto/ labeling/familles.md` (`familles.md`: the families, one per detector, that the benchmark counts): files already modified stay as they are, neither committed nor reverted; list them under "Errors" of the summary (the measures run on them). Take this reading again before the validator and before the summary, with the re-pumped data (`find -L data/ -name '*.csv' -newer <session>/benchmark-before.txt | cut -d/ -f2 | sort -u`): a file changed during the session by another session (a label touched up in the Data page, for example) goes under "Errors"; before the validator, then rerun the benchmark before at the session base (in a `mktemp -d` folder: `git -C detectors archive <base> --prefix=detectors/ | tar -x -C <folder>`, links to `data/`, `labeling/` and `agent-tools/` of the working repository, then `python3 <folder>/detectors/skills/wf-improve/scripts/regression.py`) into `<session>/benchmark-before-rerun.txt`, and add to the validator's request: `Changed during the session: <paths or stations>; benchmark before rerun: <session>/benchmark-before-rerun.txt.`
|
||||
|
||||
**The two measures** of the session, at each step:
|
||||
- **the benchmark**, the non-regression: `python3 detectors/scripts/wf-improve/regression.py > <session>/benchmark-<step>.txt 2>&1` (about 3 min). The score per family and per instant on each station of `labeling/auto/`, then their `Total`;
|
||||
- **the range score**, the goal: `python3 detectors/scripts/wf-improve/score.py data/<STATION> --from <FROM> --to <TO> > <session>/range-<step>.txt 2>&1`. The labels of `labeling/auto/<STATION>.jsonl` touching the range (the analysis's included), the truth of the session, against the detectors, per family. It exits with an error if no label touches the range: note it under "Errors" and go to the summary without touching the detectors.
|
||||
- **the benchmark**, the non-regression: `python3 detectors/skills/wf-improve/scripts/regression.py > <session>/benchmark-<step>.txt 2>&1` (about 3 min). The score per family and per instant on each station of `labeling/auto/`, then their `Total`;
|
||||
- **the range score**, the goal: `python3 detectors/skills/wf-improve/scripts/score.py data/<STATION> --from <FROM> --to <TO> > <session>/range-<step>.txt 2>&1`. The labels of `labeling/auto/<STATION>.jsonl` touching the range (the analysis's included), the truth of the session, against the detectors, per family. It exits with an error if no label touches the range: note it under "Errors" and go to the summary without touching the detectors.
|
||||
|
||||
**Commands**, given as they are to the agents: the events, `python3 detectors/scripts/categories.py data/<STATION> --from <FROM> --to <TO>`; the sensors, `python3 detectors/scripts/wf-improve/sensors.py data/<STATION> --from <FROM> --to <TO>`. For the validator only, which looks at all stations, the events command with `<STATION>`, `<from>` and `<to>` to fill from the header of the station's block in the benchmark: `python3 detectors/scripts/categories.py data/<STATION> --from <from> --to <to>`.
|
||||
**Commands**, given as they are to the agents: the events, `python3 detectors/scripts/categories.py data/<STATION> --from <FROM> --to <TO>`; the sensors, `python3 detectors/skills/wf-improve/scripts/sensors.py data/<STATION> --from <FROM> --to <TO>`. For the validator only, which looks at all stations, the events command with `<STATION>`, `<from>` and `<to>` to fill from the header of the station's block in the benchmark: `python3 detectors/scripts/categories.py data/<STATION> --from <from> --to <to>`.
|
||||
|
||||
## 2. In order
|
||||
|
||||
@@ -34,16 +34,16 @@ Session commits (the session folder, the detectors, `thresholds.md`) are made in
|
||||
1. **Measures before**: step `before`, so `<session>/benchmark-before.txt` and `<session>/range-before.txt`.
|
||||
Commit `wf-improve/<SESSION>_<ANALYSIS>/`, message `wf-improve <ANALYSIS>: benchmark`.
|
||||
2. **Upgrader**, a single pass:
|
||||
`Analysis: <A>/. Station: <STATION>, range <début> → <fin> (UTC). Events command: <command>. Sensors command: <sensors command>. Range score: python3 detectors/scripts/wf-improve/score.py data/<STATION> --from <FROM> --to <TO> (before: <session>/range-before.txt). Benchmark before: <session>/benchmark-before.txt, to rerun with python3 detectors/scripts/wf-improve/regression.py. Session folder: <session>. Files already modified at the start of the session: <list | none>. Write <session>/modification.md.`
|
||||
`Analysis: <A>/. Station: <STATION>, range <début> → <fin> (UTC). Events command: <command>. Sensors command: <sensors command>. Range score: python3 detectors/skills/wf-improve/scripts/score.py data/<STATION> --from <FROM> --to <TO> (before: <session>/range-before.txt). Benchmark before: <session>/benchmark-before.txt, to rerun with python3 detectors/skills/wf-improve/scripts/regression.py. Session folder: <session>. Files already modified at the start of the session: <list | none>. Write <session>/modification.md.`
|
||||
The upgrader commits each fix it keeps itself, in `detectors/`, with its benchmark `<session>/benchmark-S<n>.txt`, message `wf-improve <ANALYSIS>: S<n>`: the validator can thus have a single one removed.
|
||||
Then the measures, step `after`, in `<session>/benchmark-after.txt` and `<session>/range-after.txt`.
|
||||
Commit of the session folder, message `wf-improve <ANALYSIS>: modification (<verdict>)`. No fix kept (verdict `nothing to change`, or no `, S<n>` commit: `git -C detectors log --format=%h --grep ': S[0-9]*$' <base>..HEAD`): go to the summary.
|
||||
3. **Validator**, which reads neither the diagnosis nor the detail of the modification:
|
||||
`Analysis: <ANALYSIS>. Modification: <session>/modification.md. Benchmark: <session>/benchmark-before.txt → <session>/benchmark-after.txt, to rerun with python3 detectors/scripts/wf-improve/regression.py; after each fix: <session>/benchmark-S*.txt. Range score: <session>/range-before.txt → <session>/range-after.txt. Events command: <validator's command>. Base (in detectors/): <session base>. Write <session>/validation.md.`
|
||||
`Analysis: <ANALYSIS>. Modification: <session>/modification.md. Benchmark: <session>/benchmark-before.txt → <session>/benchmark-after.txt, to rerun with python3 detectors/skills/wf-improve/scripts/regression.py; after each fix: <session>/benchmark-S*.txt. Range score: <session>/range-before.txt → <session>/range-after.txt. Events command: <validator's command>. Base (in detectors/): <session base>. Write <session>/validation.md.`
|
||||
Commit of the session folder, message `wf-improve <ANALYSIS>: validator (<verdict>)`.
|
||||
- `keep`: go to the summary.
|
||||
- `remove`: for each fix of the `**Remove**:` line, its session commits (`git -C detectors log --format=%h --grep ': S<n>$' <session base>..HEAD`), from the most recent to the oldest: `git -C detectors revert --no-edit <commit>`, then amend its message so that it ends with the `Co-Authored-By` line (`git -C detectors commit --amend -m "wf-improve <ANALYSIS>: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`). A conflict: `git -C detectors revert --abort`, and do as for `cancel`. Then go to the summary.
|
||||
- `cancel`, or any other word: `git -C detectors checkout <session base> -- 'scripts/*.py' thresholds.md` (without the files already modified noted in §1; `scripts/wf-improve/` is not touched), commit of these paths, message `wf-improve <ANALYSIS>: modification cancelled`.
|
||||
- `cancel`, or any other word: `git -C detectors checkout <session base> -- 'scripts/*.py' thresholds.md` (without the files already modified noted in §1; `skills/wf-improve/` is not touched), commit of these paths, message `wf-improve <ANALYSIS>: modification cancelled`.
|
||||
|
||||
An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to the summary. Do not rerun an agent more than once.
|
||||
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Benchmark of the wf-improve workflow (D-116): the detectors against all the labeller's labels (labeling/auto/).
|
||||
|
||||
For each station of labeling/auto/ that has its folder in data/, the score per family and per instant of score.py
|
||||
(D-112), over the range of its labels: from the day of the first to the midnight after the last. Then the total, family
|
||||
by family, which says whether the detectors do better overall from one session to the next. The block of the session's
|
||||
station is its score; the other blocks, the non-regression: a family that goes back on a station is a regression, even
|
||||
if the total rises.
|
||||
Usage: python3 detectors/skills/wf-improve/scripts/regression.py [--stations A,B]
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
from datetime import timedelta
|
||||
|
||||
import score # next to it
|
||||
|
||||
from data import iso, parse_time # on the path through score
|
||||
|
||||
|
||||
def label_range(path):
|
||||
"""labeling/auto/<STATION>.jsonl -> (start, end) iso, in whole UTC days, end excluded."""
|
||||
with open(path, encoding="utf-8") as f:
|
||||
labels = [json.loads(l) for l in f if l.strip()]
|
||||
start = min(e["debut"] for e in labels)[:10]
|
||||
# ponytail: the end of the batch is not written; midnight closest to the last bound (a label bound overflows by one
|
||||
# measurement after midnight); the real end of the batch in labeling/auto/ if the gap matters
|
||||
end = iso(parse_time(max(e["fin"] for e in labels)) + timedelta(hours=12))[:10]
|
||||
return iso(parse_time(start)), iso(parse_time(end))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
p = argparse.ArgumentParser(usage=__doc__)
|
||||
p.add_argument("--stations", help="A,B: these stations only (a shorter run)")
|
||||
a = p.parse_args()
|
||||
total = {}
|
||||
for path in sorted((score.WORK / "labeling" / "auto").glob("*.jsonl")):
|
||||
station = path.stem
|
||||
if a.stations and station not in a.stations.split(","):
|
||||
continue
|
||||
folder = score.WORK / "data" / station
|
||||
if not folder.is_dir():
|
||||
print(f"{station}: no data/{station}/, skipped\n")
|
||||
continue
|
||||
start, end = label_range(path)
|
||||
n, labels = score.family_scores(str(folder), start, end)
|
||||
score.show(f"{station}: {start} → {end}, {len(labels)} labels", n)
|
||||
print()
|
||||
for k, v in n.items():
|
||||
total[k] = [x + y for x, y in zip(total.get(k, [0, 0, 0]), v)]
|
||||
score.show("Total", total)
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Score of the detectors on a range: the detectors' events against the labels of labeling/auto/<STATION>.jsonl that
|
||||
touch the range, per family (labeling/familles.md) and per instant (D-112). An instant: an hour of the range where at
|
||||
least one sensor has a measurement; it receives, on one side, the families of the labels covering it, bounds excluded
|
||||
(except for a hole, without a measurement inside), on the other, those of the detectors' events (categories.py, the
|
||||
range classified with its 3 days of history). The category of an event is its family.
|
||||
found: labelled instants that the detectors also flagged (share of the labelled ones);
|
||||
correct: flagged instants that a label also puts in the family (share of the flagged ones).
|
||||
The score is a fact, not a verdict: the labels can be wrong too. No label on the range: exits with an error.
|
||||
Label fields (written by Diagnostic-app): debut, fin, categorie (the family), description.
|
||||
Run from the working repository, where detectors/ is mounted.
|
||||
Usage: python3 detectors/skills/wf-improve/scripts/score.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(os.path.abspath(__file__)).parents[3] # the detectors repository
|
||||
WORK = REPO.parent # the working repository (labeling/, data/)
|
||||
sys.path.insert(0, str(REPO / "scripts"))
|
||||
import categories # noqa: E402
|
||||
from data import iso, load, parse_time # noqa: E402
|
||||
|
||||
HISTORY = timedelta(days=3) # as the batch command (D-056)
|
||||
|
||||
|
||||
def events(measurements, start, end, folder=None):
|
||||
"""Detectors' events touching the range, classified with its days of history (as the batch command)."""
|
||||
h, b = parse_time(start) - HISTORY, parse_time(end)
|
||||
chunk = {s: k for s, ms in measurements.items() if (k := [m for m in ms if h <= m["t"] < b])}
|
||||
return [v for v in (categories.events(chunk, folder=folder) if chunk else []) if v["end"] >= start]
|
||||
|
||||
|
||||
def count(pairs):
|
||||
"""[(labelled keys, flagged keys)], one pair per instant -> {key: [labelled, flagged, both]}."""
|
||||
n = {}
|
||||
for a, d in pairs:
|
||||
for k in a | d:
|
||||
x = n.setdefault(k, [0, 0, 0])
|
||||
x[0], x[1], x[2] = x[0] + (k in a), x[1] + (k in d), x[2] + (k in a and k in d)
|
||||
return n
|
||||
|
||||
|
||||
def auto_labels(station, start, end):
|
||||
"""labeling/auto/<STATION>.jsonl -> the labeller's labels touching [start, end)."""
|
||||
path = WORK / "labeling" / "auto" / f"{station}.jsonl"
|
||||
if not path.exists():
|
||||
return []
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return [e for e in map(json.loads, filter(str.strip, f)) if e["debut"] < end and e["fin"] >= start]
|
||||
|
||||
|
||||
def family_scores(folder, start, end):
|
||||
"""Score of a range per family and per instant -> ({family: [labelled, flagged, both]}, labels)."""
|
||||
# ponytail: per instant and not per sensor, the labels only give their sensors in the text; a sensors field in the
|
||||
# label if this precision is missing
|
||||
measurements = load(folder)
|
||||
labels = auto_labels(os.path.basename(os.path.normpath(folder)), start, end)
|
||||
evts = events(measurements, start, end, folder)
|
||||
instants = sorted({iso(m["t"]) for ms in measurements.values() for m in ms if start <= iso(m["t"]) < end})
|
||||
# bounds excluded: they are the healthy measurement before and after (D-111); a hole, without a measurement
|
||||
# inside, keeps them
|
||||
empty = {i for i, e in enumerate(labels) if not any(e["debut"] < t < e["fin"] for t in instants)}
|
||||
pairs = []
|
||||
for t in instants:
|
||||
a = {e.get("categorie", "no family") for i, e in enumerate(labels)
|
||||
if e["debut"] < t < e["fin"] or (i in empty and e["debut"] <= t <= e["fin"])} - {"None"}
|
||||
d = {v["category"] for v in evts if v["start"] <= t <= v["end"]}
|
||||
pairs.append((a, d))
|
||||
return count(pairs), labels
|
||||
|
||||
|
||||
def pct(a, b):
|
||||
return f"{a}/{b} ({round(100 * a / b)} %)" if b else "—"
|
||||
|
||||
|
||||
def show(title, n):
|
||||
print(title)
|
||||
print(f" {'':<22}{'found':>20}{'correct':>20}")
|
||||
for k in sorted(n):
|
||||
labelled, flagged, both = n[k]
|
||||
print(f" {k:<22}{pct(both, labelled):>20}{pct(both, flagged):>20}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
p = argparse.ArgumentParser(usage=__doc__)
|
||||
p.add_argument("folder")
|
||||
p.add_argument("--from", dest="start", required=True)
|
||||
p.add_argument("--to", dest="end", required=True)
|
||||
a = p.parse_args()
|
||||
station, start, end = os.path.basename(os.path.normpath(a.folder)), iso(parse_time(a.start)), iso(parse_time(a.end))
|
||||
n, labels = family_scores(a.folder, start, end)
|
||||
if not labels:
|
||||
sys.exit(f"No label of {station} on {start} → {end} in labeling/auto/{station}.jsonl: analyse the range first.")
|
||||
show(f"{station}: {start} → {end}, {len(labels)} labels of labeling/auto/{station}.jsonl, per family and per instant", n)
|
||||
@@ -0,0 +1,31 @@
|
||||
"""For the upgrader: the cadence of the measurements of the range, the number of targets and references, then each event
|
||||
of the range with its sensors (categories.py only gives their number). Same computation as categories.py without
|
||||
--chunk (3 days of history).
|
||||
Usage: python3 detectors/skills/wf-improve/scripts/sensors.py <folder> --from YYYY-MM-DD --to YYYY-MM-DD
|
||||
"""
|
||||
import argparse
|
||||
import statistics
|
||||
import sys
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts"))
|
||||
import categories # noqa: E402
|
||||
from data import iso, load # noqa: E402
|
||||
|
||||
p = argparse.ArgumentParser(usage=__doc__)
|
||||
p.add_argument("folder")
|
||||
p.add_argument("--from", dest="start", required=True)
|
||||
p.add_argument("--to", dest="end", required=True)
|
||||
a = p.parse_args()
|
||||
day = lambda d: datetime.strptime(d[:10], "%Y-%m-%d").replace(tzinfo=timezone.utc) # noqa: E731
|
||||
start, end = day(a.start), day(a.end)
|
||||
batch = {s: k for s, ms in load(a.folder).items() if (k := [m for m in ms if start - timedelta(days=3) <= m["t"] < end])}
|
||||
ts = sorted({m["t"] for ms in batch.values() for m in ms if m["t"] >= start})
|
||||
steps = [(y - x).total_seconds() / 60 for x, y in zip(ts, ts[1:])]
|
||||
print(f"cadence: {statistics.median(steps):.0f} min (median of the steps between measurements)" if steps else "cadence: no measurement")
|
||||
refs = sum(ms[0]["pattern_type"] == "global_ref" for ms in batch.values())
|
||||
print(f"sensors: {len(batch) - refs} targets, {refs} references")
|
||||
for v in (v for v in categories.events(batch, end, a.folder) if v["end"] >= iso(start)) if batch else []:
|
||||
print(v["start"], "→", v["end"], f"{v['category']:<20}", "|", v["episodes"][0]["label"], "|",
|
||||
", ".join(sorted(v["sensors"])) or "all sensors")
|
||||
Reference in new issue
Block a user