Label keys start/end/family in the bench and the docs (benchmark unchanged); knowledge path in data.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b01a9116d0
commit
0abba2d0ed
5 files changed
+10
-10
No files matched your search
@@ -6,7 +6,7 @@ AIMsight anomaly detectors (Python 3, standard library only) and their benchmark
|
|||||||
|
|
||||||
- One detector per family, in `scripts/`: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py` (D-119). `categories.py` loads every `scripts/*.py` that has `detect`: never put another script with a `detect` function, or with side effects at import, directly in `scripts/`; tools go in `bench/`.
|
- One detector per family, in `scripts/`: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py` (D-119). `categories.py` loads every `scripts/*.py` that has `detect`: never put another script with a `detect` function, or with side effects at import, directly in `scripts/`; tools go in `bench/`.
|
||||||
- `detect(measurements)` returns episodes with the keys `start`, `end`, `kind` (the family), `label` (the variety), `text`, `ignore`, `sensors`. A detector may also declare `temperature`, `camera`, `metrics` (`categories.CONTEXT`) or `end`; `categories.events` passes them by introspection.
|
- `detect(measurements)` returns episodes with the keys `start`, `end`, `kind` (the family), `label` (the variety), `text`, `ignore`, `sensors`. A detector may also declare `temperature`, `camera`, `metrics` (`categories.CONTEXT`) or `end`; `categories.events` passes them by introspection.
|
||||||
- Label files are external data and keep their French keys: `debut`, `fin`, `categorie`, `description`.
|
- Label files (JSONL): `id`, `start`, `end`, `family`, `description` (and `review`, `analysis`); the project uses `labeling/auto` only.
|
||||||
- Two repositories use this one through a `detectors -> ~/detectors` link (one copy on the workstation: a commit here is live for both at once); a rename here needs the matching change there:
|
- Two repositories use this one through a `detectors -> ~/detectors` link (one copy on the workstation: a commit here is live for both at once); a rename here needs the matching change there:
|
||||||
- Diagnostic-app: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the `--from/--to` CLIs, `bench/score.py` (the command shown in the page);
|
- Diagnostic-app: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the `--from/--to` CLIs, `bench/score.py` (the command shown in the page);
|
||||||
- Optimizer (`improve` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix.
|
- Optimizer (`improve` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix.
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ python3 scripts/visibility.py <station folder>
|
|||||||
|
|
||||||
## Measuring them: `bench/`
|
## Measuring them: `bench/`
|
||||||
|
|
||||||
The detectors against labels (JSONL, one per line, `{"id", "debut", "fin", "categorie", "description"}`, UTC ISO times, `categorie` = the family):
|
The detectors against labels (JSONL, one per line, `{"id", "start", "end", "family", "description"}`, UTC ISO times):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 bench/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD # the score of a range
|
python3 bench/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD # the score of a range
|
||||||
|
|||||||
+2
-2
@@ -20,10 +20,10 @@ from data import iso, parse_time # on the path through score
|
|||||||
def label_range(path):
|
def label_range(path):
|
||||||
"""Labels JSONL -> (start, end) iso, in whole UTC days, end excluded."""
|
"""Labels JSONL -> (start, end) iso, in whole UTC days, end excluded."""
|
||||||
labels = score.read_labels(path)
|
labels = score.read_labels(path)
|
||||||
start = min(e["debut"] for e in labels)[:10]
|
start = min(e["start"] for e in labels)[:10]
|
||||||
# ponytail: the end of the labelled batch is not written; midnight closest to the last bound (a label bound
|
# ponytail: the end of the labelled batch is not written; midnight closest to the last bound (a label bound
|
||||||
# overflows by one measurement after midnight)
|
# overflows by one measurement after midnight)
|
||||||
end = iso(parse_time(max(e["fin"] for e in labels)) + timedelta(hours=12))[:10]
|
end = iso(parse_time(max(e["end"] for e in labels)) + timedelta(hours=12))[:10]
|
||||||
return iso(parse_time(start)), iso(parse_time(end))
|
return iso(parse_time(start)), iso(parse_time(end))
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+5
-5
@@ -6,7 +6,7 @@ event is its family.
|
|||||||
found: labelled instants that the detectors also flagged (share of the labelled ones);
|
found: labelled instants that the detectors also flagged (share of the labelled ones);
|
||||||
correct: flagged instants that a label also puts in the family (share of the flagged ones).
|
correct: flagged instants that a label also puts in the family (share of the flagged ones).
|
||||||
The score is a fact, not a verdict: the labels can be wrong too. No label on the range: exits with an error.
|
The score is a fact, not a verdict: the labels can be wrong too. No label on the range: exits with an error.
|
||||||
Labels: a JSONL file, one label per line, {"id", "debut", "fin", "categorie", "description"} (UTC ISO times, categorie =
|
Labels: a JSONL file, one label per line, {"id", "start", "end", "family", "description"} (UTC ISO times);
|
||||||
the family); its bounds are the healthy measurement before and after the anomaly.
|
the family); its bounds are the healthy measurement before and after the anomaly.
|
||||||
Usage: python3 score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD
|
Usage: python3 score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD
|
||||||
"""
|
"""
|
||||||
@@ -48,7 +48,7 @@ def read_labels(path, start=None, end=None):
|
|||||||
return []
|
return []
|
||||||
with open(path, encoding="utf-8") as f:
|
with open(path, encoding="utf-8") as f:
|
||||||
return [e for e in map(json.loads, filter(str.strip, f))
|
return [e for e in map(json.loads, filter(str.strip, f))
|
||||||
if (end is None or e["debut"] < end) and (start is None or e["fin"] >= start)]
|
if (end is None or e["start"] < end) and (start is None or e["end"] >= start)]
|
||||||
|
|
||||||
|
|
||||||
def family_scores(folder, labels_path, start, end):
|
def family_scores(folder, labels_path, start, end):
|
||||||
@@ -60,11 +60,11 @@ def family_scores(folder, labels_path, start, end):
|
|||||||
evts = events(measurements, start, end, folder)
|
evts = events(measurements, start, end, folder)
|
||||||
instants = sorted({iso(m["t"]) for ms in measurements.values() for m in ms if start <= iso(m["t"]) < end})
|
instants = sorted({iso(m["t"]) for ms in measurements.values() for m in ms if start <= iso(m["t"]) < end})
|
||||||
# bounds excluded: they are the healthy measurement before and after; a hole, without a measurement inside, keeps them
|
# bounds excluded: they are the healthy measurement before and after; a hole, without a measurement inside, keeps them
|
||||||
empty = {i for i, e in enumerate(labels) if not any(e["debut"] < t < e["fin"] for t in instants)}
|
empty = {i for i, e in enumerate(labels) if not any(e["start"] < t < e["end"] for t in instants)}
|
||||||
pairs = []
|
pairs = []
|
||||||
for t in instants:
|
for t in instants:
|
||||||
a = {e.get("categorie", "no family") for i, e in enumerate(labels)
|
a = {e.get("family", "no family") for i, e in enumerate(labels)
|
||||||
if e["debut"] < t < e["fin"] or (i in empty and e["debut"] <= t <= e["fin"])} - {"None"}
|
if e["start"] < t < e["end"] or (i in empty and e["start"] <= t <= e["end"])} - {"None"}
|
||||||
d = {v["category"] for v in evts if v["start"] <= t <= v["end"]}
|
d = {v["category"] for v in evts if v["start"] <= t <= v["end"]}
|
||||||
pairs.append((a, d))
|
pairs.append((a, d))
|
||||||
return count(pairs), labels
|
return count(pairs), labels
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
"""Reading a station folder (standard library only). Every time is UTC.
|
"""Reading a station folder (standard library only). Every time is UTC.
|
||||||
|
|
||||||
Column formats: connaissances/colonnes-des-deplacements.md in the labeling repository.
|
Column formats: knowledge/displacement-columns.md in the labeling repository.
|
||||||
"""
|
"""
|
||||||
import csv
|
import csv
|
||||||
import glob
|
import glob
|
||||||
|
|||||||
Reference in new issue
Block a user