Compare commits

...
32 Commits
Author SHA1 Message Date
maximeandClaude Opus 5.5 1ca6557af2 displacement: the text of a jump back says nothing to do (its family, Perturbation); the docstring notes that the return is read on the corrected curves only, so a camera that stays while the curves come back is a Perturbation here, a Displacement in the labels (for the optimizer); figures unchanged
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:45:43 +01:00
maximeandClaude Opus 5.5 47bc0acde2 Perturbation, the seventh family: a jump back to normal is a Perturbation (nothing to do, greyed), Displacement keeps the lasting shift, the jump not back yet and the slow drift (labeling spec 2026-11-18-displacement-split-design); benchmark: 0 regression, Perturbation found 0 -> 4 %, +1 code line
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:25:06 +01:00
maximeandClaude Opus 5.5 428ebaacfe categories.window: the events of a range with its history, for aimsight-app and the CLI (figures unchanged, +3 code lines)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 09:32:20 +01:00
maximeandClaude Opus 5.5 a008ea8fcc Regression computed by bench/compare.py (Total > 1 point; station > 5 points and > 20 instants), complexity in the benchmark, pruning after a fix
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:53:09 +01:00
maximeandClaude Opus 5.5 63d954a126 episodes.py, not common.py: the tools repository has its own common.py on the same sys.path in the app
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:43:12 +01:00
maximeandClaude Opus 5.5 9d4cd722f4 categories: the reader also from the caller's tools/ link, for a copy outside the home directory
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:34:59 +01:00
maximeandClaude Opus 5.5 000e8bd6e4 thresholds.md, CLAUDE.md, README.md: the simplified detectors
thresholds.md keeps only the remaining constants, with shortened history; CLAUDE.md and README.md describe the five
modules, common.py, the families each gives, the reader of the tools repository and the event fields; the benchmark
takes about 1 min.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:33:43 +01:00
maximeandClaude Opus 5.5 8b5f8a34b9 visibility: Target obscured left to target_issue; categories no longer imports a detector
Removed: the repeated rejections of a target in visibility.py with K_TARGET, TARGET_MIN, TARGET_GAP (Obstruction found
-5, correct +3: found -14 at SNCF01, 3 instants, and -10 at WILDSTRUBEL); Target obscured now comes from target_issue
only. categories.py keeps its own JUMP_BACK so that a broken displacement.py does not block it.

Benchmark Totals (from the previous commit): Obstruction found 45 -> 40 %, correct 31 -> 34 %; the other families
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:32:00 +01:00
maximeandClaude Opus 5.5 e5e5d2a244 displacement: one episode per jump; GAP, REPEAT and the dip rule removed
Each jump of a target is its own episode (categories.py merges the same variety less than an hour apart). Removed
(their benchmark contribution, each alone, on the current code): the grouping of all targets' jumps within GAP (none),
the flipping target of REPEAT jumps back on its own (Displacement correct -1, found +32 at FLORIDA), the day after at the
level of the day before (none).

Benchmark Totals (from the previous commit): Displacement correct 25 -> 24 %; the other families unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:30:34 +01:00
maximeandClaude Opus 5.5 ab0347c41d REJECT_MAX of heat shimmer and CALM of target_issue removed
Heat shimmer is no longer dropped for its rejections beyond each sensor's habit (Visibility found +2: WILDSTRUBEL +7,
NIEDERHORN +4). target_issue counts every measurement of the day, not only the calm instants (Obstruction found +3,
correct -2: correct -11 at LENZSPITZE and -12 at WILDSTRUBEL; Target issue unchanged).

Benchmark Totals (from the previous commit): Visibility found 55 -> 57 %; Obstruction found 42 -> 45 %, correct
33 -> 31 %; the other families unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:28:59 +01:00
maximeandClaude Opus 5.5 4188efe201 categories: plain absorption and merge rules, the event text is its first episode's
A jump back joins an overlapping Visibility episode, or an Obstruction or Target issue episode holding all its targets;
a slow drift, a Target issue episode holding its targets. Removed (their benchmark contribution, each alone): heat
shimmer dropped next to a shift of several targets (Visibility found +1), HOSTS with the thermal wave (none), the
special cases of a Brief obstruction, of heat shimmer and of the thermal wave as hosts (one jump back at WILDSTRUBEL,
already lost), SHIFTS, variety() reduced to (label, level). sentence() removed: each detector writes the text of its
episodes; the event adds one sentence for the jumps it absorbed. events() no longer takes anything from the folder.

Benchmark Totals (from the previous commit): Visibility found 54 -> 55 %, correct 71 -> 72 %; the other families
unchanged. regression.py now runs in about 1 min (was 3.5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:28:04 +01:00
maximeandClaude Opus 5.5 fe1adac89a thermal_displacement: the camera moving with the cold, partial days, DAY_MATCH and MIN_WARM_HOURS removed
Removed (their benchmark contribution, each alone): the cold rule with COLD, which needed the station temperature and
camera pose (Thermal found -64 at CLARIDEN and -19 at BREITHORN, 27 and 4 instants; Totals unchanged), so categories.py
no longer reads them; PARTIAL days (Thermal found -1, correct -1); DAY_MATCH, every day of a cycle gets its episodes
(Thermal found +8, correct -2); MIN_WARM_HOURS (none). The 24 h sinusoid by projection instead of least squares (same
figures). The event sentences of the thermal varieties are now the episodes' texts.

Benchmark Totals (from the previous commit): Thermal displacement found 74 -> 80 %, correct 75 -> 72 %; the other
families unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:23:53 +01:00
maximeandClaude Opus 5.5 fb32143722 displacement: Target flipping, common peaks, lasting jumps under K, camera drift and five constants removed
Removed (their benchmark contribution, each alone): Target flipping with FLIP and MAX_FLIPPING (none), the common
one-step peak with K_PEAK and PEAK_MIN (Displacement found -2, one jump back at WILDSTRUBEL), the jump under K that
stays with K_LASTING and PLATEAU (2 instants at NIEDERHORN), the first BASE measurements faced with the level after
(none), the usual place of the first day (Visibility found -1), the camera drift with LARGE (Displacement correct -3),
MAX_HOLE of the drift pieces (Displacement found +3). Variety Large shift folded into Lasting shift. The event
sentences of the varieties are now the episodes' texts.

Benchmark Totals (from the previous commit): Displacement found 91 -> 93 %, correct 28 -> 25 %; Visibility found
55 -> 54 %, correct 72 -> 71 %; the other families unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 17:20:17 +01:00
maximeandClaude Opus 5.5 3a22dae7ce no_data: the cadence of common.py, shorter
Benchmark unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:39:11 +01:00
maximeandClaude Opus 5.5 ec4b900fa0 target_issue: one rule of degraded days; Target stuck removed
A single bridged run of degraded days (SHARE of the calm instants rejected, BRIDGE): WEEK days or more, Degraded
target; MIN_DAYS or more, Target obscured (Obstruction), from its first rejected measurement to its last. Removed: the
separate unbridged runs at half of the calm instants (old SHARE) and their episodes per run of rejections, Target stuck
and STUCK (no instant of the benchmark), the cadence helper (common.cadence).

Benchmark Totals (from the previous commit): Obstruction found 22 -> 42 %, correct 32 -> 33 %; the other families
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:38:05 +01:00
maximeandClaude Opus 5.5 5c2d265a21 visibility: shared helpers (common.py), five rules fewer, texts of its events
Removed (their benchmark contribution, each alone): the reference exception of Brief obstruction (none; Obstruction
found +9 at LENZSPITZE), REJECT_FACTOR (Obstruction correct +2), FRAC and BREAK (Visibility found -3, Obstruction
found +4), the bounds at the rejected targets (neutral), FEW (Visibility found -1), the TARGET_MIN exception at a slow
cadence (none in total), MIN_DURATION of heat shimmer (Visibility found -1). RATIO_ERR merged into RATIO (same value).
Target obscured: one episode per target. The event sentences of its varieties are now the episodes' texts.
common.py: episode, runs, noise, level, cadence, targets.

Benchmark Totals: Visibility found 47 -> 55 %, correct 77 -> 72 %; Obstruction found 26 -> 22 %, correct 28 -> 32 %;
the other families unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:28:52 +01:00
maximeandClaude Opus 5.5 5f085bf5ff One computation per module, an explicit list of detectors; obstruction.py removed
Each module returns every family it finds (kind); categories.py imports DETECTORS by name (no introspection, no
CONTEXT) and drops from Obstruction the sensors already in an overlapping Target issue episode (formerly
obstruction.py). No detector computed twice. Benchmark identical to the baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:06:24 +01:00
maximeandClaude Opus 5.5 5e9a7aa9d1 The single reader: data.py of the tools repository (sibling ~/aimsight-tools), located once in categories.py
scripts/data.py removed; the detectors' standalone CLIs go (categories.py runs them).
Benchmark identical to the baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:25:01 +01:00
maximeandClaude Opus 5.5 908369b9e9 Dépôts renommés avec le préfixe aimsight- : chemins et noms mis à jour
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:56:41 +01:00
maximeandClaude Opus 5.5 8e65ee3e08 CLAUDE.md: thresholds.md "Tuned on" follows a changed constant
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:48:44 +01:00
maximeandClaude Opus 5.5 c4fca737dc CLAUDE.md: the rules the upgrader and validator read (what changes, regression, everything is detected)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:41:17 +01:00
maximeandClaude Opus 5.5 2bba21c7e6 Optimize: the skill is /optimize
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:16:00 +01:00
maximeandClaude Opus 5.5 3aff4d154a data.py: the column formats are in tools/DATA.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:46:55 +01:00
maximeandClaude Opus 5.5 0abba2d0ed Label keys start/end/family in the bench and the docs (benchmark unchanged); knowledge path in data.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:40:03 +01:00
maximeandClaude Opus 5.5 b01a9116d0 README: the detectors and their benchmark (the workflow lives in Optimizer)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 12:13:17 +01:00
maximeandClaude Opus 5.5 3ea6b3e419 improve: S1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 11:56:56 +01:00
maximeandClaude Opus 5.5 e7c03e7304 Le benchmark dans bench/ ; plus de plugin : l'amélioration vit dans Optimizer, ses agents dans le dépôt agents
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 11:00:27 +01:00
maximeandClaude Opus 5.5 5438b0b469 wf-improve: S2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:03:02 +01:00
maximeandClaude Opus 5.5 7e3043d3a0 wf-improve: S1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:59:41 +01:00
maximeandClaude Opus 5.5 887b313ba2 wf-improve: several stations on the same range (DATA, STATIONS, a labels folder); regression.py --from/--to; the agents keep their scripts in OUT/tools/
Optimizer's daily batch labels every station on one day: one session for all of them. The single-station request (Diagnostic-app) is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:53:19 +01:00
maximeandClaude Opus 5.5 c7e34b49f7 Used through a detectors -> ~/detectors link by Diagnostic-app and Optimizer: one copy, no submodule
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 17:10:19 +01:00
maximeandClaude Opus 5.5 21a35d90aa Optimizer, second user of the contract (categories, scripts/data.py); labels and data from the labeling and aimsight-data repositories
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 17:01:14 +01:00
20 changed files with 628 additions and 1732 deletions

No files matched your search

-4
View File
@@ -1,4 +0,0 @@
{
"name": "detectors",
"description": "AIMsight anomaly detectors and their improvement on a range of labelled data (/detectors:wf-improve)."
}
+18 -12
View File
@@ -1,21 +1,27 @@
# detectors
AIMsight anomaly detectors (Python 3, standard library only) and the Claude Code plugin that improves them (`/detectors:wf-improve`). Everything is in English: code, identifiers, docs, outputs. Layout and usage: `README.md`.
AIMsight anomaly detectors (Python 3, standard library only) and their benchmark (`bench/`); nothing else. They are improved by Optimizer (`optimize` skill) with the upgrader and validator of the agents repository. Everything is in English: code, identifiers, docs, outputs. Layout and usage: `README.md`.
## Contract (do not break)
- One detector per family, in `scripts/`: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py` (D-119). `categories.py` loads every `scripts/*.py` that has `detect`: never put another script with a `detect` function, or with side effects at import, directly in `scripts/`; tools go in `skills/wf-improve/scripts/`.
- `detect(measurements)` returns episodes with the keys `start`, `end`, `kind` (the family), `label` (the variety), `text`, `ignore`, `sensors`. A detector may also declare `temperature`, `camera`, `metrics` (`categories.CONTEXT`) or `end`; `categories.events` passes them by introspection.
- Label files are external data and keep their French keys: `debut`, `fin`, `categorie`, `description`.
- Diagnostic-app uses this repository as its `detectors/` submodule: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the `--from/--to` CLIs, `/detectors:wf-improve`. A rename here needs the matching change there.
## The workflow stays generic
`skills/wf-improve/SKILL.md` and `agents/` describe only the workflow (range + labels → upgrader → validator → summary). Everything specific to a caller (its paths, its files, its git rules, where the summary goes) is given in the request at launch, never written here. The tools take their paths as arguments.
- `scripts/categories.py` runs the detectors of `DETECTORS` (modules of `scripts/`: `visibility`, `target_issue`, `thermal_displacement`, `no_data`, `displacement`) and makes their episodes events. A module returns every family it finds: `visibility` also gives Brief obstruction and `target_issue` Target obscured (Obstruction). `episodes.py` holds the helpers they share.
- `detect(measurements)` (`no_data`: also `end`, the end of the window read) returns episodes with the keys `start`, `end`, `kind` (the family), `label` (the variety), `text` (the sentence shown), `ignore`, `sensors`; a Poor visibility also has `level`.
- The measurements are read by `scripts/data.py` of the tools repository, the sibling `~/aimsight-tools` (`categories.py` puts it on the path); a reading fix is made there.
- Label files (JSONL): `id`, `start`, `end`, `family`, `description` (and `review`, `analysis`); the project uses `labeling/auto` only.
- Two repositories use this one through a `detectors -> ~/aimsight-detectors` link (one copy on the workstation: a commit here is live for both at once); a rename here needs the matching change there:
- aimsight-app: `categories.window` (the events of a range, with its history), `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the event fields (`start`, `end`, `category`, `text`, `sensors`, `ignore`, `episodes`), the `--from/--to` CLIs, `bench/score.py` (the command shown in the page);
- Optimizer (`optimize` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `compare.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix.
## Rules
- A constant added or changed updates `thresholds.md` in the same change; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
- A change to improve the detectors touches `scripts/*.py` and `thresholds.md` only; never `bench/`, which is the measure. Anything is possible there: add, remove or change a constant, a rule or a whole detector, as long as the seven families remain.
- **Simple first**: a rule earns its place by its score on the benchmark, not by one case; prefer removing or widening a rule to adding a special case. The benchmark ends with the complexity of the detectors (code lines and constants of `scripts/`), `bench/compare.py` shows it before → after. A fix that adds a constant or more than 20 code lines earns them with at least 1 point on the `Total` of a family ("found" or "correct"); otherwise it fits one range: simplify it or drop it. A change that removes code without a regression is a gain in itself.
- **Pruning**: after a fix, each rule of the detector it touched (a condition, a branch, a constant a plainer value could replace) is tried without it, on a copy in the working folder (the copy's `bench/regression.py`, then `bench/compare.py`); a rule whose removal makes no regression is removed, one commit per rule, with its line of `thresholds.md`.
- **Everything is detected**: what is shown or not to a client is not the detectors' business. A setting chosen not to show something may be changed, saying so.
- A constant added or changed updates its line of `thresholds.md` ("Tuned on": the run that changed it) and the detector's docstring in the same change; a constant or rule removed removes its line; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
- A detector that fails is skipped: a `skipped` line in the output of `categories.py` or of the benchmark means the change is broken.
- No rule specific to a station, a sensor or a date.
- A change to a detector is checked with the benchmark: `python3 skills/wf-improve/scripts/regression.py --labels <labels folder> --data <data folder>` (Diagnostic-app: `labeling/auto`, `data`; about 3 min). A refactoring must leave every figure unchanged.
- History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in Diagnostic-app/Support-Agent; keep them when editing a rule.
- A change to a detector is checked with the benchmark: `python3 bench/regression.py --labels <labels folder> --data <data folder>` (aimsight-app and Optimizer: `labeling/auto`, `data`, links to the shared `~/aimsight-labeling` and `~/aimsight-data/data`; about 1 min). A refactoring must leave every figure unchanged.
- **The benchmark's scores**, per family of each station: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family. **A regression**, computed by `python3 bench/compare.py <before.txt> <after.txt>` (its constants): a family `Total` whose "found" or "correct" loses more than 1 point; a family of a station whose "found" or "correct" loses more than 5 points and more than 20 instants (labelled instants no longer flagged, or wrongly flagged instants added), even if the `Total` rises; below, it is the noise of a few instants (a station at one measurement every 4 h, a family with few labels). A rise is a plus, never a requirement. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain: labels contradicting each other from one station to another (the same state in two families), the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score (per instant, without the sensors) counted as found by an event on another sensor. A drop counted as not a regression is said, with the labels (id). Labels have their ambiguities: the goal is detectors that do their best, not perfect, and not overtrained on the labels.
- History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in aimsight-app/Support-Agent; keep them short when editing a rule.
- The bench tools take their paths as arguments; the architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`.
+20 -35
View File
@@ -1,57 +1,42 @@
# detectors
The AIMsight anomaly detectors, and the workflow that improves them. Python 3, standard library only.
The AIMsight anomaly detectors and their benchmark. Python 3, standard library only.
## The detectors
One detector per anomaly family of the displacements, in `scripts/`:
Five modules in `scripts/`, one per kind of reading; each gives episodes of one or more of the seven families:
| Family | Detector |
|---|---|
| Visibility | `visibility.py` |
| Obstruction | `obstruction.py` |
| Target issue | `target_issue.py` |
| Thermal displacement | `thermal_displacement.py` |
| No data | `no_data.py` |
| Displacement | `displacement.py` |
| Module | Reads | Families (varieties) |
|---|---|---|
| `visibility.py` | error_value of each sensor; fast noise of the curves | Visibility (Poor visibility "none" or "reduced", Heat shimmer), Obstruction (Brief obstruction) |
| `target_issue.py` | rejections day after day | Target issue (Degraded target), Obstruction (Target obscured) |
| `thermal_displacement.py` | the daily wave of the curves | Thermal displacement (Thermal expansion / contraction, Camera thermal expansion / contraction, Camera thermal movement) |
| `no_data.py` | the cadence of the measurements | No data |
| `displacement.py` | jumps and slow drift of the corrected curves | Displacement (Lasting shift; Jump, not back yet; Slow drift), Perturbation (Sudden jump, back to normal) |
`categories.py` runs them all and classifies their episodes into events, one line per event; `data.py` reads a station folder (`displacement/`, `metrics/`, `camera/`).
`episodes.py` holds their shared helpers. `categories.py` runs them all and classifies their episodes into events (a jump back joins the visibility loss or the obstruction that explains it; the same variety less than an hour apart, one event), one line per event. The measurements are read by `scripts/data.py` of the tools repository, the sibling `~/aimsight-tools`.
```bash
python3 scripts/categories.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD # --chunk N, --history N
python3 scripts/visibility.py <station folder> # one detector alone
```
`thresholds.md` is the registry of the detectors' thresholds and what each was tuned on.
## Improving them: `/detectors:wf-improve`
## Measuring them: `bench/`
The repository is also a Claude Code plugin (`.claude-plugin/plugin.json`). Its workflow improves the detectors on a range of data where they perform poorly, given labels as the truth on that range:
1. measures before: the score on the range, and a benchmark on a reference truth (every labelled station);
2. the `upgrader` agent compares the detectors' events with the labels and fixes the detectors with general rules, one commit per fix;
3. the `validator` agent, an LLM-as-judge that does not read the upgrader's diagnosis, keeps the fixes, removes some, or cancels them: no station may go back, no overfitting;
4. a summary.
The result: detectors a little better on the range without going back elsewhere, or unchanged. Nothing is pushed.
The caller gives the context of each run: the range, the labels, the reference truth, the families document, any context documents, the output folder, and its own rules (git branch, what else to commit). For example:
The detectors against labels (JSONL, one per line, `{"id", "start", "end", "family", "description"}`, UTC ISO times):
```bash
claude --plugin-dir detectors -p "/detectors:wf-improve DETECTORS: detectors/. STATION: data/NORDEND. \
FROM: 2026-11-04. TO: 2026-11-08. LABELS: labeling/auto/NORDEND.jsonl. \
REFERENCE: --labels labeling/auto --data data. FAMILIES: labeling/familles.md. \
CONTEXT: labeling/auto/analyses/NORDEND_x/rapport.md. OUT: detectors/runs/NORDEND_x/."
python3 bench/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD # the score of a range
python3 bench/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> # the benchmark; --from/--to: one range for all
python3 bench/compare.py <before.txt> <after.txt> # two benchmarks: the regressions, the complexity
python3 bench/sensors.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD # the sensors of each event
```
Labels: JSONL, one per line, `{"id", "debut", "fin", "categorie", "description"}` (UTC ISO times, `categorie` = the family).
A change to a detector is checked with the benchmark; a refactoring leaves every figure unchanged.
- `skills/wf-improve/SKILL.md`: the workflow; its tools in `skills/wf-improve/scripts/`: `score.py` (the score of a range), `regression.py` (the benchmark), `sensors.py` (the sensors of each event).
- `agents/`: `upgrader`, `validator` (called `detectors:<name>`).
## Improving them
```bash
python3 skills/wf-improve/scripts/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD
python3 skills/wf-improve/scripts/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/>
```
The detectors are improved by Optimizer (its `optimize` skill, from one analysis), with the `upgrader` and `validator` agents of the agents repository: on the range of an analysis, its labels as the truth, the upgrader fixes the detectors (one commit per fix here) and the validator keeps, removes or cancels the fixes against overfitting, judged by this benchmark. Nothing is pushed.
Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here. Used by Diagnostic-app, where it is the `detectors/` submodule.
Copied from Support-Agent on 2026-11-13 (commit 48a3092), simplified on 2026-11-16 (branch `simplify`); the detectors now evolve here. Used by aimsight-app and Optimizer, through a `detectors -> ~/aimsight-detectors` link (one copy on the workstation); their labels come from the `labeling` repository, their data from `aimsight-data`. The architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`.
-111
View File
@@ -1,111 +0,0 @@
---
name: upgrader
description: Upgrader of the wf-improve workflow — on a range of data where the detectors perform poorly, compares what they find, all families, with the labels of the range (the truth), then fixes the detectors (adds, removes, changes) so that they do better there without overfitting, one commit per fix. Called by the orchestrator with the range, the labels, the families, the commands, the base commit and its output folder.
tools: Read, Edit, Write, Bash
model: inherit
---
You are the upgrader of the wf-improve workflow: you make the detectors better on a range of data of a station where they perform poorly. The labels of the range are the truth. You compare what the detectors find with these labels, event by event, all families, say what is wrong and which detector causes it, then fix it. Afterwards, the `validator` judges your modification on the reference truth (all labelled stations, all families): a gain on the range at the cost of another station or another family is refused.
**Avoid overfitting**: a small step. A single range must not dictate the detectors: do better here with rules that would hold on any station, without aiming to reproduce exactly these labels. A constant tuned to make these bounds fall exactly, a rule that only recognises the exact shape of these events: no. What counts is the mechanism behind a label (a rejected reference that makes all targets jump, for example), recognisable wherever it happens. What the detectors cannot see with these measurements (a cause only an image reveals) is not a fix: say it in "Limits".
## Your request
The detectors repository (`DETECTORS`: the detectors `scripts/*.py`, their threshold registry `thresholds.md`), the station's data folder and the range (UTC), the labels file, the families document, the context documents if any, the events, sensors, range score and benchmark commands with their outputs before, the base commit, the files already modified, the output folder and the path of your output `modification.md`; sometimes the caller's context, which comes before the instructions below, not before "What you change".
A label: one line of JSONL, `{"id", "debut", "fin", "categorie", "description"}` (cited by its `id`), a single anomaly, seen on `corr_x` and `corr_y` (they move or are missing); `categorie` is its **family**; `description` says the sensors touched, the cause and how it was known. Its bounds go from the measurement just before the first point touched to the measurement just after the last. An instant of the range without a label: the corrected curves show nothing there.
## 1. Diagnosis
Read:
- the labels touching the range (`debut` < `TO` and `fin` ≥ `FROM`), and the context documents: the mechanism of each event, the sensors touched and those that do not move, the level of proof; a document may say that a label is wrong: a limit to report, not a target;
- the families document **in full**: the families and their boundaries; the category of an event is its family (one detector per family);
- the outputs of the events command (run with `2>&1`) and of the sensors command (the cadence, then the sensors of each event); the range score; the benchmark before;
- the code of the detectors, `scripts/categories.py` (how episodes become events: a jump back absorbed by its cause, the varieties) and `thresholds.md`.
Compare, event by event and label by label: correct; wrong family; extra (flagged by the detectors, without a label); wrong bounds (late, too early, too wide); wrong sensors (the event's, by the sensors command, against those the description names); missing. A few measurements of gap at the bounds are not a defect. Several varieties (the column after the family: Poor visibility, Heat shimmer…) fall into one family: between them, the label does not decide. A reading of the detectors other than the label's but consistent (a boundary of the families document that allows both) is not a defect either: an "alternative reading", no fix.
Then the fixes to try, from the most important to the least: a significant event missed or badly bounded before a small nuisance, a mechanism that comes back several times on the range before an isolated case. A fix is one you can make with a simple, general rule, on the measurements alone, without fitting the range and without making other stations or families go back; otherwise it goes to "Not fixed" or "Limits". You may read the station's data to understand why a rule fails.
If nothing important is to be fixed: verdict `nothing to change`, write your output and stop.
## 2. What you change
Only the detectors (`<DETECTORS>/scripts/*.py`) and `<DETECTORS>/thresholds.md`; never `skills/`, `agents/`, nor anything outside `DETECTORS` but your output folder. On them, anything is possible: add, remove or change a constant, a rule or a whole detector. You never commit a file the request says was already modified.
**Everything is detected**: what is shown or not to a client is not the detectors' business. A setting chosen not to show something does not stop you: you may change it, saying so in "What changed".
**The simplest first**, in this order; you only go to the next if the previous cannot fix the defect:
1. remove or simplify: a rule, a constant or a detector that causes the defect, or that another rule makes useless;
2. change a constant;
3. adjust an existing rule;
4. add a new, short rule;
5. as a last resort, a new variety (a `label`) in the family's detector.
A small gain is not worth one more rule: better to fix half simply than everything with a contraption.
**Forbidden.**
- One detector per family: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py`, no more, no less; the `kind` of an episode is its family. A fix that needs a new family goes to "Not fixed".
- No rule specific to a station, a sensor or a date.
- The contract `detect(measurements)` (a detector may also declare `temperature`, `[(t, °C)]`; `camera`, `[{t, pan, tilt, rotation, scale, sigma0}]`; `metrics`, `{name: [(t, value)]}`; `end` for `no_data.py`; a station may have none of them: the rule then holds without them) and the keys of the episodes (`start`, `end`, `kind`, `label`, `text`, `ignore`, `sensors`) do not change.
- Standard library only.
**In the same change**: a constant added or changed updates its line of `thresholds.md` ("Tuned on": this run) and the detector's docstring; a constant or detector removed removes its line. Check: `grep -nE "^[A-Z_0-9]+ = " <DETECTORS>/scripts/*.py`.
## 3. Check
**On the range**, you alone check event by event: the validator only reads the scores and the code. Run the events command with `2>&1` **before** changing and **after**, and the range score command. After, no `skipped` line: a detector that no longer loads silently disappears.
Then the **benchmark** against the benchmark before: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family.
- **The range score**: what you make rise, without losing a whole event. A hole has no measurement: it only counts its two bounds; for a hole, read the events command. Do not aim for 100 %.
- **All lines of all stations**: the non-regression, the same rule as the validator. **A regression**, on a family of a station: "found" or "correct" loses more than 2 points; it is a regression even if the `Total` rises. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain (labels contradicting each other from one station to another, the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score counted as found by an event on another sensor). A regression: rework your change before committing it, or drop the fix. A drop from another consistent reading: keep it, and say it in "For the user to decide".
Your working folder is yours alone: `mktemp -d`. To try a variant without touching the repository, copy `DETECTORS` there: the copy's scripts measure the copy, with the same arguments. For a shorter run, `--stations A,B` measures only these stations; the whole benchmark runs in the background, output to a file.
## 4. One fix, one commit
The validator can have a single fix removed without touching the others: each has its commit. For each fix, in order:
1. change, then check as above, against the benchmark of the last committed state (the benchmark before, or that of your previous kept fix); before your output, the whole against the benchmark before; keep the output of the benchmark in `<output folder>/benchmark-S<n>.txt`;
2. kept: `git -C <DETECTORS> add <the changed files> && git -C <DETECTORS> commit -m "wf-improve: S<n>" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" -- <the same>`;
3. not kept: `git -C <DETECTORS> checkout -- <the files you changed for it>`, only them, and it goes to "Not fixed".
Never `git add -A`, `git stash`, or `git checkout` on other files: another session may be working there.
## Your output, `modification.md`
```markdown
# Modification
**Verdict**: changed | nothing to change
## Diagnosis
<station, range, cadence>; | Family | labelled | found | flagged | wrongly |
<one line per event or label: judgement (correct, wrong family, extra, wrong bounds, wrong sensors, missing, alternative reading) and why; same judgement and cause grouped>
## What changed
<per fix kept, S<n> and its commit: file, constant, rule or detector added, removed or changed; old and new value>
## Why
<the defects fixed, one or two sentences each, without station data>
## Before / after on the range
<the changed lines of the two outputs of the events command; the range score before → after, per family>
## Benchmark
<the lines that changed, station and family, before → after, then the changed lines of the `Total` block>
## For the user to decide
none | - <a reading of the detectors other than the labels', kept: the labels (id, station, moment, family), the detectors' reading, what it changes in the benchmark>
## Not fixed
none | <what remains, and why>
## Limits
none | - <what the detectors cannot see with the measurements alone: the label (id, bounds, family), what it would take to see it (image…); a label the context says is wrong (id)>
```
The Verdict line carries one of these two words; the orchestrator reads it as it is. The validator only reads "What changed" and "Why", with the diff: these two parts stand alone, without referring to a station's data. Times in UTC.
## Your answer
In 3 lines: the verdict and the fixes kept with their commits, what was not fixed, the path of the output.
-67
View File
@@ -1,67 +0,0 @@
---
name: validator
description: Validator of the wf-improve workflow — the LLM-as-judge against overfitting: judges a modification of the upgrader, made for one range of data, on the reference truth (all labelled stations and families) and on the code of the detectors, without reading the diagnosis; keeps it, has part of it removed, or cancels it. Called by the orchestrator with the modification, the benchmarks, the range score, the events command and the base commit.
tools: Read, Bash, Write
model: inherit
---
You are the validator of the wf-improve workflow, its safeguard. The upgrader has just adapted the detectors to better find the events of a single range of a single station. The risk is overfitting: detectors glued to this range, which lose precision elsewhere, on another station or another family. You check it, and you decide in a single pass: keep the modification, have part of it removed, or cancel it.
**You stay independent**: you do not read the upgrader's diagnosis. Your conclusions come from the benchmark on the reference truth and from the code of the detectors.
## Your request
The detectors repository (`DETECTORS`: the detectors `scripts/*.py`, `thresholds.md`), the families document, the modification (`modification.md`), the range score before and after, the benchmark before and after and after each fix (`benchmark-S<n>.txt`) with its command, the events command, the base commit, the path of your output `validation.md`; sometimes the caller's context, which comes before the instructions below (including the reasons of "Your decision"), not before "What you read".
## What you read, only
- the range score before → after: the gain the modification sought;
- in `modification.md`, the parts "What changed" and "Why", nothing else: `sed -n '/^## What changed/,/^## Before/p' <modification>`;
- the diff of the detectors since the base: `git -C <DETECTORS> diff <base> HEAD -- 'scripts/*.py' thresholds.md`, and the code it touches; each fix has its commit (`git -C <DETECTORS> log --oneline <base>..HEAD`, message `wf-improve: S<n>`), its diff by `git -C <DETECTORS> show <commit>`;
- the benchmarks (the score per family on each station of the reference truth, then their `Total`), all stations in full; the benchmarks after each fix, in order, say which one makes a station go back; you may rerun the benchmark command;
- to understand a regression on a station: the events command on its range (from the header of the station's block in the benchmark), with `2>&1`, at HEAD and, to compare, at the base (`git -C <DETECTORS> archive <base> | tar -x -C <folder>` in your working folder, then the same command with the copy's `scripts/categories.py`), the station's labels, and the families document.
Nothing else: neither the diagnosis nor earlier runs.
## What you judge
1. **The detectors load**: a `skipped` line in a benchmark (a detector that no longer loads silently disappears) means "cancel".
2. **No station goes back.** The benchmark compares the detectors with the labels of each station, per family: "found" (the share of the labelled instants the detectors flag) and "correct" (the share of the flagged instants a label puts in the family). Compare them before → after, station by station and family by family, the range's station included:
- **A regression**, on a family of a station: "found" or "correct" loses more than 2 points; it is a regression even if the `Total` rises. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain: labels contradicting each other from one station to another (the same state in two families), the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score (per instant, without the sensors) counted as found by an event on another sensor. Labels have their ambiguities: the goal is detectors that do their best, not perfect, and not overtrained on the labels. A drop you do not count as a regression goes to "Reservations";
- under 2 points, it is not a regression;
- a rise is a plus, never a requirement.
3. **The code stays simple, the gain justifies it**: the modification is in proportion to what it fixes ("Why") and to what the range score gains; a range gain of nothing or almost, at the price of one more rule, means "cancel" (or "remove" for the fix in question); a rule removed is better than a rule added. **No overfitting**: a rule or a constant that only holds by fitting the range (values cut for its bounds, the exact shape of its events) is a defect, even without a regression elsewhere.
4. **The forbidden**: one detector per family (`visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py`, no more, no less, and `kind` is the family); no rule specific to a station, a sensor or a date (look for names or instants in the diff); contract `detect(measurements)` (plus `temperature`, `camera`, `metrics` or `end` if the detector declares them) and episode keys (`start`, `end`, `kind`, `label`, `text`, `ignore`, `sensors`) unchanged; standard library; `thresholds.md` up to date (`grep -nE "^[A-Z_0-9]+ = " <DETECTORS>/scripts/*.py`, against the registry).
You change no file but your output. Your working folder is yours alone: `mktemp -d`.
## Your decision
A single pass: no sending back to the upgrader. The upgrader must be able to progress: do not cancel for perfection. **Keep** if nothing goes back, the code stays simple and the range gain justifies it; **remove** when the defect (a regression, a disproportionate complexity, an overfitting, a forbidden) is due to some fixes only, and the others are good without them: the benchmarks after each fix say which one; **cancel** otherwise, or for a skipped detector.
## Your output, `validation.md`
```markdown
# Validation
**Verdict**: keep | remove | cancel
**Remove**: S<n>, S<m> (only with the verdict remove)
## Answers
<the four points judged, one line each>
## The range
<the range score before → after, per family>
## Non-regression
<the lines that changed, station and family, before → after, then the changed lines of the `Total` block>
## Reservations
none | - <what you keep without counting it as a regression, for the user: the station, the family, before → after, the detectors' reading against the labels (id)>
```
The Verdict line carries a single one of these three words, the Remove line only the fixes to remove; the orchestrator reads them as they are.
## Your answer
In 2 lines: the verdict, the path of the output.
+61
View File
@@ -0,0 +1,61 @@
"""Two benchmark outputs (regression.py), before and after: what changed, and the regressions by the definition of
CLAUDE.md, computed.
- A family Total: "found" or "correct" loses more than TOTAL_POINTS.
- A family of a station: "found" or "correct" loses more than POINTS and more than INSTANTS instants ("found": labelled
instants no longer flagged; "correct": wrongly flagged instants added). Below, the loss is noise of a few instants.
Then the complexity before → after. The verdict is the reader's: a drop the labels explain (CLAUDE.md, "Not a
regression") is said, not hidden.
Usage: python3 compare.py <before.txt> <after.txt>
"""
import re
import sys
TOTAL_POINTS = 1
POINTS = 5
INSTANTS = 20
CELL = r"(\d+)/(\d+) \(\d+ %\)|—"
def parse(path):
"""Benchmark output -> ({(station, family): (both, labelled, flagged)}, complexity line or '')."""
out, station, cx = {}, None, ""
for line in open(path, encoding="utf-8"):
if line.startswith("Complexity:"):
cx = line.split(":", 1)[1].strip()
elif line.strip() and not line.startswith(" "):
station = line.split(":")[0].strip()
elif m := re.match(rf" (\S.*?)\s{{2,}}({CELL})\s+({CELL})\s*$", line):
g = m.groups() # family, found (both, labelled), correct (both, flagged)
out[(station, g[0])] = (int(g[2] or g[5] or 0), int(g[3] or 0), int(g[6] or 0))
return out, cx
def pct(a, b):
return 100 * a / b if b else None
def changes(before, after):
"""-> [(station, family, measure, before %, after %, lost instants, regression)] for every measure that moved."""
rows = []
for key in sorted(set(before) | set(after)):
(b0, l0, f0), (b1, l1, f1) = before.get(key, (0, 0, 0)), after.get(key, (0, 0, 0))
for measure, p0, p1, lost in (("found", pct(b0, l0), pct(b1, l1), b0 - b1),
("correct", pct(b0, f0), pct(b1, f1), (f1 - b1) - (f0 - b0))):
if p0 is None or p1 is None or round(p0) == round(p1):
continue
drop = p0 - p1
bad = drop > TOTAL_POINTS if key[0] == "Total" else drop > POINTS and lost > INSTANTS
rows.append((*key, measure, p0, p1, lost, bad))
return rows
if __name__ == "__main__":
if len(sys.argv) != 3:
sys.exit(__doc__)
(before, cx0), (after, cx1) = parse(sys.argv[1]), parse(sys.argv[2])
rows = changes(before, after)
for station, family, measure, p0, p1, lost, bad in rows:
print(f"{'REGRESSION' if bad else '':<11}{station:<13}{family:<22}{measure:<8}{p0:4.0f} → {p1:3.0f} %"
+ (f" ({lost} instants lost)" if p1 < p0 else ""))
n = sum(r[-1] for r in rows)
print(f"\n{n} regression(s); complexity: {cx0 or '?'} → {cx1 or '?'}")
@@ -1,12 +1,15 @@
"""Benchmark of the detectors against a reference truth: every labelled station.
For each <STATION>.jsonl of --labels that has its folder <STATION>/ in --data, the score per family and per instant of
score.py, over the range of its labels: from the day of the first to the midnight after the last. Then the total,
family by family, which says whether the detectors do better overall. The block of a station is its score; the
others, the non-regression: a family that goes back on a station is a regression, even if the total rises.
score.py, over the range of its labels: from the day of the first to the midnight after the last; with --from/--to, over
that range for every station, a station without a label on it included (several stations labelled the same days). Then the total,
family by family, which says whether the detectors do better overall, and the complexity of the detectors (code lines
and constants of scripts/). Two outputs are compared by compare.py (the regressions, CLAUDE.md).
Usage: python3 regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> [--stations A,B]
[--from YYYY-MM-DD --to YYYY-MM-DD]
"""
import argparse
import re
from datetime import timedelta
from pathlib import Path
@@ -15,13 +18,20 @@ import score # next to it
from data import iso, parse_time # on the path through score
def complexity(scripts=Path(__file__).resolve().parents[1] / "scripts"):
"""-> (code lines, constants) of the detectors: non-blank lines that are not a comment; NAME = lines at module level."""
lines = [x for f in sorted(scripts.glob("*.py")) for x in f.read_text().splitlines()]
code = [x for x in lines if x.strip() and not x.strip().startswith("#")]
return len(code), sum(bool(re.match(r"[A-Z][A-Z_0-9]* = ", x)) for x in lines)
def label_range(path):
"""Labels JSONL -> (start, end) iso, in whole UTC days, end excluded."""
labels = score.read_labels(path)
start = min(e["debut"] for e in labels)[:10]
start = min(e["start"] for e in labels)[:10]
# ponytail: the end of the labelled batch is not written; midnight closest to the last bound (a label bound
# overflows by one measurement after midnight)
end = iso(parse_time(max(e["fin"] for e in labels)) + timedelta(hours=12))[:10]
end = iso(parse_time(max(e["end"] for e in labels)) + timedelta(hours=12))[:10]
return iso(parse_time(start)), iso(parse_time(end))
@@ -30,7 +40,11 @@ if __name__ == "__main__":
p.add_argument("--labels", required=True, help="folder of the labels, one <STATION>.jsonl per station")
p.add_argument("--data", required=True, help="folder of the station folders, <STATION>/")
p.add_argument("--stations", help="A,B: these stations only (a shorter run)")
p.add_argument("--from", dest="start", help="YYYY-MM-DD: this range for every station, not the range of its labels")
p.add_argument("--to", dest="end", help="YYYY-MM-DD, excluded (with --from)")
a = p.parse_args()
if bool(a.start) != bool(a.end):
p.error("--from and --to go together")
total = {}
for path in sorted(Path(a.labels).glob("*.jsonl")):
station = path.stem
@@ -40,10 +54,11 @@ if __name__ == "__main__":
if not folder.is_dir():
print(f"{station}: no {folder}/, skipped\n")
continue
start, end = label_range(path)
start, end = (iso(parse_time(a.start)), iso(parse_time(a.end))) if a.start else label_range(path)
n, labels = score.family_scores(str(folder), str(path), start, end)
score.show(f"{station}: {start} → {end}, {len(labels)} labels", n)
print()
for k, v in n.items():
total[k] = [x + y for x, y in zip(total.get(k, [0, 0, 0]), v)]
score.show("Total", total)
print("\nComplexity: %d code lines, %d constants" % complexity())
@@ -6,7 +6,7 @@ event is its family.
found: labelled instants that the detectors also flagged (share of the labelled ones);
correct: flagged instants that a label also puts in the family (share of the flagged ones).
The score is a fact, not a verdict: the labels can be wrong too. No label on the range: exits with an error.
Labels: a JSONL file, one label per line, {"id", "debut", "fin", "categorie", "description"} (UTC ISO times, categorie =
Labels: a JSONL file, one label per line, {"id", "start", "end", "family", "description"} (UTC ISO times);
the family); its bounds are the healthy measurement before and after the anomaly.
Usage: python3 score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD
"""
@@ -17,9 +17,9 @@ import sys
from datetime import timedelta
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts")) # the detectors
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts")) # the detectors
import categories # noqa: E402
from data import iso, load, parse_time # noqa: E402
from data import iso, load, parse_time # noqa: E402 (the tools repository's reader, on the path through categories)
HISTORY = timedelta(days=3) # as the batch command of categories.py
@@ -48,7 +48,7 @@ def read_labels(path, start=None, end=None):
return []
with open(path, encoding="utf-8") as f:
return [e for e in map(json.loads, filter(str.strip, f))
if (end is None or e["debut"] < end) and (start is None or e["fin"] >= start)]
if (end is None or e["start"] < end) and (start is None or e["end"] >= start)]
def family_scores(folder, labels_path, start, end):
@@ -60,11 +60,11 @@ def family_scores(folder, labels_path, start, end):
evts = events(measurements, start, end, folder)
instants = sorted({iso(m["t"]) for ms in measurements.values() for m in ms if start <= iso(m["t"]) < end})
# bounds excluded: they are the healthy measurement before and after; a hole, without a measurement inside, keeps them
empty = {i for i, e in enumerate(labels) if not any(e["debut"] < t < e["fin"] for t in instants)}
empty = {i for i, e in enumerate(labels) if not any(e["start"] < t < e["end"] for t in instants)}
pairs = []
for t in instants:
a = {e.get("categorie", "no family") for i, e in enumerate(labels)
if e["debut"] < t < e["fin"] or (i in empty and e["debut"] <= t <= e["fin"])} - {"None"}
a = {e.get("family", "no family") for i, e in enumerate(labels)
if e["start"] < t < e["end"] or (i in empty and e["start"] <= t <= e["end"])} - {"None"}
d = {v["category"] for v in evts if v["start"] <= t <= v["end"]}
pairs.append((a, d))
return count(pairs), labels
@@ -9,9 +9,9 @@ import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts"))
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
import categories # noqa: E402
from data import iso, load # noqa: E402
from data import iso, load # noqa: E402 (the tools repository's reader, on the path through categories)
p = argparse.ArgumentParser(usage=__doc__)
p.add_argument("folder")
+75 -171
View File
@@ -1,58 +1,36 @@
"""From the detectors' episodes to events, one per family.
"""From the detectors' episodes to events, one family each.
One detector per label family (the families document): visibility.py, obstruction.py, target_issue.py,
thermal_displacement.py, no_data.py, displacement.py. The category of an event is its family, the kind field of its
episodes; their label gives the variety (Poor visibility, Heat shimmer, Brief obstruction, Slow drift…). Rules:
- a jump back to normal (JUMP_BACK) that overlaps an episode of HOSTS (loss of visibility, something passing, heat
shimmer, and last the thermal effect, if it touches several targets) is absorbed into it: it takes its cause and
family; otherwise it stays Displacement, cause not established, to be checked on the images (the families document, "A jump then
the return"); a Brief obstruction (one instant) only absorbs the jumps starting at its instant, not those of several
days it crosses (wf-improve session Displacement, mod. 1, S5);
- a slow drift or a jump back whose targets are all in an overlapping Target issue episode is absorbed into it (the
curve of a problem target is to be ignored, the families document);
- a shift that lasts, or too recent to know (SHIFTS), is never absorbed;
- heat shimmer overlapping a shift of several targets is not kept: a real movement also shakes the curves; the shift of
a single target does not remove the heat shimmer of the whole scene, and heat shimmer only absorbs the jumps back
starting during it, not the one of the day before coming back within its day (wf-improve session Visibility, mod. 1,
S1: WILDSTRUBEL);
- two events of the same variety less than MERGE apart make one; two Obstruction (Target obscured, Brief obstruction)
only if they share a sensor (wf-improve session Obstruction, mod. 1: S1, different targets chained into an event of
several days; S3, passings of a few minutes into a whole hour); two Target issue too: each problem target keeps its
bounds (wf-improve session Target issue, mod. 1, S3); a Brief obstruction only absorbs the jumps back within MERGE after
its instant (S3: not the 30 h jump of another target).
Each event gets its sentence, in English (the site): what happened and what to do about it, as the "For the client"
column of the families document says; the cause is only given if the variety establishes it, otherwise the images show it.
No detect function: the loader does not take this file for a detector.
Usage: python3 scripts/categories.py <station folder> (one line per event: family, variety, sensors)
... <folder> [--from YYYY-MM-DD] [--to YYYY-MM-DD] [--chunk 3] [--history N] (batch run: each chunk of N days
classified alone, with the days of history before it if they exist; only the events touching the chunk are
shown; "nothing" for a quiet chunk)
Each module of DETECTORS (next to this file) returns episodes: start, end, kind (the family), label (the variety), text,
ignore, sensors (a Poor visibility also has its level). classify() makes them events:
- a target already in an overlapping Target issue episode is dropped from an Obstruction episode: its curve is already
to be ignored;
- a jump back to normal (JUMP_BACK) overlapping a Visibility episode joins its event: rain, fog or heat shimmer
explains it; otherwise it is a Perturbation: nothing to do;
- a jump back whose targets are all in an overlapping Obstruction or Target issue episode joins it (something in front
of them explains it, or the curve of a problem target is to be ignored); a slow drift, in a Target issue episode;
- episodes of the same variety less than MERGE apart make one event (validated rule, 2026-10-24); an Obstruction or a
Target issue only with an episode of the same target (wf-improve Obstruction, mod. 1, S1; Target issue, mod. 1, S3).
The text of an event is that of its first episode, plus a sentence for the jumps it absorbed.
Usage: python3 scripts/categories.py <station folder> [--from YYYY-MM-DD] [--to YYYY-MM-DD] [--chunk N] [--history N]
one line per event; --chunk: each chunk of N days classified alone, with the days of history before it; only the
events touching the chunk are shown, "nothing" for a quiet chunk.
"""
import functools
import importlib.util
import inspect
import os
import importlib
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
from data import iso, load, load_camera, load_metrics, load_temperature # noqa: E402
# The detectors next to this file, and the single reader of the measurements: data.py of the tools repository, the
# sibling of this one (one copy of each repository in the home directory), else the tools/ link of the caller's root
# (a copy of this repository elsewhere, e.g. extracted by git archive).
HERE = Path(__file__).resolve().parent
sys.path[:0] = [str(HERE), str(HERE.parents[1] / "aimsight-tools" / "scripts"), str(Path.cwd() / "tools" / "scripts")]
from episodes import targets # noqa: E402
from data import iso, load # noqa: E402
# What a detector can read from the station besides the measurements, if it declares it in detect (D-124, D-134):
# temperature [(t, °C)], hourly median of the components; camera [pose] (pan, tilt, rotation, scale, sigma0);
# metrics {name: [(t, value)]}, all numeric metrics (sensor_CPU_deg_c, sensor_GPU_deg_c, BusVoltage…).
CONTEXT = {"temperature": load_temperature, "camera": load_camera, "metrics": load_metrics}
@functools.lru_cache(maxsize=None)
def _context(folder, key):
return CONTEXT[key](folder)
# The families, in the order of the timeline rows; ignore: greyed zone (None: as its episodes
# say: a total loss of visibility, strong heat shimmer); process: Images button, nothing, or Images and a link to
# Diagnosis.
DETECTORS = ("displacement", "no_data", "target_issue", "thermal_displacement", "visibility")
# The families, in the order of the timeline rows; ignore: greyed zone (None: as its episodes say); process: Images
# button, nothing, or Images and a link to Diagnosis.
CATEGORIES = {
"Visibility": {"ignore": None, "process": "images"},
"Obstruction": {"ignore": True, "process": "images"},
@@ -60,154 +38,82 @@ CATEGORIES = {
"Thermal displacement": {"ignore": False, "process": "none"},
"No data": {"ignore": False, "process": "none"},
"Displacement": {"ignore": False, "process": "check"},
"Perturbation": {"ignore": True, "process": "none"},
}
JUMP_BACK = "Sudden jump, back to normal" # a jump back: its cause, if it overlaps a host; otherwise Displacement
SHIFTS = {"Lasting shift", "Jump, not back yet", "Large shift"} # to be checked: never absorbed
HOSTS = {"Poor visibility", "Brief obstruction", "Heat shimmer", "Thermal expansion", "Thermal contraction",
"Camera thermal expansion", "Camera thermal contraction", "Camera thermal movement"} # give the cause of an
# overlapping jump back
MERGE = timedelta(hours=1) # same variety less than an hour apart: a single event (validated rule, 2026-10-24)
def detectors():
"""Modules of the *.py next to this file that have detect; a file that does not load is skipped with a message."""
mods = []
for path in sorted(Path(__file__).resolve().parent.glob("*.py")):
if path == Path(__file__).resolve():
continue
try:
spec = importlib.util.spec_from_file_location(f"detector_{path.stem}", path)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
except Exception as e: # noqa: BLE001 (a broken detector does not block the page)
print(f"Detectors: {path.name} skipped ({type(e).__name__}: {e})", file=sys.stderr)
continue
if hasattr(mod, "detect"):
mods.append(mod)
return mods
JUMP_BACK = "Sudden jump, back to normal" # the label of displacement.py
MERGE = timedelta(hours=1)
def variety(ep):
"""The variety of an episode within its family: shifts together, the thermal effect without "Camera", the loss of
visibility by level."""
label = "shift" if ep["label"] in SHIFTS else ep["label"].removeprefix("Camera ").capitalize()
return ep["kind"], label, ep.get("level")
def is_thermal(ep):
return ep["kind"] == "Thermal displacement"
return ep["label"], ep.get("level")
def overlap(a, b):
return a["start"] <= b["end"] and a["end"] >= b["start"] # data.iso times: text order is time order
def targets(n):
return "one target" if n == 1 else f"{n} targets"
def host(e, eps):
"""The episode whose event a jump back or a slow drift joins, or None."""
kinds = {JUMP_BACK: ("Obstruction", "Target issue"), "Slow drift": ("Target issue",)}.get(e["label"], ())
return next((h for h in eps if overlap(e, h) and (h["kind"] == "Visibility" and e["label"] == JUMP_BACK
or h["kind"] in kinds and set(e["sensors"]) <= set(h["sensors"]))), None)
def sentence(group, sensors, references):
"""Sentence of an event (validated by the user on 2026-10-24), from the variety of its main episode; an absorbed
jump adds that it has the same cause."""
n, one, ep = len(sensors), len(sensors) == 1, group[0]
jumps = [e for e in group[1:] if e["label"] == JUMP_BACK]
jumped = len({s for e in jumps for s in e["sensors"]})
lasting = len({s for e in group if e["label"] in SHIFTS for s in e.get("lasting", [])})
refs = ", including the reference targets" if references & set(sensors) else ""
camera = ep["label"].startswith("Camera")
large = any(e["label"] == "Large shift" for e in group)
sort = "shift" if ep["label"] in SHIFTS else f"{ep['label']} {ep['level']}" if ep.get("level") else ep["label"]
text = {
"Poor visibility none": f"The camera could not see {targets(n)} clearly{refs}. "
"The curves in this period are not reliable: ignore them.",
"Poor visibility reduced": f"The camera saw {targets(n)} less clearly than usual{refs}. "
"The data in this period are less precise: use them with care.",
"Brief obstruction": f"Something passed in front of {targets(n)} for a single measurement. Ignore this point on the curves.",
"Target obscured": f"{', '.join(sensors)} {'was' if one else 'were'} hidden several times while the targets around "
f"{'it' if one else 'them'} stayed clear. Ignore {'its curve' if one else 'their curves'} in this period.",
"Heat shimmer": "Warm air made the readings shake. The small fast wiggles are not movement.",
"expansion": ("The camera support expands in the day heat; the curves are corrected for it."
if camera else "The structure expands in the day heat.") + " This daily wave is normal: nothing to do.",
"contraction": ("The camera support contracts as it cools down; the curves are corrected for it."
if camera else "The structure contracts as it cools down.") + " This daily wave is normal: nothing to do.",
JUMP_BACK: f"{targets(n).capitalize()} jumped and came back. The cause is not established: look at the images.",
"shift": (f"{targets(n).capitalize()}, most of the targets, shifted together: everything moved." if large
else f"{targets(n).capitalize()} shifted and {'has' if one else 'have'} not come back yet." if not lasting
else f"{targets(n).capitalize()} shifted and stayed shifted." if lasting == n
else f"{targets(n).capitalize()} shifted; {lasting} of them stayed shifted.")
+ " Check whether the structure moved: look at the images.",
}.get(sort.split()[-1] if is_thermal(ep) else sort, ep["text"]) # No data, Target issue, Slow drift: the detector's text
if ep["kind"] == "Target issue":
text += f" Ignore {'its curve' if one else 'their curves'} in this period and look at the images of the target."
if not jumped:
return text
return text + (f" The jump of {targets(jumped)} in this period has the same cause." if len(jumps) == 1
else f" The jumps of {targets(jumped)} in this period have the same cause.")
def event(group):
"""Episodes of an event, the main one first -> event."""
cat, c = group[0]["kind"], CATEGORIES[group[0]["kind"]]
main = [e for e in group if variety(e) == variety(group[0])] # the duration is theirs, not the absorbed jumps'
jumped = len({s for e in group if e["label"] == JUMP_BACK and e not in main for s in e["sensors"]})
text = group[0]["text"] + (f" The jumps of {targets(jumped)} in this period have the same cause." if jumped else "")
return {"start": min(e["start"] for e in main), "end": max(e["end"] for e in main), "category": cat, "label": cat,
"text": text, "ignore": c["ignore"] if c["ignore"] is not None else any(e["ignore"] for e in main),
"sensors": [] if any(not e["sensors"] for e in group) else sorted({s for e in group for s in e["sensors"]}),
"process": c["process"], "episodes": group}
def event(group, references):
"""Episodes of the same event, the main one first -> event."""
cat, var = group[0]["kind"], variety(group[0])
c = CATEGORIES.get(cat, {"ignore": None, "process": "none"})
sensors = [] if any(not e["sensors"] for e in group) else list(dict.fromkeys(s for e in group for s in e["sensors"]))
main = [e for e in group if variety(e) == var] # the duration is the main episodes', not the absorbed jumps'
return {"start": min(e["start"] for e in main), "end": max(e["end"] for e in main), "category": cat,
"label": cat, "text": sentence(group, sensors, references),
"ignore": c["ignore"] if c["ignore"] is not None else any(e["ignore"] for e in main),
"sensors": sensors, "process": c["process"], "episodes": group}
def classify(episodes, references=frozenset()):
"""Episodes of all detectors -> events sorted by start (rules: module docstring); references: the global_ref
sensors, for the sentence."""
def classify(episodes):
"""Episodes of all detectors -> events sorted by start (rules: module docstring)."""
eps = sorted(episodes, key=lambda e: e["start"])
# a real movement is never taken for heat shimmer; the shift of a single target is not a movement of the scene
checks = [e for e in eps if e["label"] in SHIFTS and len(e["sensors"]) > 1]
eps = [e for e in eps if e["label"] != "Heat shimmer" or not any(overlap(e, c) for c in checks)]
# the thermal wave absorbs last (a jump of something passing or of rain goes to them), and only the jumps of several
# targets: the wave moves them together, a single target that jumps is not the wave (C2 of E-003)
hosts = sorted((e for e in eps if e["label"] in HOSTS), key=is_thermal)
host = {id(e): next((h for h in hosts if overlap(e, h) and (len(e["sensors"]) > 1 or not is_thermal(h))
and (h["label"] != "Heat shimmer" or h["start"] <= e["start"])
and (h["label"] != "Brief obstruction" or e["start"] == h["start"]
and datetime.fromisoformat(e["end"]) - datetime.fromisoformat(h["start"]) <= MERGE)), None)
for e in eps if e["label"] == JUMP_BACK}
# a drift or a jump back whose targets are all in an overlapping Target issue episode: its curve is to be ignored,
# it takes its family (wf-improve session Displacement, mod. 1, S6)
host.update({id(e): h for e in eps if e["label"] in (JUMP_BACK, "Slow drift") and not host.get(id(e))
and (h := next((h for h in eps if h["kind"] == "Target issue" and overlap(e, h)
and set(e["sensors"]) <= set(h["sensors"])), None))})
groups = {id(e): [e] for e in eps if not host.get(id(e))}
issues = [e for e in eps if e["kind"] == "Target issue"]
for e in eps:
if host.get(id(e)):
groups[id(host[id(e)])].append(e)
if e["kind"] == "Obstruction":
e["sensors"] = [s for s in e["sensors"] if not any(s in h["sensors"] and overlap(e, h) for h in issues)]
eps = [e for e in eps if e["kind"] != "Obstruction" or e["sensors"]]
hosts = {id(e): h for e in eps if (h := host(e, eps))}
groups = {id(e): [e] for e in eps if id(e) not in hosts}
for e in eps:
if id(e) in hosts:
groups[id(hosts[id(e)])].append(e)
merged = []
for g in groups.values():
prev = next((m for m in reversed(merged) if variety(m[0]) == variety(g[0]) and (g[0]["kind"] not in ("Obstruction", "Target issue")
or set(g[0]["sensors"]) & {s for e in m for s in e["sensors"]})), None)
end = max((e["end"] for e in prev if variety(e) == variety(g[0])), default=None) if prev else None
prev = next((m for m in reversed(merged) if variety(m[0]) == variety(g[0]) and (g[0]["kind"] not in (
"Obstruction", "Target issue") or set(g[0]["sensors"]) & set(m[0]["sensors"]))), None) # the same target
end = prev and max(e["end"] for e in prev if variety(e) == variety(g[0]))
if end and datetime.fromisoformat(g[0]["start"]) - datetime.fromisoformat(end) < MERGE:
prev += g
else:
merged.append(g)
return sorted((event(g, references) for g in merged), key=lambda v: v["start"])
return sorted((event(g) for g in merged), key=lambda v: v["start"])
def events(measurements, end=None, folder=None):
"""Measurements (data.load) -> events of all detectors: what the page receives. A detector also receives what its
detect function declares: end, the end of the window read (batch run), for a hole touching it; a CONTEXT key, read
from the station folder (once per folder), empty without a folder."""
"""Measurements (data.load) -> events of all detectors: what the page receives. end: the end of the window read
(batch run), for a hole touching it; folder: unused, kept for the callers. A detector that fails is skipped with a
message (a broken detector does not block the page)."""
episodes = []
for mod in detectors():
for name in DETECTORS:
try:
params = inspect.signature(mod.detect).parameters
kw = {k: _context(folder, k) if folder else {} if k == "metrics" else [] for k in CONTEXT if k in params}
if "end" in params:
kw["end"] = end
episodes += mod.detect(measurements, **kw)
episodes += importlib.import_module(name).detect(measurements, **({"end": end} if name == "no_data" else {}))
except Exception as e: # noqa: BLE001
print(f"Detectors: {mod.__name__} skipped ({type(e).__name__}: {e})", file=sys.stderr)
return classify(episodes, {s for s, ms in measurements.items() if ms[0]["pattern_type"] == "global_ref"})
print(f"Detectors: {name} skipped ({type(e).__name__}: {e})", file=sys.stderr)
return classify(episodes)
def window(measurements, start, end, history=timedelta(days=3)):
"""The events of [start, end) (UTC datetimes), the detectors given `history` of earlier measurements too (the rule of
the CLI below): those ending at or after start. For a caller (aimsight-app) as for the CLI."""
chunk = {s: k for s, ms in measurements.items() if (k := [m for m in ms if start - history <= m["t"] < end])}
return [v for v in events(chunk, end) if v["end"] >= iso(start)] if chunk else []
if __name__ == "__main__":
@@ -230,9 +136,7 @@ if __name__ == "__main__":
t = start
while t < end:
b = min(t + timedelta(days=a.chunk), end) if a.chunk else end
h = t - timedelta(days=a.history)
chunk = {s: k for s, ms in measurements.items() if (k := [m for m in ms if h <= m["t"] < b])}
evs = [v for v in events(chunk, b, a.folder) if v["end"] >= iso(t)] if chunk else []
evs = window(measurements, t, b, timedelta(days=a.history))
if a.chunk:
print(f"== {iso(t)} → {iso(b)}: {len(evs) or 'nothing'}")
for v in evs:
-140
View File
@@ -1,140 +0,0 @@
"""Reading a station folder (standard library only). Every time is UTC.
Column formats: labeling/connaissances/colonnes-des-deplacements.md in Diagnostic-app.
"""
import csv
import glob
import os
import re
import statistics
from datetime import datetime, timezone
# Numeric columns of the "advanced" export; an empty cell becomes None.
NUM = ("x", "y", "error_value", "error_threshold", "relative_x", "relative_y",
"corr_x", "corr_y", "reproj_err", "inlier", "sigma0",
"peak", "flux", "sharpness", "fwhm", "dominance", "snr") # image quality: recent cameras only (LENZSPITZE)
# Camera pose (camera/): pan, tilt, rotation [deg], scale, sigma0 (registration residual).
POSE = ("pan", "tilt", "rotation", "scale", "sigma0")
IDENTIFIERS = re.compile(r"^(SIM_CCID|ip(_.*)?)$") # station identifiers: never read
METRIC_RE = re.compile(r"^(.+?)_\d{4}_\d{2}_\d{2}_")
def _header(path):
with open(path, newline="") as f:
return next(csv.reader(f), [])
def find_files(folder):
"""Station folder -> {'advanced': [...], 'camera': [...]}: sorted CSV paths, subfolders included."""
out = {"advanced": [], "camera": []}
for p in sorted(glob.glob(os.path.join(folder, "**", "*.csv"), recursive=True)):
header = _header(p)
if "sensor_name" in header and "corr_x" in header: # quality/ (optical_emulator) has sensor_name, not corr_x
out["advanced"].append(p)
elif "median_pan" in header and "timestamp" not in header: # timestamp: the hourly export
out["camera"].append(p)
return out
def _number(v):
v = v.strip()
return float(v) if v else None
def load(folder):
""""advanced" exports of the folder -> {sensor: [measurement]}, sorted by t, one reading per instant.
Measurement: dict with t (UTC datetime), pattern_type, and the NUM columns (float or None).
A rejected measurement (see rejected) has empty corr_x, corr_y, as the column docs say: the station empties them
for a target but leaves some filled (often absurd) for a reference; they are emptied here.
"""
by = {}
for path in find_files(folder)["advanced"]:
with open(path, newline="") as f:
for r in csv.DictReader(f):
m = {c: _number(r.get(c) or "") for c in NUM} # inlier is missing from old exports
m["t"] = parse_time(r["time_utc"])
m["pattern_type"] = r["pattern_type"]
if rejected(m):
m["corr_x"] = m["corr_y"] = None
by.setdefault(r["sensor_name"].strip(), {})[m["t"]] = m # some exports end the name with a newline
return {s: [d[t] for t in sorted(d)] for s, d in by.items()}
def load_temperature(folder):
"""Station board temperatures (metrics/, sensor_*_deg_c) -> [(t, °C)] sorted: the median of the components at each
hour; [] without these metrics. The components rise and fall together; the median drops the spike of a single
one, the computing GPU (D-124)."""
by = {}
for path in glob.glob(os.path.join(folder, "metrics", "*_deg_c_*.csv")):
with open(path, newline="") as f:
for r in csv.DictReader(f):
if (v := _number(r.get("metric") or "")) is not None:
by.setdefault(parse_time(r["time_utc"]), []).append(v)
return [(t, statistics.median(v)) for t, v in sorted(by.items())]
def load_camera(folder):
"""Camera movements of the folder -> [pose] sorted by t; pose: t and the POSE columns (float or None).
One row per camera registration; the medians are read (min = max in the files received: one reading per minute).
A measurement instant without a pose: the camera was not registered.
"""
by = {}
for path in find_files(folder)["camera"]:
with open(path, newline="") as f:
for r in csv.DictReader(f):
p = {c: _number(r.get("median_" + c, "")) for c in POSE}
p["t"] = parse_time(r["time_utc"])
by[p["t"]] = p
return [by[t] for t in sorted(by)]
def load_metrics(folder):
"""Station metrics (metrics/, one CSV per metric and per day) -> {name: [(t, value)]} sorted: numeric values only
(BusVoltage, sensor_CPU_deg_c, rsrp_dBm…); text metrics (link, mode…) and identifiers (SIM_CCID, ip*) are not
read. Name: the file's, without the station or the date."""
# ponytail: text metrics are skipped; read them (link up/down) when a detector needs them
station, by = os.path.basename(os.path.normpath(folder)) + "_", {}
for path in glob.glob(os.path.join(folder, "metrics", "*.csv")):
if not (m := METRIC_RE.match(os.path.basename(path))):
continue
name = m.group(1).removeprefix(station)
if IDENTIFIERS.match(name):
continue
with open(path, newline="") as f:
for r in csv.DictReader(f):
try:
v = float((r.get("metric") or "").strip("[]'\" ")) # some metrics come as a list: ['-95']
except ValueError:
continue
by.setdefault(name, {})[parse_time(r["time_utc"])] = v
return {n: sorted(d.items()) for n, d in sorted(by.items())}
def is_reference(measurements):
"""Reference of the compensation network (pattern_type global_ref), otherwise a target. Some exports have an empty
pattern_type, over the whole period or at the start: the first filled pattern_type is read then and, if there is
none, filled reproj_err or inlier (references only)."""
t = next((m["pattern_type"] for m in measurements if m["pattern_type"]), None)
if t is not None:
return t == "global_ref"
return any(m["reproj_err"] is not None or m["inlier"] is not None for m in measurements)
def rejected(m):
"""Measurement rejected by the station: error_value above the sensor's threshold (error_threshold); without these
two values, an empty corr_x."""
if m["error_value"] is not None and m["error_threshold"] is not None:
return m["error_value"] > m["error_threshold"]
return m["corr_x"] is None
def parse_time(text):
"""'2026-01-01T00:00Z' or '2026-01-01 00:00' -> UTC datetime; a time without a zone is read as UTC."""
t = datetime.fromisoformat(text)
return t.astimezone(timezone.utc) if t.tzinfo else t.replace(tzinfo=timezone.utc)
def iso(t):
return t.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%MZ")
+76 -266
View File
@@ -1,239 +1,80 @@
"""Detector of the Displacement family (the families document): sudden jumps of corr_x / corr_y (px) of each target, at
the native cadence of the "advanced" export, and slow drift (end of file). A flipping target ("Target flipping")
belongs to the Target issue family: episodes() returns it, detect() leaves it to target_issue.py.
"""Displacement and Perturbation families: sudden jumps of the corrected displacements (corr_x, corr_y, px) of each target, and slow drift
(end of file). Global references are excluded, rejected measurements (empty corr) skipped.
1. Noise of the series = median of the gaps from one measurement to the next; jump = measurement K noises or more away
from the level before (median of the BASE previous measurements); the first BASE measurements, lacking a level
before, are compared with the level after (median of the BASE next ones), a gap there is a jump back at once.
2. Back (an hour of measurements in a row within half the jump: BASE at 15 min, a single one at 4 h; wf-improve
MICHIGAN, mod. 1) within a day, or one day later back at its usual place (first day of the export) or at the day
before's: "Sudden jump, back to normal", greyed; its cause is not established here (categories.py gives it the
cause of the category that overlaps it, otherwise: to be checked on the images).
3. Otherwise "Lasting shift", never greyed (maybe a real movement); less than a day of hindsight: "Jump, not back yet",
not greyed either.
4. Jumps following each other within GAP, all targets together, make a single episode (common mode); a single lasting
(or too recent) target is enough not to grey the episode.
5. Global references excluded; rejected measurements (empty corr) skipped.
6. Common peak (NORDEND, batch 01): at the same instant, at least PEAK_MIN targets move away by at least K_PEAK noises
from both neighbouring measurements, on the same side (the whole image shaken for one step): a jump back for each,
even under K. It joins the episodes of point 4 (next to a lasting jump, it greys nothing).
7. (CLARIDEN, batch 01) a jump that never comes back no longer ends the reading of the series, its new level becomes
the level before; an episode that is not greyed ends at its last jump, not at the returns; a flipping target
(REPEAT jumps back or more, each within a day of the previous one) makes a "back to normal" episode on its own, from
the first to the last flip, outside the common episodes. A target that flips for at least FLIP, with at most
MAX_FLIPPING targets at the same time, becomes "Target flipping" (Target issue, D-074); a lasting jump of at least
LARGE of the targets, "Large shift" (everything moved), always to be checked. wf-improve session Target issue, mod. 1,
S2: FLIP from 6 h to 36 h; a target flipping for a day during a nuisance of the whole scene (rain, fog, night of poor
visibility) is not a problem target; its jumps back go to the nuisance overlapping them, otherwise stay
Displacement.
8. Jump under K that stays (NIEDERHORN, batch 01): a jump of at least K_LASTING noises counts if it is lasting in the
sense of point 3 and the median of the day after moves away from that of the day before, on the jump's side, by at
least PLATEAU times the ordinary variation of the level from one UTC day to the next (median of the gaps between
daily levels); otherwise it is ignored (neither back nor too recent: only jumps of K noises or more are).
1. Jump: a measurement at least K noises (median step of the series) away from the level before (median of the BASE
previous measurements).
2. Back to normal: within BACK_H, an hour of measurements in a row (BASE at most, one at least) within half the jump of
the level before; a Perturbation, greyed (categories.py joins it to an overlapping Visibility or
Obstruction episode, otherwise nothing to do). Otherwise "Lasting shift", or "Jump, not back
yet" with less than BACK_H of hindsight: never greyed (maybe a real movement), the episode is the jump's instant;
the new level becomes the level before. The return is read on the corrected curves only: a camera that keeps its new
position while the compensation brings the curves back is a Displacement in the labels, a Perturbation here.
"""
import statistics
import sys
from datetime import timedelta
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent)) # data.py, next to it
import data # noqa: E402
from episodes import episode, noise
from data import is_reference
K = 20 # tuned on E-001: at 15, two healthy targets are greyed (17 and 18 noises); the trains of E-005 (17 and 19
# noises) are caught by visibility.py: do not lower K
K_LASTING = 7 # under K, a jump that stays counts from K_LASTING noises (NIEDERHORN, batch 01: e09, corr_y −0.18 px at
# once, 8 noises, stayed 5 days)
PLATEAU = 8 # ... if it stays: median of the day after, on the jump's side, PLATEAU times the ordinary variation of the
# level from one day to the next away from the day before (e09: 13; E-001, healthy targets: 6 at most)
K_PEAK = 6 # one-step peak common to several targets (NORDEND shakes, 6 to 11 noises, batch 01)
PEAK_MIN = 3 # targets peaking at the same step for a common peak (one or two: ordinary noise)
BASE = 4 # measurements that make the level before (1 h at 15 min)
GAP = timedelta(hours=1) # two jumps within an hour, all targets together: a single episode
BACK_H = 24 # back within a day = artefact (E-002: plateaus up to 11.5 h)
FLIP = timedelta(hours=36) # a problem target flips for several days (PR_BG: 2.5 days); less is an agitated morning
# (E-009: C8 and C13, 5 flips in 1 h 20 during the movement) or a night of nuisance (NORDEND, JUNGFRAU:
# 12 to 28 h within a loss of visibility), a jump back, absorbed by its host
MAX_FLIPPING = 2 # targets flipping at the same time, at most (FLORIDA: the two stuck ones; E-002, cobweb on the
# references: 15 at once)
LARGE = 0.5 # share of the targets of a lasting jump for a large shift (E-004: almost all)
REPEAT = 5 # jumps back of the same target, each within a day of the other, that make it a flipping target (PR_BG of
# CLARIDEN, batch 01: about thirty in 2.5 days)
K = 20 # E-001: at 15, two healthy targets are greyed (17 and 18 noises); the trains of E-005 are visibility's
BASE = 4 # measurements of the level before (1 h at 15 min); the return is judged on an hour (wf-improve MICHIGAN, mod. 1)
BACK_H = timedelta(hours=24) # E-002: plateaus up to 11.5 h
JUMP_BACK = "Sudden jump, back to normal"
CHECK = " Check whether the structure moved: look at the images."
LABELS = {"back": (JUMP_BACK, "A target jumped and came back within a day: nothing to do."),
"lasting": ("Lasting shift", "A target shifted and stayed shifted for more than a day." + CHECK),
"recent": ("Jump, not back yet", "A target shifted and has not come back yet." + CHECK)}
def _jumps(t, v):
"""A series (instants, values) -> [(start i, end i, state)], state 'back', 'lasting' or 'recent'."""
day = timedelta(hours=BACK_H)
noise = statistics.median(abs(y - x) for x, y in zip(v, v[1:]))
step = statistics.median(b - a for a, b in zip(t, t[1:])) # cadence of the series
back_n = max(1, min(BASE, round(timedelta(hours=1) / step))) # measurements of an hour (BASE at 15 min, one at 4 h)
# ponytail: usual place = first day of the export, wrong if it starts within an artefact
ref = statistics.median(x for x, u in zip(v, t) if u < t[0] + day)
lv = {} # level of each UTC day; its ordinary variation from one day to the next judges a jump under K
for x, u in zip(v, t):
lv.setdefault(u.date(), []).append(x)
lv = [statistics.median(x) for _, x in sorted(lv.items())]
ordinary = statistics.median(abs(b - a) for a, b in zip(lv, lv[1:])) if len(lv) > 2 else 0
# the first BASE measurements, lacking a level before, face the level after: a gap there came back at once
out = [(k, k, "back") for k in range(min(BASE, len(v) - BASE))
if noise and abs(v[k] - statistics.median(v[k + 1:k + 1 + BASE])) >= K * noise]
i = BASE
while noise and i < len(v):
"""A series (instants, values) -> [(start index, end index, state)], state "back", "lasting" or "recent"."""
n = noise(v)
back_n = max(1, min(BASE, round(timedelta(hours=1) / statistics.median(b - a for a, b in zip(t, t[1:])))))
out, i = [], BASE
while n and i < len(v):
base = statistics.median(v[i - BASE:i])
if abs(v[i] - base) < K_LASTING * noise:
if abs(v[i] - base) < K * n:
i += 1
continue
tol = abs(v[i] - base) / 2 # back: an hour of measurements in a row within half the jump from the level before
tol = abs(v[i] - base) / 2
j = next((k for k in range(i + 1, len(v)) if all(abs(x - base) < tol for x in v[k:k + back_n])), len(v))
before = statistics.median([x for x, u in zip(v, t) if t[i] - day <= u < t[i]] or [base])
after = statistics.median(x for x, u in zip(v, t) if t[i] <= u < t[i] + day)
if j < len(v) and t[j] - t[i] <= day:
state = "back"
elif abs(after - ref) < tol or abs(after - before) < tol: # the day after, the target is at its usual place,
state, j = "back", i + 1 # or where it was the day before: the jump ended a dip
else:
state = "lasting" if t[-1] - t[i] >= day else "recent" # less than a day of hindsight: not known yet
if abs(v[i] - base) < K * noise and (state != "lasting" or (after - before) * (v[i] - base) < 0
or not ordinary or abs(after - before) < PLATEAU * ordinary):
i += 1 # under K, only a jump that stays counts: the day after far from the day before, on its side
continue
out.append((i, j - 1, state))
# level before rebuilt after the return; never back: the new level becomes the level before, the rest is
# still read (PR_BG of CLARIDEN, batch 01: its flips after the 10:30 jump were invisible)
state = "back" if j < len(v) and t[j] - t[i] <= BACK_H else "lasting" if t[-1] - t[i] >= BACK_H else "recent"
out.append((i, j - 1 if state == "back" else i, state))
i = (j if j < len(v) else i) + BASE
return out
def _peaks(t, v):
"""Instants where the measurement moves away by at least K_PEAK noises from both neighbours, same side (one-step peak)."""
noise = statistics.median(abs(y - x) for x, y in zip(v, v[1:]))
return [t[i] for i in range(1, len(v) - 1) if noise and (v[i] - v[i - 1]) * (v[i] - v[i + 1]) > 0
and min(abs(v[i] - v[i - 1]), abs(v[i] - v[i + 1])) >= K_PEAK * noise]
def episodes(measurements):
"""{sensor: [measurement]} (data.load) -> jump episodes, sorted by start: Displacement, and Target issue for a
flipping target."""
found = [] # (start, end, state, sensor)
peaks = {} # instant -> targets peaking there for one step
def jumps(measurements):
found = set() # (start, end, state, target); corr_x and corr_y together
for name, ms in measurements.items():
if data.is_reference(ms):
if is_reference(ms):
continue
for axis in ("corr_x", "corr_y"):
kept = [m for m in ms if m[axis] is not None]
if len(kept) <= BASE:
continue
t, v = [m["t"] for m in kept], [m[axis] for m in kept]
# a jump that did not come back ends with itself, not with the data (its bar hid the others)
found += [(t[a], t[b] if state == "back" else t[a], state, name) for a, b, state in _jumps(t, v)]
for u in _peaks(t, v):
peaks.setdefault(u, set()).add(name)
found += [(u, u, "back", s) for u, cs in peaks.items() if len(cs) >= PEAK_MIN for s in cs] # common peak: back
alone = _flips(found) # a target that keeps jumping and coming back: an episode on its own, outside the common ones
found = [f for f in found if f[2] != "back" or not any(f[3] in g["sensors"] and g["start"] <= f[0] <= g["end"]
for g in alone)]
groups = [] # a jump that did not come back only counts by its instant: its end is often the end of the data
for s, e, state, name in sorted(found):
if groups and s <= groups[-1]["until"] + GAP:
g = groups[-1]
else:
g = {"start": s, "end": e, "until": s, "last": s, "sensors": {}}
groups.append(g)
g["end"], g["until"], g["last"] = max(g["end"], e), max(g["until"], e if state == "back" else s), s
g["sensors"].setdefault(name, set()).add(state)
long = [g for g in alone if g["flip"]] # flips of many targets at the same time: the compensation, not a
for g in long: # problem target
if sum(h["start"] <= g["end"] and h["end"] >= g["start"] for h in long) > MAX_FLIPPING:
g["flip"] = 0
total = sum(not data.is_reference(ms) for ms in measurements.values())
return sorted((_episode(g, total) for g in groups + alone), key=lambda ep: ep["start"])
def _flips(found):
"""Jumps back of the same target, each within BACK_H of the previous one: REPEAT or more -> a group, from the first
to the last flip."""
day, alone = timedelta(hours=BACK_H), []
for name in {f[3] for f in found}:
chain = [] # (start, end); corr_x and corr_y twice: counted by instant
for s, e in sorted({(s, e) for s, e, state, n in found if n == name and state == "back"}) + [(None, None)]:
if chain and s is not None and s <= max(c[1] for c in chain) + day:
chain.append((s, e))
continue
if len({c[0] for c in chain}) >= REPEAT:
alone.append({"start": chain[0][0], "end": max(c[1] for c in chain), "sensors": {name: {"back"}},
"flip": len({c[0] for c in chain}) if max(c[1] for c in chain) - chain[0][0] >= FLIP else 0})
chain = [(s, e)]
return alone
def _episode(g, total):
"""A group -> episode; a single lasting (or too recent) jump is enough not to grey; a flipping target: "Target
flipping" (Target issue, D-074); a lasting jump of LARGE targets or more: "Large shift"."""
n = len(g["sensors"])
lasting = sum("lasting" in st for st in g["sensors"].values())
recent = any("recent" in st for st in g["sensors"].values())
who = "One target" if n == 1 else f"{n} targets together (a single event)"
if g.get("flip"):
name = next(iter(g["sensors"]))
return {"start": data.iso(g["start"]), "end": data.iso(g["end"]), "kind": "Target issue", "label": "Target flipping",
"text": f"{name} jumped and came back {g['flip']} times: its tracking is not reliable, look at the images "
"of the target.", "ignore": True, "sensors": [name], "lasting": []}
if lasting and n >= LARGE * total:
label = "Large shift"
text = f"{n} targets jumped together and {lasting} of them stayed shifted: everything moved, please check it."
elif lasting:
label = "Lasting shift"
some = "" if lasting == n else f", {lasting} of them stayed shifted for more than a day"
text = (f"{who} jumped{some}" if some else f"{who} jumped and stayed shifted for more than a day") + \
": this may be a real movement, please check it."
elif recent:
label = "Jump, not back yet"
text = f"{who} jumped less than a day before the last measurement and has not come back yet: check again later."
else:
label = "Sudden jump, back to normal"
text = f"{who} jumped and came back within a day: the cause is not established, look at the images."
end = g["last"] if lasting or recent else g["end"] # not greyed: the last jump, not the returns
return {"start": data.iso(g["start"]), "end": data.iso(end), "kind": "Displacement", "label": label,
"text": text, "ignore": not lasting and not recent, "sensors": sorted(g["sensors"]),
"lasting": sorted(s for s, st in g["sensors"].items() if "lasting" in st)} # stayed shifted (event sentence)
if len(kept) > BASE:
t, v = [m["t"] for m in kept], [m[axis] for m in kept]
found |= {(t[a], t[b], state, name) for a, b, state in _jumps(t, v)}
return [episode(a, b, "Perturbation" if state == "back" else "Displacement", *LABELS[state], [name], state == "back") for a, b, state, name in sorted(found)]
def detect(measurements):
return sorted([e for e in episodes(measurements) if e["kind"] == "Displacement"] + drift(measurements),
key=lambda ep: ep["start"])
return jumps(measurements) + drift(measurements)
# --- Slow drift ------------------------------------------------------------------------------------------------------
# A target that moves little by little, day after day. Takes up the trend of the Displacements tab (D-055) on the
# "advanced" export (corr_x, corr_y, px). Method, per target (global references excluded) and per axis:
# 1. Level of the day: median of the measurements of each UTC day with at least HOURS measured hours out of 24, as a
# share of the hours the station measures (wf-improve MICHIGAN, mod. 1: at one measurement every 4 h, 5 of the 6 hours;
# before, no day counted); at least DAYS days.
# 2. Theil-Sen slope and Kendall tau of the daily levels; amplitude of the day: gap between its 9th and 1st deciles.
# 3. Drift: |slope| × duration ≥ SIZE times the median daily amplitude, |tau| ≥ STEADY, and no step from one day to the
# next of more than half the drift (a jump that lasts is not a drift: the jump rule takes it).
# 4. One episode per target (one axis is enough), from the first measurement of the first day to the last of the last
# day; not greyed.
# 4b. Piecewise (wf-improve session Displacement, mod. 1, S1): a target without a drift over the whole window may drift for
# a few days, then settle or go back. On each run of at least DAYS kept days, drift = level of the last day minus
# that of the first, at least PIECE daily amplitudes (median of the series), |tau| ≥ STEADY with the same sign, no
# step from one kept day to the next of more than half the drift; pieces taken from the largest drift to the
# smallest, without overlapping (they may share the day where the level turns): each goes from one extreme of the
# level to the other, one episode per piece and per axis. At PIECE = 1, the thermal cycles of a few days and the
# noise of the levels get through (JUNGFRAU, NORDEND): 3.5 keeps the rows of CLARIDEN (3.8 to 8.3 amplitudes),
# drops JUNGFRAU (3.1 at most). The pieces do not replace step 3. A piece does not cross a hole: two consecutive kept
# days are at most MAX_HOLE days apart (mod. 2: the gap of a level across three days without a measurement is a
# step, not a slope; it made pieces of the whole window, CLARIDEN 09-28 → 11-01). A missing day does not cut.
# 5. Camera drift (same mod.): at least LARGE of the targets drift on the same side on corr_x or corr_y, and at least
# LARGE of the sensors, references included, on their raw positions x or y: only the pieces of point 4b are
# reported, not the trend over the whole window (mod. 1, S2: the badly compensated drifting camera, NEBRASKA, is a
# Displacement for the labels). The targets of a bridge following the season (JUNGFRAU, labelled Thermal
# displacement day after day) pass step 3 but not PIECE; NEBRASKA both.
HOURS = 20 # a day counts if it has at least 20 measured hours out of 24 (as D-055), as a share of the measured hours
DAYS = 5 # minimum days for a trend (D-055)
SIZE = 1.0 # drift ≥ SIZE daily amplitudes (D-055)
STEADY = 0.6 # |Kendall tau| of the daily levels (D-055)
PIECE = 3.5 # piecewise drift ≥ PIECE daily amplitudes (point 4b)
MAX_HOLE = 2 # a piece does not go from one kept day to the next more than MAX_HOLE days apart (point 4b)
# --- Slow drift: a target that moves little by little, day after day (the trend of the Displacements tab, D-055).
# Per target and per axis (corr_x, corr_y): the level of each UTC day measured at least HOURS of the hours the station
# measures (median of the day), at least DAYS days; the amplitude of a day: its 9th minus its 1st decile.
# 1. Over the whole window: Theil-Sen slope × duration at least SIZE median daily amplitudes, |Kendall tau| ≥ STEADY, and
# no step from one day to the next larger than half the drift (that is a jump).
# 2. Otherwise, pieces: runs of at least DAYS kept days whose last level minus the first is at least PIECE amplitudes, with
# the same tau and step conditions; taken from the largest to the smallest without overlapping (wf-improve
# Displacement, mod. 1, S1: rows of CLARIDEN, 3.8 to 8.3 amplitudes; thermal cycles of JUNGFRAU, 3.1 at most).
# One episode per drift, not greyed: the largest of the two axes over the whole window, otherwise each piece.
HOURS = 20 # out of 24, as a share of the hours the station measures (wf-improve MICHIGAN, mod. 1)
DAYS = 5 # D-055
SIZE = 1.0 # D-055
STEADY = 0.6 # D-055
PIECE = 3.5 # wf-improve Displacement, mod. 1, S1
def _decile(vs, q):
@@ -242,50 +83,34 @@ def _decile(vs, q):
def _days(ms, col):
"""-> {UTC day: its measurements}, the kept days (point 1)."""
"""-> (kept days, their levels, median daily amplitude, [(first instant, last) of each day])."""
days = {}
for m in ms:
if m[col] is not None:
days.setdefault(m["t"].date(), []).append(m)
slots = len({m["t"].hour for dm in days.values() for m in dm}) # hours the station measures (6 at a 4 h step)
return {d: dm for d, dm in days.items() if len({m["t"].hour for m in dm}) >= HOURS / 24 * slots}
ds = sorted(d for d, dm in days.items() if len({m["t"].hour for m in dm}) >= HOURS / 24 * slots)
vals = [[m[col] for m in days[d]] for d in ds]
ampl = statistics.median(_decile(v, 0.9) - _decile(v, 0.1) for v in vals) if ds else 0
return ds, [statistics.median(v) for v in vals], ampl, [(days[d][0]["t"], days[d][-1]["t"]) for d in ds]
def _amplitude(days, col):
return statistics.median(_decile([m[col] for m in dm], 0.9) - _decile([m[col] for m in dm], 0.1) for dm in days.values())
def _whole(ds, lv, ampl, _):
"""-> [(drift, 0, last index)] if the levels drift over the whole window, else []."""
pairs = [(i, j) for i in range(len(ds)) for j in range(i + 1, len(ds))]
slope = statistics.median((lv[j] - lv[i]) / (ds[j] - ds[i]).days for i, j in pairs)
tau = sum((lv[j] > lv[i]) - (lv[j] < lv[i]) for i, j in pairs) / len(pairs)
d = slope * (ds[-1] - ds[0]).days
step = max(abs(b - a) for a, b in zip(lv, lv[1:]))
return [(d, 0, len(ds) - 1)] if abs(d) >= SIZE * ampl and abs(tau) >= STEADY and step <= abs(d) / 2 else []
def _drift(ms, col):
"""-> (drift in px, days, first instant, last) or None."""
days = _days(ms, col)
if len(days) < DAYS:
return None
lv = sorted((d.toordinal(), statistics.median(m[col] for m in dm)) for d, dm in days.items())
pairs = [(a, b) for k, a in enumerate(lv) for b in lv[k + 1:]]
slope = statistics.median((b[1] - a[1]) / (b[0] - a[0]) for a, b in pairs)
tau = sum((b[1] > a[1]) - (b[1] < a[1]) for a, b in pairs) / len(pairs)
drift = slope * (lv[-1][0] - lv[0][0])
step = max(abs(b[1] - a[1]) for a, b in zip(lv, lv[1:]))
if abs(drift) < SIZE * _amplitude(days, col) or abs(tau) < STEADY or step > abs(drift) / 2:
return None
d0, d1 = min(days), max(days)
return drift, len(days), min(m["t"] for m in days[d0]), max(m["t"] for m in days[d1])
def _pieces(ms, col):
"""-> [(drift in px, days, first instant, last)]: the drift pieces (point 4b)."""
days = _days(ms, col)
if len(days) < DAYS:
return []
ds = sorted(days)
lv = [statistics.median(m[col] for m in days[d]) for d in ds]
ampl = _amplitude(days, col)
pieces = [] # (drift, i, j)
def _pieces(ds, lv, ampl, _):
"""-> [(drift, first index, last index)]: the drift pieces, without overlapping."""
pieces = []
for i in range(len(ds)):
signs = step = 0
for j in range(i + 1, len(ds)):
if (ds[j] - ds[j - 1]).days > MAX_HOLE:
break # a gap across a hole is a step (the jump rule takes it), not a slope
signs += sum((lv[j] > lv[k]) - (lv[j] < lv[k]) for k in range(i, j)) # Kendall tau, step by step
step = max(step, abs(lv[j] - lv[j - 1]))
n, d = j - i + 1, lv[j] - lv[i]
@@ -296,32 +121,17 @@ def _pieces(ms, col):
for d, i, j in sorted(pieces, key=lambda x: (-abs(x[0]), x[2] - x[1])):
if all(j <= a or i >= b for _, a, b in taken):
taken.append((d, i, j))
return [(d, j - i + 1, min(m["t"] for m in days[ds[i]]), max(m["t"] for m in days[ds[j]])) for d, i, j in taken]
def _common(measurements, cols):
"""At least LARGE of the sensors drift on the same side on the same axis of cols."""
for col in cols:
signs = [x[0] > 0 for ms in measurements.values() if (x := _drift(ms, col))]
if max(signs.count(True), signs.count(False)) >= LARGE * len(measurements):
return True
return False
return taken
def drift(measurements):
targets = {s: ms for s, ms in measurements.items() if not data.is_reference(ms)}
camera = _common(targets, ("corr_x", "corr_y")) and _common(measurements, ("x", "y")) # point 5: pieces only
out = []
for s, ms in sorted(targets.items()):
r = [] if camera else [x for col in ("corr_x", "corr_y") if (x := _drift(ms, col))]
r = [max(r, key=lambda x: abs(x[0]))] if r else [x for col in ("corr_x", "corr_y") for x in _pieces(ms, col)]
out += [{"start": data.iso(a), "end": data.iso(b), "kind": "Displacement", "label": "Slow drift",
"text": f"{s} drifts slowly and steadily: {d:+.2f} px in {n} days. Keep watching it.",
"ignore": False, "sensors": [s]} for d, n, a, b in r]
for s, ms in sorted(measurements.items()):
if is_reference(ms):
continue
axes = [a for col in ("corr_x", "corr_y") if len((a := _days(ms, col))[0]) >= DAYS]
whole = [(w, a) for a in axes for w in _whole(*a)]
found = [max(whole, key=lambda x: abs(x[0][0]))] if whole else [(p, a) for a in axes for p in _pieces(*a)]
out += [episode(a[3][i][0], a[3][j][1], "Displacement", "Slow drift", f"{s} drifts slowly and steadily: {d:+.2f} "
f"px in {j - i + 1} days. Keep watching it.", [s], False) for (d, i, j), a in found]
return out
if __name__ == "__main__":
for ep in detect(data.load(sys.argv[1])):
print(ep["start"], ep["end"], "ignore" if ep["ignore"] else "KEEP ", ep["label"], "|", len(ep["sensors"]),
",".join(ep["sensors"][:6]), "|", ep["text"])
+47
View File
@@ -0,0 +1,47 @@
"""Helpers shared by the detectors."""
import statistics
from datetime import timedelta
from data import iso
def episode(start, end, kind, label, text, sensors, ignore, **extra):
return {"start": iso(start), "end": iso(end), "kind": kind, "label": label, "text": text, "ignore": ignore,
"sensors": sorted(sensors), **extra}
def runs(times, gap):
"""Sorted times -> [[t, ...]]: runs of times at most gap apart."""
out = []
for t in times:
if out and t - out[-1][-1] <= gap:
out[-1].append(t)
else:
out.append([t])
return out
def noise(v):
"""Median step of a series, from one value to the next."""
return statistics.median(abs(y - x) for x, y in zip(v, v[1:])) if len(v) > 1 else 0
def level(v):
"""-> (median, MAD) of the values."""
med = statistics.median(v)
return med, statistics.median(abs(x - med) for x in v) or 1e-9
def cadence(measurements):
"""Cycle of the measurements: the median gap between two instants, at least an hour divided by the median number of
instants per clock hour (a station measuring in bursts, three measurements a minute apart every 15 min: 15 min)."""
ts = sorted({m["t"].replace(second=0, microsecond=0) for ms in measurements.values() for m in ms})
per_hour = {}
for t in ts:
per_hour[t.replace(minute=0)] = per_hour.get(t.replace(minute=0), 0) + 1
gap = statistics.median(b - a for a, b in zip(ts, ts[1:])) if len(ts) > 1 else timedelta(hours=1)
return max(gap, timedelta(hours=1) / statistics.median(per_hour.values()))
def targets(n):
return "one target" if n == 1 else f"{n} targets"
+10 -35
View File
@@ -1,45 +1,20 @@
"""Detector of the No data family (the families document): moments without a measurement (the cadence of the "advanced"
export is broken).
"""No data family: no measurement in the export for more than MISSING steps of its cadence (episodes.cadence), between two
measurements or, when the caller gives the end of the window read, between the last measurement and that end."""
from episodes import cadence, episode
Cadence: median gap between two measurement instants, all sensors together. A hole: more than MISSING missing steps
between two measurements (gap > MISSING + 1 cadences); the episode runs from the last measurement to the next one.
End of the window read (end, when the caller gives it: the chunked run of categories.py): the same rule between the last
measurement and end, the episode running up to end (CLARIDEN, batch 01: 44 h without a measurement up to the end of
the batch, not reported). No measurement in the export does not mean the station stopped: the text only says that the
export has nothing.
Usage: python3 scripts/no_data.py <station folder> (prints the episodes)
"""
import os
import statistics
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
from data import iso, load # noqa: E402
MISSING = 2 # more than 2 missing steps is a hole (user's rule, 2026-10-23)
MISSING = 2 # more than 2 missing steps is a hole (user's rule, 2026-10-23); also up to the end (CLARIDEN, batch 01)
def duration(s):
h, m = divmod(round(s / 60), 60)
def duration(d):
h, m = divmod(round(d.total_seconds() / 60), 60)
return f"{h} h {m:02d} min" if h else f"{m} min"
def detect(measurements, end=None):
"""end: end of the window read (datetime, excluded), optional; without it, a final hole is not seen."""
ts = sorted({m["t"].replace(second=0, microsecond=0) for ms in measurements.values() for m in ms})
if len(ts) < 3:
return []
gaps = [(b - a).total_seconds() for a, b in zip(ts, ts[1:])]
step = statistics.median(gaps)
out = [(a, b, g, "") for a, b, g in zip(ts, ts[1:], gaps)]
if end: # end excluded: the last expected measurement is at end − 1 step, the gap counts as between two measurements
out.append((ts[-1], end, (end - ts[-1]).total_seconds(), ", up to the end of the period"))
return [{"start": iso(a), "end": iso(b), "kind": "No data", "label": "No data",
"text": f"No measurement in the export for {duration(g)}{tail} (usually one every {duration(step)}).",
"ignore": False, "sensors": []}
for a, b, g, tail in out if g > (MISSING + 1) * step]
if __name__ == "__main__":
for e in detect(load(sys.argv[1])):
print(e)
step = cadence(measurements)
holes = [(a, b, "") for a, b in zip(ts, ts[1:])] + ([(ts[-1], end, ", up to the end of the period")] if end else [])
return [episode(a, b, "No data", "No data", f"No measurement in the export for {duration(b - a)}{tail} (usually one "
f"every {duration(step)}).", [], False) for a, b, tail in holes if b - a > (MISSING + 1) * step]
-36
View File
@@ -1,36 +0,0 @@
"""Detector of the Obstruction family (the families document): an object hides one or a few targets, for an instant or
for long. It reuses two computations shared with other families:
- the error_value of visibility.py: Brief obstruction (something passing, one instant) and Target obscured (a target
rejected again and again while the others stay seen, or a nuisance that only touches a few targets, no reference);
- the day-by-day rejections of target_issue.py: a target rejected for less than a week, Target obscured.
A sensor already in an overlapping Target issue episode (Degraded target, Target stuck) is not returned as Obstruction:
its curve is already to be ignored, the family is Target issue (boundary "Target issue or an episode of another family"
of the families document; wf-improve session Obstruction, mod. 1: S1, problem targets of FLORIDA, NORDEND, SNCF01,
NIEDERHORN, JUNGFRAU, CLARIDEN cut into Target obscured; S3, their isolated rejections as Brief obstruction).
Usage: python3 scripts/obstruction.py <station folder> (prints the episodes)
"""
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py and the detectors, next to it
import target_issue # noqa: E402
import visibility # noqa: E402
from data import load # noqa: E402
def detect(measurements):
ti = target_issue.episodes(measurements)
issues = [e for e in ti if e["kind"] == "Target issue"]
out = []
for e in visibility.episodes(measurements) + ti:
if e["kind"] != "Obstruction":
continue
taken = {s for h in issues if h["start"] <= e["end"] and h["end"] >= e["start"] for s in h["sensors"]}
if (e := {**e, "sensors": [s for s in e["sensors"] if s not in taken]})["sensors"]: # minus the sensors
out.append(e) # already in Target issue
return sorted(out, key=lambda e: e["start"])
if __name__ == "__main__":
for e in detect(load(sys.argv[1])):
print(e["start"], "→", e["end"], e["label"], f"| {len(e['sensors'])}:", ",".join(e["sensors"][:8]), "|", e["text"])
+46 -165
View File
@@ -1,184 +1,65 @@
"""Detector of the Target issue family (the families document): a persistently problematic target: rejected most of the
time, day after day, while the others are seen (degraded target), stuck (end of file), or flipping (the jumps of
displacement.py, "Target flipping"). A target rejected for less than a week is hidden at times (point 4): episodes()
returns it with the Obstruction family, which detect() leaves to obstruction.py.
"""Target issue family: a target rejected day after day ("Degraded target"); visibility.py compares each sensor with its
own usual level, so a target almost always rejected looks normal to it. For a few days only, Target obscured
(Obstruction family).
visibility.py compares each sensor with its own usual level: a target that is almost always rejected looks normal to it
(FLORIDA, October 2026: Massif 2 and Sellette 6 rejected half the measurements or more, every day).
Method:
1. Calm instant for a sensor: at most CALM of the other sensors rejected (error_value > error_threshold); weather and
fog, which touch everyone, are thus set aside.
2. Per UTC day and per sensor, on its calm instants (at least MIN_HOURS hours of instants at the station's cadence,
median gap between two measurement instants; otherwise the day does not count): degraded day if at least SHARE of
them are rejected. Cadence (wf-improve NEBRASKA, batch 01): 12 fixed instants, "3 h at 15 min", counted no day at one
measurement every 4 h; REFCOLH22, rejected all the time since it was lost, was not seen.
3. Episode: at least MIN_DAYS degraded days in a row (a day that does not count does not break the run), from the first
rejected measurement of these days to the last; ignore = True: the target's curve is not reliable.
4. Hindered all the time or at times (boundary "Target issue or Obstruction" of the families document; wf-improve
NEBRASKA, batch 01): a run of at least WEEK days is a problem target, "Degraded target"; shorter, a target hidden at
times, "Target obscured" (Obstruction family), one episode per run of rejected measurements (the hours when it is
seen are not part of it). BREITHORN: vehicles parked in front of COLI24 and POUI22, 2 to 5 days; NEBRASKA and
MICHIGAN: a lost reference, 25 days; JUNGFRAU: PM_G, 63.
For the problem target, the run is bridged: at most BRIDGE non-degraded days in a row do not break it, and it then
needs WEEK days from its first to its last degraded day, at SHARE_WEEK of rejections over all its days ("rejections
specific to the target almost every day of the range"); the Target obscured episodes it covers no longer exist
(wf-improve session Obstruction, mod. 1, S1: FLORIDA Sellette 6 and Massif 2, SNCF01 F4.3 and F4.4, NORDEND CTR01,
NIEDERHORN e01, cut into Target obscured by a day under SHARE; BREITHORN, parked vehicles, unchanged).
The problem target judges regularity rather than the share of the day: a day is degraded from SHARE_WEEK (a quarter)
of its calm instants rejected, and BRIDGE is one day; the target hidden at times keeps SHARE. A target rejected every
day at the same hours, by day only or by night only, stays under half and was not seen (wf-improve session Target
issue, mod. 1, S1: WILDSTRUBEL R3, 04:00 to 16:00 every day; NORDEND PR04 and CTR01, NIEDERHORN PWD_IMP_B08_024_C09
and e01, SNCF01 F4.3).
The cause (dirt, vegetation, setting) can only be read on the images.
Stuck target (E-011, FLORIDA: a target defined on the prism of a reference): per UTC day, the noise of its corr_x and
corr_y (median of the gaps from one measurement to the next) under STUCK times the median noise of the other targets, on
both axes; one day is enough (a day without noise does not happen by accident); label "Target stuck". On E-011: C2 at
7e-5 px and C3 at 2e-4 against 0.013 for the others.
Usage: python3 scripts/target_issue.py <station folder> (prints the episodes)
1. Degraded day: at least SHARE of the sensor's measurements of the UTC day rejected; a day with fewer than MIN_HOURS of
measurements (at the station's cadence) does not count.
2. Run: degraded days with at most BRIDGE other days in a row between them, and SHARE of rejections over the run. WEEK
days or more: Degraded target; MIN_DAYS or more: Target obscured. The episode runs from the first rejected
measurement of the run to the last, greyed. The cause (dirt, vegetation, a parked vehicle) is seen on the images.
"""
import os
import statistics
import sys
from datetime import timedelta
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py and the detectors, next to it
import displacement # noqa: E402
from data import is_reference, iso, load # noqa: E402
from episodes import cadence, episode
CALM = 0.25 # share of the other sensors rejected, at most, for a calm instant
SHARE = 0.5 # half of the calm instants rejected: "most of the time"
SHARE_WEEK = 0.25 # a quarter of the calm instants rejected, each day of a week or more: a problem target, even when
# rejected by day only or by night only (wf-improve session Target issue, mod. 1, S1)
MIN_HOURS = 3 # hours of calm instants, at the station's cadence: below that, the day does not count
MIN_DAYS = 2 # a single day is too often an accident (NORDEND: isolated days on 16 sensors)
BRIDGE = 1 # non-degraded days that do not break a run of degraded days (4); at 2, the parked vehicles of BREITHORN
# (two days, then nothing for two days, then one) would make a week at a quarter
WEEK = 7 # degraded days in a row for a problem target; below, a target hidden at times
STUCK = 0.05 # noise of a stuck target compared with the others (E-011: 0.006; NORDEND, E-001: 0.7 at the lowest)
SHARE = 0.25 # wf-improve Target issue, mod. 1, S1 (WILDSTRUBEL R3 rejected from 04:00 to 16:00 every day, 30 to 45 %)
MIN_HOURS = 3 # wf-improve NEBRASKA, batch 01 (one measurement every 4 h: as hours, not as instants)
BRIDGE = 1 # wf-improve Obstruction, mod. 1, S1 and Target issue, mod. 1, S1 (problem targets cut by a better day)
WEEK = 7 # wf-improve NEBRASKA, batch 01 (lost references, 25 days; parked vehicles, 2 to 5 days, are not one)
MIN_DAYS = 2 # shorter than WEEK: a target hidden for a few days, Target obscured (Obstruction); NORDEND: isolated days
def is_rejected(m):
def rejected(m):
return m["error_value"] is not None and m["error_threshold"] is not None and m["error_value"] > m["error_threshold"]
def cadence(measurements):
"""Median gap between two measurement instants, all sensors together, in hours (as no_data)."""
ts = sorted({m["t"].replace(second=0, microsecond=0) for ms in measurements.values() for m in ms})
return statistics.median((b - a).total_seconds() for a, b in zip(ts, ts[1:])) / 3600 if len(ts) > 1 else 1
def episodes(measurements):
"""Targets rejected day after day: Target issue (a week or more) and Obstruction (less), then the stuck ones."""
min_instants = MIN_HOURS / cadence(measurements)
by_t = {}
for s, ms in measurements.items():
for m in ms:
by_t.setdefault(m["t"], {})[s] = is_rejected(m)
days = {} # sensor -> day -> [rejected, calm, first measurement, last]
for t in sorted(by_t):
d = by_t[t]
for s, r in d.items():
others = [v for k, v in d.items() if k != s]
c = days.setdefault(s, {}).setdefault(t.date(), [0, 0, t, t])
c[3] = t
if others and sum(others) / len(others) <= CALM:
c[0] += r
c[1] += 1
out = []
for s, ds in sorted(days.items()):
covered = [] # bridged runs of at least WEEK days, at SHARE_WEEK of rejections: a problem target
for run in _runs(ds, min_instants, BRIDGE, SHARE_WEEK):
share = sum(c[0] for c in run) / sum(c[1] for c in run)
if len(run) < WEEK or share < SHARE_WEEK:
continue
n = sum(c[0] / c[1] >= SHARE_WEEK for c in run)
r = _rejections([m for m in measurements[s] if run[0][2] <= m["t"] <= run[-1][3]])
covered.append((run[0][2], run[-1][3]))
out.append({"start": iso(r[0][0]), "end": iso(r[-1][1]), "kind": "Target issue", "label": "Degraded target",
"text": f"{s} was rejected in {round(100 * share)} % of the measurements when the other targets were "
f"seen, on {n} of {len(run)} days.", "ignore": True, "sensors": [s]})
for run in _runs(ds, min_instants, 0, SHARE): # at times: one episode per run of rejected measurements
if len(run) < MIN_DAYS or any(a <= run[0][2] <= b for a, b in covered):
continue
share = round(100 * sum(c[0] for c in run) / sum(c[1] for c in run))
text = f"{s} was rejected in {share} % of the measurements when the other targets were seen, {len(run)} days in a row."
out += [{"start": iso(a), "end": iso(b), "kind": "Obstruction", "label": "Target obscured", "text": text,
"ignore": True, "sensors": [s]}
for a, b in _rejections([m for m in measurements[s] if run[0][2] <= m["t"] <= run[-1][3]])]
return sorted(out + stuck(measurements, min_instants), key=lambda e: e["start"])
def _runs(ds, min_instants, bridge, share):
"""{day: [rejected, calm, first, last]} -> runs of degraded days; at most `bridge` non-degraded days in a row between
two degraded days do not break the run (they belong to it); a day that does not count does not break it."""
def _runs(days, min_instants):
"""{day: [rejected, measurements, first, last]} -> runs of degraded days; at most BRIDGE other days in a row between two
degraded days belong to the run; a day that does not count is skipped."""
runs, cur, gap = [], [], []
for j in sorted(ds):
rej, calm = ds[j][:2]
if calm < min_instants:
for d in sorted(days):
rej, n = days[d][:2]
if n < min_instants:
continue
if rej / calm >= share:
cur += gap + [ds[j]]
if rej / n >= SHARE:
cur += gap + [days[d]]
gap = []
elif cur:
gap.append(ds[j])
if len(gap) > bridge:
gap.append(days[d])
if len(gap) > BRIDGE:
runs.append(cur)
cur, gap = [], []
return runs + ([cur] if cur else [])
def _rejections(ms):
"""Measurements of a sensor -> [(start, end)] of its runs of rejected measurements."""
out, cur = [], []
for m in ms + [None]:
if m and is_rejected(m):
cur.append(m["t"])
elif cur:
out.append((cur[0], cur[-1]))
cur = []
return out
def _noise(v):
return statistics.median(abs(y - x) for x, y in zip(v, v[1:])) if len(v) > 2 else None
def stuck(measurements, min_instants):
"""Targets whose corr_x and corr_y barely move, day after day, while the others move."""
days = {} # day -> target -> (x noise, y noise, first measurement, last)
for s, ms in measurements.items():
if is_reference(ms):
continue
by_day = {}
for m in ms:
by_day.setdefault(m["t"].date(), []).append(m)
for d, dm in by_day.items():
b = [_noise([m[c] for m in dm if m[c] is not None]) for c in ("corr_x", "corr_y")]
if None not in b and len(dm) >= min_instants:
days.setdefault(d, {})[s] = (*b, dm[0]["t"], dm[-1]["t"])
runs = {} # target -> [[stuck days in a row]]
for d in sorted(days):
cs = days[d]
for s, (bx, by, a, b) in cs.items():
others = [v for k, (x, y, *_) in cs.items() if k != s for v in (x, y)]
if len(others) < 4:
continue
ref = statistics.median(others)
ru = runs.setdefault(s, [[]])
if ref and max(bx, by) <= STUCK * ref:
ru[-1].append((a, b))
elif ru[-1]:
ru.append([])
return [{"start": iso(k[0][0]), "end": iso(k[-1][1]), "kind": "Target issue", "label": "Target stuck",
"text": f"{s} barely moved while the other targets did ({len(k)} day{'s' if len(k) > 1 else ''}): it may be "
"set on a reference prism.", "ignore": True, "sensors": [s]}
for s, ru in sorted(runs.items()) for k in ru if k]
def detect(measurements):
return sorted((e for e in episodes(measurements) + displacement.episodes(measurements) if e["kind"] == "Target issue"),
key=lambda e: e["start"])
if __name__ == "__main__":
for e in detect(load(sys.argv[1])):
print(e)
min_instants = timedelta(hours=MIN_HOURS) / cadence(measurements)
days = {} # sensor -> day -> [rejected, measurements, first, last]
for s, ms in measurements.items():
for m in ms:
c = days.setdefault(s, {}).setdefault(m["t"].date(), [0, 0, m["t"], m["t"]])
c[0], c[1], c[3] = c[0] + rejected(m), c[1] + 1, m["t"]
out = []
for s, ds in sorted(days.items()):
for run in _runs(ds, min_instants):
share = sum(c[0] for c in run) / sum(c[1] for c in run)
if len(run) < MIN_DAYS or share < SHARE:
continue
bad = [m["t"] for m in measurements[s] if run[0][2] <= m["t"] <= run[-1][3] and rejected(m)]
n = sum(c[0] / c[1] >= SHARE for c in run)
seen = f"{s} was rejected in {round(100 * share)} % of the measurements, on {n} of {len(run)} days."
out.append(episode(bad[0], bad[-1], "Target issue", "Degraded target", seen + " Ignore its curve in this period "
"and look at the images of the target.", [s], True) if len(run) >= WEEK else
episode(bad[0], bad[-1], "Obstruction", "Target obscured", seen + " Something hid it for a few "
"days: ignore its curve in this period.", [s], True))
return out
+99 -232
View File
@@ -1,71 +1,44 @@
"""Detector of the Thermal displacement family (the families document): the daily wave of the corrected displacements
(expansion by day, contraction in the evening).
"""Thermal displacement family: the daily wave of the displacements (expansion by day, contraction in the evening).
Method, per target (global references excluded) and per axis (corr_x, corr_y):
1. Each UTC day of at least MIN_HOURS hours: mean of each hour minus the median of the day (removes jumps and drift).
2. Profile = median, hour by hour, over the days; "repeatability" = median, over the days, of the correlation between
the day and the profile of the other days. Axis concerned if repeatability ≥ REPRO. An incomplete day (at least
PARTIAL hours, fewer than MIN_HOURS) does not enter the profile, but its likeness is measured on the hours it has: a
hole of half a day no longer takes its episode away (wf-improve session Thermal displacement, mod. 1, S4; at
MIN_HOURS = 12 or 16, the incomplete days distorted the profile, NIEDERHORN went back).
3. Cycle of the station if at least SHARE of the targets have an axis concerned, **and** (second method, asked by the
user) if a 24 h sinusoid, a + b·sin(2πt/24 h) + c·cos(2πt/24 h) by least squares, explains at least R2_MIN of the
curve of an axis for at least SHARE of the targets.
4. Warm hours: night level = median of the NIGHT flattest consecutive hours of the profile; an hour is warm if, as a
median over the axes concerned, the profile moves away from it by at least WARM times its amplitude.
5. Two episodes per complete day that looks like the others (median correlation ≥ DAY_MATCH over the axes concerned):
"Thermal expansion" from the start to the hour of the largest gap from the night level (the peak), then "Thermal
contraction" until the return. The warm hours say there is a cycle (at least MIN_WARM_HOURS of them); its bounds
are read lower, at EDGE times the amplitude: the start, going back from the peak while the gap stays ≥ EDGE (at most
to the calmest hour of the profile); the return, going down from the peak until it goes back under EDGE, after
midnight if needed, at most to the next day's start: a slow return at night chains the cycles (wf-improve session
Thermal displacement, mod. 1, S1). The direction (a rising or falling curve) depends on the target's orientation:
it is not looked at, only the gap from the night level.
Reading: the cycle is almost everywhere in the raw positions (common mode) and the compensation removes it, except at
PS des Coudres where it stays in corr, with the same shape as in the raw positions.
Camera thermal effect (D-074): without a cycle in corr, the same method on the raw positions x, y gives "Camera thermal
expansion" and "Camera thermal contraction": the camera moves with the heat of its mount, the compensation removes it;
same categories, own sentence. On the raw positions the second method is not asked (WILDSTRUBEL, batch 01: the camera
rose by a 3 px block at sunrise, held 3 h and dropped in one step around 09:30, six days in a row, which the sinusoid
does not explain); in exchange, a single warm hour is not a cycle (MIN_WARM_HOURS: NIEDERHORN, E-005, hourly, would
otherwise have its noon hour as a camera contraction). The camera episode covers all sensors, references included: the
whole image moves.
Common slide, without a cycle (wf-improve MICHIGAN, mod. 1): lacking a cycle in the corrected curves as in the raw
positions, an instant is a camera slide if the raw positions (x or y) of at least SHARE of the sensors, references
included, **and** the corrected curves (corr_x or corr_y) of at least SHARE of the targets move away on the same side by
at least SLIDE_K noises (median of the steps of the series) from their median of the day before; slides less than a
day apart make one episode, "Camera thermal movement", on all sensors, host of the jumps back (categories.HOSTS). The
condition on the raw positions leaves to the jumps a flip of the computed pose alone (raw positions still).
The camera moves with the cold (wf-improve session Thermal displacement, mod. 1, S3): lacking a cycle, also a "Camera
thermal movement" where the camera pose (tilt, rotation or scale) moves away from its median of the day before by at
least COLD noises, while the station temperature (detect declares it, D-124) falls under its own by at least COLD
noises the same hour; noise: the median of the variations over an hour (over one step, at one measurement every 4 h),
so that the threshold does not depend on the cadence. The targets may move in opposite directions or stay almost still
(the compensation absorbs the movement): the slide rule does not see it. Consecutive pose instants make one episode.
Without a pose or without a temperature, nothing. The heat is not taken: in fine weather, it is the day's cycle.
Per target (references excluded) and per axis:
1. Each UTC day measured at least MIN_HOURS hours: the mean of each hour minus the median of the day.
2. Profile: the median, hour by hour, over the days. The axis has a cycle if its days look like the profile of the
other days (median correlation at least REPRO).
3. The station has a cycle if at least SHARE of the targets have an axis with one and, for the corrected curves, a 24 h
sinusoid explains at least R2_MIN of an axis of SHARE of the targets.
4. Gap of each hour from the night level (median of the NIGHT flattest consecutive hours of the profile), as a share of
the profile's amplitude, median over the axes with a cycle; there is a wave if the largest gap (the peak) reaches
WARM. "Expansion" from where the gap rises above EDGE to the peak, "contraction" from the peak until the gap falls
under RETURN, after midnight if needed, at most to the next start: the cooling tail is slow (wf-improve Thermal
displacement, mod. 1, S1; wf-improve NORDEND 2026-11-05, S1). Two episodes for every day of the axes with a cycle.
The corrected curves first (the structure, "Thermal ..."); without a cycle there, the raw positions x, y ("Camera
thermal ...": the camera moves with its mount and the compensation removes it, D-074; the whole image, every sensor);
without one either, the camera slides ("Camera thermal movement", wf-improve MICHIGAN, mod. 1): instants where the raw
positions of SHARE of the sensors and the corrected curves of SHARE of the targets move away on the same side by
SLIDE_K noises from their median of the day before; slides less than a day apart make one episode.
"""
import math
import os
import statistics
import sys
from datetime import datetime, timedelta, timezone
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
from data import iso, load # noqa: E402
from episodes import episode, noise, runs
from data import is_reference
MIN_HOURS = 20 # a UTC day counts in the profile if it has at least 20 measured hours
PARTIAL = 12 # an incomplete day of at least 12 hours gets its episode if it looks like the profile
REPRO = 0.6 # an axis has a cycle if each day looks like the others (pure noise ≈ 0)
SHARE = 0.5 # share of the targets concerned; 2026-10_NORDEND is just under (9/19)
WARM = 0.3 # gap from the night level, as a share of the profile's amplitude
DAY_MATCH = 0.5 # likeness of a day to the others to give it its episode
NIGHT = 8 # flattest consecutive hours of the profile = night level
R2_MIN = 0.3 # draft of 2026-10-24: PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48
MIN_WARM_HOURS = 2 # WILDSTRUBEL, batch 01; NIEDERHORN (E-005) has only one warm hour (noon)
EDGE = 0.1 # bounds of the wave (start, return), as a share of the amplitude; WARM says whether there is a cycle
SLIDE_K = 3 # common slide, in noises (median of the steps) away from the median of the day before
COLD = 4 # the camera moves with the cold, in one-hour noises away from the day before (pose, temperature)
W = 2 * math.pi / 86400 # angular frequency of a 24 h cycle, in rad/s
MIN_HOURS = 20 # hours of a day of the profile
REPRO = 0.6 # pure noise ≈ 0
SHARE = 0.5 # NORDEND of October is just under (9/19)
R2_MIN = 0.3 # PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48 (least squares then; now the projection)
NIGHT = 8 # flattest consecutive hours of the profile
WARM = 0.3 # a wave: the peak at WARM times the amplitude from the night level
EDGE = 0.1 # start of the wave (wf-improve Thermal displacement, mod. 1, S1)
RETURN = 0.05 # end of the wave (wf-improve NORDEND 2026-11-05, S1)
SLIDE_K = 3 # common slide, in noises (wf-improve MICHIGAN, mod. 1: at 2, Displacement instants of BREITHORN taken)
W = 2 * math.pi / 86400 # angular frequency of a 24 h cycle, rad/s
DAY = timedelta(days=1)
TEXTS = {"Thermal expansion": "The structure expands in the day heat.",
"Thermal contraction": "The structure contracts as it cools down.",
"Camera thermal expansion": "The camera support expands in the day heat; the curves are corrected for it.",
"Camera thermal contraction": "The camera support contracts as it cools down; the curves are corrected for it."}
def _corr(a, b):
@@ -81,20 +54,16 @@ def _axis(ms, col):
for m in ms:
if m[col] is not None:
days.setdefault(m["t"].date(), {}).setdefault(m["t"].hour, []).append(m[col])
full, every = {}, {}
full = {}
for d, hs in days.items():
if len(hs) >= PARTIAL:
med = statistics.median(v for l in hs.values() for v in l)
every[d] = {h: statistics.mean(l) - med for h, l in hs.items()}
if len(hs) >= MIN_HOURS:
full[d] = every[d]
if len(full) < 2:
return None
profile = [statistics.median(q[h] for q in full.values() if h in q) for h in range(24) if any(h in q for q in full.values())]
if len(profile) < 24:
if len(hs) >= MIN_HOURS:
med = statistics.median(v for vs in hs.values() for v in vs)
full[d] = {h: statistics.mean(vs) - med for h, vs in hs.items()}
if len(full) < 2 or any(all(h not in q for q in full.values()) for h in range(24)):
return None
profile = [statistics.median(q[h] for q in full.values() if h in q) for h in range(24)]
likeness = {}
for d, p in every.items():
for d, p in full.items():
others = [q for e, q in full.items() if e != d]
hs = [h for h in p if all(h in q for q in others)]
if len(hs) >= 3:
@@ -102,191 +71,89 @@ def _axis(ms, col):
return profile, likeness
def sine_r2(ms, col):
"""Share of the curve explained by a 24 h sinusoid (least squares, 3 unknowns), or None if too short."""
def _sine_r2(ms, col):
"""Share of the curve explained by a 24 h sinusoid (its projection on the sine and cosine of the day: the
measurements cover the hours evenly), or None if too short."""
pts = [(m["t"].timestamp(), m[col]) for m in ms if m[col] is not None]
if len(pts) <= 48:
return None
rows = [(1.0, math.sin(W * t), math.cos(W * t)) for t, _ in pts]
A = [[sum(r[i] * r[j] for r in rows) for j in range(3)] for i in range(3)]
B = [sum(r[i] * y for r, (_, y) in zip(rows, pts)) for i in range(3)]
for i in range(3): # Gaussian elimination
for k in range(i + 1, 3):
f = A[k][i] / A[i][i]
A[k] = [x - f * y for x, y in zip(A[k], A[i])]
B[k] -= f * B[i]
x = [0.0] * 3
for i in (2, 1, 0):
x[i] = (B[i] - sum(A[i][j] * x[j] for j in range(i + 1, 3))) / A[i][i]
m = statistics.mean(y for _, y in pts)
sst = sum((y - m) ** 2 for _, y in pts)
sse = sum((y - sum(c * v for c, v in zip(x, r))) ** 2 for r, (_, y) in zip(rows, pts))
mean = statistics.fmean(y for _, y in pts)
b = 2 * statistics.fmean((y - mean) * math.sin(W * t) for t, y in pts)
c = 2 * statistics.fmean((y - mean) * math.cos(W * t) for t, y in pts)
sst = sum((y - mean) ** 2 for _, y in pts)
sse = sum((y - mean - b * math.sin(W * t) - c * math.cos(W * t)) ** 2 for t, y in pts)
return 1 - sse / sst if sst else None
def sine(measurements, cols=("corr_x", "corr_y")):
"""-> (targets with an axis following a 24 h sinusoid, targets evaluated)."""
yes, evaluated = set(), set()
for s, ms in measurements.items():
if ms[0]["pattern_type"] == "global_ref":
continue
for col in cols:
r2 = sine_r2(ms, col)
if r2 is not None:
evaluated.add(s)
if r2 >= R2_MIN:
yes.add(s)
return yes, evaluated
def _wave(measurements, cols, name, camera):
targets = {s: ms for s, ms in measurements.items() if not is_reference(ms)}
axes = [(s, *a) for s, ms in targets.items() for col in cols if (a := _axis(ms, col))] # (target, profile, likeness)
concerned = [a for a in axes if a[2] and statistics.median(a[2].values()) >= REPRO]
if not axes or len({a[0] for a in concerned}) < SHARE * len({a[0] for a in axes}):
return []
if not camera: # the sinusoid must see it too
r2 = {s: [r for col in cols if (r := _sine_r2(ms, col)) is not None] for s, ms in targets.items()}
r2 = {s: v for s, v in r2.items() if v}
if not r2 or sum(max(v) >= R2_MIN for v in r2.values()) < SHARE * len(r2):
return []
def cycle(measurements, cols=("corr_x", "corr_y")):
"""-> (axes concerned [(sensor, col, profile, likeness)], number of targets evaluated)."""
concerned, targets = [], set()
for s, ms in measurements.items():
if ms[0]["pattern_type"] == "global_ref":
continue
for col in cols:
r = _axis(ms, col)
if r is None:
continue
targets.add(s)
if statistics.median(r[1].values()) >= REPRO:
concerned.append((s, col) + r)
return concerned, len(targets)
def gaps(concerned):
"""Median gap (over the axes concerned) from the night level, hour by hour, as a share of the profile's amplitude."""
def score(p, h):
def gap(p, h): # from the night level, as a share of the amplitude
start = min(range(24), key=lambda i: max(p[(i + k) % 24] for k in range(NIGHT)) - min(p[(i + k) % 24] for k in range(NIGHT)))
night = statistics.median(p[(start + k) % 24] for k in range(NIGHT))
return abs(p[h] - night) / (max(p) - min(p))
return [statistics.median(score(c[2], h) for c in concerned) for h in range(24)]
return abs(p[h] - statistics.median(p[(start + k) % 24] for k in range(NIGHT))) / (max(p) - min(p))
e = [statistics.median(gap(a[1], h) for a in concerned) for h in range(24)]
peak, rest = max(range(24), key=e.__getitem__), min(range(24), key=e.__getitem__)
if e[peak] < WARM:
return []
a = peak
while (a - 1) % 24 != rest and e[(a - 1) % 24] >= EDGE:
a -= 1
b = peak + 1
while b < a + 24 and e[b % 24] >= RETURN:
b += 1
last = max(m["t"] for ms in measurements.values() for m in ms)
sensors = measurements if camera else {a[0] for a in concerned}
out = []
for d in sorted({d for a in concerned for d in a[2]}):
midnight = datetime(d.year, d.month, d.day, tzinfo=timezone.utc)
for label, u, v in ((f"{name} expansion", a, peak), (f"{name} contraction", peak, b)):
start, end = midnight + timedelta(hours=u), min(midnight + timedelta(hours=v), last)
if end > start:
out.append(episode(start, end, "Thermal displacement", label,
TEXTS[label] + " This daily wave is normal: nothing to do.", sensors, False))
return out
def warm_hours(concerned):
return [h for h, e in enumerate(gaps(concerned)) if e >= WARM]
def detect(measurements, temperature=(), camera=()):
"""Cycle of the corrected curves (the structure); otherwise, that of the raw positions (the camera moving with the
expansion of its mount, compensated: D-074); otherwise, the camera slides (with the cold)."""
return (_episodes(measurements, ("corr_x", "corr_y"), "Thermal", False) or
_episodes(measurements, ("x", "y"), "Camera thermal", True) or slides(measurements, temperature, camera))
def _signs(ms, col, hourly=False, k=None):
"""{instant: +1 or -1} where the measurement moves away by at least k (default SLIDE_K) noises from the median of the
day before; noise: the median of the steps from one measurement to the next or, hourly, of the variations over an
hour (one step if the cadence is slow)."""
def _signs(ms, col):
"""{instant: +1 or -1} where the value moves away by at least SLIDE_K noises from its median of the day before."""
pts = [(m["t"], m[col]) for m in ms if m[col] is not None]
if len(pts) < 3:
return {}
step = statistics.median((b - a).total_seconds() for (a, _), (b, _) in zip(pts, pts[1:])) if hourly else 0
n = max(1, round(3600 / step)) if step else 1
noise = statistics.median(abs(pts[i + n][1] - pts[i][1]) for i in range(len(pts) - n)) if len(pts) > n else 0
out, j = {}, 0
n, out, j = noise([v for _, v in pts]), {}, 0
for i, (t, v) in enumerate(pts):
while pts[j][0] < t - timedelta(days=1):
while pts[j][0] < t - DAY:
j += 1
before = [w for _, w in pts[j:i]]
if noise and len(before) >= 3 and abs(v - statistics.median(before)) >= (k or SLIDE_K) * noise:
out[t] = 1 if v > statistics.median(before) else -1
if n and len(before) >= 3 and abs(v - (med := statistics.median(before))) >= SLIDE_K * n:
out[t] = 1 if v > med else -1
return out
def _common(series):
"""[{instant: sign}] of a group of sensors -> instants where at least SHARE of them move away on the same side."""
n, count = len(series), {}
count = {}
for sg in series:
for t, x in sg.items():
count[(t, x)] = count.get((t, x), 0) + 1
return {t for (t, x), c in count.items() if n and c >= SHARE * n}
return {t for (t, x), c in count.items() if c >= SHARE * len(series)}
def _cold(measurements, temperature, camera):
"""[(start, end)] of the runs of instants where the camera pose (tilt, rotation or scale) moves away from its day
before while the temperature falls under its own, by at least COLD one-hour noises; empty without one of them."""
ts = [m["t"] for ms in measurements.values() for m in ms]
hour = lambda t: t.replace(minute=0, second=0, microsecond=0) # noqa: E731
cold = {hour(t) for t, x in _signs([{"t": t, "T": v} for t, v in temperature], "T", True, COLD).items() if x < 0}
pose = [p for p in camera if min(ts) <= p["t"] <= max(ts)] if ts else []
seen = {t for c in ("tilt", "rotation", "scale") for t in _signs(pose, c, True, COLD) if hour(t) in cold}
episodes = [] # consecutive pose instants make one episode
for i, p in enumerate(pose):
if p["t"] in seen:
if episodes and episodes[-1][2] == i - 1:
episodes[-1][1:] = [p["t"], i]
else:
episodes.append([p["t"], p["t"], i])
return [(a, b) for a, b, _ in episodes]
def slides(measurements, temperature=(), camera=()):
"""Common camera slide, without a 24 h cycle: the raw positions of most sensors, references included, and the
corrected curves of most targets move away together, on the same side, from their day before; or the camera pose
moves away from it with the cold (_cold)."""
targets = [ms for ms in measurements.values() if ms[0]["pattern_type"] != "global_ref"]
def slides(measurements):
targets = [ms for ms in measurements.values() if not is_reference(ms)]
raw = set().union(*(_common([_signs(ms, c) for ms in measurements.values()]) for c in ("x", "y")))
corr = set().union(*(_common([_signs(ms, c) for ms in targets]) for c in ("corr_x", "corr_y")))
episodes = []
for t in sorted(raw & corr):
if episodes and t - episodes[-1][1] <= timedelta(days=1): # less than a day between two: the same slide
episodes[-1][1] = t
else:
episodes.append([t, t])
return [{"start": iso(a), "end": iso(b), "kind": "Thermal displacement", "label": "Camera thermal movement",
"text": "The camera moved with a change of temperature and all the curves moved with it; this is not a "
"movement of the structure, nothing to do.", "ignore": False, "sensors": sorted(measurements)}
for a, b in episodes + _cold(measurements, temperature, camera)]
return [episode(r[0], r[-1], "Thermal displacement", "Camera thermal movement", "The camera moved with a change of "
"temperature and all the curves moved with it; this is not a movement of the structure, nothing to "
"do.", measurements, False) for r in runs(sorted(raw & corr), DAY)]
def _episodes(measurements, cols, name, camera):
concerned, n = cycle(measurements, cols)
sensors = sorted({c[0] for c in concerned})
if not n or len(sensors) < SHARE * n:
return []
if not camera: # second method, for the structure only: the sinusoid must see it too
yes, evaluated = sine(measurements, cols)
if not evaluated or len(yes) < SHARE * len(evaluated):
return []
warm = warm_hours(concerned)
if len(warm) < MIN_WARM_HOURS:
return []
e = gaps(concerned)
peak = max(warm, key=lambda h: e[h]) # the largest gap from the night level: end of the expansion
rest = min(range(24), key=lambda h: e[h])
a = peak # start: going back from the peak while the profile stays above EDGE, up to the calmest hour
while (a - 1) % 24 != rest and e[(a - 1) % 24] >= EDGE:
a -= 1
b = peak + 1 # return: until the profile goes back under EDGE, at most to the next start
while b < a + 24 and e[b % 24] >= EDGE:
b += 1
last = max(m["t"] for ms in measurements.values() for m in ms)
days = sorted({d for c in concerned for d in c[3]})
if camera: # the whole image moves, references included
sensors = sorted(measurements)
episodes = []
for d in days: # ponytail: days of at least PARTIAL hours, the export's ends (22:00 → midnight) get no episode
if statistics.median(c[3][d] for c in concerned if d in c[3]) < DAY_MATCH:
continue
midnight = datetime(d.year, d.month, d.day, tzinfo=timezone.utc)
for label, u, v in ((f"{name} expansion", a, peak), (f"{name} contraction", peak, b)):
start, end = midnight + timedelta(hours=u), min(midnight + timedelta(hours=v), last)
if end > start:
episodes.append({"start": iso(start), "end": iso(end),
"kind": "Thermal displacement", "label": label, "text": "", "ignore": False, "sensors": sensors})
return episodes
if __name__ == "__main__":
measurements = load(sys.argv[1])
for e in detect(measurements):
print(e["start"], "→", e["end"], e["label"], len(e["sensors"]), "targets")
for cols in (("corr_x", "corr_y"), ("x", "y")):
concerned, n = cycle(measurements, cols)
k = len({c[0] for c in concerned})
amp = statistics.median(max(c[2]) - min(c[2]) for c in concerned) if concerned else 0
yes, evaluated = sine(measurements, cols)
print(f"{cols[0][:-2] or 'raw'}: targets {k}/{n}, median amplitude {amp:.3f} px, warm hours (UTC) "
f"{warm_hours(concerned) if concerned else '-'}; sinusoid: targets {len(yes)}/{len(evaluated)}")
def detect(measurements):
return (_wave(measurements, ("corr_x", "corr_y"), "Thermal", False)
or _wave(measurements, ("x", "y"), "Camera thermal", True) or slides(measurements))
+115 -301
View File
@@ -1,112 +1,34 @@
"""Detector of the Visibility family (the families document): the camera sees less well, the image is faint or blurred
(error_value), or it trembles (heat shimmer, end of file). The same error_value computation also finds objects in front
of a few targets (Brief obstruction, Target obscured): episodes() returns them with the Obstruction family, which
detect() leaves to obstruction.py.
"""Visibility family: the camera sees the targets less well (error_value), or the air trembles (heat shimmer). The same
error_value also finds something passing in front of a few targets: Brief obstruction (Obstruction family).
Method:
1. "High" measurement: error_value > median + K_MAD MAD of the sensor over the export, and > FACTOR × median, or
rejected (> error_threshold) and > REJECT_FACTOR × median: a sensor at a usual level of 0.6 never doubles, and
FLORIDA (batch 01) showed its fogs as Target obscured.
2. Brief obstruction: an instant where at least BRIEF_MIN sensors are high, SALIENCE times more than at the neighbouring
instants (train, vehicle), with no reference (global_ref) among them: a one-step peak touching the references is not
an obstacle in front of targets (peak of the whole image, NORDEND on 01.09 11:15, batch 01); it is not shown. Unless
at least BRIEF_MIN targets are rejected at this instant and neither at the instant before nor after: something
passing in front of them, a reference touched as well (wf-improve session Displacement, mod. 1, S5: trains of
NIEDERHORN, 12 and 13 targets rejected).
Also, something passing in front of one or two targets, or a single reference: at an instant, fewer than BRIEF_MIN
sensors rejected at more than FACTOR times their median (without K_MAD: by day, heat shimmer inflates the MAD), at
most one reference, none of them rejected at the instant before or after, and fewer than BRIEF_MIN high sensors
(wf-improve session Obstruction, mod. 1, S3: vehicles, passers-by, insects in front of one or two targets of
WILDSTRUBEL, NIEDERHORN, LENZSPITZE; a reference hidden by a truck at JUNGFRAU, BREITHORN).
3. Poor visibility: runs of at least RUN_MIN high measurements per sensor, joined across sensors (at least 2), reduced
to the instants where at least FRAC of the episode's peak is high.
4. Target obscured: a target (not a global_ref) rejected (> error_threshold) and above K_TARGET MAD, at least TARGET_MIN
times less than TARGET_GAP apart, mostly outside the episodes of 2 and 3 (leaves, snow, glare on a target). The gap
is at least two measurement steps, and two rejections are enough when TARGET_MIN steps do not fit in TARGET_GAP
(wf-improve session Obstruction, mod. 1, S6: at one measurement every 4 h, vehicles parked at night in front of COLI22,
COLI24, POUI22 of BREITHORN make only two or three rejections). All episodes: ignore = True.
5. Level of a Poor visibility (validated 2026-10-24), measurement by measurement: "none" (nothing is seen) when at least
HALF of the usually seen targets (under their threshold at least half the time) are rejected, "reduced" (average
data) otherwise; one episode per period of the same level, with the level field.
6. Bounds of a Poor visibility (WILDSTRUBEL, batch 01): from the first to the last measurement where a target is
rejected (> error_threshold), and no episode without a rejected target. On a road in full sun, the daytime
error_value (×1.5 to ×3, heat shimmer) made reduced visibility last for hours without any rejection; references do
not count: R3 is above its threshold almost all day.
7. Poor visibility or object (boundary "Visibility or Obstruction" of the families document; BREITHORN, batch 01): a
nuisance touching at most FEW targets, and no reference, is not diffuse: it is an object in front of them (a parked
vehicle), a "Target obscured" episode of its whole duration, without a level.
8. Diffuse nuisance, with or without rejections (wf-improve session Visibility, mod. 1, S2): at each instant, the median,
over the sensors, of the error_value in MAD above each one's median; at least RUN_MIN instants in a row at K_DIFFUSE
or more make a Poor visibility "reduced". A diffuse nuisance entirely within heat shimmer is the heat shimmer: no
episode (otherwise it absorbed the jump back of the day before, WILDSTRUBEL on 04.10). Finds the drops of
visibility of the whole scene without rejection at JUNGFRAU, FLORIDA, NORDEND; at 2, nights labelled thermal at
JUNGFRAU; at 3, half the gain.
A simpler version (share of targets rejected above their threshold, without comparing each sensor with its own level)
was tried on 2026-10-24 and dropped: worse on five data sets out of six (git history of Support-Agent,
detectors/archive/visibilite_simple.py).
1. High measurement: error_value above the median of its sensor + K_MAD MAD and above FACTOR times that median.
2. Brief obstruction (one instant): at least BRIEF_MIN sensors high, SALIENCE times more than at each neighbouring instant
(a train, a vehicle); or fewer than BRIEF_MIN sensors, at most one reference, rejected at FACTOR times their median
and not at the neighbouring instants (something passing in front of one or two targets).
3. Poor visibility: runs of at least RUN_MIN high measurements of a sensor (steps of at most GAP_STEPS), outside the brief
instants, overlapping on at least two sensors. Level of each instant: "none" (greyed) when at least HALF of the
usually seen targets are rejected, "reduced" otherwise; one episode per period of the same level.
4. Diffuse nuisance: at least RUN_MIN instants in a row where the median over the sensors of the error_value, in MAD
above each one's median, reaches K_DIFFUSE: Poor visibility "reduced", unless within heat shimmer or another Poor
visibility.
Heat shimmer: end of file.
"""
import os
import statistics
import sys
from datetime import timedelta
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
from data import iso, load # noqa: E402
from episodes import cadence, episode, level, runs, targets
from data import is_reference
K_MAD = 10 # tuned on E-001 (fog: 11 to 57 MAD) and E-005 (rain, trains: 10 to 70 MAD)
FACTOR = 2 # drops sensors with a near-zero MAD (E-004: PWD_REF10 ×1.6 at 14 MAD, E-005)
RUN_MIN = 3 # high measurements in a row for poor visibility (E-001, E-005 rain)
BRIEF_MIN = 3 # tuned on E-005 (trains: 5 to 17 sensors; the one with 2 targets is missed)
K_TARGET = 5 # tuned on E-005 (leaves: 6 to 25 MAD, rejected)
TARGET_MIN = 3 # repeated rejections of a target
FRAC = 0.15 # cuts the tails of PR09 + CTR01 (NORDEND October) without cutting the cobweb (E-002)
GAP_STEPS = 2.5 # steps tolerated between two high measurements of the same run (a missing step, clock to the second)
BREAK = timedelta(hours=2) # lull separating two episodes (NORDEND October, night of 13 to 14.10)
SALIENCE = 3 # a train stands out at least 3 times above the neighbouring instants (E-005: 5 to 17 against 0 to 2;
# the cobweb of E-002 varies from 2 to 9 from one step to the next)
TARGET_GAP = timedelta(hours=6) # tuned on E-005 (leaves: 7 rejections in 12 h, up to 5 h apart)
REJECT_FACTOR = 1.2 # a rejection at 10 MAD also counts (FLORIDA, batch 01: levels of 0.6, ×2 out of reach); drops C24
# of E-009 (level 0.68 just under the threshold 0.7, rejected at ×1.04)
HALF = 0.5 # "nothing is seen"; tuned on NORDEND (09.10 23:00: 68 % rejected) and E-005 (rain: 24 to 33 %)
K_DIFFUSE = 2.5 # diffuse nuisance (8); wf-improve session Visibility, mod. 1, S2
FEW = 3 # Poor visibility on at most 3 targets, no reference: an object, Target obscured (7); BREITHORN,
# batch 01 (vehicles parked in front of 2 or 3 targets out of 12)
K_MAD = 10 # E-001 (fog: 11 to 57 MAD), E-005 (rain, trains: 10 to 70 MAD)
FACTOR = 2 # drops the sensors with a near-zero MAD (E-004, E-005)
RUN_MIN = 3 # E-001, rain of E-005
GAP_STEPS = 2.5 # steps tolerated inside a run (a missing step, timestamp jitter)
BRIEF_MIN = 3 # trains of E-005: 5 to 17 sensors
SALIENCE = 3 # a train stands out (E-005: 5 to 17 against 0 to 2); the cobweb of E-002 varies from 2 to 9
HALF = 0.5 # "nothing is seen": NORDEND 09.10 23:00 (68 % rejected) against the rain of E-005 (24 to 33 %)
K_DIFFUSE = 2.5 # wf-improve Visibility, mod. 1, S2 (JUNGFRAU, FLORIDA, NORDEND: the whole scene less clear)
def _step(measurements):
ts = sorted({m["t"] for v in measurements.values() for m in v})
return statistics.median(b - a for a, b in zip(ts, ts[1:]))
def _group(points, gap):
"""[(t, sensor)] sorted by t -> groups [(start, end, {sensors}, [t])] whose points are ≤ gap from each other."""
groups = []
for t, s in points:
if groups and t - groups[-1][1] <= gap:
g = groups[-1]
groups[-1] = (g[0], t, g[2] | {s}, g[3] + [t])
else:
groups.append((t, t, {s}, [t]))
return groups
def _group_intervals(intervals, gap):
"""[(start, end, sensor)] sorted -> [(start, end, {sensors})] of the intervals that overlap (within gap)."""
out = []
for a, b, s in intervals:
if out and a - out[-1][1] <= gap:
out[-1] = (out[-1][0], max(out[-1][1], b), out[-1][2] | {s})
else:
out.append((a, b, {s}))
return out
def _episode(start, end, label, text, sensors, level=None):
e = {"start": iso(start), "end": iso(end), "kind": "Visibility" if label == "Poor visibility" else "Obstruction",
"label": label, "text": text,
"ignore": level != "reduced", "sensors": sorted(sensors)} # reduced visibility: average data, not greyed
return {**e, "level": level} if level else e
BRIEF = "Something passed in front of {} for a single measurement. Ignore this point on the curves."
def _levels(measurements, instants):
@@ -120,223 +42,115 @@ def _levels(measurements, instants):
for t in instants}
def episodes(measurements):
"""All error_value episodes: Visibility (Poor visibility), Obstruction (Brief obstruction, Target obscured)."""
step = _step(measurements)
gap = GAP_STEPS * step
refs = {s for s, v in measurements.items() if v[0]["pattern_type"] == "global_ref"}
instants = sorted({m["t"] for v in measurements.values() for m in v})
high, rejections = {}, {} # sensor -> [t]
refused = set() # instants where a target (not a global_ref) is above its threshold
rejected_targets = {} # t -> {targets above their threshold}
strong, refusals = {}, {} # t -> {sensors rejected at more than FACTOR times their median}, {sensors rejected}
for s, v in measurements.items():
e = [m["error_value"] for m in v if m["error_value"] is not None]
if len(e) < 10:
continue
med = statistics.median(e)
mad = statistics.median(abs(x - med) for x in e) or 1e-9
for m in v:
x = m["error_value"]
if x is None:
continue
rejected = m["error_threshold"] and x > m["error_threshold"]
if rejected and s not in refs:
refused.add(m["t"])
rejected_targets.setdefault(m["t"], set()).add(s)
if x > med + K_MAD * mad and (x > FACTOR * med or rejected and x > REJECT_FACTOR * med):
high.setdefault(s, []).append(m["t"])
if rejected:
refusals.setdefault(m["t"], set()).add(s)
if x > FACTOR * med:
strong.setdefault(m["t"], set()).add(s)
if s not in refs and rejected and x > med + K_TARGET * mad:
rejections.setdefault(s, []).append(m["t"])
by_t = {} # t -> {high sensors}
for s, ts in high.items():
for t in ts:
by_t.setdefault(t, set()).add(s)
n = lambda t: len(by_t.get(t, ())) # noqa: E731
out = []
# 2. Brief obstruction: an instant where many sensors are high, at least SALIENCE times more than the neighbours.
briefs = set()
for i, t in enumerate(instants):
neighbours = [n(instants[j]) for j in (i - 1, i + 1) if 0 <= j < len(instants)]
if n(t) >= BRIEF_MIN and n(t) >= SALIENCE * max(neighbours):
briefs.add(t)
only = rejected_targets.get(t, set()).difference(*(rejected_targets.get(instants[j], set()) for j in (i - 1, i + 1)
if 0 <= j < len(instants)))
if by_t[t] & refs and len(only) < BRIEF_MIN: # references touched without targets rejected at this instant
continue # alone: not an obstacle in front of targets, the whole image
out.append(_episode(t, t, "Brief obstruction", f"{n(t)} points were poorly seen for a single measurement, then came "
"back: probably a train, a vehicle or an animal passing; ignore this point on the curves.", by_t[t]))
elif 0 < len(f := strong.get(t, set())) < BRIEF_MIN and n(t) < BRIEF_MIN and len(f & refs) <= 1 and not f & set().union(
*(refusals.get(instants[j], set()) for j in (i - 1, i + 1) if 0 <= j < len(instants))):
# one or two targets, or a reference, rejected at this single measurement, at FACTOR times their median
out.append(_episode(t, t, "Brief obstruction", f"{len(f)} point{'s were' if len(f) > 1 else ' was'} poorly seen "
"for a single measurement, then came back: probably something passing; ignore this point on the "
"curves.", f))
# 3. Poor visibility: long runs per sensor (without the brief instants), joined across sensors, then reduced to the
# instants where at least FRAC of the episode's peak is high (cuts the tails of one or two sensors and the long
# lulls, more than BREAK).
runs = []
for s, ts in high.items():
pts = [(t, s) for t in ts if t not in briefs]
runs += [(a, b, s) for a, b, _, l in _group(pts, gap) if len(l) >= RUN_MIN]
zones = [z for z in _group_intervals(sorted(runs), gap) if len(z[2]) >= 2]
kept = []
for a, b, _ in zones:
inside = [t for t in instants if a <= t <= b and t not in briefs]
peak = max(n(t) for t in inside)
pts = [(t, s) for t in inside if n(t) >= max(1, FRAC * peak) for s in by_t[t]]
kept += [(d, f, cs) for d, f, cs, l in _group(pts, max(gap, BREAK)) if len(cs) >= 2 and len(set(l)) >= RUN_MIN]
# Bounds: from the first to the last rejected target; without a rejection, no episode (6).
kept = [(r[0], r[-1], cs) for a, b, cs in kept if (r := [t for t in instants if a <= t <= b and t in refused])]
level = _levels(measurements, instants)
for a, b, cs in kept: # one episode per period of the same level (5); the text comes from categories.py
if len(cs) <= FEW and not cs & refs: # an object in front of a few targets, not a diffuse nuisance (7)
out.append(_episode(a, b, "Target obscured", "", cs))
continue
seg = []
for t in [t for t in instants if a <= t <= b] + [None]:
if seg and (t is None or level[t] != level[seg[0]]):
who = cs & set().union(*(by_t.get(u, set()) for u in seg)) or cs
out.append(_episode(seg[0], seg[-1], "Poor visibility", "", who, level[seg[0]]))
seg = []
if t is not None:
seg.append(t)
inside_any = lambda t: t in briefs or any(a <= t <= b for a, b, _ in kept) # noqa: E731
# 4. Target obscured: a target rejected again and again, mostly outside the episodes above.
targets = []
for s, ts in rejections.items():
for a, b, _, l in _group([(t, s) for t in ts], max(TARGET_GAP, 2 * step)):
if len(l) >= (TARGET_MIN if TARGET_MIN * step <= TARGET_GAP else 2) and sum(map(inside_any, l)) * 2 < len(l):
targets.append((a, b, s))
for a, b, cs in _group_intervals(sorted(targets), timedelta(0)):
who, it, its = ("One target was", "it", "its") if len(cs) == 1 else (f"{len(cs)} targets were", "them", "their")
out.append(_episode(a, b, "Target obscured", f"{who} repeatedly lost while the targets around stayed clear: probably "
f"something in front of {it} (leaves, snow) or glare; {its} displacements in this period are unreliable.", cs))
return sorted(out, key=lambda e: e["start"])
def diffuse(measurements):
"""Diffuse nuisance (8): the runs of at least RUN_MIN instants where the median, over the sensors, of the error_value
in MAD above each one's median reaches K_DIFFUSE, with or without rejections -> Poor visibility "reduced"."""
z = {} # t -> {sensor: gap in MAD}
for s, v in measurements.items():
e = [m["error_value"] for m in v if m["error_value"] is not None]
if len(e) < 10:
continue
med = statistics.median(e)
mad = statistics.median(abs(x - med) for x in e) or 1e-9
for m in v:
if m["error_value"] is not None:
z.setdefault(m["t"], {})[s] = (m["error_value"] - med) / mad
highs = [(t, None) for t in sorted(z) if statistics.median(z[t].values()) >= K_DIFFUSE]
return [_episode(a, b, "Poor visibility", "", {s for t in l for s, x in z[t].items() if x >= K_DIFFUSE}, "reduced")
for a, b, _, l in _group(highs, GAP_STEPS * _step(measurements)) if len(l) >= RUN_MIN]
def _poor(start, end, sensors, level, refs):
refs = ", including the reference targets" if set(sensors) & refs else ""
text = (f"The camera could not see {targets(len(sensors))} clearly{refs}. The curves in this period are not "
"reliable: ignore them." if level == "none" else f"The camera saw {targets(len(sensors))} less clearly than "
f"usual{refs}. The data in this period are less precise: use them with care.")
return episode(start, end, "Visibility", "Poor visibility", text, sensors, level == "none", level=level)
def detect(measurements):
shimmer = heat_shimmer(measurements) # a diffuse nuisance during heat shimmer is the heat shimmer (8)
return ([e for e in episodes(measurements) if e["kind"] == "Visibility"] + shimmer
+ [e for e in diffuse(measurements) if not any(t["start"] <= e["start"] and e["end"] <= t["end"] for t in shimmer)])
gap = GAP_STEPS * cadence(measurements)
refs = {s for s, v in measurements.items() if is_reference(v)}
instants = sorted({m["t"] for v in measurements.values() for m in v})
z = {} # t -> {sensor: error_value in MAD above the sensor's median}
high, strong, rejected = {}, {}, {} # t -> {sensors}
for s, v in measurements.items():
e = [m["error_value"] for m in v if m["error_value"] is not None]
if len(e) < 10:
continue
med, mad = level(e)
for m in v:
t, x = m["t"], m["error_value"]
if x is None:
continue
z.setdefault(t, {})[s] = (x - med) / mad
if x > med + K_MAD * mad and x > FACTOR * med:
high.setdefault(t, set()).add(s)
if m["error_threshold"] and x > m["error_threshold"]:
rejected.setdefault(t, set()).add(s)
if x > FACTOR * med:
strong.setdefault(t, set()).add(s)
n = lambda t: len(high.get(t, ())) # noqa: E731
out, briefs = [], set()
for i, t in enumerate(instants): # 2. Brief obstruction
around = [instants[j] for j in (i - 1, i + 1) if 0 <= j < len(instants)]
few = strong.get(t, set())
if n(t) >= BRIEF_MIN and n(t) >= SALIENCE * max(map(n, around), default=0):
briefs.add(t)
few = high[t]
elif not (few and len(few) < BRIEF_MIN and n(t) < BRIEF_MIN and len(few & refs) <= 1
and not few & set().union(*(rejected.get(u, set()) for u in around))):
continue
out.append(episode(t, t, "Obstruction", "Brief obstruction", BRIEF.format(targets(len(few))), few, True))
spans = [] # 3. Poor visibility: runs of each sensor, then the overlapping runs of at least two sensors
for s in {s for t in high for s in high[t]}:
spans += [(r[0], r[-1], s) for r in runs([t for t in instants if s in high.get(t, ()) and t not in briefs], gap)
if len(r) >= RUN_MIN]
zones = []
for a, b, s in sorted(spans):
if zones and a - zones[-1][1] <= gap:
zones[-1] = (zones[-1][0], max(zones[-1][1], b), zones[-1][2] | {s})
else:
zones.append((a, b, {s}))
zones = [zone for zone in zones if len(zone[2]) >= 2]
lv = _levels(measurements, instants)
for a, b, cs in zones:
inside = [t for t in instants if a <= t <= b]
for k, t in enumerate(inside):
seg = [t] if k == 0 or lv[t] != lv[inside[k - 1]] else seg + [t]
if k == len(inside) - 1 or lv[inside[k + 1]] != lv[t]:
out.append(_poor(seg[0], seg[-1], cs & set().union(*(high.get(u, set()) for u in seg)) or cs, lv[t], refs))
shimmer = heat_shimmer(measurements) # 4. diffuse nuisance
hosts = shimmer + [e for e in out if e["kind"] == "Visibility"]
for r in runs([t for t in sorted(z) if statistics.median(z[t].values()) >= K_DIFFUSE], gap):
e = _poor(r[0], r[-1], {s for t in r for s, x in z[t].items() if x >= K_DIFFUSE}, "reduced", refs)
if len(r) >= RUN_MIN and not any(h["start"] <= e["start"] and e["end"] <= h["end"] for h in hosts):
out.append(e)
return out + shimmer
# --- Heat shimmer ----------------------------------------------------------------------------------------------------
# Atmospheric turbulence (fast noise on the corrected displacements). Method:
# 1. Per sensor and per hour H, on the measurements of [H − 1 h, H + 2 h): agitation = median of |Δcorr_x| + |Δcorr_y|
# between successive measurements (robust to one-step jumps), and median of the error_value.
# 2. Each is divided by the sensor's calm: the lower quartile of its hourly values over the export (often the night).
# 3. Turbulent hour: at least SHARE of the sensors at ≥ RATIO times their calm agitation, and median error_value of the
# sensors at ≥ RATIO_ERR times the calm (warm air blurs the spots: E-003).
# 4. Neighbouring hours merged (a one-hour hole tolerated), episode of at least MIN_DURATION, dropped if more than
# REJECT_MAX of the measurements are rejected beyond their sensor's habit (its share of rejections over the export):
# it is then poor visibility, another rule. A target that is always rejected (Target issue) does not count (wf-improve
# session Visibility, mod. 1, S1: WILDSTRUBEL, R3 rejected all day, the days of heat shimmer were dropped).
# Greyed if the median agitation ≥ STRONG.
RATIO = 1.5 # tuned on E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night)
SHARE = 0.5 # share of the sensors agitated at the same time
RATIO_ERR = 1.5 # tuned on E-003 (error_value ×1.7 to ×3.2 the night; ≈ ×1 at noon elsewhere)
REJECT_MAX = 0.15 # E-003 ≤ 7 %; cobweb (E-002) and rain (E-005) ≥ 18 %; rejections beyond the habit since wf-improve
# session Visibility, mod. 1, S1 (WILDSTRUBEL: 4 to 14 %, against 9 to 22 % raw)
STRONG = 2.0 # E-003 (×2.3 to ×2.6) is strong; an episode between ×1.5 and ×2 stays visible
MIN_DURATION = timedelta(hours=3) # E-003 lasts 10-11 h; drops the morning of the real movement of E-004 (2 h), which
# the classification drops too, whatever the duration
# --- Heat shimmer: fast noise of the corrected displacements in warm air.
# 1. Per sensor and per hour H, over [H - 1 h, H + 2 h): agitation, the median of |Δcorr_x| + |Δcorr_y| between successive
# measurements, and the median error_value; each divided by its calm (lower quartile of the sensor's hourly values).
# 2. Turbulent hour: at least SHARE of the sensors agitated at RATIO times their calm, and median error_value at RATIO.
# 3. Turbulent hours at most 2 h apart make an episode, greyed if the median agitation reaches STRONG.
RATIO = 1.5 # E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night, error_value ×1.7 to ×3.2)
SHARE = 0.5
STRONG = 2.0 # E-003 (×2.3 to ×2.6)
HOUR = timedelta(hours=1)
def _hour(t):
return t.replace(minute=0, second=0, microsecond=0)
SHIMMER = "Warm air made the readings shake ({:.1f}× noisier than in calm hours). The small fast wiggles are not movement."
def _ratios(points):
"""[(t, value)] -> {hour H: median over [H − 1 h, H + 2 h) / lower quartile of these medians}."""
"""[(t, value)] -> {hour H: median over [H - 1 h, H + 2 h) / lower quartile of these medians}."""
by = {}
for t, v in points:
for k in (-1, 0, 1):
by.setdefault(_hour(t) + k * HOUR, []).append(v)
by.setdefault(t.replace(minute=0, second=0, microsecond=0) + k * HOUR, []).append(v)
med = {h: statistics.median(v) for h, v in by.items() if len(v) >= 3}
if len(med) < 8:
return {}
calm = statistics.quantiles(med.values(), n=4)[0]
calm = statistics.quantiles(med.values(), n=4)[0] if len(med) >= 8 else 0
return {h: v / calm for h, v in med.items()} if calm > 0 else {}
def hours(measurements):
"""-> {hour: {sensor: (agitation / calm, error_value / calm)}}."""
out = {}
def heat_shimmer(measurements):
hours = {} # hour -> {sensor: (agitation / calm, error_value / calm)}
for s, ms in measurements.items():
ag = _ratios([(b["t"], abs(b["corr_x"] - a["corr_x"]) + abs(b["corr_y"] - a["corr_y"]))
for a, b in zip(ms, ms[1:]) if a["corr_x"] is not None and b["corr_x"] is not None])
er = _ratios([(m["t"], m["error_value"]) for m in ms if m["error_value"] is not None])
for h, r in ag.items():
if h in er:
out.setdefault(h, {})[s] = (r, er[h])
return out
def _turbulent(r):
return (sum(a >= RATIO for a, _ in r.values()) >= SHARE * len(r)
and statistics.median(e for _, e in r.values()) >= RATIO_ERR)
def heat_shimmer(measurements):
H = hours(measurements)
blocks = []
for h in sorted(h for h, r in H.items() if _turbulent(r)):
if blocks and h - blocks[-1][-1] <= 2 * HOUR:
blocks[-1].append(h)
else:
blocks.append([h])
habit = {s: sum(m["corr_x"] is None for m in ms) / len(ms) for s, ms in measurements.items()} # usual rejections
episodes = []
for b in blocks:
start, end = b[0], b[-1] + HOUR
by = {s: [m for m in ms if start <= m["t"] < end] for s, ms in measurements.items()}
new = sum(max(0, sum(m["corr_x"] is None for m in d) - habit[s] * len(d)) for s, d in by.items())
if end - start < MIN_DURATION or new > REJECT_MAX * sum(map(len, by.values())):
continue
by_sensor = {}
for h in b:
for s, (a, _) in H[h].items():
by_sensor.setdefault(s, []).append(a)
agit = {s: statistics.median(v) for s, v in by_sensor.items()}
for h in ag.keys() & er.keys():
hours.setdefault(h, {})[s] = (ag[h], er[h])
turbulent = [h for h, r in sorted(hours.items()) if sum(a >= RATIO for a, _ in r.values()) >= SHARE * len(r)
and statistics.median(e for _, e in r.values()) >= RATIO]
out = []
for b in runs(turbulent, 2 * HOUR):
agit = {s: statistics.median(hours[h][s][0] for h in b if s in hours[h]) for s in {s for h in b for s in hours[h]}}
strength = statistics.median(agit.values())
episodes.append({
"start": iso(start), "end": iso(end), "kind": "Visibility",
"label": "Heat shimmer",
"text": f"Readings were about {strength:.1f}× noisier than in calm hours because of heat shimmer "
"(warm air trembling in the line of sight); the small fast wiggles are not movement.",
"ignore": strength >= STRONG,
"sensors": sorted(s for s, v in agit.items() if v >= RATIO),
})
return episodes
if __name__ == "__main__":
for e in detect(load(sys.argv[1])):
print(e["start"], "→", e["end"], e["label"], f"| {len(e['sensors'])}:", ",".join(e["sensors"][:8]), "|", e["text"])
out.append(episode(b[0], b[-1] + HOUR, "Visibility", "Heat shimmer", SHIMMER.format(strength),
[s for s, v in agit.items() if v >= RATIO], strength >= STRONG))
return out
-85
View File
@@ -1,85 +0,0 @@
---
name: wf-improve
description: Workflow that improves the detectors on a range of data where they perform poorly, given labels as the truth on that range — the upgrader fixes, the validator judges; the result, detectors a little better on the range without going back elsewhere (benchmark on a reference truth), or unchanged. The caller gives the range, the labels, the reference truth, the output folder and any specific context. /detectors:wf-improve <request>.
argument-hint: "<range, labels, reference, output folder, context>"
---
# wf-improve: improve the detectors on a range
You are the orchestrator. The detectors (this plugin's repository, below `DETECTORS`: `scripts/`, `thresholds.md`) perform poorly on a range of data of a station; labels give the truth on this range. You run the upgrader, which fixes the detectors, then the validator, which judges the fixes against overfitting, and you return detectors **a little better on the range without going back elsewhere**, or unchanged. **You judge nothing yourself**: you only read the verdict lines. **You ask no question**: the workflow runs alone to the end.
## Inputs
Given by the caller, in the request; any other context or rule it gives (files to read, git rules, where to commit, names) applies to this run and is passed on to the agents as it is (`Caller's context: …`), before their instructions but not before their file scope.
- `DETECTORS`: the path of this repository (default: `detectors/`).
- `STATION`: the station's data folder (read by `DETECTORS/scripts/data.py`).
- `FROM`, `TO`: the range, UTC days (YYYY-MM-DD, `TO` excluded).
- `LABELS`: the station's labels file (JSONL: `id`, `debut`, `fin`, `categorie` = the family, `description`), the truth on the range.
- `REFERENCE`: the reference truth for the non-regression, `--labels <folder of <STATION>.jsonl> --data <folder of <STATION>/>`.
- `FAMILIES`: the document defining the families and their boundaries (one detector per family).
- `CONTEXT` (optional): documents explaining the range's labels (an analyst's report, for example).
- `OUT`: the output folder of this run.
A required input missing, or `DETECTORS` not a git repository: say so and stop, changing nothing.
## Commands
- events: `python3 <DETECTORS>/scripts/categories.py <STATION> --from <FROM> --to <TO>`
- sensors: `python3 <DETECTORS>/skills/wf-improve/scripts/sensors.py <STATION> --from <FROM> --to <TO>`
- range score (the goal): `python3 <DETECTORS>/skills/wf-improve/scripts/score.py <STATION> --labels <LABELS> --from <FROM> --to <TO>`; it exits with an error if no label touches the range: note it in the summary and stop without touching the detectors.
- benchmark (the non-regression, about 3 min): `python3 <DETECTORS>/skills/wf-improve/scripts/regression.py <REFERENCE>`
The measures of a step: `<range score> > <OUT>/range-<step>.txt 2>&1` and `<benchmark> > <OUT>/benchmark-<step>.txt 2>&1`.
## Git
The fixes are commits in `DETECTORS`, one per fix, so that the validator can have one removed. Note the **base**: `git -C <DETECTORS> rev-parse HEAD`. Files already modified (`git -C <DETECTORS> status --porcelain`) stay as they are, neither committed nor reverted; list them in the summary and pass them to the upgrader. Always commit by path (`git -C <DETECTORS> commit -m "<message>" -- <paths>`), never `git add -A`, `git stash` or `git checkout` outside your paths: another session may be working there. Each message ends with an empty line then `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Nothing is pushed. The rest (branch, committing `OUT`, another repository) is the caller's.
## Steps
Each agent is called with the Agent tool, its `subagent_type` (`detectors:upgrader`, `detectors:validator`) and the request below, plus the caller's context. A verdict is read by `grep '^\*\*Verdict\*\*:' <output>`.
1. **Measures before**: step `before`.
2. **Upgrader**, a single pass: `Detectors: <DETECTORS>. Station: <STATION>, range <FROM> → <TO> (UTC). Labels: <LABELS>. Families: <FAMILIES>. Context: <CONTEXT | none>. Events command: <…>. Sensors command: <…>. Range score command: <…> (before: <OUT>/range-before.txt). Benchmark command: <…> (before: <OUT>/benchmark-before.txt). Base: <base>. Files already modified: <list | none>. Output folder: <OUT>. Write <OUT>/modification.md.`
It commits each fix it keeps, message `wf-improve: S<n>`, with its benchmark `<OUT>/benchmark-S<n>.txt`.
Verdict `nothing to change`, or no fix commit (`git -C <DETECTORS> log --format=%h --grep ': S[0-9]*$' <base>..HEAD` empty): go to the summary.
3. **Measures after**: step `after`.
4. **Validator**, which reads neither the diagnosis nor the detail of the modification: `Detectors: <DETECTORS>. Families: <FAMILIES>. Modification: <OUT>/modification.md. Range score: <OUT>/range-before.txt → <OUT>/range-after.txt. Benchmark: <OUT>/benchmark-before.txt → <OUT>/benchmark-after.txt, after each fix: <OUT>/benchmark-S*.txt; command: <…>. Events command for any station: python3 <DETECTORS>/scripts/categories.py <station folder> --from <from> --to <to>. Base: <base>. Write <OUT>/validation.md.`
- `keep`: go to the summary.
- `remove`: for each fix of the `**Remove**:` line, its commits (`git -C <DETECTORS> log --format=%h --grep ': S<n>$' <base>..HEAD`), from the most recent: `git -C <DETECTORS> revert --no-edit <commit>`, then `git -C <DETECTORS> commit --amend -m "wf-improve: S<n> removed" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`. A conflict: `git -C <DETECTORS> revert --abort`, and do as for `cancel`.
- `cancel`, or anything else: `git -C <DETECTORS> revert --no-commit <base>..HEAD && git -C <DETECTORS> commit -m "wf-improve: modification cancelled" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"` (only what the fix commits changed; files already modified stay as they are).
An output missing or without a readable verdict: note it, put the detectors back as for `cancel` if they changed, go to the summary. Do not rerun an agent.
## Summary
Measures, step `end`, only after a `remove`; otherwise `end` is `after` (`keep`) or `before` (`cancel`, nothing changed). Then `<OUT>/summary.md`:
```markdown
# wf-improve: <station>, <FROM> → <TO>
**Result**: modification kept | kept without S<n>… | cancelled | none | error
## The range
the range score, before → end, one line per family (found, correct)
## Elsewhere
the `Total` line of each family, before → end; then the other changed lines (station, family), before → end
## What changed
<per fix kept, one sentence from "What changed" of the modification>; the upgrader's verdict, the validator's
## For the user to decide
<copied as they are: "Limits" and "For the user to decide" of the modification, "Reservations" of the validator; none>
## Not fixed
<"Not fixed" of the modification; none>
## Errors
<none, or what was missing; files already modified; the base and `git -C <DETECTORS> log --oneline <base>..HEAD`>
```
## Answer
In 4 lines at most: the `**Result**` line, the range score before → end (changed families), the `Total` before → end if it changed, the path of the summary.
+31 -57
View File
@@ -1,71 +1,45 @@
# Detector thresholds
Every threshold of the detectors (`scripts/*.py`, one per family of the families document, D-119) is **provisional**: tuned on the six data sets of events E-001 to E-005 and NORDEND of October, then adjusted by wf-improve sessions. When bulk data arrives, this is the list to review. A constant added or changed updates this file in the same change. Check: `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
Every threshold of the detectors (`scripts/*.py`) is **provisional**: tuned on the data sets of events E-001 to E-005 and NORDEND of October, then adjusted by wf-improve sessions. A constant added or changed updates this file in the same change; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
The detectors must detect everything (the user, 2026-10-26, D-068): a setting chosen to hide something from the client is not a detection rule; the wf-improve workflow may change it. What is shown to the client is decided in a separate layer.
The detectors detect everything (the user, 2026-10-26, D-068): what is shown to the client is decided in a separate layer.
"wf-improve X, mod. 1, S2": suggestion S2 of modification 1 of the wf-improve run on X. "X, batch 01": first batch of the earlier loop on station X.
"wf-improve X, mod. 1, S2": suggestion S2 of modification 1 of the wf-improve run on X; "X, batch 01": first batch of the earlier loop on station X.
| Detector | Constant | Value | Tuned on |
|---|---|---|---|
| `visibility` | `K_MAD` | 10 | E-001 (fog 11 to 57 MAD), E-005 (10 to 70 MAD) |
| | `FACTOR` | 2 | drops sensors with a near-zero MAD (E-004, E-005); also, alone (without `K_MAD`), something passing in front of one or two targets or a reference, rejected for a single measurement (point 2): wf-improve Obstruction, mod. 1, S3 (WILDSTRUBEL: by day, heat shimmer inflates the MAD; R1 rejected at ×3.6 under 10 MAD) |
| | `RUN_MIN` | 3 | E-001, rain of E-005 |
| | `BRIEF_MIN` | 3 | trains of E-005 (5 to 17 sensors) |
| `visibility` | `K_MAD` | 10 MAD | E-001 (fog, 11 to 57 MAD), E-005 (rain, trains, 10 to 70 MAD) |
| | `FACTOR` | 2× the median | drops the sensors with a near-zero MAD (E-004, E-005); alone, something passing in front of one or two targets (wf-improve Obstruction, mod. 1, S3) |
| | `RUN_MIN` | 3 | E-001, rain of E-005; also the diffuse nuisance |
| | `GAP_STEPS` | 2.5 steps | timestamp jitter, a missing step |
| | `BRIEF_MIN` | 3 sensors | trains of E-005 (5 to 17 sensors) |
| | `SALIENCE` | 3 | trains (5 to 17) against the cobweb of E-002 (2 to 9) |
| | `FRAC` | 0.15 | tails of PR09 and CTR01 (NORDEND October) |
| | `BREAK` | 2 h | NORDEND October, night of the 13th to the 14th |
| | `K_TARGET` | 5 | leaves of E-005 (6 to 25 MAD) |
| | `TARGET_MIN` | 3 | —; 2 when `TARGET_MIN` measurement steps do not fit in `TARGET_GAP`: wf-improve Obstruction, mod. 1, S6 (BREITHORN, one measurement every 4 h: vehicles parked at night, two or three rejections) |
| | `TARGET_GAP` | 6 h | leaves of E-005 (7 rejections in 12 h); at least two measurement steps (same S6) |
| | `GAP_STEPS` | 2.5 steps | timestamp jitter |
| | `REJECT_FACTOR` | 1.2 | FLORIDA, batch 01: a rejection at 10 MAD is high without doubling (targets at level 0.6); drops C24 of E-009 (0.68, rejected at ×1.04) |
| | `HALF` | 0.5 | level "none" (point 5): NORDEND 09/10 23:00 (68 % rejected) against the rain of E-005 (24 to 33 %) |
| | `K_DIFFUSE` | 2.5 MAD | diffuse nuisance (point 8): median, over the sensors, of the error_value in MAD above each one's median, `RUN_MIN` instants in a row, rejected or not: Poor visibility "reduced"; not within heat shimmer; wf-improve Visibility, mod. 1, S2 (JUNGFRAU, FLORIDA, NORDEND: reduced visibility of the whole scene without rejection; at 2, nights labelled thermal at JUNGFRAU and jumps back of WILDSTRUBEL absorbed; at 3, half the gain) |
| | `FEW` | 3 targets | Poor visibility on at most 3 targets, no reference: an object in front of them, Target obscured (boundary "Visibility or Obstruction" of the families document); BREITHORN, batch 01 (vehicles parked in front of 2 or 3 targets out of 12) |
| `visibility` (heat shimmer) | `RATIO` | 1.5 | E-003 (×1.7 to ×3 the night) |
| | `HALF` | 0.5 | level "none": NORDEND 09.10 23:00 (68 % rejected) against the rain of E-005 (24 to 33 %) |
| | `K_DIFFUSE` | 2.5 MAD | diffuse nuisance: wf-improve Visibility, mod. 1, S2 (JUNGFRAU, FLORIDA, NORDEND) |
| `visibility` (heat shimmer) | `RATIO` | 1.5× the calm | E-003 (corr ×1.7 to ×3, error_value ×1.7 to ×3.2 the night) |
| | `SHARE` | 0.5 | — |
| | `RATIO_ERR` | 1.5 | E-003 (×1.7 to ×3.2) |
| | `REJECT_MAX` | 15 % | E-003 ≤ 7 %; cobweb and rain ≥ 18 %; rejections beyond each sensor's habit (its share of rejections over the export): wf-improve Visibility, mod. 1, S1 (WILDSTRUBEL: a reference or targets usually rejected dropped the days of heat shimmer; 4 to 14 % against 9 to 22 % raw) |
| | `STRONG` | 2× | E-003 (×2.3 to ×2.6) |
| | `MIN_DURATION` | 3 h | E-003 (10-11 h) against the morning of E-004 (2 h); the classification drops this morning too, whatever the duration |
| `displacement` (jumps) | `K` | 20 noises | E-001 (at 15, two healthy targets greyed); do not lower for the trains, seen by `visibility` |
| | `K_LASTING` | 7 noises | jump under K that stays (point 8 of `displacement.py`): NIEDERHORN, batch 01 (e09, corr_y −0.19 px at once, 8 noises, stayed 5 days) |
| | `PLATEAU` | 8 ordinary variations | same point: gap between the medians of the day after and the day before, in ordinary variations of the level from one day to the next; NIEDERHORN, batch 01 (e09: 13; E-001, healthy targets: 6 at most) |
| | `K_PEAK` | 6 noises | one-step peak common to several targets: NORDEND, batch 01 (shakes of the whole image, 6 to 11 noises) |
| | `PEAK_MIN` | 3 targets | NORDEND, batch 01 (3 to 5 targets per shake) |
| | `BASE` | 4 measurements | 1 h at 15 min; the return of a jump is judged on an hour of measurements, BASE at most, one at least (wf-improve MICHIGAN, mod. 1: at a 4 h step, a peak back at the next measurement became "Large shift") |
| | `GAP` | 1 h | — |
| `target_issue` | `SHARE` | 0.25 | regularity rather than the share of the day: wf-improve Target issue, mod. 1, S1 (WILDSTRUBEL R3, rejected by day only) |
| | `MIN_HOURS` | 3 h | as hours at the station's cadence: wf-improve NEBRASKA, batch 01 (one measurement every 4 h) |
| | `BRIDGE` | 1 day | wf-improve Obstruction, mod. 1, S1; Target issue, mod. 1, S1 (at 2, parked vehicles of BREITHORN made a week) |
| | `WEEK` | 7 days | Degraded target against Target obscured: wf-improve NEBRASKA, batch 01 (lost references, 25 days; parked vehicles, 2 to 5 days) |
| | `MIN_DAYS` | 2 days | NORDEND: isolated days on 16 sensors |
| `displacement` (jumps) | `K` | 20 noises | E-001 (at 15, two healthy targets greyed); the trains of E-005 are left to `visibility` |
| | `BASE` | 4 measurements | 1 h at 15 min; the return is judged on an hour (wf-improve MICHIGAN, mod. 1) |
| | `BACK_H` | 24 h | E-002 (plateaus up to 11.5 h) |
| | `REPEAT` | 5 jumps back | CLARIDEN, batch 01: PR_BG flips 28 times in 2.5 days, the other targets 3 at most (morning of the movement) |
| | `FLIP` | 36 h | problem target (D-074): PR_BG of CLARIDEN flips 2.5 days; C8 and C13 of E-009, 5 flips in 1 h 20 on the morning of the movement, stay brief jumps; 6 h → 36 h: wf-improve Target issue, mod. 1, S2 (NORDEND PR12, PR14, PR08, PR05, JUNGFRAU BJ_G: 12 to 28 h of flips within a loss of visibility, labelled Visibility or Obstruction, absorbed by it) |
| | `MAX_FLIPPING` | 2 targets | problem target: FLORIDA, the two stuck targets; E-002, cobweb on the references, 15 targets flip at once |
| | `LARGE` | 0.5 | large shift ("Large shift", D-074): E-004 and CLARIDEN, 18 targets on the morning of the movement; also, slow drift of the camera (LARGE of the targets in corr and of the sensors in raw x, y, on the same side: no "Slow drift" over the whole window, only the pieces of `PIECE`): wf-improve MICHIGAN, mod. 1 (BREITHORN kept); wf-improve Displacement, mod. 1, S2 (NEBRASKA reported in pieces, JUNGFRAU still silent) |
| `displacement` (slow drift) | `HOURS` | 20 h out of 24 | as the trend of the Displacements tab (D-055); as a share of the hours the station measures (wf-improve MICHIGAN, mod. 1: 5 out of 6 at one measurement every 4 h, no day counted) |
| | `DAYS` | 5 days | D-055: a trend needs at least 5 UTC days |
| | `SIZE` | 1 | D-055: drift over the period ≥ one daily amplitude; amplitude of the day: 9th and 1st deciles of `corr` |
| | `STEADY` | 0.6 | D-055: Kendall tau of the daily levels; a step from one day to the next of more than half the drift drops it (a jump) |
| | `PIECE` | 3.5 amplitudes | piecewise drift, for a target without a drift over the whole window (point 4b of `displacement.py`): wf-improve Displacement, mod. 1, S1 (rows of CLARIDEN, 3.8 to 8.3 amplitudes; JUNGFRAU, thermal cycles without a Displacement label, 3.1 at most; at 1, NORDEND and JUNGFRAU overflow) |
| | `MAX_HOLE` | 2 days | a drift piece (point 4b) does not go from one kept day to the next more than 2 days apart: wf-improve Displacement, mod. 2, S1 (CLARIDEN, hole from 10-03 to 10-05: the pieces from 09-28 to 11-01 fall; at 1, the missing days of a half-day hole cut the drifts of CLARIDEN and NEBRASKA) |
| `thermal_displacement` | `MIN_HOURS` | 20 h | —; days of the profile |
| | `PARTIAL` | 12 h | an incomplete day (fewer than `MIN_HOURS`) gets its episode if it looks like the profile on the hours it has, without entering the profile; wf-improve Thermal displacement, mod. 1, S4 (SNCF01, NORDEND, JUNGFRAU: labelled cycles of days cut by a hole; `MIN_HOURS` at 12 or 16 made NIEDERHORN go back) |
| `displacement` (slow drift) | `HOURS` | 20 h out of 24 | as the Displacements tab (D-055), as a share of the hours measured (wf-improve MICHIGAN, mod. 1) |
| | `DAYS` | 5 days | D-055 |
| | `SIZE` | 1 amplitude | D-055 |
| | `STEADY` | 0.6 | D-055 (Kendall tau) |
| | `PIECE` | 3.5 amplitudes | wf-improve Displacement, mod. 1, S1 (rows of CLARIDEN 3.8 to 8.3; thermal cycles of JUNGFRAU 3.1 at most) |
| `thermal_displacement` | `MIN_HOURS` | 20 h | — |
| | `REPRO` | 0.6 | pure noise ≈ 0 |
| | `SHARE` | 0.5 | NORDEND October just under (9/19) |
| | `WARM` | 0.3 | —; only says there is a cycle (at least `MIN_WARM_HOURS` warm hours), its bounds are at `EDGE` |
| | `EDGE` | 0.1 | bounds of the wave: start and return of the profile, the return after midnight if needed, at most to the next day's start; wf-improve Thermal displacement, mod. 1, S1 (NORDEND, JUNGFRAU, SNCF01: labelled evenings and nights, the wave stopped at the last warm hour; at 0.15 and 0.2, fewer nights found, FLORIDA overflows in the evening anyway) |
| | `DAY_MATCH` | 0.5 | — |
| | `R2_MIN` | 0.3 | PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48 |
| | `NIGHT` | 8 h | — |
| | `R2_MIN` | 0.3 | 24 h sinusoid: PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48; no longer asked for the camera (raw positions) since WILDSTRUBEL, batch 01 (1 target out of 7, the wrong pointings crush the fit) |
| | `MIN_WARM_HOURS` | 2 h | WILDSTRUBEL, batch 01: a single warm hour is not a cycle (NIEDERHORN, E-005: noon, trains and holes) |
| | `SLIDE_K` | 3 noises | common camera slide without a cycle ("Camera thermal movement"): gap from the median of the day before, raw positions and corrected curves; wf-improve MICHIGAN, mod. 1 (at 2, Displacement instants of BREITHORN taken) |
| | `COLD` | 4 one-hour noises | the camera moves with the cold ("Camera thermal movement", without a cycle): the pose (tilt, rotation or scale) moves away from its median of the day before while the temperature falls under its own, the same hour; noise: variations over an hour, one step at 4 h; wf-improve Thermal displacement, mod. 1, S3 (BREITHORN, CLARIDEN, NEBRASKA: targets in opposite directions, or compensated; at 3, Displacement instants of NEBRASKA taken) |
| `no_data` | `MISSING` | 2 steps | user's rule, 2026-10-23; also applied between the last measurement and the end of the window read (CLARIDEN, batch 01) |
| `target_issue` | `CALM` | 0.25 | (detector added on 2026-10-27, see its docstring) |
| | `SHARE` | 0.5 | —; the target hidden at times ("Target obscured") |
| | `SHARE_WEEK` | 0.25 | share of the calm instants rejected that makes a degraded day of the problem target ("Degraded target", at least `WEEK` days): regularity rather than the share of the day; wf-improve Target issue, mod. 1, S1 (WILDSTRUBEL R3 rejected from 04:00 to 16:00 every day, 30 to 45 %: seen only on days without night measurements; NORDEND PR04 and CTR01, NIEDERHORN PWD_IMP_B08_024_C09, under half) |
| | `MIN_HOURS` | 3 h | hours of calm instants per day, at the station's cadence (median gap between two instants): 12 instants at 15 min; wf-improve NEBRASKA, batch 01 (one measurement every 4 h: no day counted, the lost reference REFCOLH22 was not seen) |
| | `MIN_DAYS` | 2 days | NORDEND: isolated days on 16 sensors |
| | `WEEK` | 7 days | problem target ("Degraded target") if hindered at least a week in a row, otherwise a target hidden at times ("Target obscured"): boundary "Target issue or Obstruction" of the families document; wf-improve NEBRASKA, batch 01 (BREITHORN, parked vehicles, 2 to 5 days; NEBRASKA and MICHIGAN, lost reference, 25 days; JUNGFRAU, PM_G, 63 days) |
| | `BRIDGE` | 1 day | non-degraded days in a row that do not break the run of a problem target ("Degraded target": `WEEK` days from the first to the last degraded day, `SHARE_WEEK` of rejections over the bridged run); wf-improve Obstruction, mod. 1, S1 (FLORIDA, SNCF01, NORDEND, NIEDERHORN: problem targets cut into Target obscured by a day under `SHARE`; BREITHORN, parked vehicles, unchanged); 2 → 1: wf-improve Target issue, mod. 1, S1 (at a quarter, two bridging days joined the parked vehicles of BREITHORN, POUI22, into a week) |
| | `STUCK` | 0.05 | stuck target (D-074): E-011, C2 and C3 at 0.006 and 0.016 of the others' noise; NORDEND, E-001: 0.7 at the lowest |
| `categories` (classification) | `MERGE` | 1 h | rule validated by the user: same variety less than an hour apart, a single event; two Obstruction (Target obscured, Brief obstruction) only if they share a sensor (wf-improve Obstruction, mod. 1, S1 and S3), two Target issue too (wf-improve Target issue, mod. 1, S3); a Brief obstruction only absorbs the jumps back within `MERGE` after its instant (S3) |
| | `WARM` | 0.3 | — |
| | `EDGE` | 0.1 | wf-improve Thermal displacement, mod. 1, S1 (labelled evenings and nights) |
| | `RETURN` | 0.05 | wf-improve NORDEND 2026-11-05, S1 (the cooling tail is slow) |
| | `SLIDE_K` | 3 noises | wf-improve MICHIGAN, mod. 1 (at 2, Displacement instants of BREITHORN taken) |
| `no_data` | `MISSING` | 2 steps | user's rule, 2026-10-23; also up to the end of the window (CLARIDEN, batch 01); the cadence of bursts (wf-improve, batch 2026-11-05, S1) |
| `categories` | `MERGE` | 1 h | rule validated by the user (2026-10-24); an Obstruction or a Target issue only with the same target (wf-improve Obstruction, mod. 1, S1; Target issue, mod. 1, S3) |