Compare commits
23
Commits
0abba2d0ed
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1ca6557af2 | ||
|
|
47bc0acde2 | ||
|
|
428ebaacfe | ||
|
|
a008ea8fcc | ||
|
|
63d954a126 | ||
|
|
9d4cd722f4 | ||
|
|
000e8bd6e4 | ||
|
|
8b5f8a34b9 | ||
|
|
e5e5d2a244 | ||
|
|
ab0347c41d | ||
|
|
4188efe201 | ||
|
|
fe1adac89a | ||
|
|
fb32143722 | ||
|
|
3a22dae7ce | ||
|
|
ec4b900fa0 | ||
|
|
5c2d265a21 | ||
|
|
5f085bf5ff | ||
|
|
5e9a7aa9d1 | ||
|
|
908369b9e9 | ||
|
|
8e65ee3e08 | ||
|
|
c4fca737dc | ||
|
|
2bba21c7e6 | ||
|
|
3aff4d154a |
No files matched your search
@@ -1,20 +1,27 @@
|
|||||||
# detectors
|
# detectors
|
||||||
|
|
||||||
AIMsight anomaly detectors (Python 3, standard library only) and their benchmark (`bench/`); nothing else. They are improved by Optimizer (`improve` skill) with the upgrader and validator of the agents repository. Everything is in English: code, identifiers, docs, outputs. Layout and usage: `README.md`.
|
AIMsight anomaly detectors (Python 3, standard library only) and their benchmark (`bench/`); nothing else. They are improved by Optimizer (`optimize` skill) with the upgrader and validator of the agents repository. Everything is in English: code, identifiers, docs, outputs. Layout and usage: `README.md`.
|
||||||
|
|
||||||
## Contract (do not break)
|
## Contract (do not break)
|
||||||
|
|
||||||
- One detector per family, in `scripts/`: `visibility.py`, `obstruction.py`, `target_issue.py`, `thermal_displacement.py`, `no_data.py`, `displacement.py` (D-119). `categories.py` loads every `scripts/*.py` that has `detect`: never put another script with a `detect` function, or with side effects at import, directly in `scripts/`; tools go in `bench/`.
|
- `scripts/categories.py` runs the detectors of `DETECTORS` (modules of `scripts/`: `visibility`, `target_issue`, `thermal_displacement`, `no_data`, `displacement`) and makes their episodes events. A module returns every family it finds: `visibility` also gives Brief obstruction and `target_issue` Target obscured (Obstruction). `episodes.py` holds the helpers they share.
|
||||||
- `detect(measurements)` returns episodes with the keys `start`, `end`, `kind` (the family), `label` (the variety), `text`, `ignore`, `sensors`. A detector may also declare `temperature`, `camera`, `metrics` (`categories.CONTEXT`) or `end`; `categories.events` passes them by introspection.
|
- `detect(measurements)` (`no_data`: also `end`, the end of the window read) returns episodes with the keys `start`, `end`, `kind` (the family), `label` (the variety), `text` (the sentence shown), `ignore`, `sensors`; a Poor visibility also has `level`.
|
||||||
|
- The measurements are read by `scripts/data.py` of the tools repository, the sibling `~/aimsight-tools` (`categories.py` puts it on the path); a reading fix is made there.
|
||||||
- Label files (JSONL): `id`, `start`, `end`, `family`, `description` (and `review`, `analysis`); the project uses `labeling/auto` only.
|
- Label files (JSONL): `id`, `start`, `end`, `family`, `description` (and `review`, `analysis`); the project uses `labeling/auto` only.
|
||||||
- Two repositories use this one through a `detectors -> ~/detectors` link (one copy on the workstation: a commit here is live for both at once); a rename here needs the matching change there:
|
- Two repositories use this one through a `detectors -> ~/aimsight-detectors` link (one copy on the workstation: a commit here is live for both at once); a rename here needs the matching change there:
|
||||||
- Diagnostic-app: `categories.events`, `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the `--from/--to` CLIs, `bench/score.py` (the command shown in the page);
|
- aimsight-app: `categories.window` (the events of a range, with its history), `categories.CATEGORIES` (`process`: `images`, `none`, `check`), the event fields (`start`, `end`, `category`, `text`, `sensors`, `ignore`, `episodes`), the `--from/--to` CLIs, `bench/score.py` (the command shown in the page);
|
||||||
- Optimizer (`improve` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix.
|
- Optimizer (`optimize` skill and its agents): `scripts/categories.py --from/--to`, `bench/` (`score.py`, `regression.py`, `compare.py`, `sensors.py`); its upgrader changes `scripts/*.py` and `thresholds.md`, one commit per fix.
|
||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
|
|
||||||
- A constant added or changed updates `thresholds.md` in the same change; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
|
- A change to improve the detectors touches `scripts/*.py` and `thresholds.md` only; never `bench/`, which is the measure. Anything is possible there: add, remove or change a constant, a rule or a whole detector, as long as the seven families remain.
|
||||||
|
- **Simple first**: a rule earns its place by its score on the benchmark, not by one case; prefer removing or widening a rule to adding a special case. The benchmark ends with the complexity of the detectors (code lines and constants of `scripts/`), `bench/compare.py` shows it before → after. A fix that adds a constant or more than 20 code lines earns them with at least 1 point on the `Total` of a family ("found" or "correct"); otherwise it fits one range: simplify it or drop it. A change that removes code without a regression is a gain in itself.
|
||||||
|
- **Pruning**: after a fix, each rule of the detector it touched (a condition, a branch, a constant a plainer value could replace) is tried without it, on a copy in the working folder (the copy's `bench/regression.py`, then `bench/compare.py`); a rule whose removal makes no regression is removed, one commit per rule, with its line of `thresholds.md`.
|
||||||
|
- **Everything is detected**: what is shown or not to a client is not the detectors' business. A setting chosen not to show something may be changed, saying so.
|
||||||
|
- A constant added or changed updates its line of `thresholds.md` ("Tuned on": the run that changed it) and the detector's docstring in the same change; a constant or rule removed removes its line; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
|
||||||
|
- A detector that fails is skipped: a `skipped` line in the output of `categories.py` or of the benchmark means the change is broken.
|
||||||
- No rule specific to a station, a sensor or a date.
|
- No rule specific to a station, a sensor or a date.
|
||||||
- A change to a detector is checked with the benchmark: `python3 bench/regression.py --labels <labels folder> --data <data folder>` (Diagnostic-app and Optimizer: `labeling/auto`, `data`, links to the shared `~/labeling` and `~/aimsight-data/data`; about 3 min). A refactoring must leave every figure unchanged.
|
- A change to a detector is checked with the benchmark: `python3 bench/regression.py --labels <labels folder> --data <data folder>` (aimsight-app and Optimizer: `labeling/auto`, `data`, links to the shared `~/aimsight-labeling` and `~/aimsight-data/data`; about 1 min). A refactoring must leave every figure unchanged.
|
||||||
- History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in Diagnostic-app/Support-Agent; keep them when editing a rule.
|
- **The benchmark's scores**, per family of each station: "found", the share of the labelled instants the detectors flag; "correct", the share of the flagged instants a label puts in the family. **A regression**, computed by `python3 bench/compare.py <before.txt> <after.txt>` (its constants): a family `Total` whose "found" or "correct" loses more than 1 point; a family of a station whose "found" or "correct" loses more than 5 points and more than 20 instants (labelled instants no longer flagged, or wrongly flagged instants added), even if the `Total` rises; below, it is the noise of a few instants (a station at one measurement every 4 h, a family with few labels). A rise is a plus, never a requirement. **Not a regression**: a drop that comes from another consistent reading of the detectors that the labels do not retain: labels contradicting each other from one station to another (the same state in two families), the gaps of a few measurements between two labels of the same phenomenon, the bounds of a label, an instant the score (per instant, without the sensors) counted as found by an event on another sensor. A drop counted as not a regression is said, with the labels (id). Labels have their ambiguities: the goal is detectors that do their best, not perfect, and not overtrained on the labels.
|
||||||
- The bench tools take their paths as arguments; the architecture of the repositories: `~/Diagnostic-App/docs/architecture.drawio`.
|
- History references in docstrings and `thresholds.md` ("wf-improve X, mod. 1, S2", "X, batch 01", D-xxx, E-xxx) point to past runs and to decisions recorded in aimsight-app/Support-Agent; keep them short when editing a rule.
|
||||||
|
- The bench tools take their paths as arguments; the architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`.
|
||||||
@@ -4,22 +4,20 @@ The AIMsight anomaly detectors and their benchmark. Python 3, standard library o
|
|||||||
|
|
||||||
## The detectors
|
## The detectors
|
||||||
|
|
||||||
One detector per anomaly family of the displacements, in `scripts/`:
|
Five modules in `scripts/`, one per kind of reading; each gives episodes of one or more of the seven families:
|
||||||
|
|
||||||
| Family | Detector |
|
| Module | Reads | Families (varieties) |
|
||||||
|---|---|
|
|---|---|---|
|
||||||
| Visibility | `visibility.py` |
|
| `visibility.py` | error_value of each sensor; fast noise of the curves | Visibility (Poor visibility "none" or "reduced", Heat shimmer), Obstruction (Brief obstruction) |
|
||||||
| Obstruction | `obstruction.py` |
|
| `target_issue.py` | rejections day after day | Target issue (Degraded target), Obstruction (Target obscured) |
|
||||||
| Target issue | `target_issue.py` |
|
| `thermal_displacement.py` | the daily wave of the curves | Thermal displacement (Thermal expansion / contraction, Camera thermal expansion / contraction, Camera thermal movement) |
|
||||||
| Thermal displacement | `thermal_displacement.py` |
|
| `no_data.py` | the cadence of the measurements | No data |
|
||||||
| No data | `no_data.py` |
|
| `displacement.py` | jumps and slow drift of the corrected curves | Displacement (Lasting shift; Jump, not back yet; Slow drift), Perturbation (Sudden jump, back to normal) |
|
||||||
| Displacement | `displacement.py` |
|
|
||||||
|
|
||||||
`categories.py` runs them all and classifies their episodes into events, one line per event; `data.py` reads a station folder (`displacement/`, `metrics/`, `camera/`).
|
`episodes.py` holds their shared helpers. `categories.py` runs them all and classifies their episodes into events (a jump back joins the visibility loss or the obstruction that explains it; the same variety less than an hour apart, one event), one line per event. The measurements are read by `scripts/data.py` of the tools repository, the sibling `~/aimsight-tools`.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 scripts/categories.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD # --chunk N, --history N
|
python3 scripts/categories.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD # --chunk N, --history N
|
||||||
python3 scripts/visibility.py <station folder> # one detector alone
|
|
||||||
```
|
```
|
||||||
|
|
||||||
`thresholds.md` is the registry of the detectors' thresholds and what each was tuned on.
|
`thresholds.md` is the registry of the detectors' thresholds and what each was tuned on.
|
||||||
@@ -31,6 +29,7 @@ The detectors against labels (JSONL, one per line, `{"id", "start", "end", "fami
|
|||||||
```bash
|
```bash
|
||||||
python3 bench/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD # the score of a range
|
python3 bench/score.py <station folder> --labels <labels.jsonl> --from YYYY-MM-DD --to YYYY-MM-DD # the score of a range
|
||||||
python3 bench/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> # the benchmark; --from/--to: one range for all
|
python3 bench/regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> # the benchmark; --from/--to: one range for all
|
||||||
|
python3 bench/compare.py <before.txt> <after.txt> # two benchmarks: the regressions, the complexity
|
||||||
python3 bench/sensors.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD # the sensors of each event
|
python3 bench/sensors.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD # the sensors of each event
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -38,6 +37,6 @@ A change to a detector is checked with the benchmark; a refactoring leaves every
|
|||||||
|
|
||||||
## Improving them
|
## Improving them
|
||||||
|
|
||||||
The detectors are improved by Optimizer (its `improve` skill), with the `upgrader` and `validator` agents of the agents repository: on the range of an analysis, its labels as the truth, the upgrader fixes the detectors (one commit per fix here) and the validator keeps, removes or cancels the fixes against overfitting, judged by this benchmark. Nothing is pushed.
|
The detectors are improved by Optimizer (its `optimize` skill, from one analysis), with the `upgrader` and `validator` agents of the agents repository: on the range of an analysis, its labels as the truth, the upgrader fixes the detectors (one commit per fix here) and the validator keeps, removes or cancels the fixes against overfitting, judged by this benchmark. Nothing is pushed.
|
||||||
|
|
||||||
Copied from Support-Agent on 2026-11-13 (commit 48a3092); the detectors now evolve here. Used by Diagnostic-app and Optimizer, through a `detectors -> ~/detectors` link (one copy on the workstation); their labels come from the `labeling` repository, their data from `aimsight-data`. The architecture of the repositories: `~/Diagnostic-App/docs/architecture.drawio`.
|
Copied from Support-Agent on 2026-11-13 (commit 48a3092), simplified on 2026-11-16 (branch `simplify`); the detectors now evolve here. Used by aimsight-app and Optimizer, through a `detectors -> ~/aimsight-detectors` link (one copy on the workstation); their labels come from the `labeling` repository, their data from `aimsight-data`. The architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`.
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
"""Two benchmark outputs (regression.py), before and after: what changed, and the regressions by the definition of
|
||||||
|
CLAUDE.md, computed.
|
||||||
|
- A family Total: "found" or "correct" loses more than TOTAL_POINTS.
|
||||||
|
- A family of a station: "found" or "correct" loses more than POINTS and more than INSTANTS instants ("found": labelled
|
||||||
|
instants no longer flagged; "correct": wrongly flagged instants added). Below, the loss is noise of a few instants.
|
||||||
|
Then the complexity before → after. The verdict is the reader's: a drop the labels explain (CLAUDE.md, "Not a
|
||||||
|
regression") is said, not hidden.
|
||||||
|
Usage: python3 compare.py <before.txt> <after.txt>
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
TOTAL_POINTS = 1
|
||||||
|
POINTS = 5
|
||||||
|
INSTANTS = 20
|
||||||
|
CELL = r"(\d+)/(\d+) \(\d+ %\)|—"
|
||||||
|
|
||||||
|
|
||||||
|
def parse(path):
|
||||||
|
"""Benchmark output -> ({(station, family): (both, labelled, flagged)}, complexity line or '')."""
|
||||||
|
out, station, cx = {}, None, ""
|
||||||
|
for line in open(path, encoding="utf-8"):
|
||||||
|
if line.startswith("Complexity:"):
|
||||||
|
cx = line.split(":", 1)[1].strip()
|
||||||
|
elif line.strip() and not line.startswith(" "):
|
||||||
|
station = line.split(":")[0].strip()
|
||||||
|
elif m := re.match(rf" (\S.*?)\s{{2,}}({CELL})\s+({CELL})\s*$", line):
|
||||||
|
g = m.groups() # family, found (both, labelled), correct (both, flagged)
|
||||||
|
out[(station, g[0])] = (int(g[2] or g[5] or 0), int(g[3] or 0), int(g[6] or 0))
|
||||||
|
return out, cx
|
||||||
|
|
||||||
|
|
||||||
|
def pct(a, b):
|
||||||
|
return 100 * a / b if b else None
|
||||||
|
|
||||||
|
|
||||||
|
def changes(before, after):
|
||||||
|
"""-> [(station, family, measure, before %, after %, lost instants, regression)] for every measure that moved."""
|
||||||
|
rows = []
|
||||||
|
for key in sorted(set(before) | set(after)):
|
||||||
|
(b0, l0, f0), (b1, l1, f1) = before.get(key, (0, 0, 0)), after.get(key, (0, 0, 0))
|
||||||
|
for measure, p0, p1, lost in (("found", pct(b0, l0), pct(b1, l1), b0 - b1),
|
||||||
|
("correct", pct(b0, f0), pct(b1, f1), (f1 - b1) - (f0 - b0))):
|
||||||
|
if p0 is None or p1 is None or round(p0) == round(p1):
|
||||||
|
continue
|
||||||
|
drop = p0 - p1
|
||||||
|
bad = drop > TOTAL_POINTS if key[0] == "Total" else drop > POINTS and lost > INSTANTS
|
||||||
|
rows.append((*key, measure, p0, p1, lost, bad))
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
if len(sys.argv) != 3:
|
||||||
|
sys.exit(__doc__)
|
||||||
|
(before, cx0), (after, cx1) = parse(sys.argv[1]), parse(sys.argv[2])
|
||||||
|
rows = changes(before, after)
|
||||||
|
for station, family, measure, p0, p1, lost, bad in rows:
|
||||||
|
print(f"{'REGRESSION' if bad else '':<11}{station:<13}{family:<22}{measure:<8}{p0:4.0f} → {p1:3.0f} %"
|
||||||
|
+ (f" ({lost} instants lost)" if p1 < p0 else ""))
|
||||||
|
n = sum(r[-1] for r in rows)
|
||||||
|
print(f"\n{n} regression(s); complexity: {cx0 or '?'} → {cx1 or '?'}")
|
||||||
+11
-2
@@ -3,12 +3,13 @@
|
|||||||
For each <STATION>.jsonl of --labels that has its folder <STATION>/ in --data, the score per family and per instant of
|
For each <STATION>.jsonl of --labels that has its folder <STATION>/ in --data, the score per family and per instant of
|
||||||
score.py, over the range of its labels: from the day of the first to the midnight after the last; with --from/--to, over
|
score.py, over the range of its labels: from the day of the first to the midnight after the last; with --from/--to, over
|
||||||
that range for every station, a station without a label on it included (several stations labelled the same days). Then the total,
|
that range for every station, a station without a label on it included (several stations labelled the same days). Then the total,
|
||||||
family by family, which says whether the detectors do better overall. The block of a station is its score; the
|
family by family, which says whether the detectors do better overall, and the complexity of the detectors (code lines
|
||||||
others, the non-regression: a family that goes back on a station is a regression, even if the total rises.
|
and constants of scripts/). Two outputs are compared by compare.py (the regressions, CLAUDE.md).
|
||||||
Usage: python3 regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> [--stations A,B]
|
Usage: python3 regression.py --labels <folder of <STATION>.jsonl> --data <folder of <STATION>/> [--stations A,B]
|
||||||
[--from YYYY-MM-DD --to YYYY-MM-DD]
|
[--from YYYY-MM-DD --to YYYY-MM-DD]
|
||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
|
import re
|
||||||
from datetime import timedelta
|
from datetime import timedelta
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -17,6 +18,13 @@ import score # next to it
|
|||||||
from data import iso, parse_time # on the path through score
|
from data import iso, parse_time # on the path through score
|
||||||
|
|
||||||
|
|
||||||
|
def complexity(scripts=Path(__file__).resolve().parents[1] / "scripts"):
|
||||||
|
"""-> (code lines, constants) of the detectors: non-blank lines that are not a comment; NAME = lines at module level."""
|
||||||
|
lines = [x for f in sorted(scripts.glob("*.py")) for x in f.read_text().splitlines()]
|
||||||
|
code = [x for x in lines if x.strip() and not x.strip().startswith("#")]
|
||||||
|
return len(code), sum(bool(re.match(r"[A-Z][A-Z_0-9]* = ", x)) for x in lines)
|
||||||
|
|
||||||
|
|
||||||
def label_range(path):
|
def label_range(path):
|
||||||
"""Labels JSONL -> (start, end) iso, in whole UTC days, end excluded."""
|
"""Labels JSONL -> (start, end) iso, in whole UTC days, end excluded."""
|
||||||
labels = score.read_labels(path)
|
labels = score.read_labels(path)
|
||||||
@@ -53,3 +61,4 @@ if __name__ == "__main__":
|
|||||||
for k, v in n.items():
|
for k, v in n.items():
|
||||||
total[k] = [x + y for x, y in zip(total.get(k, [0, 0, 0]), v)]
|
total[k] = [x + y for x, y in zip(total.get(k, [0, 0, 0]), v)]
|
||||||
score.show("Total", total)
|
score.show("Total", total)
|
||||||
|
print("\nComplexity: %d code lines, %d constants" % complexity())
|
||||||
+1
-1
@@ -19,7 +19,7 @@ from pathlib import Path
|
|||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts")) # the detectors
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts")) # the detectors
|
||||||
import categories # noqa: E402
|
import categories # noqa: E402
|
||||||
from data import iso, load, parse_time # noqa: E402
|
from data import iso, load, parse_time # noqa: E402 (the tools repository's reader, on the path through categories)
|
||||||
|
|
||||||
HISTORY = timedelta(days=3) # as the batch command of categories.py
|
HISTORY = timedelta(days=3) # as the batch command of categories.py
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -11,7 +11,7 @@ from pathlib import Path
|
|||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
|
||||||
import categories # noqa: E402
|
import categories # noqa: E402
|
||||||
from data import iso, load # noqa: E402
|
from data import iso, load # noqa: E402 (the tools repository's reader, on the path through categories)
|
||||||
|
|
||||||
p = argparse.ArgumentParser(usage=__doc__)
|
p = argparse.ArgumentParser(usage=__doc__)
|
||||||
p.add_argument("folder")
|
p.add_argument("folder")
|
||||||
|
|||||||
+75
-171
@@ -1,58 +1,36 @@
|
|||||||
"""From the detectors' episodes to events, one per family.
|
"""From the detectors' episodes to events, one family each.
|
||||||
|
|
||||||
One detector per label family (the families document): visibility.py, obstruction.py, target_issue.py,
|
Each module of DETECTORS (next to this file) returns episodes: start, end, kind (the family), label (the variety), text,
|
||||||
thermal_displacement.py, no_data.py, displacement.py. The category of an event is its family, the kind field of its
|
ignore, sensors (a Poor visibility also has its level). classify() makes them events:
|
||||||
episodes; their label gives the variety (Poor visibility, Heat shimmer, Brief obstruction, Slow drift…). Rules:
|
- a target already in an overlapping Target issue episode is dropped from an Obstruction episode: its curve is already
|
||||||
- a jump back to normal (JUMP_BACK) that overlaps an episode of HOSTS (loss of visibility, something passing, heat
|
to be ignored;
|
||||||
shimmer, and last the thermal effect, if it touches several targets) is absorbed into it: it takes its cause and
|
- a jump back to normal (JUMP_BACK) overlapping a Visibility episode joins its event: rain, fog or heat shimmer
|
||||||
family; otherwise it stays Displacement, cause not established, to be checked on the images (the families document, "A jump then
|
explains it; otherwise it is a Perturbation: nothing to do;
|
||||||
the return"); a Brief obstruction (one instant) only absorbs the jumps starting at its instant, not those of several
|
- a jump back whose targets are all in an overlapping Obstruction or Target issue episode joins it (something in front
|
||||||
days it crosses (wf-improve session Displacement, mod. 1, S5);
|
of them explains it, or the curve of a problem target is to be ignored); a slow drift, in a Target issue episode;
|
||||||
- a slow drift or a jump back whose targets are all in an overlapping Target issue episode is absorbed into it (the
|
- episodes of the same variety less than MERGE apart make one event (validated rule, 2026-10-24); an Obstruction or a
|
||||||
curve of a problem target is to be ignored, the families document);
|
Target issue only with an episode of the same target (wf-improve Obstruction, mod. 1, S1; Target issue, mod. 1, S3).
|
||||||
- a shift that lasts, or too recent to know (SHIFTS), is never absorbed;
|
The text of an event is that of its first episode, plus a sentence for the jumps it absorbed.
|
||||||
- heat shimmer overlapping a shift of several targets is not kept: a real movement also shakes the curves; the shift of
|
Usage: python3 scripts/categories.py <station folder> [--from YYYY-MM-DD] [--to YYYY-MM-DD] [--chunk N] [--history N]
|
||||||
a single target does not remove the heat shimmer of the whole scene, and heat shimmer only absorbs the jumps back
|
one line per event; --chunk: each chunk of N days classified alone, with the days of history before it; only the
|
||||||
starting during it, not the one of the day before coming back within its day (wf-improve session Visibility, mod. 1,
|
events touching the chunk are shown, "nothing" for a quiet chunk.
|
||||||
S1: WILDSTRUBEL);
|
|
||||||
- two events of the same variety less than MERGE apart make one; two Obstruction (Target obscured, Brief obstruction)
|
|
||||||
only if they share a sensor (wf-improve session Obstruction, mod. 1: S1, different targets chained into an event of
|
|
||||||
several days; S3, passings of a few minutes into a whole hour); two Target issue too: each problem target keeps its
|
|
||||||
bounds (wf-improve session Target issue, mod. 1, S3); a Brief obstruction only absorbs the jumps back within MERGE after
|
|
||||||
its instant (S3: not the 30 h jump of another target).
|
|
||||||
Each event gets its sentence, in English (the site): what happened and what to do about it, as the "For the client"
|
|
||||||
column of the families document says; the cause is only given if the variety establishes it, otherwise the images show it.
|
|
||||||
No detect function: the loader does not take this file for a detector.
|
|
||||||
Usage: python3 scripts/categories.py <station folder> (one line per event: family, variety, sensors)
|
|
||||||
... <folder> [--from YYYY-MM-DD] [--to YYYY-MM-DD] [--chunk 3] [--history N] (batch run: each chunk of N days
|
|
||||||
classified alone, with the days of history before it if they exist; only the events touching the chunk are
|
|
||||||
shown; "nothing" for a quiet chunk)
|
|
||||||
"""
|
"""
|
||||||
import functools
|
import importlib
|
||||||
import importlib.util
|
|
||||||
import inspect
|
|
||||||
import os
|
|
||||||
import sys
|
import sys
|
||||||
from datetime import datetime, timedelta, timezone
|
from datetime import datetime, timedelta, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
|
# The detectors next to this file, and the single reader of the measurements: data.py of the tools repository, the
|
||||||
from data import iso, load, load_camera, load_metrics, load_temperature # noqa: E402
|
# sibling of this one (one copy of each repository in the home directory), else the tools/ link of the caller's root
|
||||||
|
# (a copy of this repository elsewhere, e.g. extracted by git archive).
|
||||||
|
HERE = Path(__file__).resolve().parent
|
||||||
|
sys.path[:0] = [str(HERE), str(HERE.parents[1] / "aimsight-tools" / "scripts"), str(Path.cwd() / "tools" / "scripts")]
|
||||||
|
from episodes import targets # noqa: E402
|
||||||
|
from data import iso, load # noqa: E402
|
||||||
|
|
||||||
# What a detector can read from the station besides the measurements, if it declares it in detect (D-124, D-134):
|
DETECTORS = ("displacement", "no_data", "target_issue", "thermal_displacement", "visibility")
|
||||||
# temperature [(t, °C)], hourly median of the components; camera [pose] (pan, tilt, rotation, scale, sigma0);
|
# The families, in the order of the timeline rows; ignore: greyed zone (None: as its episodes say); process: Images
|
||||||
# metrics {name: [(t, value)]}, all numeric metrics (sensor_CPU_deg_c, sensor_GPU_deg_c, BusVoltage…).
|
# button, nothing, or Images and a link to Diagnosis.
|
||||||
CONTEXT = {"temperature": load_temperature, "camera": load_camera, "metrics": load_metrics}
|
|
||||||
|
|
||||||
|
|
||||||
@functools.lru_cache(maxsize=None)
|
|
||||||
def _context(folder, key):
|
|
||||||
return CONTEXT[key](folder)
|
|
||||||
|
|
||||||
|
|
||||||
# The families, in the order of the timeline rows; ignore: greyed zone (None: as its episodes
|
|
||||||
# say: a total loss of visibility, strong heat shimmer); process: Images button, nothing, or Images and a link to
|
|
||||||
# Diagnosis.
|
|
||||||
CATEGORIES = {
|
CATEGORIES = {
|
||||||
"Visibility": {"ignore": None, "process": "images"},
|
"Visibility": {"ignore": None, "process": "images"},
|
||||||
"Obstruction": {"ignore": True, "process": "images"},
|
"Obstruction": {"ignore": True, "process": "images"},
|
||||||
@@ -60,154 +38,82 @@ CATEGORIES = {
|
|||||||
"Thermal displacement": {"ignore": False, "process": "none"},
|
"Thermal displacement": {"ignore": False, "process": "none"},
|
||||||
"No data": {"ignore": False, "process": "none"},
|
"No data": {"ignore": False, "process": "none"},
|
||||||
"Displacement": {"ignore": False, "process": "check"},
|
"Displacement": {"ignore": False, "process": "check"},
|
||||||
|
"Perturbation": {"ignore": True, "process": "none"},
|
||||||
}
|
}
|
||||||
JUMP_BACK = "Sudden jump, back to normal" # a jump back: its cause, if it overlaps a host; otherwise Displacement
|
JUMP_BACK = "Sudden jump, back to normal" # the label of displacement.py
|
||||||
SHIFTS = {"Lasting shift", "Jump, not back yet", "Large shift"} # to be checked: never absorbed
|
MERGE = timedelta(hours=1)
|
||||||
HOSTS = {"Poor visibility", "Brief obstruction", "Heat shimmer", "Thermal expansion", "Thermal contraction",
|
|
||||||
"Camera thermal expansion", "Camera thermal contraction", "Camera thermal movement"} # give the cause of an
|
|
||||||
# overlapping jump back
|
|
||||||
MERGE = timedelta(hours=1) # same variety less than an hour apart: a single event (validated rule, 2026-10-24)
|
|
||||||
|
|
||||||
|
|
||||||
def detectors():
|
|
||||||
"""Modules of the *.py next to this file that have detect; a file that does not load is skipped with a message."""
|
|
||||||
mods = []
|
|
||||||
for path in sorted(Path(__file__).resolve().parent.glob("*.py")):
|
|
||||||
if path == Path(__file__).resolve():
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
spec = importlib.util.spec_from_file_location(f"detector_{path.stem}", path)
|
|
||||||
mod = importlib.util.module_from_spec(spec)
|
|
||||||
spec.loader.exec_module(mod)
|
|
||||||
except Exception as e: # noqa: BLE001 (a broken detector does not block the page)
|
|
||||||
print(f"Detectors: {path.name} skipped ({type(e).__name__}: {e})", file=sys.stderr)
|
|
||||||
continue
|
|
||||||
if hasattr(mod, "detect"):
|
|
||||||
mods.append(mod)
|
|
||||||
return mods
|
|
||||||
|
|
||||||
|
|
||||||
def variety(ep):
|
def variety(ep):
|
||||||
"""The variety of an episode within its family: shifts together, the thermal effect without "Camera", the loss of
|
return ep["label"], ep.get("level")
|
||||||
visibility by level."""
|
|
||||||
label = "shift" if ep["label"] in SHIFTS else ep["label"].removeprefix("Camera ").capitalize()
|
|
||||||
return ep["kind"], label, ep.get("level")
|
|
||||||
|
|
||||||
|
|
||||||
def is_thermal(ep):
|
|
||||||
return ep["kind"] == "Thermal displacement"
|
|
||||||
|
|
||||||
|
|
||||||
def overlap(a, b):
|
def overlap(a, b):
|
||||||
return a["start"] <= b["end"] and a["end"] >= b["start"] # data.iso times: text order is time order
|
return a["start"] <= b["end"] and a["end"] >= b["start"] # data.iso times: text order is time order
|
||||||
|
|
||||||
|
|
||||||
def targets(n):
|
def host(e, eps):
|
||||||
return "one target" if n == 1 else f"{n} targets"
|
"""The episode whose event a jump back or a slow drift joins, or None."""
|
||||||
|
kinds = {JUMP_BACK: ("Obstruction", "Target issue"), "Slow drift": ("Target issue",)}.get(e["label"], ())
|
||||||
|
return next((h for h in eps if overlap(e, h) and (h["kind"] == "Visibility" and e["label"] == JUMP_BACK
|
||||||
|
or h["kind"] in kinds and set(e["sensors"]) <= set(h["sensors"]))), None)
|
||||||
|
|
||||||
|
|
||||||
def sentence(group, sensors, references):
|
def event(group):
|
||||||
"""Sentence of an event (validated by the user on 2026-10-24), from the variety of its main episode; an absorbed
|
"""Episodes of an event, the main one first -> event."""
|
||||||
jump adds that it has the same cause."""
|
cat, c = group[0]["kind"], CATEGORIES[group[0]["kind"]]
|
||||||
n, one, ep = len(sensors), len(sensors) == 1, group[0]
|
main = [e for e in group if variety(e) == variety(group[0])] # the duration is theirs, not the absorbed jumps'
|
||||||
jumps = [e for e in group[1:] if e["label"] == JUMP_BACK]
|
jumped = len({s for e in group if e["label"] == JUMP_BACK and e not in main for s in e["sensors"]})
|
||||||
jumped = len({s for e in jumps for s in e["sensors"]})
|
text = group[0]["text"] + (f" The jumps of {targets(jumped)} in this period have the same cause." if jumped else "")
|
||||||
lasting = len({s for e in group if e["label"] in SHIFTS for s in e.get("lasting", [])})
|
return {"start": min(e["start"] for e in main), "end": max(e["end"] for e in main), "category": cat, "label": cat,
|
||||||
refs = ", including the reference targets" if references & set(sensors) else ""
|
"text": text, "ignore": c["ignore"] if c["ignore"] is not None else any(e["ignore"] for e in main),
|
||||||
camera = ep["label"].startswith("Camera")
|
"sensors": [] if any(not e["sensors"] for e in group) else sorted({s for e in group for s in e["sensors"]}),
|
||||||
large = any(e["label"] == "Large shift" for e in group)
|
"process": c["process"], "episodes": group}
|
||||||
sort = "shift" if ep["label"] in SHIFTS else f"{ep['label']} {ep['level']}" if ep.get("level") else ep["label"]
|
|
||||||
text = {
|
|
||||||
"Poor visibility none": f"The camera could not see {targets(n)} clearly{refs}. "
|
|
||||||
"The curves in this period are not reliable: ignore them.",
|
|
||||||
"Poor visibility reduced": f"The camera saw {targets(n)} less clearly than usual{refs}. "
|
|
||||||
"The data in this period are less precise: use them with care.",
|
|
||||||
"Brief obstruction": f"Something passed in front of {targets(n)} for a single measurement. Ignore this point on the curves.",
|
|
||||||
"Target obscured": f"{', '.join(sensors)} {'was' if one else 'were'} hidden several times while the targets around "
|
|
||||||
f"{'it' if one else 'them'} stayed clear. Ignore {'its curve' if one else 'their curves'} in this period.",
|
|
||||||
"Heat shimmer": "Warm air made the readings shake. The small fast wiggles are not movement.",
|
|
||||||
"expansion": ("The camera support expands in the day heat; the curves are corrected for it."
|
|
||||||
if camera else "The structure expands in the day heat.") + " This daily wave is normal: nothing to do.",
|
|
||||||
"contraction": ("The camera support contracts as it cools down; the curves are corrected for it."
|
|
||||||
if camera else "The structure contracts as it cools down.") + " This daily wave is normal: nothing to do.",
|
|
||||||
JUMP_BACK: f"{targets(n).capitalize()} jumped and came back. The cause is not established: look at the images.",
|
|
||||||
"shift": (f"{targets(n).capitalize()}, most of the targets, shifted together: everything moved." if large
|
|
||||||
else f"{targets(n).capitalize()} shifted and {'has' if one else 'have'} not come back yet." if not lasting
|
|
||||||
else f"{targets(n).capitalize()} shifted and stayed shifted." if lasting == n
|
|
||||||
else f"{targets(n).capitalize()} shifted; {lasting} of them stayed shifted.")
|
|
||||||
+ " Check whether the structure moved: look at the images.",
|
|
||||||
}.get(sort.split()[-1] if is_thermal(ep) else sort, ep["text"]) # No data, Target issue, Slow drift: the detector's text
|
|
||||||
if ep["kind"] == "Target issue":
|
|
||||||
text += f" Ignore {'its curve' if one else 'their curves'} in this period and look at the images of the target."
|
|
||||||
if not jumped:
|
|
||||||
return text
|
|
||||||
return text + (f" The jump of {targets(jumped)} in this period has the same cause." if len(jumps) == 1
|
|
||||||
else f" The jumps of {targets(jumped)} in this period have the same cause.")
|
|
||||||
|
|
||||||
|
|
||||||
def event(group, references):
|
def classify(episodes):
|
||||||
"""Episodes of the same event, the main one first -> event."""
|
"""Episodes of all detectors -> events sorted by start (rules: module docstring)."""
|
||||||
cat, var = group[0]["kind"], variety(group[0])
|
|
||||||
c = CATEGORIES.get(cat, {"ignore": None, "process": "none"})
|
|
||||||
sensors = [] if any(not e["sensors"] for e in group) else list(dict.fromkeys(s for e in group for s in e["sensors"]))
|
|
||||||
main = [e for e in group if variety(e) == var] # the duration is the main episodes', not the absorbed jumps'
|
|
||||||
return {"start": min(e["start"] for e in main), "end": max(e["end"] for e in main), "category": cat,
|
|
||||||
"label": cat, "text": sentence(group, sensors, references),
|
|
||||||
"ignore": c["ignore"] if c["ignore"] is not None else any(e["ignore"] for e in main),
|
|
||||||
"sensors": sensors, "process": c["process"], "episodes": group}
|
|
||||||
|
|
||||||
|
|
||||||
def classify(episodes, references=frozenset()):
|
|
||||||
"""Episodes of all detectors -> events sorted by start (rules: module docstring); references: the global_ref
|
|
||||||
sensors, for the sentence."""
|
|
||||||
eps = sorted(episodes, key=lambda e: e["start"])
|
eps = sorted(episodes, key=lambda e: e["start"])
|
||||||
# a real movement is never taken for heat shimmer; the shift of a single target is not a movement of the scene
|
issues = [e for e in eps if e["kind"] == "Target issue"]
|
||||||
checks = [e for e in eps if e["label"] in SHIFTS and len(e["sensors"]) > 1]
|
|
||||||
eps = [e for e in eps if e["label"] != "Heat shimmer" or not any(overlap(e, c) for c in checks)]
|
|
||||||
# the thermal wave absorbs last (a jump of something passing or of rain goes to them), and only the jumps of several
|
|
||||||
# targets: the wave moves them together, a single target that jumps is not the wave (C2 of E-003)
|
|
||||||
hosts = sorted((e for e in eps if e["label"] in HOSTS), key=is_thermal)
|
|
||||||
host = {id(e): next((h for h in hosts if overlap(e, h) and (len(e["sensors"]) > 1 or not is_thermal(h))
|
|
||||||
and (h["label"] != "Heat shimmer" or h["start"] <= e["start"])
|
|
||||||
and (h["label"] != "Brief obstruction" or e["start"] == h["start"]
|
|
||||||
and datetime.fromisoformat(e["end"]) - datetime.fromisoformat(h["start"]) <= MERGE)), None)
|
|
||||||
for e in eps if e["label"] == JUMP_BACK}
|
|
||||||
# a drift or a jump back whose targets are all in an overlapping Target issue episode: its curve is to be ignored,
|
|
||||||
# it takes its family (wf-improve session Displacement, mod. 1, S6)
|
|
||||||
host.update({id(e): h for e in eps if e["label"] in (JUMP_BACK, "Slow drift") and not host.get(id(e))
|
|
||||||
and (h := next((h for h in eps if h["kind"] == "Target issue" and overlap(e, h)
|
|
||||||
and set(e["sensors"]) <= set(h["sensors"])), None))})
|
|
||||||
groups = {id(e): [e] for e in eps if not host.get(id(e))}
|
|
||||||
for e in eps:
|
for e in eps:
|
||||||
if host.get(id(e)):
|
if e["kind"] == "Obstruction":
|
||||||
groups[id(host[id(e)])].append(e)
|
e["sensors"] = [s for s in e["sensors"] if not any(s in h["sensors"] and overlap(e, h) for h in issues)]
|
||||||
|
eps = [e for e in eps if e["kind"] != "Obstruction" or e["sensors"]]
|
||||||
|
hosts = {id(e): h for e in eps if (h := host(e, eps))}
|
||||||
|
groups = {id(e): [e] for e in eps if id(e) not in hosts}
|
||||||
|
for e in eps:
|
||||||
|
if id(e) in hosts:
|
||||||
|
groups[id(hosts[id(e)])].append(e)
|
||||||
merged = []
|
merged = []
|
||||||
for g in groups.values():
|
for g in groups.values():
|
||||||
prev = next((m for m in reversed(merged) if variety(m[0]) == variety(g[0]) and (g[0]["kind"] not in ("Obstruction", "Target issue")
|
prev = next((m for m in reversed(merged) if variety(m[0]) == variety(g[0]) and (g[0]["kind"] not in (
|
||||||
or set(g[0]["sensors"]) & {s for e in m for s in e["sensors"]})), None)
|
"Obstruction", "Target issue") or set(g[0]["sensors"]) & set(m[0]["sensors"]))), None) # the same target
|
||||||
end = max((e["end"] for e in prev if variety(e) == variety(g[0])), default=None) if prev else None
|
end = prev and max(e["end"] for e in prev if variety(e) == variety(g[0]))
|
||||||
if end and datetime.fromisoformat(g[0]["start"]) - datetime.fromisoformat(end) < MERGE:
|
if end and datetime.fromisoformat(g[0]["start"]) - datetime.fromisoformat(end) < MERGE:
|
||||||
prev += g
|
prev += g
|
||||||
else:
|
else:
|
||||||
merged.append(g)
|
merged.append(g)
|
||||||
return sorted((event(g, references) for g in merged), key=lambda v: v["start"])
|
return sorted((event(g) for g in merged), key=lambda v: v["start"])
|
||||||
|
|
||||||
|
|
||||||
def events(measurements, end=None, folder=None):
|
def events(measurements, end=None, folder=None):
|
||||||
"""Measurements (data.load) -> events of all detectors: what the page receives. A detector also receives what its
|
"""Measurements (data.load) -> events of all detectors: what the page receives. end: the end of the window read
|
||||||
detect function declares: end, the end of the window read (batch run), for a hole touching it; a CONTEXT key, read
|
(batch run), for a hole touching it; folder: unused, kept for the callers. A detector that fails is skipped with a
|
||||||
from the station folder (once per folder), empty without a folder."""
|
message (a broken detector does not block the page)."""
|
||||||
episodes = []
|
episodes = []
|
||||||
for mod in detectors():
|
for name in DETECTORS:
|
||||||
try:
|
try:
|
||||||
params = inspect.signature(mod.detect).parameters
|
episodes += importlib.import_module(name).detect(measurements, **({"end": end} if name == "no_data" else {}))
|
||||||
kw = {k: _context(folder, k) if folder else {} if k == "metrics" else [] for k in CONTEXT if k in params}
|
|
||||||
if "end" in params:
|
|
||||||
kw["end"] = end
|
|
||||||
episodes += mod.detect(measurements, **kw)
|
|
||||||
except Exception as e: # noqa: BLE001
|
except Exception as e: # noqa: BLE001
|
||||||
print(f"Detectors: {mod.__name__} skipped ({type(e).__name__}: {e})", file=sys.stderr)
|
print(f"Detectors: {name} skipped ({type(e).__name__}: {e})", file=sys.stderr)
|
||||||
return classify(episodes, {s for s, ms in measurements.items() if ms[0]["pattern_type"] == "global_ref"})
|
return classify(episodes)
|
||||||
|
|
||||||
|
|
||||||
|
def window(measurements, start, end, history=timedelta(days=3)):
|
||||||
|
"""The events of [start, end) (UTC datetimes), the detectors given `history` of earlier measurements too (the rule of
|
||||||
|
the CLI below): those ending at or after start. For a caller (aimsight-app) as for the CLI."""
|
||||||
|
chunk = {s: k for s, ms in measurements.items() if (k := [m for m in ms if start - history <= m["t"] < end])}
|
||||||
|
return [v for v in events(chunk, end) if v["end"] >= iso(start)] if chunk else []
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
@@ -230,9 +136,7 @@ if __name__ == "__main__":
|
|||||||
t = start
|
t = start
|
||||||
while t < end:
|
while t < end:
|
||||||
b = min(t + timedelta(days=a.chunk), end) if a.chunk else end
|
b = min(t + timedelta(days=a.chunk), end) if a.chunk else end
|
||||||
h = t - timedelta(days=a.history)
|
evs = window(measurements, t, b, timedelta(days=a.history))
|
||||||
chunk = {s: k for s, ms in measurements.items() if (k := [m for m in ms if h <= m["t"] < b])}
|
|
||||||
evs = [v for v in events(chunk, b, a.folder) if v["end"] >= iso(t)] if chunk else []
|
|
||||||
if a.chunk:
|
if a.chunk:
|
||||||
print(f"== {iso(t)} → {iso(b)}: {len(evs) or 'nothing'}")
|
print(f"== {iso(t)} → {iso(b)}: {len(evs) or 'nothing'}")
|
||||||
for v in evs:
|
for v in evs:
|
||||||
|
|||||||
-140
@@ -1,140 +0,0 @@
|
|||||||
"""Reading a station folder (standard library only). Every time is UTC.
|
|
||||||
|
|
||||||
Column formats: knowledge/displacement-columns.md in the labeling repository.
|
|
||||||
"""
|
|
||||||
import csv
|
|
||||||
import glob
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import statistics
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
# Numeric columns of the "advanced" export; an empty cell becomes None.
|
|
||||||
NUM = ("x", "y", "error_value", "error_threshold", "relative_x", "relative_y",
|
|
||||||
"corr_x", "corr_y", "reproj_err", "inlier", "sigma0",
|
|
||||||
"peak", "flux", "sharpness", "fwhm", "dominance", "snr") # image quality: recent cameras only (LENZSPITZE)
|
|
||||||
# Camera pose (camera/): pan, tilt, rotation [deg], scale, sigma0 (registration residual).
|
|
||||||
POSE = ("pan", "tilt", "rotation", "scale", "sigma0")
|
|
||||||
IDENTIFIERS = re.compile(r"^(SIM_CCID|ip(_.*)?)$") # station identifiers: never read
|
|
||||||
METRIC_RE = re.compile(r"^(.+?)_\d{4}_\d{2}_\d{2}_")
|
|
||||||
|
|
||||||
|
|
||||||
def _header(path):
|
|
||||||
with open(path, newline="") as f:
|
|
||||||
return next(csv.reader(f), [])
|
|
||||||
|
|
||||||
|
|
||||||
def find_files(folder):
|
|
||||||
"""Station folder -> {'advanced': [...], 'camera': [...]}: sorted CSV paths, subfolders included."""
|
|
||||||
out = {"advanced": [], "camera": []}
|
|
||||||
for p in sorted(glob.glob(os.path.join(folder, "**", "*.csv"), recursive=True)):
|
|
||||||
header = _header(p)
|
|
||||||
if "sensor_name" in header and "corr_x" in header: # quality/ (optical_emulator) has sensor_name, not corr_x
|
|
||||||
out["advanced"].append(p)
|
|
||||||
elif "median_pan" in header and "timestamp" not in header: # timestamp: the hourly export
|
|
||||||
out["camera"].append(p)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _number(v):
|
|
||||||
v = v.strip()
|
|
||||||
return float(v) if v else None
|
|
||||||
|
|
||||||
|
|
||||||
def load(folder):
|
|
||||||
""""advanced" exports of the folder -> {sensor: [measurement]}, sorted by t, one reading per instant.
|
|
||||||
|
|
||||||
Measurement: dict with t (UTC datetime), pattern_type, and the NUM columns (float or None).
|
|
||||||
A rejected measurement (see rejected) has empty corr_x, corr_y, as the column docs say: the station empties them
|
|
||||||
for a target but leaves some filled (often absurd) for a reference; they are emptied here.
|
|
||||||
"""
|
|
||||||
by = {}
|
|
||||||
for path in find_files(folder)["advanced"]:
|
|
||||||
with open(path, newline="") as f:
|
|
||||||
for r in csv.DictReader(f):
|
|
||||||
m = {c: _number(r.get(c) or "") for c in NUM} # inlier is missing from old exports
|
|
||||||
m["t"] = parse_time(r["time_utc"])
|
|
||||||
m["pattern_type"] = r["pattern_type"]
|
|
||||||
if rejected(m):
|
|
||||||
m["corr_x"] = m["corr_y"] = None
|
|
||||||
by.setdefault(r["sensor_name"].strip(), {})[m["t"]] = m # some exports end the name with a newline
|
|
||||||
return {s: [d[t] for t in sorted(d)] for s, d in by.items()}
|
|
||||||
|
|
||||||
|
|
||||||
def load_temperature(folder):
|
|
||||||
"""Station board temperatures (metrics/, sensor_*_deg_c) -> [(t, °C)] sorted: the median of the components at each
|
|
||||||
hour; [] without these metrics. The components rise and fall together; the median drops the spike of a single
|
|
||||||
one, the computing GPU (D-124)."""
|
|
||||||
by = {}
|
|
||||||
for path in glob.glob(os.path.join(folder, "metrics", "*_deg_c_*.csv")):
|
|
||||||
with open(path, newline="") as f:
|
|
||||||
for r in csv.DictReader(f):
|
|
||||||
if (v := _number(r.get("metric") or "")) is not None:
|
|
||||||
by.setdefault(parse_time(r["time_utc"]), []).append(v)
|
|
||||||
return [(t, statistics.median(v)) for t, v in sorted(by.items())]
|
|
||||||
|
|
||||||
|
|
||||||
def load_camera(folder):
|
|
||||||
"""Camera movements of the folder -> [pose] sorted by t; pose: t and the POSE columns (float or None).
|
|
||||||
|
|
||||||
One row per camera registration; the medians are read (min = max in the files received: one reading per minute).
|
|
||||||
A measurement instant without a pose: the camera was not registered.
|
|
||||||
"""
|
|
||||||
by = {}
|
|
||||||
for path in find_files(folder)["camera"]:
|
|
||||||
with open(path, newline="") as f:
|
|
||||||
for r in csv.DictReader(f):
|
|
||||||
p = {c: _number(r.get("median_" + c, "")) for c in POSE}
|
|
||||||
p["t"] = parse_time(r["time_utc"])
|
|
||||||
by[p["t"]] = p
|
|
||||||
return [by[t] for t in sorted(by)]
|
|
||||||
|
|
||||||
|
|
||||||
def load_metrics(folder):
|
|
||||||
"""Station metrics (metrics/, one CSV per metric and per day) -> {name: [(t, value)]} sorted: numeric values only
|
|
||||||
(BusVoltage, sensor_CPU_deg_c, rsrp_dBm…); text metrics (link, mode…) and identifiers (SIM_CCID, ip*) are not
|
|
||||||
read. Name: the file's, without the station or the date."""
|
|
||||||
# ponytail: text metrics are skipped; read them (link up/down) when a detector needs them
|
|
||||||
station, by = os.path.basename(os.path.normpath(folder)) + "_", {}
|
|
||||||
for path in glob.glob(os.path.join(folder, "metrics", "*.csv")):
|
|
||||||
if not (m := METRIC_RE.match(os.path.basename(path))):
|
|
||||||
continue
|
|
||||||
name = m.group(1).removeprefix(station)
|
|
||||||
if IDENTIFIERS.match(name):
|
|
||||||
continue
|
|
||||||
with open(path, newline="") as f:
|
|
||||||
for r in csv.DictReader(f):
|
|
||||||
try:
|
|
||||||
v = float((r.get("metric") or "").strip("[]'\" ")) # some metrics come as a list: ['-95']
|
|
||||||
except ValueError:
|
|
||||||
continue
|
|
||||||
by.setdefault(name, {})[parse_time(r["time_utc"])] = v
|
|
||||||
return {n: sorted(d.items()) for n, d in sorted(by.items())}
|
|
||||||
|
|
||||||
|
|
||||||
def is_reference(measurements):
|
|
||||||
"""Reference of the compensation network (pattern_type global_ref), otherwise a target. Some exports have an empty
|
|
||||||
pattern_type, over the whole period or at the start: the first filled pattern_type is read then and, if there is
|
|
||||||
none, filled reproj_err or inlier (references only)."""
|
|
||||||
t = next((m["pattern_type"] for m in measurements if m["pattern_type"]), None)
|
|
||||||
if t is not None:
|
|
||||||
return t == "global_ref"
|
|
||||||
return any(m["reproj_err"] is not None or m["inlier"] is not None for m in measurements)
|
|
||||||
|
|
||||||
|
|
||||||
def rejected(m):
|
|
||||||
"""Measurement rejected by the station: error_value above the sensor's threshold (error_threshold); without these
|
|
||||||
two values, an empty corr_x."""
|
|
||||||
if m["error_value"] is not None and m["error_threshold"] is not None:
|
|
||||||
return m["error_value"] > m["error_threshold"]
|
|
||||||
return m["corr_x"] is None
|
|
||||||
|
|
||||||
|
|
||||||
def parse_time(text):
|
|
||||||
"""'2026-01-01T00:00Z' or '2026-01-01 00:00' -> UTC datetime; a time without a zone is read as UTC."""
|
|
||||||
t = datetime.fromisoformat(text)
|
|
||||||
return t.astimezone(timezone.utc) if t.tzinfo else t.replace(tzinfo=timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
def iso(t):
|
|
||||||
return t.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%MZ")
|
|
||||||
+76
-266
@@ -1,239 +1,80 @@
|
|||||||
"""Detector of the Displacement family (the families document): sudden jumps of corr_x / corr_y (px) of each target, at
|
"""Displacement and Perturbation families: sudden jumps of the corrected displacements (corr_x, corr_y, px) of each target, and slow drift
|
||||||
the native cadence of the "advanced" export, and slow drift (end of file). A flipping target ("Target flipping")
|
(end of file). Global references are excluded, rejected measurements (empty corr) skipped.
|
||||||
belongs to the Target issue family: episodes() returns it, detect() leaves it to target_issue.py.
|
|
||||||
|
|
||||||
1. Noise of the series = median of the gaps from one measurement to the next; jump = measurement K noises or more away
|
1. Jump: a measurement at least K noises (median step of the series) away from the level before (median of the BASE
|
||||||
from the level before (median of the BASE previous measurements); the first BASE measurements, lacking a level
|
previous measurements).
|
||||||
before, are compared with the level after (median of the BASE next ones), a gap there is a jump back at once.
|
2. Back to normal: within BACK_H, an hour of measurements in a row (BASE at most, one at least) within half the jump of
|
||||||
2. Back (an hour of measurements in a row within half the jump: BASE at 15 min, a single one at 4 h; wf-improve
|
the level before; a Perturbation, greyed (categories.py joins it to an overlapping Visibility or
|
||||||
MICHIGAN, mod. 1) within a day, or one day later back at its usual place (first day of the export) or at the day
|
Obstruction episode, otherwise nothing to do). Otherwise "Lasting shift", or "Jump, not back
|
||||||
before's: "Sudden jump, back to normal", greyed; its cause is not established here (categories.py gives it the
|
yet" with less than BACK_H of hindsight: never greyed (maybe a real movement), the episode is the jump's instant;
|
||||||
cause of the category that overlaps it, otherwise: to be checked on the images).
|
the new level becomes the level before. The return is read on the corrected curves only: a camera that keeps its new
|
||||||
3. Otherwise "Lasting shift", never greyed (maybe a real movement); less than a day of hindsight: "Jump, not back yet",
|
position while the compensation brings the curves back is a Displacement in the labels, a Perturbation here.
|
||||||
not greyed either.
|
|
||||||
4. Jumps following each other within GAP, all targets together, make a single episode (common mode); a single lasting
|
|
||||||
(or too recent) target is enough not to grey the episode.
|
|
||||||
5. Global references excluded; rejected measurements (empty corr) skipped.
|
|
||||||
6. Common peak (NORDEND, batch 01): at the same instant, at least PEAK_MIN targets move away by at least K_PEAK noises
|
|
||||||
from both neighbouring measurements, on the same side (the whole image shaken for one step): a jump back for each,
|
|
||||||
even under K. It joins the episodes of point 4 (next to a lasting jump, it greys nothing).
|
|
||||||
7. (CLARIDEN, batch 01) a jump that never comes back no longer ends the reading of the series, its new level becomes
|
|
||||||
the level before; an episode that is not greyed ends at its last jump, not at the returns; a flipping target
|
|
||||||
(REPEAT jumps back or more, each within a day of the previous one) makes a "back to normal" episode on its own, from
|
|
||||||
the first to the last flip, outside the common episodes. A target that flips for at least FLIP, with at most
|
|
||||||
MAX_FLIPPING targets at the same time, becomes "Target flipping" (Target issue, D-074); a lasting jump of at least
|
|
||||||
LARGE of the targets, "Large shift" (everything moved), always to be checked. wf-improve session Target issue, mod. 1,
|
|
||||||
S2: FLIP from 6 h to 36 h; a target flipping for a day during a nuisance of the whole scene (rain, fog, night of poor
|
|
||||||
visibility) is not a problem target; its jumps back go to the nuisance overlapping them, otherwise stay
|
|
||||||
Displacement.
|
|
||||||
8. Jump under K that stays (NIEDERHORN, batch 01): a jump of at least K_LASTING noises counts if it is lasting in the
|
|
||||||
sense of point 3 and the median of the day after moves away from that of the day before, on the jump's side, by at
|
|
||||||
least PLATEAU times the ordinary variation of the level from one UTC day to the next (median of the gaps between
|
|
||||||
daily levels); otherwise it is ignored (neither back nor too recent: only jumps of K noises or more are).
|
|
||||||
"""
|
"""
|
||||||
import statistics
|
import statistics
|
||||||
import sys
|
|
||||||
from datetime import timedelta
|
from datetime import timedelta
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent)) # data.py, next to it
|
from episodes import episode, noise
|
||||||
import data # noqa: E402
|
from data import is_reference
|
||||||
|
|
||||||
K = 20 # tuned on E-001: at 15, two healthy targets are greyed (17 and 18 noises); the trains of E-005 (17 and 19
|
K = 20 # E-001: at 15, two healthy targets are greyed (17 and 18 noises); the trains of E-005 are visibility's
|
||||||
# noises) are caught by visibility.py: do not lower K
|
BASE = 4 # measurements of the level before (1 h at 15 min); the return is judged on an hour (wf-improve MICHIGAN, mod. 1)
|
||||||
K_LASTING = 7 # under K, a jump that stays counts from K_LASTING noises (NIEDERHORN, batch 01: e09, corr_y −0.18 px at
|
BACK_H = timedelta(hours=24) # E-002: plateaus up to 11.5 h
|
||||||
# once, 8 noises, stayed 5 days)
|
JUMP_BACK = "Sudden jump, back to normal"
|
||||||
PLATEAU = 8 # ... if it stays: median of the day after, on the jump's side, PLATEAU times the ordinary variation of the
|
CHECK = " Check whether the structure moved: look at the images."
|
||||||
# level from one day to the next away from the day before (e09: 13; E-001, healthy targets: 6 at most)
|
LABELS = {"back": (JUMP_BACK, "A target jumped and came back within a day: nothing to do."),
|
||||||
K_PEAK = 6 # one-step peak common to several targets (NORDEND shakes, 6 to 11 noises, batch 01)
|
"lasting": ("Lasting shift", "A target shifted and stayed shifted for more than a day." + CHECK),
|
||||||
PEAK_MIN = 3 # targets peaking at the same step for a common peak (one or two: ordinary noise)
|
"recent": ("Jump, not back yet", "A target shifted and has not come back yet." + CHECK)}
|
||||||
BASE = 4 # measurements that make the level before (1 h at 15 min)
|
|
||||||
GAP = timedelta(hours=1) # two jumps within an hour, all targets together: a single episode
|
|
||||||
BACK_H = 24 # back within a day = artefact (E-002: plateaus up to 11.5 h)
|
|
||||||
FLIP = timedelta(hours=36) # a problem target flips for several days (PR_BG: 2.5 days); less is an agitated morning
|
|
||||||
# (E-009: C8 and C13, 5 flips in 1 h 20 during the movement) or a night of nuisance (NORDEND, JUNGFRAU:
|
|
||||||
# 12 to 28 h within a loss of visibility), a jump back, absorbed by its host
|
|
||||||
MAX_FLIPPING = 2 # targets flipping at the same time, at most (FLORIDA: the two stuck ones; E-002, cobweb on the
|
|
||||||
# references: 15 at once)
|
|
||||||
LARGE = 0.5 # share of the targets of a lasting jump for a large shift (E-004: almost all)
|
|
||||||
REPEAT = 5 # jumps back of the same target, each within a day of the other, that make it a flipping target (PR_BG of
|
|
||||||
# CLARIDEN, batch 01: about thirty in 2.5 days)
|
|
||||||
|
|
||||||
|
|
||||||
def _jumps(t, v):
|
def _jumps(t, v):
|
||||||
"""A series (instants, values) -> [(start i, end i, state)], state 'back', 'lasting' or 'recent'."""
|
"""A series (instants, values) -> [(start index, end index, state)], state "back", "lasting" or "recent"."""
|
||||||
day = timedelta(hours=BACK_H)
|
n = noise(v)
|
||||||
noise = statistics.median(abs(y - x) for x, y in zip(v, v[1:]))
|
back_n = max(1, min(BASE, round(timedelta(hours=1) / statistics.median(b - a for a, b in zip(t, t[1:])))))
|
||||||
step = statistics.median(b - a for a, b in zip(t, t[1:])) # cadence of the series
|
out, i = [], BASE
|
||||||
back_n = max(1, min(BASE, round(timedelta(hours=1) / step))) # measurements of an hour (BASE at 15 min, one at 4 h)
|
while n and i < len(v):
|
||||||
# ponytail: usual place = first day of the export, wrong if it starts within an artefact
|
|
||||||
ref = statistics.median(x for x, u in zip(v, t) if u < t[0] + day)
|
|
||||||
lv = {} # level of each UTC day; its ordinary variation from one day to the next judges a jump under K
|
|
||||||
for x, u in zip(v, t):
|
|
||||||
lv.setdefault(u.date(), []).append(x)
|
|
||||||
lv = [statistics.median(x) for _, x in sorted(lv.items())]
|
|
||||||
ordinary = statistics.median(abs(b - a) for a, b in zip(lv, lv[1:])) if len(lv) > 2 else 0
|
|
||||||
# the first BASE measurements, lacking a level before, face the level after: a gap there came back at once
|
|
||||||
out = [(k, k, "back") for k in range(min(BASE, len(v) - BASE))
|
|
||||||
if noise and abs(v[k] - statistics.median(v[k + 1:k + 1 + BASE])) >= K * noise]
|
|
||||||
i = BASE
|
|
||||||
while noise and i < len(v):
|
|
||||||
base = statistics.median(v[i - BASE:i])
|
base = statistics.median(v[i - BASE:i])
|
||||||
if abs(v[i] - base) < K_LASTING * noise:
|
if abs(v[i] - base) < K * n:
|
||||||
i += 1
|
i += 1
|
||||||
continue
|
continue
|
||||||
tol = abs(v[i] - base) / 2 # back: an hour of measurements in a row within half the jump from the level before
|
tol = abs(v[i] - base) / 2
|
||||||
j = next((k for k in range(i + 1, len(v)) if all(abs(x - base) < tol for x in v[k:k + back_n])), len(v))
|
j = next((k for k in range(i + 1, len(v)) if all(abs(x - base) < tol for x in v[k:k + back_n])), len(v))
|
||||||
before = statistics.median([x for x, u in zip(v, t) if t[i] - day <= u < t[i]] or [base])
|
state = "back" if j < len(v) and t[j] - t[i] <= BACK_H else "lasting" if t[-1] - t[i] >= BACK_H else "recent"
|
||||||
after = statistics.median(x for x, u in zip(v, t) if t[i] <= u < t[i] + day)
|
out.append((i, j - 1 if state == "back" else i, state))
|
||||||
if j < len(v) and t[j] - t[i] <= day:
|
|
||||||
state = "back"
|
|
||||||
elif abs(after - ref) < tol or abs(after - before) < tol: # the day after, the target is at its usual place,
|
|
||||||
state, j = "back", i + 1 # or where it was the day before: the jump ended a dip
|
|
||||||
else:
|
|
||||||
state = "lasting" if t[-1] - t[i] >= day else "recent" # less than a day of hindsight: not known yet
|
|
||||||
if abs(v[i] - base) < K * noise and (state != "lasting" or (after - before) * (v[i] - base) < 0
|
|
||||||
or not ordinary or abs(after - before) < PLATEAU * ordinary):
|
|
||||||
i += 1 # under K, only a jump that stays counts: the day after far from the day before, on its side
|
|
||||||
continue
|
|
||||||
out.append((i, j - 1, state))
|
|
||||||
# level before rebuilt after the return; never back: the new level becomes the level before, the rest is
|
|
||||||
# still read (PR_BG of CLARIDEN, batch 01: its flips after the 10:30 jump were invisible)
|
|
||||||
i = (j if j < len(v) else i) + BASE
|
i = (j if j < len(v) else i) + BASE
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
def _peaks(t, v):
|
def jumps(measurements):
|
||||||
"""Instants where the measurement moves away by at least K_PEAK noises from both neighbours, same side (one-step peak)."""
|
found = set() # (start, end, state, target); corr_x and corr_y together
|
||||||
noise = statistics.median(abs(y - x) for x, y in zip(v, v[1:]))
|
|
||||||
return [t[i] for i in range(1, len(v) - 1) if noise and (v[i] - v[i - 1]) * (v[i] - v[i + 1]) > 0
|
|
||||||
and min(abs(v[i] - v[i - 1]), abs(v[i] - v[i + 1])) >= K_PEAK * noise]
|
|
||||||
|
|
||||||
|
|
||||||
def episodes(measurements):
|
|
||||||
"""{sensor: [measurement]} (data.load) -> jump episodes, sorted by start: Displacement, and Target issue for a
|
|
||||||
flipping target."""
|
|
||||||
found = [] # (start, end, state, sensor)
|
|
||||||
peaks = {} # instant -> targets peaking there for one step
|
|
||||||
for name, ms in measurements.items():
|
for name, ms in measurements.items():
|
||||||
if data.is_reference(ms):
|
if is_reference(ms):
|
||||||
continue
|
continue
|
||||||
for axis in ("corr_x", "corr_y"):
|
for axis in ("corr_x", "corr_y"):
|
||||||
kept = [m for m in ms if m[axis] is not None]
|
kept = [m for m in ms if m[axis] is not None]
|
||||||
if len(kept) <= BASE:
|
if len(kept) > BASE:
|
||||||
continue
|
t, v = [m["t"] for m in kept], [m[axis] for m in kept]
|
||||||
t, v = [m["t"] for m in kept], [m[axis] for m in kept]
|
found |= {(t[a], t[b], state, name) for a, b, state in _jumps(t, v)}
|
||||||
# a jump that did not come back ends with itself, not with the data (its bar hid the others)
|
return [episode(a, b, "Perturbation" if state == "back" else "Displacement", *LABELS[state], [name], state == "back") for a, b, state, name in sorted(found)]
|
||||||
found += [(t[a], t[b] if state == "back" else t[a], state, name) for a, b, state in _jumps(t, v)]
|
|
||||||
for u in _peaks(t, v):
|
|
||||||
peaks.setdefault(u, set()).add(name)
|
|
||||||
found += [(u, u, "back", s) for u, cs in peaks.items() if len(cs) >= PEAK_MIN for s in cs] # common peak: back
|
|
||||||
alone = _flips(found) # a target that keeps jumping and coming back: an episode on its own, outside the common ones
|
|
||||||
found = [f for f in found if f[2] != "back" or not any(f[3] in g["sensors"] and g["start"] <= f[0] <= g["end"]
|
|
||||||
for g in alone)]
|
|
||||||
groups = [] # a jump that did not come back only counts by its instant: its end is often the end of the data
|
|
||||||
for s, e, state, name in sorted(found):
|
|
||||||
if groups and s <= groups[-1]["until"] + GAP:
|
|
||||||
g = groups[-1]
|
|
||||||
else:
|
|
||||||
g = {"start": s, "end": e, "until": s, "last": s, "sensors": {}}
|
|
||||||
groups.append(g)
|
|
||||||
g["end"], g["until"], g["last"] = max(g["end"], e), max(g["until"], e if state == "back" else s), s
|
|
||||||
g["sensors"].setdefault(name, set()).add(state)
|
|
||||||
long = [g for g in alone if g["flip"]] # flips of many targets at the same time: the compensation, not a
|
|
||||||
for g in long: # problem target
|
|
||||||
if sum(h["start"] <= g["end"] and h["end"] >= g["start"] for h in long) > MAX_FLIPPING:
|
|
||||||
g["flip"] = 0
|
|
||||||
total = sum(not data.is_reference(ms) for ms in measurements.values())
|
|
||||||
return sorted((_episode(g, total) for g in groups + alone), key=lambda ep: ep["start"])
|
|
||||||
|
|
||||||
|
|
||||||
def _flips(found):
|
|
||||||
"""Jumps back of the same target, each within BACK_H of the previous one: REPEAT or more -> a group, from the first
|
|
||||||
to the last flip."""
|
|
||||||
day, alone = timedelta(hours=BACK_H), []
|
|
||||||
for name in {f[3] for f in found}:
|
|
||||||
chain = [] # (start, end); corr_x and corr_y twice: counted by instant
|
|
||||||
for s, e in sorted({(s, e) for s, e, state, n in found if n == name and state == "back"}) + [(None, None)]:
|
|
||||||
if chain and s is not None and s <= max(c[1] for c in chain) + day:
|
|
||||||
chain.append((s, e))
|
|
||||||
continue
|
|
||||||
if len({c[0] for c in chain}) >= REPEAT:
|
|
||||||
alone.append({"start": chain[0][0], "end": max(c[1] for c in chain), "sensors": {name: {"back"}},
|
|
||||||
"flip": len({c[0] for c in chain}) if max(c[1] for c in chain) - chain[0][0] >= FLIP else 0})
|
|
||||||
chain = [(s, e)]
|
|
||||||
return alone
|
|
||||||
|
|
||||||
|
|
||||||
def _episode(g, total):
|
|
||||||
"""A group -> episode; a single lasting (or too recent) jump is enough not to grey; a flipping target: "Target
|
|
||||||
flipping" (Target issue, D-074); a lasting jump of LARGE targets or more: "Large shift"."""
|
|
||||||
n = len(g["sensors"])
|
|
||||||
lasting = sum("lasting" in st for st in g["sensors"].values())
|
|
||||||
recent = any("recent" in st for st in g["sensors"].values())
|
|
||||||
who = "One target" if n == 1 else f"{n} targets together (a single event)"
|
|
||||||
if g.get("flip"):
|
|
||||||
name = next(iter(g["sensors"]))
|
|
||||||
return {"start": data.iso(g["start"]), "end": data.iso(g["end"]), "kind": "Target issue", "label": "Target flipping",
|
|
||||||
"text": f"{name} jumped and came back {g['flip']} times: its tracking is not reliable, look at the images "
|
|
||||||
"of the target.", "ignore": True, "sensors": [name], "lasting": []}
|
|
||||||
if lasting and n >= LARGE * total:
|
|
||||||
label = "Large shift"
|
|
||||||
text = f"{n} targets jumped together and {lasting} of them stayed shifted: everything moved, please check it."
|
|
||||||
elif lasting:
|
|
||||||
label = "Lasting shift"
|
|
||||||
some = "" if lasting == n else f", {lasting} of them stayed shifted for more than a day"
|
|
||||||
text = (f"{who} jumped{some}" if some else f"{who} jumped and stayed shifted for more than a day") + \
|
|
||||||
": this may be a real movement, please check it."
|
|
||||||
elif recent:
|
|
||||||
label = "Jump, not back yet"
|
|
||||||
text = f"{who} jumped less than a day before the last measurement and has not come back yet: check again later."
|
|
||||||
else:
|
|
||||||
label = "Sudden jump, back to normal"
|
|
||||||
text = f"{who} jumped and came back within a day: the cause is not established, look at the images."
|
|
||||||
end = g["last"] if lasting or recent else g["end"] # not greyed: the last jump, not the returns
|
|
||||||
return {"start": data.iso(g["start"]), "end": data.iso(end), "kind": "Displacement", "label": label,
|
|
||||||
"text": text, "ignore": not lasting and not recent, "sensors": sorted(g["sensors"]),
|
|
||||||
"lasting": sorted(s for s, st in g["sensors"].items() if "lasting" in st)} # stayed shifted (event sentence)
|
|
||||||
|
|
||||||
|
|
||||||
def detect(measurements):
|
def detect(measurements):
|
||||||
return sorted([e for e in episodes(measurements) if e["kind"] == "Displacement"] + drift(measurements),
|
return jumps(measurements) + drift(measurements)
|
||||||
key=lambda ep: ep["start"])
|
|
||||||
|
|
||||||
|
|
||||||
# --- Slow drift ------------------------------------------------------------------------------------------------------
|
# --- Slow drift: a target that moves little by little, day after day (the trend of the Displacements tab, D-055).
|
||||||
# A target that moves little by little, day after day. Takes up the trend of the Displacements tab (D-055) on the
|
# Per target and per axis (corr_x, corr_y): the level of each UTC day measured at least HOURS of the hours the station
|
||||||
# "advanced" export (corr_x, corr_y, px). Method, per target (global references excluded) and per axis:
|
# measures (median of the day), at least DAYS days; the amplitude of a day: its 9th minus its 1st decile.
|
||||||
# 1. Level of the day: median of the measurements of each UTC day with at least HOURS measured hours out of 24, as a
|
# 1. Over the whole window: Theil-Sen slope × duration at least SIZE median daily amplitudes, |Kendall tau| ≥ STEADY, and
|
||||||
# share of the hours the station measures (wf-improve MICHIGAN, mod. 1: at one measurement every 4 h, 5 of the 6 hours;
|
# no step from one day to the next larger than half the drift (that is a jump).
|
||||||
# before, no day counted); at least DAYS days.
|
# 2. Otherwise, pieces: runs of at least DAYS kept days whose last level minus the first is at least PIECE amplitudes, with
|
||||||
# 2. Theil-Sen slope and Kendall tau of the daily levels; amplitude of the day: gap between its 9th and 1st deciles.
|
# the same tau and step conditions; taken from the largest to the smallest without overlapping (wf-improve
|
||||||
# 3. Drift: |slope| × duration ≥ SIZE times the median daily amplitude, |tau| ≥ STEADY, and no step from one day to the
|
# Displacement, mod. 1, S1: rows of CLARIDEN, 3.8 to 8.3 amplitudes; thermal cycles of JUNGFRAU, 3.1 at most).
|
||||||
# next of more than half the drift (a jump that lasts is not a drift: the jump rule takes it).
|
# One episode per drift, not greyed: the largest of the two axes over the whole window, otherwise each piece.
|
||||||
# 4. One episode per target (one axis is enough), from the first measurement of the first day to the last of the last
|
HOURS = 20 # out of 24, as a share of the hours the station measures (wf-improve MICHIGAN, mod. 1)
|
||||||
# day; not greyed.
|
DAYS = 5 # D-055
|
||||||
# 4b. Piecewise (wf-improve session Displacement, mod. 1, S1): a target without a drift over the whole window may drift for
|
SIZE = 1.0 # D-055
|
||||||
# a few days, then settle or go back. On each run of at least DAYS kept days, drift = level of the last day minus
|
STEADY = 0.6 # D-055
|
||||||
# that of the first, at least PIECE daily amplitudes (median of the series), |tau| ≥ STEADY with the same sign, no
|
PIECE = 3.5 # wf-improve Displacement, mod. 1, S1
|
||||||
# step from one kept day to the next of more than half the drift; pieces taken from the largest drift to the
|
|
||||||
# smallest, without overlapping (they may share the day where the level turns): each goes from one extreme of the
|
|
||||||
# level to the other, one episode per piece and per axis. At PIECE = 1, the thermal cycles of a few days and the
|
|
||||||
# noise of the levels get through (JUNGFRAU, NORDEND): 3.5 keeps the rows of CLARIDEN (3.8 to 8.3 amplitudes),
|
|
||||||
# drops JUNGFRAU (3.1 at most). The pieces do not replace step 3. A piece does not cross a hole: two consecutive kept
|
|
||||||
# days are at most MAX_HOLE days apart (mod. 2: the gap of a level across three days without a measurement is a
|
|
||||||
# step, not a slope; it made pieces of the whole window, CLARIDEN 09-28 → 11-01). A missing day does not cut.
|
|
||||||
# 5. Camera drift (same mod.): at least LARGE of the targets drift on the same side on corr_x or corr_y, and at least
|
|
||||||
# LARGE of the sensors, references included, on their raw positions x or y: only the pieces of point 4b are
|
|
||||||
# reported, not the trend over the whole window (mod. 1, S2: the badly compensated drifting camera, NEBRASKA, is a
|
|
||||||
# Displacement for the labels). The targets of a bridge following the season (JUNGFRAU, labelled Thermal
|
|
||||||
# displacement day after day) pass step 3 but not PIECE; NEBRASKA both.
|
|
||||||
HOURS = 20 # a day counts if it has at least 20 measured hours out of 24 (as D-055), as a share of the measured hours
|
|
||||||
DAYS = 5 # minimum days for a trend (D-055)
|
|
||||||
SIZE = 1.0 # drift ≥ SIZE daily amplitudes (D-055)
|
|
||||||
STEADY = 0.6 # |Kendall tau| of the daily levels (D-055)
|
|
||||||
PIECE = 3.5 # piecewise drift ≥ PIECE daily amplitudes (point 4b)
|
|
||||||
MAX_HOLE = 2 # a piece does not go from one kept day to the next more than MAX_HOLE days apart (point 4b)
|
|
||||||
|
|
||||||
|
|
||||||
def _decile(vs, q):
|
def _decile(vs, q):
|
||||||
@@ -242,50 +83,34 @@ def _decile(vs, q):
|
|||||||
|
|
||||||
|
|
||||||
def _days(ms, col):
|
def _days(ms, col):
|
||||||
"""-> {UTC day: its measurements}, the kept days (point 1)."""
|
"""-> (kept days, their levels, median daily amplitude, [(first instant, last) of each day])."""
|
||||||
days = {}
|
days = {}
|
||||||
for m in ms:
|
for m in ms:
|
||||||
if m[col] is not None:
|
if m[col] is not None:
|
||||||
days.setdefault(m["t"].date(), []).append(m)
|
days.setdefault(m["t"].date(), []).append(m)
|
||||||
slots = len({m["t"].hour for dm in days.values() for m in dm}) # hours the station measures (6 at a 4 h step)
|
slots = len({m["t"].hour for dm in days.values() for m in dm}) # hours the station measures (6 at a 4 h step)
|
||||||
return {d: dm for d, dm in days.items() if len({m["t"].hour for m in dm}) >= HOURS / 24 * slots}
|
ds = sorted(d for d, dm in days.items() if len({m["t"].hour for m in dm}) >= HOURS / 24 * slots)
|
||||||
|
vals = [[m[col] for m in days[d]] for d in ds]
|
||||||
|
ampl = statistics.median(_decile(v, 0.9) - _decile(v, 0.1) for v in vals) if ds else 0
|
||||||
|
return ds, [statistics.median(v) for v in vals], ampl, [(days[d][0]["t"], days[d][-1]["t"]) for d in ds]
|
||||||
|
|
||||||
|
|
||||||
def _amplitude(days, col):
|
def _whole(ds, lv, ampl, _):
|
||||||
return statistics.median(_decile([m[col] for m in dm], 0.9) - _decile([m[col] for m in dm], 0.1) for dm in days.values())
|
"""-> [(drift, 0, last index)] if the levels drift over the whole window, else []."""
|
||||||
|
pairs = [(i, j) for i in range(len(ds)) for j in range(i + 1, len(ds))]
|
||||||
|
slope = statistics.median((lv[j] - lv[i]) / (ds[j] - ds[i]).days for i, j in pairs)
|
||||||
|
tau = sum((lv[j] > lv[i]) - (lv[j] < lv[i]) for i, j in pairs) / len(pairs)
|
||||||
|
d = slope * (ds[-1] - ds[0]).days
|
||||||
|
step = max(abs(b - a) for a, b in zip(lv, lv[1:]))
|
||||||
|
return [(d, 0, len(ds) - 1)] if abs(d) >= SIZE * ampl and abs(tau) >= STEADY and step <= abs(d) / 2 else []
|
||||||
|
|
||||||
|
|
||||||
def _drift(ms, col):
|
def _pieces(ds, lv, ampl, _):
|
||||||
"""-> (drift in px, days, first instant, last) or None."""
|
"""-> [(drift, first index, last index)]: the drift pieces, without overlapping."""
|
||||||
days = _days(ms, col)
|
pieces = []
|
||||||
if len(days) < DAYS:
|
|
||||||
return None
|
|
||||||
lv = sorted((d.toordinal(), statistics.median(m[col] for m in dm)) for d, dm in days.items())
|
|
||||||
pairs = [(a, b) for k, a in enumerate(lv) for b in lv[k + 1:]]
|
|
||||||
slope = statistics.median((b[1] - a[1]) / (b[0] - a[0]) for a, b in pairs)
|
|
||||||
tau = sum((b[1] > a[1]) - (b[1] < a[1]) for a, b in pairs) / len(pairs)
|
|
||||||
drift = slope * (lv[-1][0] - lv[0][0])
|
|
||||||
step = max(abs(b[1] - a[1]) for a, b in zip(lv, lv[1:]))
|
|
||||||
if abs(drift) < SIZE * _amplitude(days, col) or abs(tau) < STEADY or step > abs(drift) / 2:
|
|
||||||
return None
|
|
||||||
d0, d1 = min(days), max(days)
|
|
||||||
return drift, len(days), min(m["t"] for m in days[d0]), max(m["t"] for m in days[d1])
|
|
||||||
|
|
||||||
|
|
||||||
def _pieces(ms, col):
|
|
||||||
"""-> [(drift in px, days, first instant, last)]: the drift pieces (point 4b)."""
|
|
||||||
days = _days(ms, col)
|
|
||||||
if len(days) < DAYS:
|
|
||||||
return []
|
|
||||||
ds = sorted(days)
|
|
||||||
lv = [statistics.median(m[col] for m in days[d]) for d in ds]
|
|
||||||
ampl = _amplitude(days, col)
|
|
||||||
pieces = [] # (drift, i, j)
|
|
||||||
for i in range(len(ds)):
|
for i in range(len(ds)):
|
||||||
signs = step = 0
|
signs = step = 0
|
||||||
for j in range(i + 1, len(ds)):
|
for j in range(i + 1, len(ds)):
|
||||||
if (ds[j] - ds[j - 1]).days > MAX_HOLE:
|
|
||||||
break # a gap across a hole is a step (the jump rule takes it), not a slope
|
|
||||||
signs += sum((lv[j] > lv[k]) - (lv[j] < lv[k]) for k in range(i, j)) # Kendall tau, step by step
|
signs += sum((lv[j] > lv[k]) - (lv[j] < lv[k]) for k in range(i, j)) # Kendall tau, step by step
|
||||||
step = max(step, abs(lv[j] - lv[j - 1]))
|
step = max(step, abs(lv[j] - lv[j - 1]))
|
||||||
n, d = j - i + 1, lv[j] - lv[i]
|
n, d = j - i + 1, lv[j] - lv[i]
|
||||||
@@ -296,32 +121,17 @@ def _pieces(ms, col):
|
|||||||
for d, i, j in sorted(pieces, key=lambda x: (-abs(x[0]), x[2] - x[1])):
|
for d, i, j in sorted(pieces, key=lambda x: (-abs(x[0]), x[2] - x[1])):
|
||||||
if all(j <= a or i >= b for _, a, b in taken):
|
if all(j <= a or i >= b for _, a, b in taken):
|
||||||
taken.append((d, i, j))
|
taken.append((d, i, j))
|
||||||
return [(d, j - i + 1, min(m["t"] for m in days[ds[i]]), max(m["t"] for m in days[ds[j]])) for d, i, j in taken]
|
return taken
|
||||||
|
|
||||||
|
|
||||||
def _common(measurements, cols):
|
|
||||||
"""At least LARGE of the sensors drift on the same side on the same axis of cols."""
|
|
||||||
for col in cols:
|
|
||||||
signs = [x[0] > 0 for ms in measurements.values() if (x := _drift(ms, col))]
|
|
||||||
if max(signs.count(True), signs.count(False)) >= LARGE * len(measurements):
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def drift(measurements):
|
def drift(measurements):
|
||||||
targets = {s: ms for s, ms in measurements.items() if not data.is_reference(ms)}
|
|
||||||
camera = _common(targets, ("corr_x", "corr_y")) and _common(measurements, ("x", "y")) # point 5: pieces only
|
|
||||||
out = []
|
out = []
|
||||||
for s, ms in sorted(targets.items()):
|
for s, ms in sorted(measurements.items()):
|
||||||
r = [] if camera else [x for col in ("corr_x", "corr_y") if (x := _drift(ms, col))]
|
if is_reference(ms):
|
||||||
r = [max(r, key=lambda x: abs(x[0]))] if r else [x for col in ("corr_x", "corr_y") for x in _pieces(ms, col)]
|
continue
|
||||||
out += [{"start": data.iso(a), "end": data.iso(b), "kind": "Displacement", "label": "Slow drift",
|
axes = [a for col in ("corr_x", "corr_y") if len((a := _days(ms, col))[0]) >= DAYS]
|
||||||
"text": f"{s} drifts slowly and steadily: {d:+.2f} px in {n} days. Keep watching it.",
|
whole = [(w, a) for a in axes for w in _whole(*a)]
|
||||||
"ignore": False, "sensors": [s]} for d, n, a, b in r]
|
found = [max(whole, key=lambda x: abs(x[0][0]))] if whole else [(p, a) for a in axes for p in _pieces(*a)]
|
||||||
|
out += [episode(a[3][i][0], a[3][j][1], "Displacement", "Slow drift", f"{s} drifts slowly and steadily: {d:+.2f} "
|
||||||
|
f"px in {j - i + 1} days. Keep watching it.", [s], False) for (d, i, j), a in found]
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
for ep in detect(data.load(sys.argv[1])):
|
|
||||||
print(ep["start"], ep["end"], "ignore" if ep["ignore"] else "KEEP ", ep["label"], "|", len(ep["sensors"]),
|
|
||||||
",".join(ep["sensors"][:6]), "|", ep["text"])
|
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
"""Helpers shared by the detectors."""
|
||||||
|
import statistics
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
from data import iso
|
||||||
|
|
||||||
|
|
||||||
|
def episode(start, end, kind, label, text, sensors, ignore, **extra):
|
||||||
|
return {"start": iso(start), "end": iso(end), "kind": kind, "label": label, "text": text, "ignore": ignore,
|
||||||
|
"sensors": sorted(sensors), **extra}
|
||||||
|
|
||||||
|
|
||||||
|
def runs(times, gap):
|
||||||
|
"""Sorted times -> [[t, ...]]: runs of times at most gap apart."""
|
||||||
|
out = []
|
||||||
|
for t in times:
|
||||||
|
if out and t - out[-1][-1] <= gap:
|
||||||
|
out[-1].append(t)
|
||||||
|
else:
|
||||||
|
out.append([t])
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def noise(v):
|
||||||
|
"""Median step of a series, from one value to the next."""
|
||||||
|
return statistics.median(abs(y - x) for x, y in zip(v, v[1:])) if len(v) > 1 else 0
|
||||||
|
|
||||||
|
|
||||||
|
def level(v):
|
||||||
|
"""-> (median, MAD) of the values."""
|
||||||
|
med = statistics.median(v)
|
||||||
|
return med, statistics.median(abs(x - med) for x in v) or 1e-9
|
||||||
|
|
||||||
|
|
||||||
|
def cadence(measurements):
|
||||||
|
"""Cycle of the measurements: the median gap between two instants, at least an hour divided by the median number of
|
||||||
|
instants per clock hour (a station measuring in bursts, three measurements a minute apart every 15 min: 15 min)."""
|
||||||
|
ts = sorted({m["t"].replace(second=0, microsecond=0) for ms in measurements.values() for m in ms})
|
||||||
|
per_hour = {}
|
||||||
|
for t in ts:
|
||||||
|
per_hour[t.replace(minute=0)] = per_hour.get(t.replace(minute=0), 0) + 1
|
||||||
|
gap = statistics.median(b - a for a, b in zip(ts, ts[1:])) if len(ts) > 1 else timedelta(hours=1)
|
||||||
|
return max(gap, timedelta(hours=1) / statistics.median(per_hour.values()))
|
||||||
|
|
||||||
|
|
||||||
|
def targets(n):
|
||||||
|
return "one target" if n == 1 else f"{n} targets"
|
||||||
+10
-41
@@ -1,51 +1,20 @@
|
|||||||
"""Detector of the No data family (the families document): moments without a measurement (the cadence of the "advanced"
|
"""No data family: no measurement in the export for more than MISSING steps of its cadence (episodes.cadence), between two
|
||||||
export is broken).
|
measurements or, when the caller gives the end of the window read, between the last measurement and that end."""
|
||||||
|
from episodes import cadence, episode
|
||||||
|
|
||||||
Cadence: median gap between two measurement instants, all sensors together; at least an hour divided by the median
|
MISSING = 2 # more than 2 missing steps is a hole (user's rule, 2026-10-23); also up to the end (CLARIDEN, batch 01)
|
||||||
number of instants of the clock hours that have one: a station measuring in bursts (three measurements a minute apart
|
|
||||||
every 15 min) has a median gap of a minute, but a cycle of 5 min; at the median gap, every gap between two bursts was a
|
|
||||||
hole, a whole day of No data (wf-improve, batch 2026-11-05, S1). A hole: more than MISSING missing steps between two
|
|
||||||
measurements (gap > MISSING + 1 cadences); the episode runs from the last measurement to the next one.
|
|
||||||
End of the window read (end, when the caller gives it: the chunked run of categories.py): the same rule between the last
|
|
||||||
measurement and end, the episode running up to end (CLARIDEN, batch 01: 44 h without a measurement up to the end of
|
|
||||||
the batch, not reported). No measurement in the export does not mean the station stopped: the text only says that the
|
|
||||||
export has nothing.
|
|
||||||
Usage: python3 scripts/no_data.py <station folder> (prints the episodes)
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
import statistics
|
|
||||||
import sys
|
|
||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
|
|
||||||
from data import iso, load # noqa: E402
|
|
||||||
|
|
||||||
MISSING = 2 # more than 2 missing steps is a hole (user's rule, 2026-10-23)
|
|
||||||
|
|
||||||
|
|
||||||
def duration(s):
|
def duration(d):
|
||||||
h, m = divmod(round(s / 60), 60)
|
h, m = divmod(round(d.total_seconds() / 60), 60)
|
||||||
return f"{h} h {m:02d} min" if h else f"{m} min"
|
return f"{h} h {m:02d} min" if h else f"{m} min"
|
||||||
|
|
||||||
|
|
||||||
def detect(measurements, end=None):
|
def detect(measurements, end=None):
|
||||||
"""end: end of the window read (datetime, excluded), optional; without it, a final hole is not seen."""
|
|
||||||
ts = sorted({m["t"].replace(second=0, microsecond=0) for ms in measurements.values() for m in ms})
|
ts = sorted({m["t"].replace(second=0, microsecond=0) for ms in measurements.values() for m in ms})
|
||||||
if len(ts) < 3:
|
if len(ts) < 3:
|
||||||
return []
|
return []
|
||||||
gaps = [(b - a).total_seconds() for a, b in zip(ts, ts[1:])]
|
step = cadence(measurements)
|
||||||
per_hour = {}
|
holes = [(a, b, "") for a, b in zip(ts, ts[1:])] + ([(ts[-1], end, ", up to the end of the period")] if end else [])
|
||||||
for t in ts:
|
return [episode(a, b, "No data", "No data", f"No measurement in the export for {duration(b - a)}{tail} (usually one "
|
||||||
per_hour[t.replace(minute=0)] = per_hour.get(t.replace(minute=0), 0) + 1
|
f"every {duration(step)}).", [], False) for a, b, tail in holes if b - a > (MISSING + 1) * step]
|
||||||
step = max(statistics.median(gaps), 3600 / statistics.median(per_hour.values())) # bursts: the cycle
|
|
||||||
out = [(a, b, g, "") for a, b, g in zip(ts, ts[1:], gaps)]
|
|
||||||
if end: # end excluded: the last expected measurement is at end − 1 step, the gap counts as between two measurements
|
|
||||||
out.append((ts[-1], end, (end - ts[-1]).total_seconds(), ", up to the end of the period"))
|
|
||||||
return [{"start": iso(a), "end": iso(b), "kind": "No data", "label": "No data",
|
|
||||||
"text": f"No measurement in the export for {duration(g)}{tail} (usually one every {duration(step)}).",
|
|
||||||
"ignore": False, "sensors": []}
|
|
||||||
for a, b, g, tail in out if g > (MISSING + 1) * step]
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
for e in detect(load(sys.argv[1])):
|
|
||||||
print(e)
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
"""Detector of the Obstruction family (the families document): an object hides one or a few targets, for an instant or
|
|
||||||
for long. It reuses two computations shared with other families:
|
|
||||||
- the error_value of visibility.py: Brief obstruction (something passing, one instant) and Target obscured (a target
|
|
||||||
rejected again and again while the others stay seen, or a nuisance that only touches a few targets, no reference);
|
|
||||||
- the day-by-day rejections of target_issue.py: a target rejected for less than a week, Target obscured.
|
|
||||||
A sensor already in an overlapping Target issue episode (Degraded target, Target stuck) is not returned as Obstruction:
|
|
||||||
its curve is already to be ignored, the family is Target issue (boundary "Target issue or an episode of another family"
|
|
||||||
of the families document; wf-improve session Obstruction, mod. 1: S1, problem targets of FLORIDA, NORDEND, SNCF01,
|
|
||||||
NIEDERHORN, JUNGFRAU, CLARIDEN cut into Target obscured; S3, their isolated rejections as Brief obstruction).
|
|
||||||
Usage: python3 scripts/obstruction.py <station folder> (prints the episodes)
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py and the detectors, next to it
|
|
||||||
import target_issue # noqa: E402
|
|
||||||
import visibility # noqa: E402
|
|
||||||
from data import load # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
def detect(measurements):
|
|
||||||
ti = target_issue.episodes(measurements)
|
|
||||||
issues = [e for e in ti if e["kind"] == "Target issue"]
|
|
||||||
out = []
|
|
||||||
for e in visibility.episodes(measurements) + ti:
|
|
||||||
if e["kind"] != "Obstruction":
|
|
||||||
continue
|
|
||||||
taken = {s for h in issues if h["start"] <= e["end"] and h["end"] >= e["start"] for s in h["sensors"]}
|
|
||||||
if (e := {**e, "sensors": [s for s in e["sensors"] if s not in taken]})["sensors"]: # minus the sensors
|
|
||||||
out.append(e) # already in Target issue
|
|
||||||
return sorted(out, key=lambda e: e["start"])
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
for e in detect(load(sys.argv[1])):
|
|
||||||
print(e["start"], "→", e["end"], e["label"], f"| {len(e['sensors'])}:", ",".join(e["sensors"][:8]), "|", e["text"])
|
|
||||||
+46
-165
@@ -1,184 +1,65 @@
|
|||||||
"""Detector of the Target issue family (the families document): a persistently problematic target: rejected most of the
|
"""Target issue family: a target rejected day after day ("Degraded target"); visibility.py compares each sensor with its
|
||||||
time, day after day, while the others are seen (degraded target), stuck (end of file), or flipping (the jumps of
|
own usual level, so a target almost always rejected looks normal to it. For a few days only, Target obscured
|
||||||
displacement.py, "Target flipping"). A target rejected for less than a week is hidden at times (point 4): episodes()
|
(Obstruction family).
|
||||||
returns it with the Obstruction family, which detect() leaves to obstruction.py.
|
|
||||||
|
|
||||||
visibility.py compares each sensor with its own usual level: a target that is almost always rejected looks normal to it
|
1. Degraded day: at least SHARE of the sensor's measurements of the UTC day rejected; a day with fewer than MIN_HOURS of
|
||||||
(FLORIDA, October 2026: Massif 2 and Sellette 6 rejected half the measurements or more, every day).
|
measurements (at the station's cadence) does not count.
|
||||||
Method:
|
2. Run: degraded days with at most BRIDGE other days in a row between them, and SHARE of rejections over the run. WEEK
|
||||||
1. Calm instant for a sensor: at most CALM of the other sensors rejected (error_value > error_threshold); weather and
|
days or more: Degraded target; MIN_DAYS or more: Target obscured. The episode runs from the first rejected
|
||||||
fog, which touch everyone, are thus set aside.
|
measurement of the run to the last, greyed. The cause (dirt, vegetation, a parked vehicle) is seen on the images.
|
||||||
2. Per UTC day and per sensor, on its calm instants (at least MIN_HOURS hours of instants at the station's cadence,
|
|
||||||
median gap between two measurement instants; otherwise the day does not count): degraded day if at least SHARE of
|
|
||||||
them are rejected. Cadence (wf-improve NEBRASKA, batch 01): 12 fixed instants, "3 h at 15 min", counted no day at one
|
|
||||||
measurement every 4 h; REFCOLH22, rejected all the time since it was lost, was not seen.
|
|
||||||
3. Episode: at least MIN_DAYS degraded days in a row (a day that does not count does not break the run), from the first
|
|
||||||
rejected measurement of these days to the last; ignore = True: the target's curve is not reliable.
|
|
||||||
4. Hindered all the time or at times (boundary "Target issue or Obstruction" of the families document; wf-improve
|
|
||||||
NEBRASKA, batch 01): a run of at least WEEK days is a problem target, "Degraded target"; shorter, a target hidden at
|
|
||||||
times, "Target obscured" (Obstruction family), one episode per run of rejected measurements (the hours when it is
|
|
||||||
seen are not part of it). BREITHORN: vehicles parked in front of COLI24 and POUI22, 2 to 5 days; NEBRASKA and
|
|
||||||
MICHIGAN: a lost reference, 25 days; JUNGFRAU: PM_G, 63.
|
|
||||||
For the problem target, the run is bridged: at most BRIDGE non-degraded days in a row do not break it, and it then
|
|
||||||
needs WEEK days from its first to its last degraded day, at SHARE_WEEK of rejections over all its days ("rejections
|
|
||||||
specific to the target almost every day of the range"); the Target obscured episodes it covers no longer exist
|
|
||||||
(wf-improve session Obstruction, mod. 1, S1: FLORIDA Sellette 6 and Massif 2, SNCF01 F4.3 and F4.4, NORDEND CTR01,
|
|
||||||
NIEDERHORN e01, cut into Target obscured by a day under SHARE; BREITHORN, parked vehicles, unchanged).
|
|
||||||
The problem target judges regularity rather than the share of the day: a day is degraded from SHARE_WEEK (a quarter)
|
|
||||||
of its calm instants rejected, and BRIDGE is one day; the target hidden at times keeps SHARE. A target rejected every
|
|
||||||
day at the same hours, by day only or by night only, stays under half and was not seen (wf-improve session Target
|
|
||||||
issue, mod. 1, S1: WILDSTRUBEL R3, 04:00 to 16:00 every day; NORDEND PR04 and CTR01, NIEDERHORN PWD_IMP_B08_024_C09
|
|
||||||
and e01, SNCF01 F4.3).
|
|
||||||
The cause (dirt, vegetation, setting) can only be read on the images.
|
|
||||||
Stuck target (E-011, FLORIDA: a target defined on the prism of a reference): per UTC day, the noise of its corr_x and
|
|
||||||
corr_y (median of the gaps from one measurement to the next) under STUCK times the median noise of the other targets, on
|
|
||||||
both axes; one day is enough (a day without noise does not happen by accident); label "Target stuck". On E-011: C2 at
|
|
||||||
7e-5 px and C3 at 2e-4 against 0.013 for the others.
|
|
||||||
Usage: python3 scripts/target_issue.py <station folder> (prints the episodes)
|
|
||||||
"""
|
"""
|
||||||
import os
|
from datetime import timedelta
|
||||||
import statistics
|
|
||||||
import sys
|
|
||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py and the detectors, next to it
|
from episodes import cadence, episode
|
||||||
import displacement # noqa: E402
|
|
||||||
from data import is_reference, iso, load # noqa: E402
|
|
||||||
|
|
||||||
CALM = 0.25 # share of the other sensors rejected, at most, for a calm instant
|
SHARE = 0.25 # wf-improve Target issue, mod. 1, S1 (WILDSTRUBEL R3 rejected from 04:00 to 16:00 every day, 30 to 45 %)
|
||||||
SHARE = 0.5 # half of the calm instants rejected: "most of the time"
|
MIN_HOURS = 3 # wf-improve NEBRASKA, batch 01 (one measurement every 4 h: as hours, not as instants)
|
||||||
SHARE_WEEK = 0.25 # a quarter of the calm instants rejected, each day of a week or more: a problem target, even when
|
BRIDGE = 1 # wf-improve Obstruction, mod. 1, S1 and Target issue, mod. 1, S1 (problem targets cut by a better day)
|
||||||
# rejected by day only or by night only (wf-improve session Target issue, mod. 1, S1)
|
WEEK = 7 # wf-improve NEBRASKA, batch 01 (lost references, 25 days; parked vehicles, 2 to 5 days, are not one)
|
||||||
MIN_HOURS = 3 # hours of calm instants, at the station's cadence: below that, the day does not count
|
MIN_DAYS = 2 # shorter than WEEK: a target hidden for a few days, Target obscured (Obstruction); NORDEND: isolated days
|
||||||
MIN_DAYS = 2 # a single day is too often an accident (NORDEND: isolated days on 16 sensors)
|
|
||||||
BRIDGE = 1 # non-degraded days that do not break a run of degraded days (4); at 2, the parked vehicles of BREITHORN
|
|
||||||
# (two days, then nothing for two days, then one) would make a week at a quarter
|
|
||||||
WEEK = 7 # degraded days in a row for a problem target; below, a target hidden at times
|
|
||||||
STUCK = 0.05 # noise of a stuck target compared with the others (E-011: 0.006; NORDEND, E-001: 0.7 at the lowest)
|
|
||||||
|
|
||||||
|
|
||||||
def is_rejected(m):
|
def rejected(m):
|
||||||
return m["error_value"] is not None and m["error_threshold"] is not None and m["error_value"] > m["error_threshold"]
|
return m["error_value"] is not None and m["error_threshold"] is not None and m["error_value"] > m["error_threshold"]
|
||||||
|
|
||||||
|
|
||||||
def cadence(measurements):
|
def _runs(days, min_instants):
|
||||||
"""Median gap between two measurement instants, all sensors together, in hours (as no_data)."""
|
"""{day: [rejected, measurements, first, last]} -> runs of degraded days; at most BRIDGE other days in a row between two
|
||||||
ts = sorted({m["t"].replace(second=0, microsecond=0) for ms in measurements.values() for m in ms})
|
degraded days belong to the run; a day that does not count is skipped."""
|
||||||
return statistics.median((b - a).total_seconds() for a, b in zip(ts, ts[1:])) / 3600 if len(ts) > 1 else 1
|
|
||||||
|
|
||||||
|
|
||||||
def episodes(measurements):
|
|
||||||
"""Targets rejected day after day: Target issue (a week or more) and Obstruction (less), then the stuck ones."""
|
|
||||||
min_instants = MIN_HOURS / cadence(measurements)
|
|
||||||
by_t = {}
|
|
||||||
for s, ms in measurements.items():
|
|
||||||
for m in ms:
|
|
||||||
by_t.setdefault(m["t"], {})[s] = is_rejected(m)
|
|
||||||
days = {} # sensor -> day -> [rejected, calm, first measurement, last]
|
|
||||||
for t in sorted(by_t):
|
|
||||||
d = by_t[t]
|
|
||||||
for s, r in d.items():
|
|
||||||
others = [v for k, v in d.items() if k != s]
|
|
||||||
c = days.setdefault(s, {}).setdefault(t.date(), [0, 0, t, t])
|
|
||||||
c[3] = t
|
|
||||||
if others and sum(others) / len(others) <= CALM:
|
|
||||||
c[0] += r
|
|
||||||
c[1] += 1
|
|
||||||
out = []
|
|
||||||
for s, ds in sorted(days.items()):
|
|
||||||
covered = [] # bridged runs of at least WEEK days, at SHARE_WEEK of rejections: a problem target
|
|
||||||
for run in _runs(ds, min_instants, BRIDGE, SHARE_WEEK):
|
|
||||||
share = sum(c[0] for c in run) / sum(c[1] for c in run)
|
|
||||||
if len(run) < WEEK or share < SHARE_WEEK:
|
|
||||||
continue
|
|
||||||
n = sum(c[0] / c[1] >= SHARE_WEEK for c in run)
|
|
||||||
r = _rejections([m for m in measurements[s] if run[0][2] <= m["t"] <= run[-1][3]])
|
|
||||||
covered.append((run[0][2], run[-1][3]))
|
|
||||||
out.append({"start": iso(r[0][0]), "end": iso(r[-1][1]), "kind": "Target issue", "label": "Degraded target",
|
|
||||||
"text": f"{s} was rejected in {round(100 * share)} % of the measurements when the other targets were "
|
|
||||||
f"seen, on {n} of {len(run)} days.", "ignore": True, "sensors": [s]})
|
|
||||||
for run in _runs(ds, min_instants, 0, SHARE): # at times: one episode per run of rejected measurements
|
|
||||||
if len(run) < MIN_DAYS or any(a <= run[0][2] <= b for a, b in covered):
|
|
||||||
continue
|
|
||||||
share = round(100 * sum(c[0] for c in run) / sum(c[1] for c in run))
|
|
||||||
text = f"{s} was rejected in {share} % of the measurements when the other targets were seen, {len(run)} days in a row."
|
|
||||||
out += [{"start": iso(a), "end": iso(b), "kind": "Obstruction", "label": "Target obscured", "text": text,
|
|
||||||
"ignore": True, "sensors": [s]}
|
|
||||||
for a, b in _rejections([m for m in measurements[s] if run[0][2] <= m["t"] <= run[-1][3]])]
|
|
||||||
return sorted(out + stuck(measurements, min_instants), key=lambda e: e["start"])
|
|
||||||
|
|
||||||
|
|
||||||
def _runs(ds, min_instants, bridge, share):
|
|
||||||
"""{day: [rejected, calm, first, last]} -> runs of degraded days; at most `bridge` non-degraded days in a row between
|
|
||||||
two degraded days do not break the run (they belong to it); a day that does not count does not break it."""
|
|
||||||
runs, cur, gap = [], [], []
|
runs, cur, gap = [], [], []
|
||||||
for j in sorted(ds):
|
for d in sorted(days):
|
||||||
rej, calm = ds[j][:2]
|
rej, n = days[d][:2]
|
||||||
if calm < min_instants:
|
if n < min_instants:
|
||||||
continue
|
continue
|
||||||
if rej / calm >= share:
|
if rej / n >= SHARE:
|
||||||
cur += gap + [ds[j]]
|
cur += gap + [days[d]]
|
||||||
gap = []
|
gap = []
|
||||||
elif cur:
|
elif cur:
|
||||||
gap.append(ds[j])
|
gap.append(days[d])
|
||||||
if len(gap) > bridge:
|
if len(gap) > BRIDGE:
|
||||||
runs.append(cur)
|
runs.append(cur)
|
||||||
cur, gap = [], []
|
cur, gap = [], []
|
||||||
return runs + ([cur] if cur else [])
|
return runs + ([cur] if cur else [])
|
||||||
|
|
||||||
|
|
||||||
def _rejections(ms):
|
|
||||||
"""Measurements of a sensor -> [(start, end)] of its runs of rejected measurements."""
|
|
||||||
out, cur = [], []
|
|
||||||
for m in ms + [None]:
|
|
||||||
if m and is_rejected(m):
|
|
||||||
cur.append(m["t"])
|
|
||||||
elif cur:
|
|
||||||
out.append((cur[0], cur[-1]))
|
|
||||||
cur = []
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _noise(v):
|
|
||||||
return statistics.median(abs(y - x) for x, y in zip(v, v[1:])) if len(v) > 2 else None
|
|
||||||
|
|
||||||
|
|
||||||
def stuck(measurements, min_instants):
|
|
||||||
"""Targets whose corr_x and corr_y barely move, day after day, while the others move."""
|
|
||||||
days = {} # day -> target -> (x noise, y noise, first measurement, last)
|
|
||||||
for s, ms in measurements.items():
|
|
||||||
if is_reference(ms):
|
|
||||||
continue
|
|
||||||
by_day = {}
|
|
||||||
for m in ms:
|
|
||||||
by_day.setdefault(m["t"].date(), []).append(m)
|
|
||||||
for d, dm in by_day.items():
|
|
||||||
b = [_noise([m[c] for m in dm if m[c] is not None]) for c in ("corr_x", "corr_y")]
|
|
||||||
if None not in b and len(dm) >= min_instants:
|
|
||||||
days.setdefault(d, {})[s] = (*b, dm[0]["t"], dm[-1]["t"])
|
|
||||||
runs = {} # target -> [[stuck days in a row]]
|
|
||||||
for d in sorted(days):
|
|
||||||
cs = days[d]
|
|
||||||
for s, (bx, by, a, b) in cs.items():
|
|
||||||
others = [v for k, (x, y, *_) in cs.items() if k != s for v in (x, y)]
|
|
||||||
if len(others) < 4:
|
|
||||||
continue
|
|
||||||
ref = statistics.median(others)
|
|
||||||
ru = runs.setdefault(s, [[]])
|
|
||||||
if ref and max(bx, by) <= STUCK * ref:
|
|
||||||
ru[-1].append((a, b))
|
|
||||||
elif ru[-1]:
|
|
||||||
ru.append([])
|
|
||||||
return [{"start": iso(k[0][0]), "end": iso(k[-1][1]), "kind": "Target issue", "label": "Target stuck",
|
|
||||||
"text": f"{s} barely moved while the other targets did ({len(k)} day{'s' if len(k) > 1 else ''}): it may be "
|
|
||||||
"set on a reference prism.", "ignore": True, "sensors": [s]}
|
|
||||||
for s, ru in sorted(runs.items()) for k in ru if k]
|
|
||||||
|
|
||||||
|
|
||||||
def detect(measurements):
|
def detect(measurements):
|
||||||
return sorted((e for e in episodes(measurements) + displacement.episodes(measurements) if e["kind"] == "Target issue"),
|
min_instants = timedelta(hours=MIN_HOURS) / cadence(measurements)
|
||||||
key=lambda e: e["start"])
|
days = {} # sensor -> day -> [rejected, measurements, first, last]
|
||||||
|
for s, ms in measurements.items():
|
||||||
|
for m in ms:
|
||||||
if __name__ == "__main__":
|
c = days.setdefault(s, {}).setdefault(m["t"].date(), [0, 0, m["t"], m["t"]])
|
||||||
for e in detect(load(sys.argv[1])):
|
c[0], c[1], c[3] = c[0] + rejected(m), c[1] + 1, m["t"]
|
||||||
print(e)
|
out = []
|
||||||
|
for s, ds in sorted(days.items()):
|
||||||
|
for run in _runs(ds, min_instants):
|
||||||
|
share = sum(c[0] for c in run) / sum(c[1] for c in run)
|
||||||
|
if len(run) < MIN_DAYS or share < SHARE:
|
||||||
|
continue
|
||||||
|
bad = [m["t"] for m in measurements[s] if run[0][2] <= m["t"] <= run[-1][3] and rejected(m)]
|
||||||
|
n = sum(c[0] / c[1] >= SHARE for c in run)
|
||||||
|
seen = f"{s} was rejected in {round(100 * share)} % of the measurements, on {n} of {len(run)} days."
|
||||||
|
out.append(episode(bad[0], bad[-1], "Target issue", "Degraded target", seen + " Ignore its curve in this period "
|
||||||
|
"and look at the images of the target.", [s], True) if len(run) >= WEEK else
|
||||||
|
episode(bad[0], bad[-1], "Obstruction", "Target obscured", seen + " Something hid it for a few "
|
||||||
|
"days: ignore its curve in this period.", [s], True))
|
||||||
|
return out
|
||||||
+99
-235
@@ -1,74 +1,44 @@
|
|||||||
"""Detector of the Thermal displacement family (the families document): the daily wave of the corrected displacements
|
"""Thermal displacement family: the daily wave of the displacements (expansion by day, contraction in the evening).
|
||||||
(expansion by day, contraction in the evening).
|
|
||||||
|
|
||||||
Method, per target (global references excluded) and per axis (corr_x, corr_y):
|
Per target (references excluded) and per axis:
|
||||||
1. Each UTC day of at least MIN_HOURS hours: mean of each hour minus the median of the day (removes jumps and drift).
|
1. Each UTC day measured at least MIN_HOURS hours: the mean of each hour minus the median of the day.
|
||||||
2. Profile = median, hour by hour, over the days; "repeatability" = median, over the days, of the correlation between
|
2. Profile: the median, hour by hour, over the days. The axis has a cycle if its days look like the profile of the
|
||||||
the day and the profile of the other days. Axis concerned if repeatability ≥ REPRO. An incomplete day (at least
|
other days (median correlation at least REPRO).
|
||||||
PARTIAL hours, fewer than MIN_HOURS) does not enter the profile, but its likeness is measured on the hours it has: a
|
3. The station has a cycle if at least SHARE of the targets have an axis with one and, for the corrected curves, a 24 h
|
||||||
hole of half a day no longer takes its episode away (wf-improve session Thermal displacement, mod. 1, S4; at
|
sinusoid explains at least R2_MIN of an axis of SHARE of the targets.
|
||||||
MIN_HOURS = 12 or 16, the incomplete days distorted the profile, NIEDERHORN went back).
|
4. Gap of each hour from the night level (median of the NIGHT flattest consecutive hours of the profile), as a share of
|
||||||
3. Cycle of the station if at least SHARE of the targets have an axis concerned, **and** (second method, asked by the
|
the profile's amplitude, median over the axes with a cycle; there is a wave if the largest gap (the peak) reaches
|
||||||
user) if a 24 h sinusoid, a + b·sin(2πt/24 h) + c·cos(2πt/24 h) by least squares, explains at least R2_MIN of the
|
WARM. "Expansion" from where the gap rises above EDGE to the peak, "contraction" from the peak until the gap falls
|
||||||
curve of an axis for at least SHARE of the targets.
|
under RETURN, after midnight if needed, at most to the next start: the cooling tail is slow (wf-improve Thermal
|
||||||
4. Warm hours: night level = median of the NIGHT flattest consecutive hours of the profile; an hour is warm if, as a
|
displacement, mod. 1, S1; wf-improve NORDEND 2026-11-05, S1). Two episodes for every day of the axes with a cycle.
|
||||||
median over the axes concerned, the profile moves away from it by at least WARM times its amplitude.
|
The corrected curves first (the structure, "Thermal ..."); without a cycle there, the raw positions x, y ("Camera
|
||||||
5. Two episodes per complete day that looks like the others (median correlation ≥ DAY_MATCH over the axes concerned):
|
thermal ...": the camera moves with its mount and the compensation removes it, D-074; the whole image, every sensor);
|
||||||
"Thermal expansion" from the start to the hour of the largest gap from the night level (the peak), then "Thermal
|
without one either, the camera slides ("Camera thermal movement", wf-improve MICHIGAN, mod. 1): instants where the raw
|
||||||
contraction" until the return. The warm hours say there is a cycle (at least MIN_WARM_HOURS of them); its bounds
|
positions of SHARE of the sensors and the corrected curves of SHARE of the targets move away on the same side by
|
||||||
are read lower, at EDGE times the amplitude: the start, going back from the peak while the gap stays ≥ EDGE (at most
|
SLIDE_K noises from their median of the day before; slides less than a day apart make one episode.
|
||||||
to the calmest hour of the profile); the return, going down from the peak until it goes back under RETURN, after
|
|
||||||
midnight if needed, at most to the next day's start: a slow return at night chains the cycles (wf-improve session
|
|
||||||
Thermal displacement, mod. 1, S1). RETURN is half of EDGE: the warm-up is quick, the cooling tail slow, and the
|
|
||||||
labelled nights went on after the return reached EDGE (wf-improve NORDEND 2026-11-05, S1: NORDEND, JUNGFRAU; EDGE
|
|
||||||
itself at 0.05 made FLORIDA and SNCF01 overflow in the morning). The direction (a rising or falling curve) depends on the target's orientation:
|
|
||||||
it is not looked at, only the gap from the night level.
|
|
||||||
Reading: the cycle is almost everywhere in the raw positions (common mode) and the compensation removes it, except at
|
|
||||||
PS des Coudres where it stays in corr, with the same shape as in the raw positions.
|
|
||||||
Camera thermal effect (D-074): without a cycle in corr, the same method on the raw positions x, y gives "Camera thermal
|
|
||||||
expansion" and "Camera thermal contraction": the camera moves with the heat of its mount, the compensation removes it;
|
|
||||||
same categories, own sentence. On the raw positions the second method is not asked (WILDSTRUBEL, batch 01: the camera
|
|
||||||
rose by a 3 px block at sunrise, held 3 h and dropped in one step around 09:30, six days in a row, which the sinusoid
|
|
||||||
does not explain); in exchange, a single warm hour is not a cycle (MIN_WARM_HOURS: NIEDERHORN, E-005, hourly, would
|
|
||||||
otherwise have its noon hour as a camera contraction). The camera episode covers all sensors, references included: the
|
|
||||||
whole image moves.
|
|
||||||
Common slide, without a cycle (wf-improve MICHIGAN, mod. 1): lacking a cycle in the corrected curves as in the raw
|
|
||||||
positions, an instant is a camera slide if the raw positions (x or y) of at least SHARE of the sensors, references
|
|
||||||
included, **and** the corrected curves (corr_x or corr_y) of at least SHARE of the targets move away on the same side by
|
|
||||||
at least SLIDE_K noises (median of the steps of the series) from their median of the day before; slides less than a
|
|
||||||
day apart make one episode, "Camera thermal movement", on all sensors, host of the jumps back (categories.HOSTS). The
|
|
||||||
condition on the raw positions leaves to the jumps a flip of the computed pose alone (raw positions still).
|
|
||||||
The camera moves with the cold (wf-improve session Thermal displacement, mod. 1, S3): lacking a cycle, also a "Camera
|
|
||||||
thermal movement" where the camera pose (tilt, rotation or scale) moves away from its median of the day before by at
|
|
||||||
least COLD noises, while the station temperature (detect declares it, D-124) falls under its own by at least COLD
|
|
||||||
noises the same hour; noise: the median of the variations over an hour (over one step, at one measurement every 4 h),
|
|
||||||
so that the threshold does not depend on the cadence. The targets may move in opposite directions or stay almost still
|
|
||||||
(the compensation absorbs the movement): the slide rule does not see it. Consecutive pose instants make one episode.
|
|
||||||
Without a pose or without a temperature, nothing. The heat is not taken: in fine weather, it is the day's cycle.
|
|
||||||
"""
|
"""
|
||||||
import math
|
import math
|
||||||
import os
|
|
||||||
import statistics
|
import statistics
|
||||||
import sys
|
|
||||||
from datetime import datetime, timedelta, timezone
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
|
from episodes import episode, noise, runs
|
||||||
from data import iso, load # noqa: E402
|
from data import is_reference
|
||||||
|
|
||||||
MIN_HOURS = 20 # a UTC day counts in the profile if it has at least 20 measured hours
|
MIN_HOURS = 20 # hours of a day of the profile
|
||||||
PARTIAL = 12 # an incomplete day of at least 12 hours gets its episode if it looks like the profile
|
REPRO = 0.6 # pure noise ≈ 0
|
||||||
REPRO = 0.6 # an axis has a cycle if each day looks like the others (pure noise ≈ 0)
|
SHARE = 0.5 # NORDEND of October is just under (9/19)
|
||||||
SHARE = 0.5 # share of the targets concerned; 2026-10_NORDEND is just under (9/19)
|
R2_MIN = 0.3 # PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48 (least squares then; now the projection)
|
||||||
WARM = 0.3 # gap from the night level, as a share of the profile's amplitude
|
NIGHT = 8 # flattest consecutive hours of the profile
|
||||||
DAY_MATCH = 0.5 # likeness of a day to the others to give it its episode
|
WARM = 0.3 # a wave: the peak at WARM times the amplitude from the night level
|
||||||
NIGHT = 8 # flattest consecutive hours of the profile = night level
|
EDGE = 0.1 # start of the wave (wf-improve Thermal displacement, mod. 1, S1)
|
||||||
R2_MIN = 0.3 # draft of 2026-10-24: PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48
|
RETURN = 0.05 # end of the wave (wf-improve NORDEND 2026-11-05, S1)
|
||||||
MIN_WARM_HOURS = 2 # WILDSTRUBEL, batch 01; NIEDERHORN (E-005) has only one warm hour (noon)
|
SLIDE_K = 3 # common slide, in noises (wf-improve MICHIGAN, mod. 1: at 2, Displacement instants of BREITHORN taken)
|
||||||
EDGE = 0.1 # start of the wave, as a share of the amplitude; WARM says whether there is a cycle
|
W = 2 * math.pi / 86400 # angular frequency of a 24 h cycle, rad/s
|
||||||
RETURN = 0.05 # return of the wave, lower: the cooling tail is slow (wf-improve NORDEND 2026-11-05, S1)
|
DAY = timedelta(days=1)
|
||||||
SLIDE_K = 3 # common slide, in noises (median of the steps) away from the median of the day before
|
TEXTS = {"Thermal expansion": "The structure expands in the day heat.",
|
||||||
COLD = 4 # the camera moves with the cold, in one-hour noises away from the day before (pose, temperature)
|
"Thermal contraction": "The structure contracts as it cools down.",
|
||||||
W = 2 * math.pi / 86400 # angular frequency of a 24 h cycle, in rad/s
|
"Camera thermal expansion": "The camera support expands in the day heat; the curves are corrected for it.",
|
||||||
|
"Camera thermal contraction": "The camera support contracts as it cools down; the curves are corrected for it."}
|
||||||
|
|
||||||
|
|
||||||
def _corr(a, b):
|
def _corr(a, b):
|
||||||
@@ -84,20 +54,16 @@ def _axis(ms, col):
|
|||||||
for m in ms:
|
for m in ms:
|
||||||
if m[col] is not None:
|
if m[col] is not None:
|
||||||
days.setdefault(m["t"].date(), {}).setdefault(m["t"].hour, []).append(m[col])
|
days.setdefault(m["t"].date(), {}).setdefault(m["t"].hour, []).append(m[col])
|
||||||
full, every = {}, {}
|
full = {}
|
||||||
for d, hs in days.items():
|
for d, hs in days.items():
|
||||||
if len(hs) >= PARTIAL:
|
if len(hs) >= MIN_HOURS:
|
||||||
med = statistics.median(v for l in hs.values() for v in l)
|
med = statistics.median(v for vs in hs.values() for v in vs)
|
||||||
every[d] = {h: statistics.mean(l) - med for h, l in hs.items()}
|
full[d] = {h: statistics.mean(vs) - med for h, vs in hs.items()}
|
||||||
if len(hs) >= MIN_HOURS:
|
if len(full) < 2 or any(all(h not in q for q in full.values()) for h in range(24)):
|
||||||
full[d] = every[d]
|
|
||||||
if len(full) < 2:
|
|
||||||
return None
|
|
||||||
profile = [statistics.median(q[h] for q in full.values() if h in q) for h in range(24) if any(h in q for q in full.values())]
|
|
||||||
if len(profile) < 24:
|
|
||||||
return None
|
return None
|
||||||
|
profile = [statistics.median(q[h] for q in full.values() if h in q) for h in range(24)]
|
||||||
likeness = {}
|
likeness = {}
|
||||||
for d, p in every.items():
|
for d, p in full.items():
|
||||||
others = [q for e, q in full.items() if e != d]
|
others = [q for e, q in full.items() if e != d]
|
||||||
hs = [h for h in p if all(h in q for q in others)]
|
hs = [h for h in p if all(h in q for q in others)]
|
||||||
if len(hs) >= 3:
|
if len(hs) >= 3:
|
||||||
@@ -105,191 +71,89 @@ def _axis(ms, col):
|
|||||||
return profile, likeness
|
return profile, likeness
|
||||||
|
|
||||||
|
|
||||||
def sine_r2(ms, col):
|
def _sine_r2(ms, col):
|
||||||
"""Share of the curve explained by a 24 h sinusoid (least squares, 3 unknowns), or None if too short."""
|
"""Share of the curve explained by a 24 h sinusoid (its projection on the sine and cosine of the day: the
|
||||||
|
measurements cover the hours evenly), or None if too short."""
|
||||||
pts = [(m["t"].timestamp(), m[col]) for m in ms if m[col] is not None]
|
pts = [(m["t"].timestamp(), m[col]) for m in ms if m[col] is not None]
|
||||||
if len(pts) <= 48:
|
if len(pts) <= 48:
|
||||||
return None
|
return None
|
||||||
rows = [(1.0, math.sin(W * t), math.cos(W * t)) for t, _ in pts]
|
mean = statistics.fmean(y for _, y in pts)
|
||||||
A = [[sum(r[i] * r[j] for r in rows) for j in range(3)] for i in range(3)]
|
b = 2 * statistics.fmean((y - mean) * math.sin(W * t) for t, y in pts)
|
||||||
B = [sum(r[i] * y for r, (_, y) in zip(rows, pts)) for i in range(3)]
|
c = 2 * statistics.fmean((y - mean) * math.cos(W * t) for t, y in pts)
|
||||||
for i in range(3): # Gaussian elimination
|
sst = sum((y - mean) ** 2 for _, y in pts)
|
||||||
for k in range(i + 1, 3):
|
sse = sum((y - mean - b * math.sin(W * t) - c * math.cos(W * t)) ** 2 for t, y in pts)
|
||||||
f = A[k][i] / A[i][i]
|
|
||||||
A[k] = [x - f * y for x, y in zip(A[k], A[i])]
|
|
||||||
B[k] -= f * B[i]
|
|
||||||
x = [0.0] * 3
|
|
||||||
for i in (2, 1, 0):
|
|
||||||
x[i] = (B[i] - sum(A[i][j] * x[j] for j in range(i + 1, 3))) / A[i][i]
|
|
||||||
m = statistics.mean(y for _, y in pts)
|
|
||||||
sst = sum((y - m) ** 2 for _, y in pts)
|
|
||||||
sse = sum((y - sum(c * v for c, v in zip(x, r))) ** 2 for r, (_, y) in zip(rows, pts))
|
|
||||||
return 1 - sse / sst if sst else None
|
return 1 - sse / sst if sst else None
|
||||||
|
|
||||||
|
|
||||||
def sine(measurements, cols=("corr_x", "corr_y")):
|
def _wave(measurements, cols, name, camera):
|
||||||
"""-> (targets with an axis following a 24 h sinusoid, targets evaluated)."""
|
targets = {s: ms for s, ms in measurements.items() if not is_reference(ms)}
|
||||||
yes, evaluated = set(), set()
|
axes = [(s, *a) for s, ms in targets.items() for col in cols if (a := _axis(ms, col))] # (target, profile, likeness)
|
||||||
for s, ms in measurements.items():
|
concerned = [a for a in axes if a[2] and statistics.median(a[2].values()) >= REPRO]
|
||||||
if ms[0]["pattern_type"] == "global_ref":
|
if not axes or len({a[0] for a in concerned}) < SHARE * len({a[0] for a in axes}):
|
||||||
continue
|
return []
|
||||||
for col in cols:
|
if not camera: # the sinusoid must see it too
|
||||||
r2 = sine_r2(ms, col)
|
r2 = {s: [r for col in cols if (r := _sine_r2(ms, col)) is not None] for s, ms in targets.items()}
|
||||||
if r2 is not None:
|
r2 = {s: v for s, v in r2.items() if v}
|
||||||
evaluated.add(s)
|
if not r2 or sum(max(v) >= R2_MIN for v in r2.values()) < SHARE * len(r2):
|
||||||
if r2 >= R2_MIN:
|
return []
|
||||||
yes.add(s)
|
|
||||||
return yes, evaluated
|
|
||||||
|
|
||||||
|
def gap(p, h): # from the night level, as a share of the amplitude
|
||||||
def cycle(measurements, cols=("corr_x", "corr_y")):
|
|
||||||
"""-> (axes concerned [(sensor, col, profile, likeness)], number of targets evaluated)."""
|
|
||||||
concerned, targets = [], set()
|
|
||||||
for s, ms in measurements.items():
|
|
||||||
if ms[0]["pattern_type"] == "global_ref":
|
|
||||||
continue
|
|
||||||
for col in cols:
|
|
||||||
r = _axis(ms, col)
|
|
||||||
if r is None:
|
|
||||||
continue
|
|
||||||
targets.add(s)
|
|
||||||
if statistics.median(r[1].values()) >= REPRO:
|
|
||||||
concerned.append((s, col) + r)
|
|
||||||
return concerned, len(targets)
|
|
||||||
|
|
||||||
|
|
||||||
def gaps(concerned):
|
|
||||||
"""Median gap (over the axes concerned) from the night level, hour by hour, as a share of the profile's amplitude."""
|
|
||||||
def score(p, h):
|
|
||||||
start = min(range(24), key=lambda i: max(p[(i + k) % 24] for k in range(NIGHT)) - min(p[(i + k) % 24] for k in range(NIGHT)))
|
start = min(range(24), key=lambda i: max(p[(i + k) % 24] for k in range(NIGHT)) - min(p[(i + k) % 24] for k in range(NIGHT)))
|
||||||
night = statistics.median(p[(start + k) % 24] for k in range(NIGHT))
|
return abs(p[h] - statistics.median(p[(start + k) % 24] for k in range(NIGHT))) / (max(p) - min(p))
|
||||||
return abs(p[h] - night) / (max(p) - min(p))
|
e = [statistics.median(gap(a[1], h) for a in concerned) for h in range(24)]
|
||||||
return [statistics.median(score(c[2], h) for c in concerned) for h in range(24)]
|
peak, rest = max(range(24), key=e.__getitem__), min(range(24), key=e.__getitem__)
|
||||||
|
if e[peak] < WARM:
|
||||||
|
return []
|
||||||
|
a = peak
|
||||||
|
while (a - 1) % 24 != rest and e[(a - 1) % 24] >= EDGE:
|
||||||
|
a -= 1
|
||||||
|
b = peak + 1
|
||||||
|
while b < a + 24 and e[b % 24] >= RETURN:
|
||||||
|
b += 1
|
||||||
|
last = max(m["t"] for ms in measurements.values() for m in ms)
|
||||||
|
sensors = measurements if camera else {a[0] for a in concerned}
|
||||||
|
out = []
|
||||||
|
for d in sorted({d for a in concerned for d in a[2]}):
|
||||||
|
midnight = datetime(d.year, d.month, d.day, tzinfo=timezone.utc)
|
||||||
|
for label, u, v in ((f"{name} expansion", a, peak), (f"{name} contraction", peak, b)):
|
||||||
|
start, end = midnight + timedelta(hours=u), min(midnight + timedelta(hours=v), last)
|
||||||
|
if end > start:
|
||||||
|
out.append(episode(start, end, "Thermal displacement", label,
|
||||||
|
TEXTS[label] + " This daily wave is normal: nothing to do.", sensors, False))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
def warm_hours(concerned):
|
def _signs(ms, col):
|
||||||
return [h for h, e in enumerate(gaps(concerned)) if e >= WARM]
|
"""{instant: +1 or -1} where the value moves away by at least SLIDE_K noises from its median of the day before."""
|
||||||
|
|
||||||
|
|
||||||
def detect(measurements, temperature=(), camera=()):
|
|
||||||
"""Cycle of the corrected curves (the structure); otherwise, that of the raw positions (the camera moving with the
|
|
||||||
expansion of its mount, compensated: D-074); otherwise, the camera slides (with the cold)."""
|
|
||||||
return (_episodes(measurements, ("corr_x", "corr_y"), "Thermal", False) or
|
|
||||||
_episodes(measurements, ("x", "y"), "Camera thermal", True) or slides(measurements, temperature, camera))
|
|
||||||
|
|
||||||
|
|
||||||
def _signs(ms, col, hourly=False, k=None):
|
|
||||||
"""{instant: +1 or -1} where the measurement moves away by at least k (default SLIDE_K) noises from the median of the
|
|
||||||
day before; noise: the median of the steps from one measurement to the next or, hourly, of the variations over an
|
|
||||||
hour (one step if the cadence is slow)."""
|
|
||||||
pts = [(m["t"], m[col]) for m in ms if m[col] is not None]
|
pts = [(m["t"], m[col]) for m in ms if m[col] is not None]
|
||||||
if len(pts) < 3:
|
n, out, j = noise([v for _, v in pts]), {}, 0
|
||||||
return {}
|
|
||||||
step = statistics.median((b - a).total_seconds() for (a, _), (b, _) in zip(pts, pts[1:])) if hourly else 0
|
|
||||||
n = max(1, round(3600 / step)) if step else 1
|
|
||||||
noise = statistics.median(abs(pts[i + n][1] - pts[i][1]) for i in range(len(pts) - n)) if len(pts) > n else 0
|
|
||||||
out, j = {}, 0
|
|
||||||
for i, (t, v) in enumerate(pts):
|
for i, (t, v) in enumerate(pts):
|
||||||
while pts[j][0] < t - timedelta(days=1):
|
while pts[j][0] < t - DAY:
|
||||||
j += 1
|
j += 1
|
||||||
before = [w for _, w in pts[j:i]]
|
before = [w for _, w in pts[j:i]]
|
||||||
if noise and len(before) >= 3 and abs(v - statistics.median(before)) >= (k or SLIDE_K) * noise:
|
if n and len(before) >= 3 and abs(v - (med := statistics.median(before))) >= SLIDE_K * n:
|
||||||
out[t] = 1 if v > statistics.median(before) else -1
|
out[t] = 1 if v > med else -1
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
def _common(series):
|
def _common(series):
|
||||||
"""[{instant: sign}] of a group of sensors -> instants where at least SHARE of them move away on the same side."""
|
"""[{instant: sign}] of a group of sensors -> instants where at least SHARE of them move away on the same side."""
|
||||||
n, count = len(series), {}
|
count = {}
|
||||||
for sg in series:
|
for sg in series:
|
||||||
for t, x in sg.items():
|
for t, x in sg.items():
|
||||||
count[(t, x)] = count.get((t, x), 0) + 1
|
count[(t, x)] = count.get((t, x), 0) + 1
|
||||||
return {t for (t, x), c in count.items() if n and c >= SHARE * n}
|
return {t for (t, x), c in count.items() if c >= SHARE * len(series)}
|
||||||
|
|
||||||
|
|
||||||
def _cold(measurements, temperature, camera):
|
def slides(measurements):
|
||||||
"""[(start, end)] of the runs of instants where the camera pose (tilt, rotation or scale) moves away from its day
|
targets = [ms for ms in measurements.values() if not is_reference(ms)]
|
||||||
before while the temperature falls under its own, by at least COLD one-hour noises; empty without one of them."""
|
|
||||||
ts = [m["t"] for ms in measurements.values() for m in ms]
|
|
||||||
hour = lambda t: t.replace(minute=0, second=0, microsecond=0) # noqa: E731
|
|
||||||
cold = {hour(t) for t, x in _signs([{"t": t, "T": v} for t, v in temperature], "T", True, COLD).items() if x < 0}
|
|
||||||
pose = [p for p in camera if min(ts) <= p["t"] <= max(ts)] if ts else []
|
|
||||||
seen = {t for c in ("tilt", "rotation", "scale") for t in _signs(pose, c, True, COLD) if hour(t) in cold}
|
|
||||||
episodes = [] # consecutive pose instants make one episode
|
|
||||||
for i, p in enumerate(pose):
|
|
||||||
if p["t"] in seen:
|
|
||||||
if episodes and episodes[-1][2] == i - 1:
|
|
||||||
episodes[-1][1:] = [p["t"], i]
|
|
||||||
else:
|
|
||||||
episodes.append([p["t"], p["t"], i])
|
|
||||||
return [(a, b) for a, b, _ in episodes]
|
|
||||||
|
|
||||||
|
|
||||||
def slides(measurements, temperature=(), camera=()):
|
|
||||||
"""Common camera slide, without a 24 h cycle: the raw positions of most sensors, references included, and the
|
|
||||||
corrected curves of most targets move away together, on the same side, from their day before; or the camera pose
|
|
||||||
moves away from it with the cold (_cold)."""
|
|
||||||
targets = [ms for ms in measurements.values() if ms[0]["pattern_type"] != "global_ref"]
|
|
||||||
raw = set().union(*(_common([_signs(ms, c) for ms in measurements.values()]) for c in ("x", "y")))
|
raw = set().union(*(_common([_signs(ms, c) for ms in measurements.values()]) for c in ("x", "y")))
|
||||||
corr = set().union(*(_common([_signs(ms, c) for ms in targets]) for c in ("corr_x", "corr_y")))
|
corr = set().union(*(_common([_signs(ms, c) for ms in targets]) for c in ("corr_x", "corr_y")))
|
||||||
episodes = []
|
return [episode(r[0], r[-1], "Thermal displacement", "Camera thermal movement", "The camera moved with a change of "
|
||||||
for t in sorted(raw & corr):
|
"temperature and all the curves moved with it; this is not a movement of the structure, nothing to "
|
||||||
if episodes and t - episodes[-1][1] <= timedelta(days=1): # less than a day between two: the same slide
|
"do.", measurements, False) for r in runs(sorted(raw & corr), DAY)]
|
||||||
episodes[-1][1] = t
|
|
||||||
else:
|
|
||||||
episodes.append([t, t])
|
|
||||||
return [{"start": iso(a), "end": iso(b), "kind": "Thermal displacement", "label": "Camera thermal movement",
|
|
||||||
"text": "The camera moved with a change of temperature and all the curves moved with it; this is not a "
|
|
||||||
"movement of the structure, nothing to do.", "ignore": False, "sensors": sorted(measurements)}
|
|
||||||
for a, b in episodes + _cold(measurements, temperature, camera)]
|
|
||||||
|
|
||||||
|
|
||||||
def _episodes(measurements, cols, name, camera):
|
def detect(measurements):
|
||||||
concerned, n = cycle(measurements, cols)
|
return (_wave(measurements, ("corr_x", "corr_y"), "Thermal", False)
|
||||||
sensors = sorted({c[0] for c in concerned})
|
or _wave(measurements, ("x", "y"), "Camera thermal", True) or slides(measurements))
|
||||||
if not n or len(sensors) < SHARE * n:
|
|
||||||
return []
|
|
||||||
if not camera: # second method, for the structure only: the sinusoid must see it too
|
|
||||||
yes, evaluated = sine(measurements, cols)
|
|
||||||
if not evaluated or len(yes) < SHARE * len(evaluated):
|
|
||||||
return []
|
|
||||||
warm = warm_hours(concerned)
|
|
||||||
if len(warm) < MIN_WARM_HOURS:
|
|
||||||
return []
|
|
||||||
e = gaps(concerned)
|
|
||||||
peak = max(warm, key=lambda h: e[h]) # the largest gap from the night level: end of the expansion
|
|
||||||
rest = min(range(24), key=lambda h: e[h])
|
|
||||||
a = peak # start: going back from the peak while the profile stays above EDGE, up to the calmest hour
|
|
||||||
while (a - 1) % 24 != rest and e[(a - 1) % 24] >= EDGE:
|
|
||||||
a -= 1
|
|
||||||
b = peak + 1 # return: until the profile goes back under RETURN, at most to the next start
|
|
||||||
while b < a + 24 and e[b % 24] >= RETURN:
|
|
||||||
b += 1
|
|
||||||
last = max(m["t"] for ms in measurements.values() for m in ms)
|
|
||||||
days = sorted({d for c in concerned for d in c[3]})
|
|
||||||
if camera: # the whole image moves, references included
|
|
||||||
sensors = sorted(measurements)
|
|
||||||
episodes = []
|
|
||||||
for d in days: # ponytail: days of at least PARTIAL hours, the export's ends (22:00 → midnight) get no episode
|
|
||||||
if statistics.median(c[3][d] for c in concerned if d in c[3]) < DAY_MATCH:
|
|
||||||
continue
|
|
||||||
midnight = datetime(d.year, d.month, d.day, tzinfo=timezone.utc)
|
|
||||||
for label, u, v in ((f"{name} expansion", a, peak), (f"{name} contraction", peak, b)):
|
|
||||||
start, end = midnight + timedelta(hours=u), min(midnight + timedelta(hours=v), last)
|
|
||||||
if end > start:
|
|
||||||
episodes.append({"start": iso(start), "end": iso(end),
|
|
||||||
"kind": "Thermal displacement", "label": label, "text": "", "ignore": False, "sensors": sensors})
|
|
||||||
return episodes
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
measurements = load(sys.argv[1])
|
|
||||||
for e in detect(measurements):
|
|
||||||
print(e["start"], "→", e["end"], e["label"], len(e["sensors"]), "targets")
|
|
||||||
for cols in (("corr_x", "corr_y"), ("x", "y")):
|
|
||||||
concerned, n = cycle(measurements, cols)
|
|
||||||
k = len({c[0] for c in concerned})
|
|
||||||
amp = statistics.median(max(c[2]) - min(c[2]) for c in concerned) if concerned else 0
|
|
||||||
yes, evaluated = sine(measurements, cols)
|
|
||||||
print(f"{cols[0][:-2] or 'raw'}: targets {k}/{n}, median amplitude {amp:.3f} px, warm hours (UTC) "
|
|
||||||
f"{warm_hours(concerned) if concerned else '-'}; sinusoid: targets {len(yes)}/{len(evaluated)}")
|
|
||||||
+115
-303
@@ -1,113 +1,34 @@
|
|||||||
"""Detector of the Visibility family (the families document): the camera sees less well, the image is faint or blurred
|
"""Visibility family: the camera sees the targets less well (error_value), or the air trembles (heat shimmer). The same
|
||||||
(error_value), or it trembles (heat shimmer, end of file). The same error_value computation also finds objects in front
|
error_value also finds something passing in front of a few targets: Brief obstruction (Obstruction family).
|
||||||
of a few targets (Brief obstruction, Target obscured): episodes() returns them with the Obstruction family, which
|
|
||||||
detect() leaves to obstruction.py.
|
|
||||||
|
|
||||||
Method:
|
1. High measurement: error_value above the median of its sensor + K_MAD MAD and above FACTOR times that median.
|
||||||
1. "High" measurement: error_value > median + K_MAD MAD of the sensor over the export, and > FACTOR × median, or
|
2. Brief obstruction (one instant): at least BRIEF_MIN sensors high, SALIENCE times more than at each neighbouring instant
|
||||||
rejected (> error_threshold) and > REJECT_FACTOR × median: a sensor at a usual level of 0.6 never doubles, and
|
(a train, a vehicle); or fewer than BRIEF_MIN sensors, at most one reference, rejected at FACTOR times their median
|
||||||
FLORIDA (batch 01) showed its fogs as Target obscured.
|
and not at the neighbouring instants (something passing in front of one or two targets).
|
||||||
2. Brief obstruction: an instant where at least BRIEF_MIN sensors are high, SALIENCE times more than at the neighbouring
|
3. Poor visibility: runs of at least RUN_MIN high measurements of a sensor (steps of at most GAP_STEPS), outside the brief
|
||||||
instants (train, vehicle), with no reference (global_ref) among them: a one-step peak touching the references is not
|
instants, overlapping on at least two sensors. Level of each instant: "none" (greyed) when at least HALF of the
|
||||||
an obstacle in front of targets (peak of the whole image, NORDEND on 01.09 11:15, batch 01); it is not shown. Unless
|
usually seen targets are rejected, "reduced" otherwise; one episode per period of the same level.
|
||||||
at least BRIEF_MIN targets are rejected at this instant and neither at the instant before nor after: something
|
4. Diffuse nuisance: at least RUN_MIN instants in a row where the median over the sensors of the error_value, in MAD
|
||||||
passing in front of them, a reference touched as well (wf-improve session Displacement, mod. 1, S5: trains of
|
above each one's median, reaches K_DIFFUSE: Poor visibility "reduced", unless within heat shimmer or another Poor
|
||||||
NIEDERHORN, 12 and 13 targets rejected).
|
visibility.
|
||||||
Also, something passing in front of one or two targets, or a single reference: at an instant, fewer than BRIEF_MIN
|
Heat shimmer: end of file.
|
||||||
sensors rejected at more than FACTOR times their median (without K_MAD: by day, heat shimmer inflates the MAD), at
|
|
||||||
most one reference, none of them rejected at the instant before or after, and fewer than BRIEF_MIN high sensors
|
|
||||||
(wf-improve session Obstruction, mod. 1, S3: vehicles, passers-by, insects in front of one or two targets of
|
|
||||||
WILDSTRUBEL, NIEDERHORN, LENZSPITZE; a reference hidden by a truck at JUNGFRAU, BREITHORN).
|
|
||||||
3. Poor visibility: runs of at least RUN_MIN high measurements per sensor, joined across sensors (at least 2), reduced
|
|
||||||
to the instants where at least FRAC of the episode's peak is high.
|
|
||||||
4. Target obscured: a target (not a global_ref) rejected (> error_threshold) and above K_TARGET MAD, at least TARGET_MIN
|
|
||||||
times less than TARGET_GAP apart, mostly outside the episodes of 2 and 3 (leaves, snow, glare on a target). The gap
|
|
||||||
is at least two measurement steps, and two rejections are enough when TARGET_MIN steps do not fit in TARGET_GAP
|
|
||||||
(wf-improve session Obstruction, mod. 1, S6: at one measurement every 4 h, vehicles parked at night in front of COLI22,
|
|
||||||
COLI24, POUI22 of BREITHORN make only two or three rejections). All episodes: ignore = True.
|
|
||||||
5. Level of a Poor visibility (validated 2026-10-24), measurement by measurement: "none" (nothing is seen) when at least
|
|
||||||
HALF of the usually seen targets (under their threshold at least half the time) are rejected, "reduced" (average
|
|
||||||
data) otherwise; one episode per period of the same level, with the level field.
|
|
||||||
6. Bounds of a Poor visibility (WILDSTRUBEL, batch 01): from the first to the last measurement where a target is
|
|
||||||
rejected (> error_threshold), and no episode without a rejected target. On a road in full sun, the daytime
|
|
||||||
error_value (×1.5 to ×3, heat shimmer) made reduced visibility last for hours without any rejection; references do
|
|
||||||
not count: R3 is above its threshold almost all day.
|
|
||||||
7. Poor visibility or object (boundary "Visibility or Obstruction" of the families document; BREITHORN, batch 01): a
|
|
||||||
nuisance touching at most FEW targets, and no reference, is not diffuse: it is an object in front of them (a parked
|
|
||||||
vehicle), a "Target obscured" episode of its whole duration, without a level.
|
|
||||||
8. Diffuse nuisance, with or without rejections (wf-improve session Visibility, mod. 1, S2): at each instant, the median,
|
|
||||||
over the sensors, of the error_value in MAD above each one's median; at least RUN_MIN instants in a row at K_DIFFUSE
|
|
||||||
or more make a Poor visibility "reduced". A diffuse nuisance entirely within heat shimmer is the heat shimmer, and
|
|
||||||
within a Poor visibility of point 3, that one (the same instants twice, two events: wf-improve, batch 2026-11-05, S2):
|
|
||||||
no episode (otherwise it absorbed the jump back of the day before, WILDSTRUBEL on 04.10). Finds the drops of
|
|
||||||
visibility of the whole scene without rejection at JUNGFRAU, FLORIDA, NORDEND; at 2, nights labelled thermal at
|
|
||||||
JUNGFRAU; at 3, half the gain.
|
|
||||||
|
|
||||||
A simpler version (share of targets rejected above their threshold, without comparing each sensor with its own level)
|
|
||||||
was tried on 2026-10-24 and dropped: worse on five data sets out of six (git history of Support-Agent,
|
|
||||||
detectors/archive/visibilite_simple.py).
|
|
||||||
"""
|
"""
|
||||||
import os
|
|
||||||
import statistics
|
import statistics
|
||||||
import sys
|
|
||||||
from datetime import timedelta
|
from datetime import timedelta
|
||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
|
from episodes import cadence, episode, level, runs, targets
|
||||||
from data import iso, load # noqa: E402
|
from data import is_reference
|
||||||
|
|
||||||
K_MAD = 10 # tuned on E-001 (fog: 11 to 57 MAD) and E-005 (rain, trains: 10 to 70 MAD)
|
K_MAD = 10 # E-001 (fog: 11 to 57 MAD), E-005 (rain, trains: 10 to 70 MAD)
|
||||||
FACTOR = 2 # drops sensors with a near-zero MAD (E-004: PWD_REF10 ×1.6 at 14 MAD, E-005)
|
FACTOR = 2 # drops the sensors with a near-zero MAD (E-004, E-005)
|
||||||
RUN_MIN = 3 # high measurements in a row for poor visibility (E-001, E-005 rain)
|
RUN_MIN = 3 # E-001, rain of E-005
|
||||||
BRIEF_MIN = 3 # tuned on E-005 (trains: 5 to 17 sensors; the one with 2 targets is missed)
|
GAP_STEPS = 2.5 # steps tolerated inside a run (a missing step, timestamp jitter)
|
||||||
K_TARGET = 5 # tuned on E-005 (leaves: 6 to 25 MAD, rejected)
|
BRIEF_MIN = 3 # trains of E-005: 5 to 17 sensors
|
||||||
TARGET_MIN = 3 # repeated rejections of a target
|
SALIENCE = 3 # a train stands out (E-005: 5 to 17 against 0 to 2); the cobweb of E-002 varies from 2 to 9
|
||||||
FRAC = 0.15 # cuts the tails of PR09 + CTR01 (NORDEND October) without cutting the cobweb (E-002)
|
HALF = 0.5 # "nothing is seen": NORDEND 09.10 23:00 (68 % rejected) against the rain of E-005 (24 to 33 %)
|
||||||
GAP_STEPS = 2.5 # steps tolerated between two high measurements of the same run (a missing step, clock to the second)
|
K_DIFFUSE = 2.5 # wf-improve Visibility, mod. 1, S2 (JUNGFRAU, FLORIDA, NORDEND: the whole scene less clear)
|
||||||
BREAK = timedelta(hours=2) # lull separating two episodes (NORDEND October, night of 13 to 14.10)
|
|
||||||
SALIENCE = 3 # a train stands out at least 3 times above the neighbouring instants (E-005: 5 to 17 against 0 to 2;
|
|
||||||
# the cobweb of E-002 varies from 2 to 9 from one step to the next)
|
|
||||||
TARGET_GAP = timedelta(hours=6) # tuned on E-005 (leaves: 7 rejections in 12 h, up to 5 h apart)
|
|
||||||
REJECT_FACTOR = 1.2 # a rejection at 10 MAD also counts (FLORIDA, batch 01: levels of 0.6, ×2 out of reach); drops C24
|
|
||||||
# of E-009 (level 0.68 just under the threshold 0.7, rejected at ×1.04)
|
|
||||||
HALF = 0.5 # "nothing is seen"; tuned on NORDEND (09.10 23:00: 68 % rejected) and E-005 (rain: 24 to 33 %)
|
|
||||||
K_DIFFUSE = 2.5 # diffuse nuisance (8); wf-improve session Visibility, mod. 1, S2
|
|
||||||
FEW = 3 # Poor visibility on at most 3 targets, no reference: an object, Target obscured (7); BREITHORN,
|
|
||||||
# batch 01 (vehicles parked in front of 2 or 3 targets out of 12)
|
|
||||||
|
|
||||||
|
BRIEF = "Something passed in front of {} for a single measurement. Ignore this point on the curves."
|
||||||
def _step(measurements):
|
|
||||||
ts = sorted({m["t"] for v in measurements.values() for m in v})
|
|
||||||
return statistics.median(b - a for a, b in zip(ts, ts[1:]))
|
|
||||||
|
|
||||||
|
|
||||||
def _group(points, gap):
|
|
||||||
"""[(t, sensor)] sorted by t -> groups [(start, end, {sensors}, [t])] whose points are ≤ gap from each other."""
|
|
||||||
groups = []
|
|
||||||
for t, s in points:
|
|
||||||
if groups and t - groups[-1][1] <= gap:
|
|
||||||
g = groups[-1]
|
|
||||||
groups[-1] = (g[0], t, g[2] | {s}, g[3] + [t])
|
|
||||||
else:
|
|
||||||
groups.append((t, t, {s}, [t]))
|
|
||||||
return groups
|
|
||||||
|
|
||||||
|
|
||||||
def _group_intervals(intervals, gap):
|
|
||||||
"""[(start, end, sensor)] sorted -> [(start, end, {sensors})] of the intervals that overlap (within gap)."""
|
|
||||||
out = []
|
|
||||||
for a, b, s in intervals:
|
|
||||||
if out and a - out[-1][1] <= gap:
|
|
||||||
out[-1] = (out[-1][0], max(out[-1][1], b), out[-1][2] | {s})
|
|
||||||
else:
|
|
||||||
out.append((a, b, {s}))
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _episode(start, end, label, text, sensors, level=None):
|
|
||||||
e = {"start": iso(start), "end": iso(end), "kind": "Visibility" if label == "Poor visibility" else "Obstruction",
|
|
||||||
"label": label, "text": text,
|
|
||||||
"ignore": level != "reduced", "sensors": sorted(sensors)} # reduced visibility: average data, not greyed
|
|
||||||
return {**e, "level": level} if level else e
|
|
||||||
|
|
||||||
|
|
||||||
def _levels(measurements, instants):
|
def _levels(measurements, instants):
|
||||||
@@ -121,224 +42,115 @@ def _levels(measurements, instants):
|
|||||||
for t in instants}
|
for t in instants}
|
||||||
|
|
||||||
|
|
||||||
def episodes(measurements):
|
def _poor(start, end, sensors, level, refs):
|
||||||
"""All error_value episodes: Visibility (Poor visibility), Obstruction (Brief obstruction, Target obscured)."""
|
refs = ", including the reference targets" if set(sensors) & refs else ""
|
||||||
step = _step(measurements)
|
text = (f"The camera could not see {targets(len(sensors))} clearly{refs}. The curves in this period are not "
|
||||||
gap = GAP_STEPS * step
|
"reliable: ignore them." if level == "none" else f"The camera saw {targets(len(sensors))} less clearly than "
|
||||||
refs = {s for s, v in measurements.items() if v[0]["pattern_type"] == "global_ref"}
|
f"usual{refs}. The data in this period are less precise: use them with care.")
|
||||||
instants = sorted({m["t"] for v in measurements.values() for m in v})
|
return episode(start, end, "Visibility", "Poor visibility", text, sensors, level == "none", level=level)
|
||||||
high, rejections = {}, {} # sensor -> [t]
|
|
||||||
refused = set() # instants where a target (not a global_ref) is above its threshold
|
|
||||||
rejected_targets = {} # t -> {targets above their threshold}
|
|
||||||
strong, refusals = {}, {} # t -> {sensors rejected at more than FACTOR times their median}, {sensors rejected}
|
|
||||||
for s, v in measurements.items():
|
|
||||||
e = [m["error_value"] for m in v if m["error_value"] is not None]
|
|
||||||
if len(e) < 10:
|
|
||||||
continue
|
|
||||||
med = statistics.median(e)
|
|
||||||
mad = statistics.median(abs(x - med) for x in e) or 1e-9
|
|
||||||
for m in v:
|
|
||||||
x = m["error_value"]
|
|
||||||
if x is None:
|
|
||||||
continue
|
|
||||||
rejected = m["error_threshold"] and x > m["error_threshold"]
|
|
||||||
if rejected and s not in refs:
|
|
||||||
refused.add(m["t"])
|
|
||||||
rejected_targets.setdefault(m["t"], set()).add(s)
|
|
||||||
if x > med + K_MAD * mad and (x > FACTOR * med or rejected and x > REJECT_FACTOR * med):
|
|
||||||
high.setdefault(s, []).append(m["t"])
|
|
||||||
if rejected:
|
|
||||||
refusals.setdefault(m["t"], set()).add(s)
|
|
||||||
if x > FACTOR * med:
|
|
||||||
strong.setdefault(m["t"], set()).add(s)
|
|
||||||
if s not in refs and rejected and x > med + K_TARGET * mad:
|
|
||||||
rejections.setdefault(s, []).append(m["t"])
|
|
||||||
by_t = {} # t -> {high sensors}
|
|
||||||
for s, ts in high.items():
|
|
||||||
for t in ts:
|
|
||||||
by_t.setdefault(t, set()).add(s)
|
|
||||||
n = lambda t: len(by_t.get(t, ())) # noqa: E731
|
|
||||||
out = []
|
|
||||||
|
|
||||||
# 2. Brief obstruction: an instant where many sensors are high, at least SALIENCE times more than the neighbours.
|
|
||||||
briefs = set()
|
|
||||||
for i, t in enumerate(instants):
|
|
||||||
neighbours = [n(instants[j]) for j in (i - 1, i + 1) if 0 <= j < len(instants)]
|
|
||||||
if n(t) >= BRIEF_MIN and n(t) >= SALIENCE * max(neighbours):
|
|
||||||
briefs.add(t)
|
|
||||||
only = rejected_targets.get(t, set()).difference(*(rejected_targets.get(instants[j], set()) for j in (i - 1, i + 1)
|
|
||||||
if 0 <= j < len(instants)))
|
|
||||||
if by_t[t] & refs and len(only) < BRIEF_MIN: # references touched without targets rejected at this instant
|
|
||||||
continue # alone: not an obstacle in front of targets, the whole image
|
|
||||||
out.append(_episode(t, t, "Brief obstruction", f"{n(t)} points were poorly seen for a single measurement, then came "
|
|
||||||
"back: probably a train, a vehicle or an animal passing; ignore this point on the curves.", by_t[t]))
|
|
||||||
elif 0 < len(f := strong.get(t, set())) < BRIEF_MIN and n(t) < BRIEF_MIN and len(f & refs) <= 1 and not f & set().union(
|
|
||||||
*(refusals.get(instants[j], set()) for j in (i - 1, i + 1) if 0 <= j < len(instants))):
|
|
||||||
# one or two targets, or a reference, rejected at this single measurement, at FACTOR times their median
|
|
||||||
out.append(_episode(t, t, "Brief obstruction", f"{len(f)} point{'s were' if len(f) > 1 else ' was'} poorly seen "
|
|
||||||
"for a single measurement, then came back: probably something passing; ignore this point on the "
|
|
||||||
"curves.", f))
|
|
||||||
|
|
||||||
# 3. Poor visibility: long runs per sensor (without the brief instants), joined across sensors, then reduced to the
|
|
||||||
# instants where at least FRAC of the episode's peak is high (cuts the tails of one or two sensors and the long
|
|
||||||
# lulls, more than BREAK).
|
|
||||||
runs = []
|
|
||||||
for s, ts in high.items():
|
|
||||||
pts = [(t, s) for t in ts if t not in briefs]
|
|
||||||
runs += [(a, b, s) for a, b, _, l in _group(pts, gap) if len(l) >= RUN_MIN]
|
|
||||||
zones = [z for z in _group_intervals(sorted(runs), gap) if len(z[2]) >= 2]
|
|
||||||
kept = []
|
|
||||||
for a, b, _ in zones:
|
|
||||||
inside = [t for t in instants if a <= t <= b and t not in briefs]
|
|
||||||
peak = max(n(t) for t in inside)
|
|
||||||
pts = [(t, s) for t in inside if n(t) >= max(1, FRAC * peak) for s in by_t[t]]
|
|
||||||
kept += [(d, f, cs) for d, f, cs, l in _group(pts, max(gap, BREAK)) if len(cs) >= 2 and len(set(l)) >= RUN_MIN]
|
|
||||||
# Bounds: from the first to the last rejected target; without a rejection, no episode (6).
|
|
||||||
kept = [(r[0], r[-1], cs) for a, b, cs in kept if (r := [t for t in instants if a <= t <= b and t in refused])]
|
|
||||||
level = _levels(measurements, instants)
|
|
||||||
for a, b, cs in kept: # one episode per period of the same level (5); the text comes from categories.py
|
|
||||||
if len(cs) <= FEW and not cs & refs: # an object in front of a few targets, not a diffuse nuisance (7)
|
|
||||||
out.append(_episode(a, b, "Target obscured", "", cs))
|
|
||||||
continue
|
|
||||||
seg = []
|
|
||||||
for t in [t for t in instants if a <= t <= b] + [None]:
|
|
||||||
if seg and (t is None or level[t] != level[seg[0]]):
|
|
||||||
who = cs & set().union(*(by_t.get(u, set()) for u in seg)) or cs
|
|
||||||
out.append(_episode(seg[0], seg[-1], "Poor visibility", "", who, level[seg[0]]))
|
|
||||||
seg = []
|
|
||||||
if t is not None:
|
|
||||||
seg.append(t)
|
|
||||||
inside_any = lambda t: t in briefs or any(a <= t <= b for a, b, _ in kept) # noqa: E731
|
|
||||||
|
|
||||||
# 4. Target obscured: a target rejected again and again, mostly outside the episodes above.
|
|
||||||
targets = []
|
|
||||||
for s, ts in rejections.items():
|
|
||||||
for a, b, _, l in _group([(t, s) for t in ts], max(TARGET_GAP, 2 * step)):
|
|
||||||
if len(l) >= (TARGET_MIN if TARGET_MIN * step <= TARGET_GAP else 2) and sum(map(inside_any, l)) * 2 < len(l):
|
|
||||||
targets.append((a, b, s))
|
|
||||||
for a, b, cs in _group_intervals(sorted(targets), timedelta(0)):
|
|
||||||
who, it, its = ("One target was", "it", "its") if len(cs) == 1 else (f"{len(cs)} targets were", "them", "their")
|
|
||||||
out.append(_episode(a, b, "Target obscured", f"{who} repeatedly lost while the targets around stayed clear: probably "
|
|
||||||
f"something in front of {it} (leaves, snow) or glare; {its} displacements in this period are unreliable.", cs))
|
|
||||||
return sorted(out, key=lambda e: e["start"])
|
|
||||||
|
|
||||||
|
|
||||||
def diffuse(measurements):
|
|
||||||
"""Diffuse nuisance (8): the runs of at least RUN_MIN instants where the median, over the sensors, of the error_value
|
|
||||||
in MAD above each one's median reaches K_DIFFUSE, with or without rejections -> Poor visibility "reduced"."""
|
|
||||||
z = {} # t -> {sensor: gap in MAD}
|
|
||||||
for s, v in measurements.items():
|
|
||||||
e = [m["error_value"] for m in v if m["error_value"] is not None]
|
|
||||||
if len(e) < 10:
|
|
||||||
continue
|
|
||||||
med = statistics.median(e)
|
|
||||||
mad = statistics.median(abs(x - med) for x in e) or 1e-9
|
|
||||||
for m in v:
|
|
||||||
if m["error_value"] is not None:
|
|
||||||
z.setdefault(m["t"], {})[s] = (m["error_value"] - med) / mad
|
|
||||||
highs = [(t, None) for t in sorted(z) if statistics.median(z[t].values()) >= K_DIFFUSE]
|
|
||||||
return [_episode(a, b, "Poor visibility", "", {s for t in l for s, x in z[t].items() if x >= K_DIFFUSE}, "reduced")
|
|
||||||
for a, b, _, l in _group(highs, GAP_STEPS * _step(measurements)) if len(l) >= RUN_MIN]
|
|
||||||
|
|
||||||
|
|
||||||
def detect(measurements):
|
def detect(measurements):
|
||||||
shimmer = heat_shimmer(measurements) # a diffuse nuisance during heat shimmer is the heat shimmer (8)
|
gap = GAP_STEPS * cadence(measurements)
|
||||||
poor = [e for e in episodes(measurements) if e["kind"] == "Visibility"] # a diffuse nuisance within one of them is it
|
refs = {s for s, v in measurements.items() if is_reference(v)}
|
||||||
return poor + shimmer + [e for e in diffuse(measurements)
|
instants = sorted({m["t"] for v in measurements.values() for m in v})
|
||||||
if not any(t["start"] <= e["start"] and e["end"] <= t["end"] for t in shimmer + poor)]
|
z = {} # t -> {sensor: error_value in MAD above the sensor's median}
|
||||||
|
high, strong, rejected = {}, {}, {} # t -> {sensors}
|
||||||
|
for s, v in measurements.items():
|
||||||
|
e = [m["error_value"] for m in v if m["error_value"] is not None]
|
||||||
|
if len(e) < 10:
|
||||||
|
continue
|
||||||
|
med, mad = level(e)
|
||||||
|
for m in v:
|
||||||
|
t, x = m["t"], m["error_value"]
|
||||||
|
if x is None:
|
||||||
|
continue
|
||||||
|
z.setdefault(t, {})[s] = (x - med) / mad
|
||||||
|
if x > med + K_MAD * mad and x > FACTOR * med:
|
||||||
|
high.setdefault(t, set()).add(s)
|
||||||
|
if m["error_threshold"] and x > m["error_threshold"]:
|
||||||
|
rejected.setdefault(t, set()).add(s)
|
||||||
|
if x > FACTOR * med:
|
||||||
|
strong.setdefault(t, set()).add(s)
|
||||||
|
n = lambda t: len(high.get(t, ())) # noqa: E731
|
||||||
|
out, briefs = [], set()
|
||||||
|
|
||||||
|
for i, t in enumerate(instants): # 2. Brief obstruction
|
||||||
|
around = [instants[j] for j in (i - 1, i + 1) if 0 <= j < len(instants)]
|
||||||
|
few = strong.get(t, set())
|
||||||
|
if n(t) >= BRIEF_MIN and n(t) >= SALIENCE * max(map(n, around), default=0):
|
||||||
|
briefs.add(t)
|
||||||
|
few = high[t]
|
||||||
|
elif not (few and len(few) < BRIEF_MIN and n(t) < BRIEF_MIN and len(few & refs) <= 1
|
||||||
|
and not few & set().union(*(rejected.get(u, set()) for u in around))):
|
||||||
|
continue
|
||||||
|
out.append(episode(t, t, "Obstruction", "Brief obstruction", BRIEF.format(targets(len(few))), few, True))
|
||||||
|
|
||||||
|
spans = [] # 3. Poor visibility: runs of each sensor, then the overlapping runs of at least two sensors
|
||||||
|
for s in {s for t in high for s in high[t]}:
|
||||||
|
spans += [(r[0], r[-1], s) for r in runs([t for t in instants if s in high.get(t, ()) and t not in briefs], gap)
|
||||||
|
if len(r) >= RUN_MIN]
|
||||||
|
zones = []
|
||||||
|
for a, b, s in sorted(spans):
|
||||||
|
if zones and a - zones[-1][1] <= gap:
|
||||||
|
zones[-1] = (zones[-1][0], max(zones[-1][1], b), zones[-1][2] | {s})
|
||||||
|
else:
|
||||||
|
zones.append((a, b, {s}))
|
||||||
|
zones = [zone for zone in zones if len(zone[2]) >= 2]
|
||||||
|
lv = _levels(measurements, instants)
|
||||||
|
for a, b, cs in zones:
|
||||||
|
inside = [t for t in instants if a <= t <= b]
|
||||||
|
for k, t in enumerate(inside):
|
||||||
|
seg = [t] if k == 0 or lv[t] != lv[inside[k - 1]] else seg + [t]
|
||||||
|
if k == len(inside) - 1 or lv[inside[k + 1]] != lv[t]:
|
||||||
|
out.append(_poor(seg[0], seg[-1], cs & set().union(*(high.get(u, set()) for u in seg)) or cs, lv[t], refs))
|
||||||
|
|
||||||
|
shimmer = heat_shimmer(measurements) # 4. diffuse nuisance
|
||||||
|
hosts = shimmer + [e for e in out if e["kind"] == "Visibility"]
|
||||||
|
for r in runs([t for t in sorted(z) if statistics.median(z[t].values()) >= K_DIFFUSE], gap):
|
||||||
|
e = _poor(r[0], r[-1], {s for t in r for s, x in z[t].items() if x >= K_DIFFUSE}, "reduced", refs)
|
||||||
|
if len(r) >= RUN_MIN and not any(h["start"] <= e["start"] and e["end"] <= h["end"] for h in hosts):
|
||||||
|
out.append(e)
|
||||||
|
return out + shimmer
|
||||||
|
|
||||||
|
|
||||||
# --- Heat shimmer ----------------------------------------------------------------------------------------------------
|
# --- Heat shimmer: fast noise of the corrected displacements in warm air.
|
||||||
# Atmospheric turbulence (fast noise on the corrected displacements). Method:
|
# 1. Per sensor and per hour H, over [H - 1 h, H + 2 h): agitation, the median of |Δcorr_x| + |Δcorr_y| between successive
|
||||||
# 1. Per sensor and per hour H, on the measurements of [H − 1 h, H + 2 h): agitation = median of |Δcorr_x| + |Δcorr_y|
|
# measurements, and the median error_value; each divided by its calm (lower quartile of the sensor's hourly values).
|
||||||
# between successive measurements (robust to one-step jumps), and median of the error_value.
|
# 2. Turbulent hour: at least SHARE of the sensors agitated at RATIO times their calm, and median error_value at RATIO.
|
||||||
# 2. Each is divided by the sensor's calm: the lower quartile of its hourly values over the export (often the night).
|
# 3. Turbulent hours at most 2 h apart make an episode, greyed if the median agitation reaches STRONG.
|
||||||
# 3. Turbulent hour: at least SHARE of the sensors at ≥ RATIO times their calm agitation, and median error_value of the
|
RATIO = 1.5 # E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night, error_value ×1.7 to ×3.2)
|
||||||
# sensors at ≥ RATIO_ERR times the calm (warm air blurs the spots: E-003).
|
SHARE = 0.5
|
||||||
# 4. Neighbouring hours merged (a one-hour hole tolerated), episode of at least MIN_DURATION, dropped if more than
|
STRONG = 2.0 # E-003 (×2.3 to ×2.6)
|
||||||
# REJECT_MAX of the measurements are rejected beyond their sensor's habit (its share of rejections over the export):
|
|
||||||
# it is then poor visibility, another rule. A target that is always rejected (Target issue) does not count (wf-improve
|
|
||||||
# session Visibility, mod. 1, S1: WILDSTRUBEL, R3 rejected all day, the days of heat shimmer were dropped).
|
|
||||||
# Greyed if the median agitation ≥ STRONG.
|
|
||||||
RATIO = 1.5 # tuned on E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night)
|
|
||||||
SHARE = 0.5 # share of the sensors agitated at the same time
|
|
||||||
RATIO_ERR = 1.5 # tuned on E-003 (error_value ×1.7 to ×3.2 the night; ≈ ×1 at noon elsewhere)
|
|
||||||
REJECT_MAX = 0.15 # E-003 ≤ 7 %; cobweb (E-002) and rain (E-005) ≥ 18 %; rejections beyond the habit since wf-improve
|
|
||||||
# session Visibility, mod. 1, S1 (WILDSTRUBEL: 4 to 14 %, against 9 to 22 % raw)
|
|
||||||
STRONG = 2.0 # E-003 (×2.3 to ×2.6) is strong; an episode between ×1.5 and ×2 stays visible
|
|
||||||
MIN_DURATION = timedelta(hours=3) # E-003 lasts 10-11 h; drops the morning of the real movement of E-004 (2 h), which
|
|
||||||
# the classification drops too, whatever the duration
|
|
||||||
HOUR = timedelta(hours=1)
|
HOUR = timedelta(hours=1)
|
||||||
|
SHIMMER = "Warm air made the readings shake ({:.1f}× noisier than in calm hours). The small fast wiggles are not movement."
|
||||||
|
|
||||||
def _hour(t):
|
|
||||||
return t.replace(minute=0, second=0, microsecond=0)
|
|
||||||
|
|
||||||
|
|
||||||
def _ratios(points):
|
def _ratios(points):
|
||||||
"""[(t, value)] -> {hour H: median over [H − 1 h, H + 2 h) / lower quartile of these medians}."""
|
"""[(t, value)] -> {hour H: median over [H - 1 h, H + 2 h) / lower quartile of these medians}."""
|
||||||
by = {}
|
by = {}
|
||||||
for t, v in points:
|
for t, v in points:
|
||||||
for k in (-1, 0, 1):
|
for k in (-1, 0, 1):
|
||||||
by.setdefault(_hour(t) + k * HOUR, []).append(v)
|
by.setdefault(t.replace(minute=0, second=0, microsecond=0) + k * HOUR, []).append(v)
|
||||||
med = {h: statistics.median(v) for h, v in by.items() if len(v) >= 3}
|
med = {h: statistics.median(v) for h, v in by.items() if len(v) >= 3}
|
||||||
if len(med) < 8:
|
calm = statistics.quantiles(med.values(), n=4)[0] if len(med) >= 8 else 0
|
||||||
return {}
|
|
||||||
calm = statistics.quantiles(med.values(), n=4)[0]
|
|
||||||
return {h: v / calm for h, v in med.items()} if calm > 0 else {}
|
return {h: v / calm for h, v in med.items()} if calm > 0 else {}
|
||||||
|
|
||||||
|
|
||||||
def hours(measurements):
|
def heat_shimmer(measurements):
|
||||||
"""-> {hour: {sensor: (agitation / calm, error_value / calm)}}."""
|
hours = {} # hour -> {sensor: (agitation / calm, error_value / calm)}
|
||||||
out = {}
|
|
||||||
for s, ms in measurements.items():
|
for s, ms in measurements.items():
|
||||||
ag = _ratios([(b["t"], abs(b["corr_x"] - a["corr_x"]) + abs(b["corr_y"] - a["corr_y"]))
|
ag = _ratios([(b["t"], abs(b["corr_x"] - a["corr_x"]) + abs(b["corr_y"] - a["corr_y"]))
|
||||||
for a, b in zip(ms, ms[1:]) if a["corr_x"] is not None and b["corr_x"] is not None])
|
for a, b in zip(ms, ms[1:]) if a["corr_x"] is not None and b["corr_x"] is not None])
|
||||||
er = _ratios([(m["t"], m["error_value"]) for m in ms if m["error_value"] is not None])
|
er = _ratios([(m["t"], m["error_value"]) for m in ms if m["error_value"] is not None])
|
||||||
for h, r in ag.items():
|
for h in ag.keys() & er.keys():
|
||||||
if h in er:
|
hours.setdefault(h, {})[s] = (ag[h], er[h])
|
||||||
out.setdefault(h, {})[s] = (r, er[h])
|
turbulent = [h for h, r in sorted(hours.items()) if sum(a >= RATIO for a, _ in r.values()) >= SHARE * len(r)
|
||||||
return out
|
and statistics.median(e for _, e in r.values()) >= RATIO]
|
||||||
|
out = []
|
||||||
|
for b in runs(turbulent, 2 * HOUR):
|
||||||
def _turbulent(r):
|
agit = {s: statistics.median(hours[h][s][0] for h in b if s in hours[h]) for s in {s for h in b for s in hours[h]}}
|
||||||
return (sum(a >= RATIO for a, _ in r.values()) >= SHARE * len(r)
|
|
||||||
and statistics.median(e for _, e in r.values()) >= RATIO_ERR)
|
|
||||||
|
|
||||||
|
|
||||||
def heat_shimmer(measurements):
|
|
||||||
H = hours(measurements)
|
|
||||||
blocks = []
|
|
||||||
for h in sorted(h for h, r in H.items() if _turbulent(r)):
|
|
||||||
if blocks and h - blocks[-1][-1] <= 2 * HOUR:
|
|
||||||
blocks[-1].append(h)
|
|
||||||
else:
|
|
||||||
blocks.append([h])
|
|
||||||
habit = {s: sum(m["corr_x"] is None for m in ms) / len(ms) for s, ms in measurements.items()} # usual rejections
|
|
||||||
episodes = []
|
|
||||||
for b in blocks:
|
|
||||||
start, end = b[0], b[-1] + HOUR
|
|
||||||
by = {s: [m for m in ms if start <= m["t"] < end] for s, ms in measurements.items()}
|
|
||||||
new = sum(max(0, sum(m["corr_x"] is None for m in d) - habit[s] * len(d)) for s, d in by.items())
|
|
||||||
if end - start < MIN_DURATION or new > REJECT_MAX * sum(map(len, by.values())):
|
|
||||||
continue
|
|
||||||
by_sensor = {}
|
|
||||||
for h in b:
|
|
||||||
for s, (a, _) in H[h].items():
|
|
||||||
by_sensor.setdefault(s, []).append(a)
|
|
||||||
agit = {s: statistics.median(v) for s, v in by_sensor.items()}
|
|
||||||
strength = statistics.median(agit.values())
|
strength = statistics.median(agit.values())
|
||||||
episodes.append({
|
out.append(episode(b[0], b[-1] + HOUR, "Visibility", "Heat shimmer", SHIMMER.format(strength),
|
||||||
"start": iso(start), "end": iso(end), "kind": "Visibility",
|
[s for s, v in agit.items() if v >= RATIO], strength >= STRONG))
|
||||||
"label": "Heat shimmer",
|
return out
|
||||||
"text": f"Readings were about {strength:.1f}× noisier than in calm hours because of heat shimmer "
|
|
||||||
"(warm air trembling in the line of sight); the small fast wiggles are not movement.",
|
|
||||||
"ignore": strength >= STRONG,
|
|
||||||
"sensors": sorted(s for s, v in agit.items() if v >= RATIO),
|
|
||||||
})
|
|
||||||
return episodes
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
for e in detect(load(sys.argv[1])):
|
|
||||||
print(e["start"], "→", e["end"], e["label"], f"| {len(e['sensors'])}:", ",".join(e["sensors"][:8]), "|", e["text"])
|
|
||||||
+31
-58
@@ -1,72 +1,45 @@
|
|||||||
# Detector thresholds
|
# Detector thresholds
|
||||||
|
|
||||||
Every threshold of the detectors (`scripts/*.py`, one per family of the families document, D-119) is **provisional**: tuned on the six data sets of events E-001 to E-005 and NORDEND of October, then adjusted by wf-improve sessions. When bulk data arrives, this is the list to review. A constant added or changed updates this file in the same change. Check: `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
|
Every threshold of the detectors (`scripts/*.py`) is **provisional**: tuned on the data sets of events E-001 to E-005 and NORDEND of October, then adjusted by wf-improve sessions. A constant added or changed updates this file in the same change; check with `grep -nE "^[A-Z_0-9]+ = " scripts/*.py`.
|
||||||
|
|
||||||
The detectors must detect everything (the user, 2026-10-26, D-068): a setting chosen to hide something from the client is not a detection rule; the wf-improve workflow may change it. What is shown to the client is decided in a separate layer.
|
The detectors detect everything (the user, 2026-10-26, D-068): what is shown to the client is decided in a separate layer.
|
||||||
|
|
||||||
"wf-improve X, mod. 1, S2": suggestion S2 of modification 1 of the wf-improve run on X. "X, batch 01": first batch of the earlier loop on station X.
|
"wf-improve X, mod. 1, S2": suggestion S2 of modification 1 of the wf-improve run on X; "X, batch 01": first batch of the earlier loop on station X.
|
||||||
|
|
||||||
| Detector | Constant | Value | Tuned on |
|
| Detector | Constant | Value | Tuned on |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `visibility` | `K_MAD` | 10 | E-001 (fog 11 to 57 MAD), E-005 (10 to 70 MAD) |
|
| `visibility` | `K_MAD` | 10 MAD | E-001 (fog, 11 to 57 MAD), E-005 (rain, trains, 10 to 70 MAD) |
|
||||||
| | `FACTOR` | 2 | drops sensors with a near-zero MAD (E-004, E-005); also, alone (without `K_MAD`), something passing in front of one or two targets or a reference, rejected for a single measurement (point 2): wf-improve Obstruction, mod. 1, S3 (WILDSTRUBEL: by day, heat shimmer inflates the MAD; R1 rejected at ×3.6 under 10 MAD) |
|
| | `FACTOR` | 2× the median | drops the sensors with a near-zero MAD (E-004, E-005); alone, something passing in front of one or two targets (wf-improve Obstruction, mod. 1, S3) |
|
||||||
| | `RUN_MIN` | 3 | E-001, rain of E-005 |
|
| | `RUN_MIN` | 3 | E-001, rain of E-005; also the diffuse nuisance |
|
||||||
| | `BRIEF_MIN` | 3 | trains of E-005 (5 to 17 sensors) |
|
| | `GAP_STEPS` | 2.5 steps | timestamp jitter, a missing step |
|
||||||
|
| | `BRIEF_MIN` | 3 sensors | trains of E-005 (5 to 17 sensors) |
|
||||||
| | `SALIENCE` | 3 | trains (5 to 17) against the cobweb of E-002 (2 to 9) |
|
| | `SALIENCE` | 3 | trains (5 to 17) against the cobweb of E-002 (2 to 9) |
|
||||||
| | `FRAC` | 0.15 | tails of PR09 and CTR01 (NORDEND October) |
|
| | `HALF` | 0.5 | level "none": NORDEND 09.10 23:00 (68 % rejected) against the rain of E-005 (24 to 33 %) |
|
||||||
| | `BREAK` | 2 h | NORDEND October, night of the 13th to the 14th |
|
| | `K_DIFFUSE` | 2.5 MAD | diffuse nuisance: wf-improve Visibility, mod. 1, S2 (JUNGFRAU, FLORIDA, NORDEND) |
|
||||||
| | `K_TARGET` | 5 | leaves of E-005 (6 to 25 MAD) |
|
| `visibility` (heat shimmer) | `RATIO` | 1.5× the calm | E-003 (corr ×1.7 to ×3, error_value ×1.7 to ×3.2 the night) |
|
||||||
| | `TARGET_MIN` | 3 | —; 2 when `TARGET_MIN` measurement steps do not fit in `TARGET_GAP`: wf-improve Obstruction, mod. 1, S6 (BREITHORN, one measurement every 4 h: vehicles parked at night, two or three rejections) |
|
|
||||||
| | `TARGET_GAP` | 6 h | leaves of E-005 (7 rejections in 12 h); at least two measurement steps (same S6) |
|
|
||||||
| | `GAP_STEPS` | 2.5 steps | timestamp jitter |
|
|
||||||
| | `REJECT_FACTOR` | 1.2 | FLORIDA, batch 01: a rejection at 10 MAD is high without doubling (targets at level 0.6); drops C24 of E-009 (0.68, rejected at ×1.04) |
|
|
||||||
| | `HALF` | 0.5 | level "none" (point 5): NORDEND 09/10 23:00 (68 % rejected) against the rain of E-005 (24 to 33 %) |
|
|
||||||
| | `K_DIFFUSE` | 2.5 MAD | diffuse nuisance (point 8): median, over the sensors, of the error_value in MAD above each one's median, `RUN_MIN` instants in a row, rejected or not: Poor visibility "reduced"; not within heat shimmer; wf-improve Visibility, mod. 1, S2 (JUNGFRAU, FLORIDA, NORDEND: reduced visibility of the whole scene without rejection; at 2, nights labelled thermal at JUNGFRAU and jumps back of WILDSTRUBEL absorbed; at 3, half the gain) |
|
|
||||||
| | `FEW` | 3 targets | Poor visibility on at most 3 targets, no reference: an object in front of them, Target obscured (boundary "Visibility or Obstruction" of the families document); BREITHORN, batch 01 (vehicles parked in front of 2 or 3 targets out of 12) |
|
|
||||||
| `visibility` (heat shimmer) | `RATIO` | 1.5 | E-003 (×1.7 to ×3 the night) |
|
|
||||||
| | `SHARE` | 0.5 | — |
|
| | `SHARE` | 0.5 | — |
|
||||||
| | `RATIO_ERR` | 1.5 | E-003 (×1.7 to ×3.2) |
|
|
||||||
| | `REJECT_MAX` | 15 % | E-003 ≤ 7 %; cobweb and rain ≥ 18 %; rejections beyond each sensor's habit (its share of rejections over the export): wf-improve Visibility, mod. 1, S1 (WILDSTRUBEL: a reference or targets usually rejected dropped the days of heat shimmer; 4 to 14 % against 9 to 22 % raw) |
|
|
||||||
| | `STRONG` | 2× | E-003 (×2.3 to ×2.6) |
|
| | `STRONG` | 2× | E-003 (×2.3 to ×2.6) |
|
||||||
| | `MIN_DURATION` | 3 h | E-003 (10-11 h) against the morning of E-004 (2 h); the classification drops this morning too, whatever the duration |
|
| `target_issue` | `SHARE` | 0.25 | regularity rather than the share of the day: wf-improve Target issue, mod. 1, S1 (WILDSTRUBEL R3, rejected by day only) |
|
||||||
| `displacement` (jumps) | `K` | 20 noises | E-001 (at 15, two healthy targets greyed); do not lower for the trains, seen by `visibility` |
|
| | `MIN_HOURS` | 3 h | as hours at the station's cadence: wf-improve NEBRASKA, batch 01 (one measurement every 4 h) |
|
||||||
| | `K_LASTING` | 7 noises | jump under K that stays (point 8 of `displacement.py`): NIEDERHORN, batch 01 (e09, corr_y −0.19 px at once, 8 noises, stayed 5 days) |
|
| | `BRIDGE` | 1 day | wf-improve Obstruction, mod. 1, S1; Target issue, mod. 1, S1 (at 2, parked vehicles of BREITHORN made a week) |
|
||||||
| | `PLATEAU` | 8 ordinary variations | same point: gap between the medians of the day after and the day before, in ordinary variations of the level from one day to the next; NIEDERHORN, batch 01 (e09: 13; E-001, healthy targets: 6 at most) |
|
| | `WEEK` | 7 days | Degraded target against Target obscured: wf-improve NEBRASKA, batch 01 (lost references, 25 days; parked vehicles, 2 to 5 days) |
|
||||||
| | `K_PEAK` | 6 noises | one-step peak common to several targets: NORDEND, batch 01 (shakes of the whole image, 6 to 11 noises) |
|
| | `MIN_DAYS` | 2 days | NORDEND: isolated days on 16 sensors |
|
||||||
| | `PEAK_MIN` | 3 targets | NORDEND, batch 01 (3 to 5 targets per shake) |
|
| `displacement` (jumps) | `K` | 20 noises | E-001 (at 15, two healthy targets greyed); the trains of E-005 are left to `visibility` |
|
||||||
| | `BASE` | 4 measurements | 1 h at 15 min; the return of a jump is judged on an hour of measurements, BASE at most, one at least (wf-improve MICHIGAN, mod. 1: at a 4 h step, a peak back at the next measurement became "Large shift") |
|
| | `BASE` | 4 measurements | 1 h at 15 min; the return is judged on an hour (wf-improve MICHIGAN, mod. 1) |
|
||||||
| | `GAP` | 1 h | — |
|
|
||||||
| | `BACK_H` | 24 h | E-002 (plateaus up to 11.5 h) |
|
| | `BACK_H` | 24 h | E-002 (plateaus up to 11.5 h) |
|
||||||
| | `REPEAT` | 5 jumps back | CLARIDEN, batch 01: PR_BG flips 28 times in 2.5 days, the other targets 3 at most (morning of the movement) |
|
| `displacement` (slow drift) | `HOURS` | 20 h out of 24 | as the Displacements tab (D-055), as a share of the hours measured (wf-improve MICHIGAN, mod. 1) |
|
||||||
| | `FLIP` | 36 h | problem target (D-074): PR_BG of CLARIDEN flips 2.5 days; C8 and C13 of E-009, 5 flips in 1 h 20 on the morning of the movement, stay brief jumps; 6 h → 36 h: wf-improve Target issue, mod. 1, S2 (NORDEND PR12, PR14, PR08, PR05, JUNGFRAU BJ_G: 12 to 28 h of flips within a loss of visibility, labelled Visibility or Obstruction, absorbed by it) |
|
| | `DAYS` | 5 days | D-055 |
|
||||||
| | `MAX_FLIPPING` | 2 targets | problem target: FLORIDA, the two stuck targets; E-002, cobweb on the references, 15 targets flip at once |
|
| | `SIZE` | 1 amplitude | D-055 |
|
||||||
| | `LARGE` | 0.5 | large shift ("Large shift", D-074): E-004 and CLARIDEN, 18 targets on the morning of the movement; also, slow drift of the camera (LARGE of the targets in corr and of the sensors in raw x, y, on the same side: no "Slow drift" over the whole window, only the pieces of `PIECE`): wf-improve MICHIGAN, mod. 1 (BREITHORN kept); wf-improve Displacement, mod. 1, S2 (NEBRASKA reported in pieces, JUNGFRAU still silent) |
|
| | `STEADY` | 0.6 | D-055 (Kendall tau) |
|
||||||
| `displacement` (slow drift) | `HOURS` | 20 h out of 24 | as the trend of the Displacements tab (D-055); as a share of the hours the station measures (wf-improve MICHIGAN, mod. 1: 5 out of 6 at one measurement every 4 h, no day counted) |
|
| | `PIECE` | 3.5 amplitudes | wf-improve Displacement, mod. 1, S1 (rows of CLARIDEN 3.8 to 8.3; thermal cycles of JUNGFRAU 3.1 at most) |
|
||||||
| | `DAYS` | 5 days | D-055: a trend needs at least 5 UTC days |
|
| `thermal_displacement` | `MIN_HOURS` | 20 h | — |
|
||||||
| | `SIZE` | 1 | D-055: drift over the period ≥ one daily amplitude; amplitude of the day: 9th and 1st deciles of `corr` |
|
|
||||||
| | `STEADY` | 0.6 | D-055: Kendall tau of the daily levels; a step from one day to the next of more than half the drift drops it (a jump) |
|
|
||||||
| | `PIECE` | 3.5 amplitudes | piecewise drift, for a target without a drift over the whole window (point 4b of `displacement.py`): wf-improve Displacement, mod. 1, S1 (rows of CLARIDEN, 3.8 to 8.3 amplitudes; JUNGFRAU, thermal cycles without a Displacement label, 3.1 at most; at 1, NORDEND and JUNGFRAU overflow) |
|
|
||||||
| | `MAX_HOLE` | 2 days | a drift piece (point 4b) does not go from one kept day to the next more than 2 days apart: wf-improve Displacement, mod. 2, S1 (CLARIDEN, hole from 10-03 to 10-05: the pieces from 09-28 to 11-01 fall; at 1, the missing days of a half-day hole cut the drifts of CLARIDEN and NEBRASKA) |
|
|
||||||
| `thermal_displacement` | `MIN_HOURS` | 20 h | —; days of the profile |
|
|
||||||
| | `PARTIAL` | 12 h | an incomplete day (fewer than `MIN_HOURS`) gets its episode if it looks like the profile on the hours it has, without entering the profile; wf-improve Thermal displacement, mod. 1, S4 (SNCF01, NORDEND, JUNGFRAU: labelled cycles of days cut by a hole; `MIN_HOURS` at 12 or 16 made NIEDERHORN go back) |
|
|
||||||
| | `REPRO` | 0.6 | pure noise ≈ 0 |
|
| | `REPRO` | 0.6 | pure noise ≈ 0 |
|
||||||
| | `SHARE` | 0.5 | NORDEND October just under (9/19) |
|
| | `SHARE` | 0.5 | NORDEND October just under (9/19) |
|
||||||
| | `WARM` | 0.3 | —; only says there is a cycle (at least `MIN_WARM_HOURS` warm hours), its bounds are at `EDGE` |
|
| | `R2_MIN` | 0.3 | PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48 |
|
||||||
| | `EDGE` | 0.1 | start of the wave, going back from the peak (the return: `RETURN`); wf-improve Thermal displacement, mod. 1, S1 (NORDEND, JUNGFRAU, SNCF01: labelled evenings and nights, the wave stopped at the last warm hour; at 0.15 and 0.2, fewer nights found, FLORIDA overflows in the evening anyway) |
|
|
||||||
| | `RETURN` | 0.05 | return of the wave, the return after midnight if needed, at most to the next day's start: the cooling tail is slower than the warm-up; wf-improve NORDEND 2026-11-05, S1 (NORDEND, JUNGFRAU: labelled nights after the return reached `EDGE`; `EDGE` itself at 0.05, start included, made FLORIDA and SNCF01 overflow) |
|
|
||||||
| | `DAY_MATCH` | 0.5 | — |
|
|
||||||
| | `NIGHT` | 8 h | — |
|
| | `NIGHT` | 8 h | — |
|
||||||
| | `R2_MIN` | 0.3 | 24 h sinusoid: PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48; no longer asked for the camera (raw positions) since WILDSTRUBEL, batch 01 (1 target out of 7, the wrong pointings crush the fit) |
|
| | `WARM` | 0.3 | — |
|
||||||
| | `MIN_WARM_HOURS` | 2 h | WILDSTRUBEL, batch 01: a single warm hour is not a cycle (NIEDERHORN, E-005: noon, trains and holes) |
|
| | `EDGE` | 0.1 | wf-improve Thermal displacement, mod. 1, S1 (labelled evenings and nights) |
|
||||||
| | `SLIDE_K` | 3 noises | common camera slide without a cycle ("Camera thermal movement"): gap from the median of the day before, raw positions and corrected curves; wf-improve MICHIGAN, mod. 1 (at 2, Displacement instants of BREITHORN taken) |
|
| | `RETURN` | 0.05 | wf-improve NORDEND 2026-11-05, S1 (the cooling tail is slow) |
|
||||||
| | `COLD` | 4 one-hour noises | the camera moves with the cold ("Camera thermal movement", without a cycle): the pose (tilt, rotation or scale) moves away from its median of the day before while the temperature falls under its own, the same hour; noise: variations over an hour, one step at 4 h; wf-improve Thermal displacement, mod. 1, S3 (BREITHORN, CLARIDEN, NEBRASKA: targets in opposite directions, or compensated; at 3, Displacement instants of NEBRASKA taken) |
|
| | `SLIDE_K` | 3 noises | wf-improve MICHIGAN, mod. 1 (at 2, Displacement instants of BREITHORN taken) |
|
||||||
| `no_data` | `MISSING` | 2 steps | user's rule, 2026-10-23; also applied between the last measurement and the end of the window read (CLARIDEN, batch 01); the step is at least an hour divided by the median instants per clock hour (bursts of measurements: the cycle, not the minute inside a burst; wf-improve, batch 2026-11-05, S1: SUSTENHORN) |
|
| `no_data` | `MISSING` | 2 steps | user's rule, 2026-10-23; also up to the end of the window (CLARIDEN, batch 01); the cadence of bursts (wf-improve, batch 2026-11-05, S1) |
|
||||||
| `target_issue` | `CALM` | 0.25 | (detector added on 2026-10-27, see its docstring) |
|
| `categories` | `MERGE` | 1 h | rule validated by the user (2026-10-24); an Obstruction or a Target issue only with the same target (wf-improve Obstruction, mod. 1, S1; Target issue, mod. 1, S3) |
|
||||||
| | `SHARE` | 0.5 | —; the target hidden at times ("Target obscured") |
|
|
||||||
| | `SHARE_WEEK` | 0.25 | share of the calm instants rejected that makes a degraded day of the problem target ("Degraded target", at least `WEEK` days): regularity rather than the share of the day; wf-improve Target issue, mod. 1, S1 (WILDSTRUBEL R3 rejected from 04:00 to 16:00 every day, 30 to 45 %: seen only on days without night measurements; NORDEND PR04 and CTR01, NIEDERHORN PWD_IMP_B08_024_C09, under half) |
|
|
||||||
| | `MIN_HOURS` | 3 h | hours of calm instants per day, at the station's cadence (median gap between two instants): 12 instants at 15 min; wf-improve NEBRASKA, batch 01 (one measurement every 4 h: no day counted, the lost reference REFCOLH22 was not seen) |
|
|
||||||
| | `MIN_DAYS` | 2 days | NORDEND: isolated days on 16 sensors |
|
|
||||||
| | `WEEK` | 7 days | problem target ("Degraded target") if hindered at least a week in a row, otherwise a target hidden at times ("Target obscured"): boundary "Target issue or Obstruction" of the families document; wf-improve NEBRASKA, batch 01 (BREITHORN, parked vehicles, 2 to 5 days; NEBRASKA and MICHIGAN, lost reference, 25 days; JUNGFRAU, PM_G, 63 days) |
|
|
||||||
| | `BRIDGE` | 1 day | non-degraded days in a row that do not break the run of a problem target ("Degraded target": `WEEK` days from the first to the last degraded day, `SHARE_WEEK` of rejections over the bridged run); wf-improve Obstruction, mod. 1, S1 (FLORIDA, SNCF01, NORDEND, NIEDERHORN: problem targets cut into Target obscured by a day under `SHARE`; BREITHORN, parked vehicles, unchanged); 2 → 1: wf-improve Target issue, mod. 1, S1 (at a quarter, two bridging days joined the parked vehicles of BREITHORN, POUI22, into a week) |
|
|
||||||
| | `STUCK` | 0.05 | stuck target (D-074): E-011, C2 and C3 at 0.006 and 0.016 of the others' noise; NORDEND, E-001: 0.7 at the lowest |
|
|
||||||
| `categories` (classification) | `MERGE` | 1 h | rule validated by the user: same variety less than an hour apart, a single event; two Obstruction (Target obscured, Brief obstruction) only if they share a sensor (wf-improve Obstruction, mod. 1, S1 and S3), two Target issue too (wf-improve Target issue, mod. 1, S3); a Brief obstruction only absorbs the jumps back within `MERGE` after its instant (S3) |
|
|
||||||
Reference in new issue
Block a user