Each module returns every family it finds (kind); categories.py imports DETECTORS by name (no introspection, no CONTEXT) and drops from Obstruction the sensors already in an overlapping Target issue episode (formerly obstruction.py). No detector computed twice. Benchmark identical to the baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
338 lines
21 KiB
Python
338 lines
21 KiB
Python
"""Detector of the Visibility family (the families document): the camera sees less well, the image is faint or blurred
|
||
(error_value), or it trembles (heat shimmer, end of file). The same error_value computation also finds objects in front
|
||
of a few targets (Brief obstruction, Target obscured): episodes() returns them with the Obstruction family, which
|
||
detect() leaves to obstruction.py.
|
||
|
||
Method:
|
||
1. "High" measurement: error_value > median + K_MAD MAD of the sensor over the export, and > FACTOR × median, or
|
||
rejected (> error_threshold) and > REJECT_FACTOR × median: a sensor at a usual level of 0.6 never doubles, and
|
||
FLORIDA (batch 01) showed its fogs as Target obscured.
|
||
2. Brief obstruction: an instant where at least BRIEF_MIN sensors are high, SALIENCE times more than at the neighbouring
|
||
instants (train, vehicle), with no reference (global_ref) among them: a one-step peak touching the references is not
|
||
an obstacle in front of targets (peak of the whole image, NORDEND on 01.09 11:15, batch 01); it is not shown. Unless
|
||
at least BRIEF_MIN targets are rejected at this instant and neither at the instant before nor after: something
|
||
passing in front of them, a reference touched as well (wf-improve session Displacement, mod. 1, S5: trains of
|
||
NIEDERHORN, 12 and 13 targets rejected).
|
||
Also, something passing in front of one or two targets, or a single reference: at an instant, fewer than BRIEF_MIN
|
||
sensors rejected at more than FACTOR times their median (without K_MAD: by day, heat shimmer inflates the MAD), at
|
||
most one reference, none of them rejected at the instant before or after, and fewer than BRIEF_MIN high sensors
|
||
(wf-improve session Obstruction, mod. 1, S3: vehicles, passers-by, insects in front of one or two targets of
|
||
WILDSTRUBEL, NIEDERHORN, LENZSPITZE; a reference hidden by a truck at JUNGFRAU, BREITHORN).
|
||
3. Poor visibility: runs of at least RUN_MIN high measurements per sensor, joined across sensors (at least 2), reduced
|
||
to the instants where at least FRAC of the episode's peak is high.
|
||
4. Target obscured: a target (not a global_ref) rejected (> error_threshold) and above K_TARGET MAD, at least TARGET_MIN
|
||
times less than TARGET_GAP apart, mostly outside the episodes of 2 and 3 (leaves, snow, glare on a target). The gap
|
||
is at least two measurement steps, and two rejections are enough when TARGET_MIN steps do not fit in TARGET_GAP
|
||
(wf-improve session Obstruction, mod. 1, S6: at one measurement every 4 h, vehicles parked at night in front of COLI22,
|
||
COLI24, POUI22 of BREITHORN make only two or three rejections). All episodes: ignore = True.
|
||
5. Level of a Poor visibility (validated 2026-10-24), measurement by measurement: "none" (nothing is seen) when at least
|
||
HALF of the usually seen targets (under their threshold at least half the time) are rejected, "reduced" (average
|
||
data) otherwise; one episode per period of the same level, with the level field.
|
||
6. Bounds of a Poor visibility (WILDSTRUBEL, batch 01): from the first to the last measurement where a target is
|
||
rejected (> error_threshold), and no episode without a rejected target. On a road in full sun, the daytime
|
||
error_value (×1.5 to ×3, heat shimmer) made reduced visibility last for hours without any rejection; references do
|
||
not count: R3 is above its threshold almost all day.
|
||
7. Poor visibility or object (boundary "Visibility or Obstruction" of the families document; BREITHORN, batch 01): a
|
||
nuisance touching at most FEW targets, and no reference, is not diffuse: it is an object in front of them (a parked
|
||
vehicle), a "Target obscured" episode of its whole duration, without a level.
|
||
8. Diffuse nuisance, with or without rejections (wf-improve session Visibility, mod. 1, S2): at each instant, the median,
|
||
over the sensors, of the error_value in MAD above each one's median; at least RUN_MIN instants in a row at K_DIFFUSE
|
||
or more make a Poor visibility "reduced". A diffuse nuisance entirely within heat shimmer is the heat shimmer, and
|
||
within a Poor visibility of point 3, that one (the same instants twice, two events: wf-improve, batch 2026-11-05, S2):
|
||
no episode (otherwise it absorbed the jump back of the day before, WILDSTRUBEL on 04.10). Finds the drops of
|
||
visibility of the whole scene without rejection at JUNGFRAU, FLORIDA, NORDEND; at 2, nights labelled thermal at
|
||
JUNGFRAU; at 3, half the gain.
|
||
|
||
A simpler version (share of targets rejected above their threshold, without comparing each sensor with its own level)
|
||
was tried on 2026-10-24 and dropped: worse on five data sets out of six (git history of Support-Agent,
|
||
detectors/archive/visibilite_simple.py).
|
||
"""
|
||
import statistics
|
||
from datetime import timedelta
|
||
|
||
from data import iso
|
||
|
||
K_MAD = 10 # tuned on E-001 (fog: 11 to 57 MAD) and E-005 (rain, trains: 10 to 70 MAD)
|
||
FACTOR = 2 # drops sensors with a near-zero MAD (E-004: PWD_REF10 ×1.6 at 14 MAD, E-005)
|
||
RUN_MIN = 3 # high measurements in a row for poor visibility (E-001, E-005 rain)
|
||
BRIEF_MIN = 3 # tuned on E-005 (trains: 5 to 17 sensors; the one with 2 targets is missed)
|
||
K_TARGET = 5 # tuned on E-005 (leaves: 6 to 25 MAD, rejected)
|
||
TARGET_MIN = 3 # repeated rejections of a target
|
||
FRAC = 0.15 # cuts the tails of PR09 + CTR01 (NORDEND October) without cutting the cobweb (E-002)
|
||
GAP_STEPS = 2.5 # steps tolerated between two high measurements of the same run (a missing step, clock to the second)
|
||
BREAK = timedelta(hours=2) # lull separating two episodes (NORDEND October, night of 13 to 14.10)
|
||
SALIENCE = 3 # a train stands out at least 3 times above the neighbouring instants (E-005: 5 to 17 against 0 to 2;
|
||
# the cobweb of E-002 varies from 2 to 9 from one step to the next)
|
||
TARGET_GAP = timedelta(hours=6) # tuned on E-005 (leaves: 7 rejections in 12 h, up to 5 h apart)
|
||
REJECT_FACTOR = 1.2 # a rejection at 10 MAD also counts (FLORIDA, batch 01: levels of 0.6, ×2 out of reach); drops C24
|
||
# of E-009 (level 0.68 just under the threshold 0.7, rejected at ×1.04)
|
||
HALF = 0.5 # "nothing is seen"; tuned on NORDEND (09.10 23:00: 68 % rejected) and E-005 (rain: 24 to 33 %)
|
||
K_DIFFUSE = 2.5 # diffuse nuisance (8); wf-improve session Visibility, mod. 1, S2
|
||
FEW = 3 # Poor visibility on at most 3 targets, no reference: an object, Target obscured (7); BREITHORN,
|
||
# batch 01 (vehicles parked in front of 2 or 3 targets out of 12)
|
||
|
||
|
||
def _step(measurements):
|
||
ts = sorted({m["t"] for v in measurements.values() for m in v})
|
||
return statistics.median(b - a for a, b in zip(ts, ts[1:]))
|
||
|
||
|
||
def _group(points, gap):
|
||
"""[(t, sensor)] sorted by t -> groups [(start, end, {sensors}, [t])] whose points are ≤ gap from each other."""
|
||
groups = []
|
||
for t, s in points:
|
||
if groups and t - groups[-1][1] <= gap:
|
||
g = groups[-1]
|
||
groups[-1] = (g[0], t, g[2] | {s}, g[3] + [t])
|
||
else:
|
||
groups.append((t, t, {s}, [t]))
|
||
return groups
|
||
|
||
|
||
def _group_intervals(intervals, gap):
|
||
"""[(start, end, sensor)] sorted -> [(start, end, {sensors})] of the intervals that overlap (within gap)."""
|
||
out = []
|
||
for a, b, s in intervals:
|
||
if out and a - out[-1][1] <= gap:
|
||
out[-1] = (out[-1][0], max(out[-1][1], b), out[-1][2] | {s})
|
||
else:
|
||
out.append((a, b, {s}))
|
||
return out
|
||
|
||
|
||
def _episode(start, end, label, text, sensors, level=None):
|
||
e = {"start": iso(start), "end": iso(end), "kind": "Visibility" if label == "Poor visibility" else "Obstruction",
|
||
"label": label, "text": text,
|
||
"ignore": level != "reduced", "sensors": sorted(sensors)} # reduced visibility: average data, not greyed
|
||
return {**e, "level": level} if level else e
|
||
|
||
|
||
def _levels(measurements, instants):
|
||
"""t -> "none" if at least HALF of the usually seen targets are rejected at t, "reduced" otherwise."""
|
||
ratios = []
|
||
for v in measurements.values():
|
||
r = {m["t"]: m["error_value"] / m["error_threshold"] for m in v if m["error_value"] is not None and m["error_threshold"]}
|
||
if r and statistics.median(r.values()) < 1:
|
||
ratios.append(r)
|
||
return {t: "none" if sum(r.get(t, 0) > 1 for r in ratios) >= HALF * max(1, sum(t in r for r in ratios)) else "reduced"
|
||
for t in instants}
|
||
|
||
|
||
def episodes(measurements):
|
||
"""All error_value episodes: Visibility (Poor visibility), Obstruction (Brief obstruction, Target obscured)."""
|
||
step = _step(measurements)
|
||
gap = GAP_STEPS * step
|
||
refs = {s for s, v in measurements.items() if v[0]["pattern_type"] == "global_ref"}
|
||
instants = sorted({m["t"] for v in measurements.values() for m in v})
|
||
high, rejections = {}, {} # sensor -> [t]
|
||
refused = set() # instants where a target (not a global_ref) is above its threshold
|
||
rejected_targets = {} # t -> {targets above their threshold}
|
||
strong, refusals = {}, {} # t -> {sensors rejected at more than FACTOR times their median}, {sensors rejected}
|
||
for s, v in measurements.items():
|
||
e = [m["error_value"] for m in v if m["error_value"] is not None]
|
||
if len(e) < 10:
|
||
continue
|
||
med = statistics.median(e)
|
||
mad = statistics.median(abs(x - med) for x in e) or 1e-9
|
||
for m in v:
|
||
x = m["error_value"]
|
||
if x is None:
|
||
continue
|
||
rejected = m["error_threshold"] and x > m["error_threshold"]
|
||
if rejected and s not in refs:
|
||
refused.add(m["t"])
|
||
rejected_targets.setdefault(m["t"], set()).add(s)
|
||
if x > med + K_MAD * mad and (x > FACTOR * med or rejected and x > REJECT_FACTOR * med):
|
||
high.setdefault(s, []).append(m["t"])
|
||
if rejected:
|
||
refusals.setdefault(m["t"], set()).add(s)
|
||
if x > FACTOR * med:
|
||
strong.setdefault(m["t"], set()).add(s)
|
||
if s not in refs and rejected and x > med + K_TARGET * mad:
|
||
rejections.setdefault(s, []).append(m["t"])
|
||
by_t = {} # t -> {high sensors}
|
||
for s, ts in high.items():
|
||
for t in ts:
|
||
by_t.setdefault(t, set()).add(s)
|
||
n = lambda t: len(by_t.get(t, ())) # noqa: E731
|
||
out = []
|
||
|
||
# 2. Brief obstruction: an instant where many sensors are high, at least SALIENCE times more than the neighbours.
|
||
briefs = set()
|
||
for i, t in enumerate(instants):
|
||
neighbours = [n(instants[j]) for j in (i - 1, i + 1) if 0 <= j < len(instants)]
|
||
if n(t) >= BRIEF_MIN and n(t) >= SALIENCE * max(neighbours):
|
||
briefs.add(t)
|
||
only = rejected_targets.get(t, set()).difference(*(rejected_targets.get(instants[j], set()) for j in (i - 1, i + 1)
|
||
if 0 <= j < len(instants)))
|
||
if by_t[t] & refs and len(only) < BRIEF_MIN: # references touched without targets rejected at this instant
|
||
continue # alone: not an obstacle in front of targets, the whole image
|
||
out.append(_episode(t, t, "Brief obstruction", f"{n(t)} points were poorly seen for a single measurement, then came "
|
||
"back: probably a train, a vehicle or an animal passing; ignore this point on the curves.", by_t[t]))
|
||
elif 0 < len(f := strong.get(t, set())) < BRIEF_MIN and n(t) < BRIEF_MIN and len(f & refs) <= 1 and not f & set().union(
|
||
*(refusals.get(instants[j], set()) for j in (i - 1, i + 1) if 0 <= j < len(instants))):
|
||
# one or two targets, or a reference, rejected at this single measurement, at FACTOR times their median
|
||
out.append(_episode(t, t, "Brief obstruction", f"{len(f)} point{'s were' if len(f) > 1 else ' was'} poorly seen "
|
||
"for a single measurement, then came back: probably something passing; ignore this point on the "
|
||
"curves.", f))
|
||
|
||
# 3. Poor visibility: long runs per sensor (without the brief instants), joined across sensors, then reduced to the
|
||
# instants where at least FRAC of the episode's peak is high (cuts the tails of one or two sensors and the long
|
||
# lulls, more than BREAK).
|
||
runs = []
|
||
for s, ts in high.items():
|
||
pts = [(t, s) for t in ts if t not in briefs]
|
||
runs += [(a, b, s) for a, b, _, l in _group(pts, gap) if len(l) >= RUN_MIN]
|
||
zones = [z for z in _group_intervals(sorted(runs), gap) if len(z[2]) >= 2]
|
||
kept = []
|
||
for a, b, _ in zones:
|
||
inside = [t for t in instants if a <= t <= b and t not in briefs]
|
||
peak = max(n(t) for t in inside)
|
||
pts = [(t, s) for t in inside if n(t) >= max(1, FRAC * peak) for s in by_t[t]]
|
||
kept += [(d, f, cs) for d, f, cs, l in _group(pts, max(gap, BREAK)) if len(cs) >= 2 and len(set(l)) >= RUN_MIN]
|
||
# Bounds: from the first to the last rejected target; without a rejection, no episode (6).
|
||
kept = [(r[0], r[-1], cs) for a, b, cs in kept if (r := [t for t in instants if a <= t <= b and t in refused])]
|
||
level = _levels(measurements, instants)
|
||
for a, b, cs in kept: # one episode per period of the same level (5); the text comes from categories.py
|
||
if len(cs) <= FEW and not cs & refs: # an object in front of a few targets, not a diffuse nuisance (7)
|
||
out.append(_episode(a, b, "Target obscured", "", cs))
|
||
continue
|
||
seg = []
|
||
for t in [t for t in instants if a <= t <= b] + [None]:
|
||
if seg and (t is None or level[t] != level[seg[0]]):
|
||
who = cs & set().union(*(by_t.get(u, set()) for u in seg)) or cs
|
||
out.append(_episode(seg[0], seg[-1], "Poor visibility", "", who, level[seg[0]]))
|
||
seg = []
|
||
if t is not None:
|
||
seg.append(t)
|
||
inside_any = lambda t: t in briefs or any(a <= t <= b for a, b, _ in kept) # noqa: E731
|
||
|
||
# 4. Target obscured: a target rejected again and again, mostly outside the episodes above.
|
||
targets = []
|
||
for s, ts in rejections.items():
|
||
for a, b, _, l in _group([(t, s) for t in ts], max(TARGET_GAP, 2 * step)):
|
||
if len(l) >= (TARGET_MIN if TARGET_MIN * step <= TARGET_GAP else 2) and sum(map(inside_any, l)) * 2 < len(l):
|
||
targets.append((a, b, s))
|
||
for a, b, cs in _group_intervals(sorted(targets), timedelta(0)):
|
||
who, it, its = ("One target was", "it", "its") if len(cs) == 1 else (f"{len(cs)} targets were", "them", "their")
|
||
out.append(_episode(a, b, "Target obscured", f"{who} repeatedly lost while the targets around stayed clear: probably "
|
||
f"something in front of {it} (leaves, snow) or glare; {its} displacements in this period are unreliable.", cs))
|
||
return sorted(out, key=lambda e: e["start"])
|
||
|
||
|
||
def diffuse(measurements):
|
||
"""Diffuse nuisance (8): the runs of at least RUN_MIN instants where the median, over the sensors, of the error_value
|
||
in MAD above each one's median reaches K_DIFFUSE, with or without rejections -> Poor visibility "reduced"."""
|
||
z = {} # t -> {sensor: gap in MAD}
|
||
for s, v in measurements.items():
|
||
e = [m["error_value"] for m in v if m["error_value"] is not None]
|
||
if len(e) < 10:
|
||
continue
|
||
med = statistics.median(e)
|
||
mad = statistics.median(abs(x - med) for x in e) or 1e-9
|
||
for m in v:
|
||
if m["error_value"] is not None:
|
||
z.setdefault(m["t"], {})[s] = (m["error_value"] - med) / mad
|
||
highs = [(t, None) for t in sorted(z) if statistics.median(z[t].values()) >= K_DIFFUSE]
|
||
return [_episode(a, b, "Poor visibility", "", {s for t in l for s, x in z[t].items() if x >= K_DIFFUSE}, "reduced")
|
||
for a, b, _, l in _group(highs, GAP_STEPS * _step(measurements)) if len(l) >= RUN_MIN]
|
||
|
||
|
||
def detect(measurements):
|
||
shimmer = heat_shimmer(measurements) # a diffuse nuisance during heat shimmer is the heat shimmer (8)
|
||
eps = episodes(measurements)
|
||
poor = [e for e in eps if e["kind"] == "Visibility"] # a diffuse nuisance within one of them is it
|
||
return eps + shimmer + [e for e in diffuse(measurements)
|
||
if not any(t["start"] <= e["start"] and e["end"] <= t["end"] for t in shimmer + poor)]
|
||
|
||
|
||
# --- Heat shimmer ----------------------------------------------------------------------------------------------------
|
||
# Atmospheric turbulence (fast noise on the corrected displacements). Method:
|
||
# 1. Per sensor and per hour H, on the measurements of [H − 1 h, H + 2 h): agitation = median of |Δcorr_x| + |Δcorr_y|
|
||
# between successive measurements (robust to one-step jumps), and median of the error_value.
|
||
# 2. Each is divided by the sensor's calm: the lower quartile of its hourly values over the export (often the night).
|
||
# 3. Turbulent hour: at least SHARE of the sensors at ≥ RATIO times their calm agitation, and median error_value of the
|
||
# sensors at ≥ RATIO_ERR times the calm (warm air blurs the spots: E-003).
|
||
# 4. Neighbouring hours merged (a one-hour hole tolerated), episode of at least MIN_DURATION, dropped if more than
|
||
# REJECT_MAX of the measurements are rejected beyond their sensor's habit (its share of rejections over the export):
|
||
# it is then poor visibility, another rule. A target that is always rejected (Target issue) does not count (wf-improve
|
||
# session Visibility, mod. 1, S1: WILDSTRUBEL, R3 rejected all day, the days of heat shimmer were dropped).
|
||
# Greyed if the median agitation ≥ STRONG.
|
||
RATIO = 1.5 # tuned on E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night)
|
||
SHARE = 0.5 # share of the sensors agitated at the same time
|
||
RATIO_ERR = 1.5 # tuned on E-003 (error_value ×1.7 to ×3.2 the night; ≈ ×1 at noon elsewhere)
|
||
REJECT_MAX = 0.15 # E-003 ≤ 7 %; cobweb (E-002) and rain (E-005) ≥ 18 %; rejections beyond the habit since wf-improve
|
||
# session Visibility, mod. 1, S1 (WILDSTRUBEL: 4 to 14 %, against 9 to 22 % raw)
|
||
STRONG = 2.0 # E-003 (×2.3 to ×2.6) is strong; an episode between ×1.5 and ×2 stays visible
|
||
MIN_DURATION = timedelta(hours=3) # E-003 lasts 10-11 h; drops the morning of the real movement of E-004 (2 h), which
|
||
# the classification drops too, whatever the duration
|
||
HOUR = timedelta(hours=1)
|
||
|
||
|
||
def _hour(t):
|
||
return t.replace(minute=0, second=0, microsecond=0)
|
||
|
||
|
||
def _ratios(points):
|
||
"""[(t, value)] -> {hour H: median over [H − 1 h, H + 2 h) / lower quartile of these medians}."""
|
||
by = {}
|
||
for t, v in points:
|
||
for k in (-1, 0, 1):
|
||
by.setdefault(_hour(t) + k * HOUR, []).append(v)
|
||
med = {h: statistics.median(v) for h, v in by.items() if len(v) >= 3}
|
||
if len(med) < 8:
|
||
return {}
|
||
calm = statistics.quantiles(med.values(), n=4)[0]
|
||
return {h: v / calm for h, v in med.items()} if calm > 0 else {}
|
||
|
||
|
||
def hours(measurements):
|
||
"""-> {hour: {sensor: (agitation / calm, error_value / calm)}}."""
|
||
out = {}
|
||
for s, ms in measurements.items():
|
||
ag = _ratios([(b["t"], abs(b["corr_x"] - a["corr_x"]) + abs(b["corr_y"] - a["corr_y"]))
|
||
for a, b in zip(ms, ms[1:]) if a["corr_x"] is not None and b["corr_x"] is not None])
|
||
er = _ratios([(m["t"], m["error_value"]) for m in ms if m["error_value"] is not None])
|
||
for h, r in ag.items():
|
||
if h in er:
|
||
out.setdefault(h, {})[s] = (r, er[h])
|
||
return out
|
||
|
||
|
||
def _turbulent(r):
|
||
return (sum(a >= RATIO for a, _ in r.values()) >= SHARE * len(r)
|
||
and statistics.median(e for _, e in r.values()) >= RATIO_ERR)
|
||
|
||
|
||
def heat_shimmer(measurements):
|
||
H = hours(measurements)
|
||
blocks = []
|
||
for h in sorted(h for h, r in H.items() if _turbulent(r)):
|
||
if blocks and h - blocks[-1][-1] <= 2 * HOUR:
|
||
blocks[-1].append(h)
|
||
else:
|
||
blocks.append([h])
|
||
habit = {s: sum(m["corr_x"] is None for m in ms) / len(ms) for s, ms in measurements.items()} # usual rejections
|
||
episodes = []
|
||
for b in blocks:
|
||
start, end = b[0], b[-1] + HOUR
|
||
by = {s: [m for m in ms if start <= m["t"] < end] for s, ms in measurements.items()}
|
||
new = sum(max(0, sum(m["corr_x"] is None for m in d) - habit[s] * len(d)) for s, d in by.items())
|
||
if end - start < MIN_DURATION or new > REJECT_MAX * sum(map(len, by.values())):
|
||
continue
|
||
by_sensor = {}
|
||
for h in b:
|
||
for s, (a, _) in H[h].items():
|
||
by_sensor.setdefault(s, []).append(a)
|
||
agit = {s: statistics.median(v) for s, v in by_sensor.items()}
|
||
strength = statistics.median(agit.values())
|
||
episodes.append({
|
||
"start": iso(start), "end": iso(end), "kind": "Visibility",
|
||
"label": "Heat shimmer",
|
||
"text": f"Readings were about {strength:.1f}× noisier than in calm hours because of heat shimmer "
|
||
"(warm air trembling in the line of sight); the small fast wiggles are not movement.",
|
||
"ignore": strength >= STRONG,
|
||
"sensors": sorted(s for s, v in agit.items() if v >= RATIO),
|
||
})
|
||
return episodes
|