visibility: shared helpers (common.py), five rules fewer, texts of its events
Removed (their benchmark contribution, each alone): the reference exception of Brief obstruction (none; Obstruction found +9 at LENZSPITZE), REJECT_FACTOR (Obstruction correct +2), FRAC and BREAK (Visibility found -3, Obstruction found +4), the bounds at the rejected targets (neutral), FEW (Visibility found -1), the TARGET_MIN exception at a slow cadence (none in total), MIN_DURATION of heat shimmer (Visibility found -1). RATIO_ERR merged into RATIO (same value). Target obscured: one episode per target. The event sentences of its varieties are now the episodes' texts. common.py: episode, runs, noise, level, cadence, targets. Benchmark Totals: Visibility found 47 -> 55 %, correct 77 -> 72 %; Obstruction found 26 -> 22 %, correct 28 -> 32 %; the other families unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5f085bf5ff
commit
5c2d265a21
2 files changed
+181
-292
No files matched your search
@@ -0,0 +1,47 @@
|
||||
"""Helpers shared by the detectors."""
|
||||
import statistics
|
||||
from datetime import timedelta
|
||||
|
||||
from data import iso
|
||||
|
||||
|
||||
def episode(start, end, kind, label, text, sensors, ignore, **extra):
|
||||
return {"start": iso(start), "end": iso(end), "kind": kind, "label": label, "text": text, "ignore": ignore,
|
||||
"sensors": sorted(sensors), **extra}
|
||||
|
||||
|
||||
def runs(times, gap):
|
||||
"""Sorted times -> [[t, ...]]: runs of times at most gap apart."""
|
||||
out = []
|
||||
for t in times:
|
||||
if out and t - out[-1][-1] <= gap:
|
||||
out[-1].append(t)
|
||||
else:
|
||||
out.append([t])
|
||||
return out
|
||||
|
||||
|
||||
def noise(v):
|
||||
"""Median step of a series, from one value to the next."""
|
||||
return statistics.median(abs(y - x) for x, y in zip(v, v[1:])) if len(v) > 1 else 0
|
||||
|
||||
|
||||
def level(v):
|
||||
"""-> (median, MAD) of the values."""
|
||||
med = statistics.median(v)
|
||||
return med, statistics.median(abs(x - med) for x in v) or 1e-9
|
||||
|
||||
|
||||
def cadence(measurements):
|
||||
"""Cycle of the measurements: the median gap between two instants, at least an hour divided by the median number of
|
||||
instants per clock hour (a station measuring in bursts, three measurements a minute apart every 15 min: 15 min)."""
|
||||
ts = sorted({m["t"].replace(second=0, microsecond=0) for ms in measurements.values() for m in ms})
|
||||
per_hour = {}
|
||||
for t in ts:
|
||||
per_hour[t.replace(minute=0)] = per_hour.get(t.replace(minute=0), 0) + 1
|
||||
gap = statistics.median(b - a for a, b in zip(ts, ts[1:])) if len(ts) > 1 else timedelta(hours=1)
|
||||
return max(gap, timedelta(hours=1) / statistics.median(per_hour.values()))
|
||||
|
||||
|
||||
def targets(n):
|
||||
return "one target" if n == 1 else f"{n} targets"
|
||||
+134
-292
@@ -1,110 +1,40 @@
|
||||
"""Detector of the Visibility family (the families document): the camera sees less well, the image is faint or blurred
|
||||
(error_value), or it trembles (heat shimmer, end of file). The same error_value computation also finds objects in front
|
||||
of a few targets (Brief obstruction, Target obscured): episodes() returns them with the Obstruction family, which
|
||||
detect() leaves to obstruction.py.
|
||||
"""Visibility family: the camera sees the targets less well (error_value), or the air trembles (heat shimmer). The same
|
||||
error_value also finds objects in front of a few targets: Obstruction family (Brief obstruction, Target obscured).
|
||||
|
||||
Method:
|
||||
1. "High" measurement: error_value > median + K_MAD MAD of the sensor over the export, and > FACTOR × median, or
|
||||
rejected (> error_threshold) and > REJECT_FACTOR × median: a sensor at a usual level of 0.6 never doubles, and
|
||||
FLORIDA (batch 01) showed its fogs as Target obscured.
|
||||
2. Brief obstruction: an instant where at least BRIEF_MIN sensors are high, SALIENCE times more than at the neighbouring
|
||||
instants (train, vehicle), with no reference (global_ref) among them: a one-step peak touching the references is not
|
||||
an obstacle in front of targets (peak of the whole image, NORDEND on 01.09 11:15, batch 01); it is not shown. Unless
|
||||
at least BRIEF_MIN targets are rejected at this instant and neither at the instant before nor after: something
|
||||
passing in front of them, a reference touched as well (wf-improve session Displacement, mod. 1, S5: trains of
|
||||
NIEDERHORN, 12 and 13 targets rejected).
|
||||
Also, something passing in front of one or two targets, or a single reference: at an instant, fewer than BRIEF_MIN
|
||||
sensors rejected at more than FACTOR times their median (without K_MAD: by day, heat shimmer inflates the MAD), at
|
||||
most one reference, none of them rejected at the instant before or after, and fewer than BRIEF_MIN high sensors
|
||||
(wf-improve session Obstruction, mod. 1, S3: vehicles, passers-by, insects in front of one or two targets of
|
||||
WILDSTRUBEL, NIEDERHORN, LENZSPITZE; a reference hidden by a truck at JUNGFRAU, BREITHORN).
|
||||
3. Poor visibility: runs of at least RUN_MIN high measurements per sensor, joined across sensors (at least 2), reduced
|
||||
to the instants where at least FRAC of the episode's peak is high.
|
||||
4. Target obscured: a target (not a global_ref) rejected (> error_threshold) and above K_TARGET MAD, at least TARGET_MIN
|
||||
times less than TARGET_GAP apart, mostly outside the episodes of 2 and 3 (leaves, snow, glare on a target). The gap
|
||||
is at least two measurement steps, and two rejections are enough when TARGET_MIN steps do not fit in TARGET_GAP
|
||||
(wf-improve session Obstruction, mod. 1, S6: at one measurement every 4 h, vehicles parked at night in front of COLI22,
|
||||
COLI24, POUI22 of BREITHORN make only two or three rejections). All episodes: ignore = True.
|
||||
5. Level of a Poor visibility (validated 2026-10-24), measurement by measurement: "none" (nothing is seen) when at least
|
||||
HALF of the usually seen targets (under their threshold at least half the time) are rejected, "reduced" (average
|
||||
data) otherwise; one episode per period of the same level, with the level field.
|
||||
6. Bounds of a Poor visibility (WILDSTRUBEL, batch 01): from the first to the last measurement where a target is
|
||||
rejected (> error_threshold), and no episode without a rejected target. On a road in full sun, the daytime
|
||||
error_value (×1.5 to ×3, heat shimmer) made reduced visibility last for hours without any rejection; references do
|
||||
not count: R3 is above its threshold almost all day.
|
||||
7. Poor visibility or object (boundary "Visibility or Obstruction" of the families document; BREITHORN, batch 01): a
|
||||
nuisance touching at most FEW targets, and no reference, is not diffuse: it is an object in front of them (a parked
|
||||
vehicle), a "Target obscured" episode of its whole duration, without a level.
|
||||
8. Diffuse nuisance, with or without rejections (wf-improve session Visibility, mod. 1, S2): at each instant, the median,
|
||||
over the sensors, of the error_value in MAD above each one's median; at least RUN_MIN instants in a row at K_DIFFUSE
|
||||
or more make a Poor visibility "reduced". A diffuse nuisance entirely within heat shimmer is the heat shimmer, and
|
||||
within a Poor visibility of point 3, that one (the same instants twice, two events: wf-improve, batch 2026-11-05, S2):
|
||||
no episode (otherwise it absorbed the jump back of the day before, WILDSTRUBEL on 04.10). Finds the drops of
|
||||
visibility of the whole scene without rejection at JUNGFRAU, FLORIDA, NORDEND; at 2, nights labelled thermal at
|
||||
JUNGFRAU; at 3, half the gain.
|
||||
|
||||
A simpler version (share of targets rejected above their threshold, without comparing each sensor with its own level)
|
||||
was tried on 2026-10-24 and dropped: worse on five data sets out of six (git history of Support-Agent,
|
||||
detectors/archive/visibilite_simple.py).
|
||||
1. High measurement: error_value above the median of its sensor + K_MAD MAD and above FACTOR times that median.
|
||||
2. Brief obstruction (one instant): at least BRIEF_MIN sensors high, SALIENCE times more than at each neighbouring instant
|
||||
(a train, a vehicle); or fewer than BRIEF_MIN sensors, at most one reference, rejected at FACTOR times their median
|
||||
and not at the neighbouring instants (something passing in front of one or two targets).
|
||||
3. Poor visibility: runs of at least RUN_MIN high measurements of a sensor (steps of at most GAP_STEPS), outside the brief
|
||||
instants, overlapping on at least two sensors. Level of each instant: "none" (greyed) when at least HALF of the
|
||||
usually seen targets are rejected, "reduced" otherwise; one episode per period of the same level.
|
||||
4. Diffuse nuisance: at least RUN_MIN instants in a row where the median over the sensors of the error_value, in MAD
|
||||
above each one's median, reaches K_DIFFUSE: Poor visibility "reduced", unless within heat shimmer or another Poor
|
||||
visibility.
|
||||
5. Target obscured: a target rejected above K_TARGET MAD at least TARGET_MIN times, less than TARGET_GAP apart, mostly
|
||||
outside the episodes above (leaves, snow, glare on one target).
|
||||
Heat shimmer: end of file.
|
||||
"""
|
||||
import statistics
|
||||
from datetime import timedelta
|
||||
|
||||
from data import iso
|
||||
from common import cadence, episode, level, runs, targets
|
||||
from data import is_reference
|
||||
|
||||
K_MAD = 10 # tuned on E-001 (fog: 11 to 57 MAD) and E-005 (rain, trains: 10 to 70 MAD)
|
||||
FACTOR = 2 # drops sensors with a near-zero MAD (E-004: PWD_REF10 ×1.6 at 14 MAD, E-005)
|
||||
RUN_MIN = 3 # high measurements in a row for poor visibility (E-001, E-005 rain)
|
||||
BRIEF_MIN = 3 # tuned on E-005 (trains: 5 to 17 sensors; the one with 2 targets is missed)
|
||||
K_TARGET = 5 # tuned on E-005 (leaves: 6 to 25 MAD, rejected)
|
||||
TARGET_MIN = 3 # repeated rejections of a target
|
||||
FRAC = 0.15 # cuts the tails of PR09 + CTR01 (NORDEND October) without cutting the cobweb (E-002)
|
||||
GAP_STEPS = 2.5 # steps tolerated between two high measurements of the same run (a missing step, clock to the second)
|
||||
BREAK = timedelta(hours=2) # lull separating two episodes (NORDEND October, night of 13 to 14.10)
|
||||
SALIENCE = 3 # a train stands out at least 3 times above the neighbouring instants (E-005: 5 to 17 against 0 to 2;
|
||||
# the cobweb of E-002 varies from 2 to 9 from one step to the next)
|
||||
TARGET_GAP = timedelta(hours=6) # tuned on E-005 (leaves: 7 rejections in 12 h, up to 5 h apart)
|
||||
REJECT_FACTOR = 1.2 # a rejection at 10 MAD also counts (FLORIDA, batch 01: levels of 0.6, ×2 out of reach); drops C24
|
||||
# of E-009 (level 0.68 just under the threshold 0.7, rejected at ×1.04)
|
||||
HALF = 0.5 # "nothing is seen"; tuned on NORDEND (09.10 23:00: 68 % rejected) and E-005 (rain: 24 to 33 %)
|
||||
K_DIFFUSE = 2.5 # diffuse nuisance (8); wf-improve session Visibility, mod. 1, S2
|
||||
FEW = 3 # Poor visibility on at most 3 targets, no reference: an object, Target obscured (7); BREITHORN,
|
||||
# batch 01 (vehicles parked in front of 2 or 3 targets out of 12)
|
||||
K_MAD = 10 # E-001 (fog: 11 to 57 MAD), E-005 (rain, trains: 10 to 70 MAD)
|
||||
FACTOR = 2 # drops the sensors with a near-zero MAD (E-004, E-005)
|
||||
RUN_MIN = 3 # E-001, rain of E-005
|
||||
GAP_STEPS = 2.5 # steps tolerated inside a run (a missing step, timestamp jitter)
|
||||
BRIEF_MIN = 3 # trains of E-005: 5 to 17 sensors
|
||||
SALIENCE = 3 # a train stands out (E-005: 5 to 17 against 0 to 2); the cobweb of E-002 varies from 2 to 9
|
||||
HALF = 0.5 # "nothing is seen": NORDEND 09.10 23:00 (68 % rejected) against the rain of E-005 (24 to 33 %)
|
||||
K_DIFFUSE = 2.5 # wf-improve Visibility, mod. 1, S2 (JUNGFRAU, FLORIDA, NORDEND: the whole scene less clear)
|
||||
K_TARGET = 5 # leaves of E-005: 6 to 25 MAD, rejected
|
||||
TARGET_MIN = 3
|
||||
TARGET_GAP = timedelta(hours=6) # leaves of E-005: 7 rejections in 12 h, up to 5 h apart
|
||||
|
||||
|
||||
def _step(measurements):
|
||||
ts = sorted({m["t"] for v in measurements.values() for m in v})
|
||||
return statistics.median(b - a for a, b in zip(ts, ts[1:]))
|
||||
|
||||
|
||||
def _group(points, gap):
|
||||
"""[(t, sensor)] sorted by t -> groups [(start, end, {sensors}, [t])] whose points are ≤ gap from each other."""
|
||||
groups = []
|
||||
for t, s in points:
|
||||
if groups and t - groups[-1][1] <= gap:
|
||||
g = groups[-1]
|
||||
groups[-1] = (g[0], t, g[2] | {s}, g[3] + [t])
|
||||
else:
|
||||
groups.append((t, t, {s}, [t]))
|
||||
return groups
|
||||
|
||||
|
||||
def _group_intervals(intervals, gap):
|
||||
"""[(start, end, sensor)] sorted -> [(start, end, {sensors})] of the intervals that overlap (within gap)."""
|
||||
out = []
|
||||
for a, b, s in intervals:
|
||||
if out and a - out[-1][1] <= gap:
|
||||
out[-1] = (out[-1][0], max(out[-1][1], b), out[-1][2] | {s})
|
||||
else:
|
||||
out.append((a, b, {s}))
|
||||
return out
|
||||
|
||||
|
||||
def _episode(start, end, label, text, sensors, level=None):
|
||||
e = {"start": iso(start), "end": iso(end), "kind": "Visibility" if label == "Poor visibility" else "Obstruction",
|
||||
"label": label, "text": text,
|
||||
"ignore": level != "reduced", "sensors": sorted(sensors)} # reduced visibility: average data, not greyed
|
||||
return {**e, "level": level} if level else e
|
||||
BRIEF = "Something passed in front of {} for a single measurement. Ignore this point on the curves."
|
||||
OBSCURED = "{} was hidden several times while the targets around it stayed clear. Ignore its curve in this period."
|
||||
|
||||
|
||||
def _levels(measurements, instants):
|
||||
@@ -118,220 +48,132 @@ def _levels(measurements, instants):
|
||||
for t in instants}
|
||||
|
||||
|
||||
def episodes(measurements):
|
||||
"""All error_value episodes: Visibility (Poor visibility), Obstruction (Brief obstruction, Target obscured)."""
|
||||
step = _step(measurements)
|
||||
gap = GAP_STEPS * step
|
||||
refs = {s for s, v in measurements.items() if v[0]["pattern_type"] == "global_ref"}
|
||||
instants = sorted({m["t"] for v in measurements.values() for m in v})
|
||||
high, rejections = {}, {} # sensor -> [t]
|
||||
refused = set() # instants where a target (not a global_ref) is above its threshold
|
||||
rejected_targets = {} # t -> {targets above their threshold}
|
||||
strong, refusals = {}, {} # t -> {sensors rejected at more than FACTOR times their median}, {sensors rejected}
|
||||
for s, v in measurements.items():
|
||||
e = [m["error_value"] for m in v if m["error_value"] is not None]
|
||||
if len(e) < 10:
|
||||
continue
|
||||
med = statistics.median(e)
|
||||
mad = statistics.median(abs(x - med) for x in e) or 1e-9
|
||||
for m in v:
|
||||
x = m["error_value"]
|
||||
if x is None:
|
||||
continue
|
||||
rejected = m["error_threshold"] and x > m["error_threshold"]
|
||||
if rejected and s not in refs:
|
||||
refused.add(m["t"])
|
||||
rejected_targets.setdefault(m["t"], set()).add(s)
|
||||
if x > med + K_MAD * mad and (x > FACTOR * med or rejected and x > REJECT_FACTOR * med):
|
||||
high.setdefault(s, []).append(m["t"])
|
||||
if rejected:
|
||||
refusals.setdefault(m["t"], set()).add(s)
|
||||
if x > FACTOR * med:
|
||||
strong.setdefault(m["t"], set()).add(s)
|
||||
if s not in refs and rejected and x > med + K_TARGET * mad:
|
||||
rejections.setdefault(s, []).append(m["t"])
|
||||
by_t = {} # t -> {high sensors}
|
||||
for s, ts in high.items():
|
||||
for t in ts:
|
||||
by_t.setdefault(t, set()).add(s)
|
||||
n = lambda t: len(by_t.get(t, ())) # noqa: E731
|
||||
out = []
|
||||
|
||||
# 2. Brief obstruction: an instant where many sensors are high, at least SALIENCE times more than the neighbours.
|
||||
briefs = set()
|
||||
for i, t in enumerate(instants):
|
||||
neighbours = [n(instants[j]) for j in (i - 1, i + 1) if 0 <= j < len(instants)]
|
||||
if n(t) >= BRIEF_MIN and n(t) >= SALIENCE * max(neighbours):
|
||||
briefs.add(t)
|
||||
only = rejected_targets.get(t, set()).difference(*(rejected_targets.get(instants[j], set()) for j in (i - 1, i + 1)
|
||||
if 0 <= j < len(instants)))
|
||||
if by_t[t] & refs and len(only) < BRIEF_MIN: # references touched without targets rejected at this instant
|
||||
continue # alone: not an obstacle in front of targets, the whole image
|
||||
out.append(_episode(t, t, "Brief obstruction", f"{n(t)} points were poorly seen for a single measurement, then came "
|
||||
"back: probably a train, a vehicle or an animal passing; ignore this point on the curves.", by_t[t]))
|
||||
elif 0 < len(f := strong.get(t, set())) < BRIEF_MIN and n(t) < BRIEF_MIN and len(f & refs) <= 1 and not f & set().union(
|
||||
*(refusals.get(instants[j], set()) for j in (i - 1, i + 1) if 0 <= j < len(instants))):
|
||||
# one or two targets, or a reference, rejected at this single measurement, at FACTOR times their median
|
||||
out.append(_episode(t, t, "Brief obstruction", f"{len(f)} point{'s were' if len(f) > 1 else ' was'} poorly seen "
|
||||
"for a single measurement, then came back: probably something passing; ignore this point on the "
|
||||
"curves.", f))
|
||||
|
||||
# 3. Poor visibility: long runs per sensor (without the brief instants), joined across sensors, then reduced to the
|
||||
# instants where at least FRAC of the episode's peak is high (cuts the tails of one or two sensors and the long
|
||||
# lulls, more than BREAK).
|
||||
runs = []
|
||||
for s, ts in high.items():
|
||||
pts = [(t, s) for t in ts if t not in briefs]
|
||||
runs += [(a, b, s) for a, b, _, l in _group(pts, gap) if len(l) >= RUN_MIN]
|
||||
zones = [z for z in _group_intervals(sorted(runs), gap) if len(z[2]) >= 2]
|
||||
kept = []
|
||||
for a, b, _ in zones:
|
||||
inside = [t for t in instants if a <= t <= b and t not in briefs]
|
||||
peak = max(n(t) for t in inside)
|
||||
pts = [(t, s) for t in inside if n(t) >= max(1, FRAC * peak) for s in by_t[t]]
|
||||
kept += [(d, f, cs) for d, f, cs, l in _group(pts, max(gap, BREAK)) if len(cs) >= 2 and len(set(l)) >= RUN_MIN]
|
||||
# Bounds: from the first to the last rejected target; without a rejection, no episode (6).
|
||||
kept = [(r[0], r[-1], cs) for a, b, cs in kept if (r := [t for t in instants if a <= t <= b and t in refused])]
|
||||
level = _levels(measurements, instants)
|
||||
for a, b, cs in kept: # one episode per period of the same level (5); the text comes from categories.py
|
||||
if len(cs) <= FEW and not cs & refs: # an object in front of a few targets, not a diffuse nuisance (7)
|
||||
out.append(_episode(a, b, "Target obscured", "", cs))
|
||||
continue
|
||||
seg = []
|
||||
for t in [t for t in instants if a <= t <= b] + [None]:
|
||||
if seg and (t is None or level[t] != level[seg[0]]):
|
||||
who = cs & set().union(*(by_t.get(u, set()) for u in seg)) or cs
|
||||
out.append(_episode(seg[0], seg[-1], "Poor visibility", "", who, level[seg[0]]))
|
||||
seg = []
|
||||
if t is not None:
|
||||
seg.append(t)
|
||||
inside_any = lambda t: t in briefs or any(a <= t <= b for a, b, _ in kept) # noqa: E731
|
||||
|
||||
# 4. Target obscured: a target rejected again and again, mostly outside the episodes above.
|
||||
targets = []
|
||||
for s, ts in rejections.items():
|
||||
for a, b, _, l in _group([(t, s) for t in ts], max(TARGET_GAP, 2 * step)):
|
||||
if len(l) >= (TARGET_MIN if TARGET_MIN * step <= TARGET_GAP else 2) and sum(map(inside_any, l)) * 2 < len(l):
|
||||
targets.append((a, b, s))
|
||||
for a, b, cs in _group_intervals(sorted(targets), timedelta(0)):
|
||||
who, it, its = ("One target was", "it", "its") if len(cs) == 1 else (f"{len(cs)} targets were", "them", "their")
|
||||
out.append(_episode(a, b, "Target obscured", f"{who} repeatedly lost while the targets around stayed clear: probably "
|
||||
f"something in front of {it} (leaves, snow) or glare; {its} displacements in this period are unreliable.", cs))
|
||||
return sorted(out, key=lambda e: e["start"])
|
||||
|
||||
|
||||
def diffuse(measurements):
|
||||
"""Diffuse nuisance (8): the runs of at least RUN_MIN instants where the median, over the sensors, of the error_value
|
||||
in MAD above each one's median reaches K_DIFFUSE, with or without rejections -> Poor visibility "reduced"."""
|
||||
z = {} # t -> {sensor: gap in MAD}
|
||||
for s, v in measurements.items():
|
||||
e = [m["error_value"] for m in v if m["error_value"] is not None]
|
||||
if len(e) < 10:
|
||||
continue
|
||||
med = statistics.median(e)
|
||||
mad = statistics.median(abs(x - med) for x in e) or 1e-9
|
||||
for m in v:
|
||||
if m["error_value"] is not None:
|
||||
z.setdefault(m["t"], {})[s] = (m["error_value"] - med) / mad
|
||||
highs = [(t, None) for t in sorted(z) if statistics.median(z[t].values()) >= K_DIFFUSE]
|
||||
return [_episode(a, b, "Poor visibility", "", {s for t in l for s, x in z[t].items() if x >= K_DIFFUSE}, "reduced")
|
||||
for a, b, _, l in _group(highs, GAP_STEPS * _step(measurements)) if len(l) >= RUN_MIN]
|
||||
def _poor(start, end, sensors, level, refs):
|
||||
refs = ", including the reference targets" if set(sensors) & refs else ""
|
||||
text = (f"The camera could not see {targets(len(sensors))} clearly{refs}. The curves in this period are not "
|
||||
"reliable: ignore them." if level == "none" else f"The camera saw {targets(len(sensors))} less clearly than "
|
||||
f"usual{refs}. The data in this period are less precise: use them with care.")
|
||||
return episode(start, end, "Visibility", "Poor visibility", text, sensors, level == "none", level=level)
|
||||
|
||||
|
||||
def detect(measurements):
|
||||
shimmer = heat_shimmer(measurements) # a diffuse nuisance during heat shimmer is the heat shimmer (8)
|
||||
eps = episodes(measurements)
|
||||
poor = [e for e in eps if e["kind"] == "Visibility"] # a diffuse nuisance within one of them is it
|
||||
return eps + shimmer + [e for e in diffuse(measurements)
|
||||
if not any(t["start"] <= e["start"] and e["end"] <= t["end"] for t in shimmer + poor)]
|
||||
gap = GAP_STEPS * cadence(measurements)
|
||||
refs = {s for s, v in measurements.items() if is_reference(v)}
|
||||
instants = sorted({m["t"] for v in measurements.values() for m in v})
|
||||
z = {} # t -> {sensor: error_value in MAD above the sensor's median}
|
||||
high, strong, rejected, repeated = {}, {}, {}, {} # t -> {sensors}; repeated: target -> [t]
|
||||
for s, v in measurements.items():
|
||||
e = [m["error_value"] for m in v if m["error_value"] is not None]
|
||||
if len(e) < 10:
|
||||
continue
|
||||
med, mad = level(e)
|
||||
for m in v:
|
||||
t, x = m["t"], m["error_value"]
|
||||
if x is None:
|
||||
continue
|
||||
z.setdefault(t, {})[s] = (x - med) / mad
|
||||
if x > med + K_MAD * mad and x > FACTOR * med:
|
||||
high.setdefault(t, set()).add(s)
|
||||
if m["error_threshold"] and x > m["error_threshold"]:
|
||||
rejected.setdefault(t, set()).add(s)
|
||||
if x > FACTOR * med:
|
||||
strong.setdefault(t, set()).add(s)
|
||||
if s not in refs and x > med + K_TARGET * mad:
|
||||
repeated.setdefault(s, []).append(t)
|
||||
n = lambda t: len(high.get(t, ())) # noqa: E731
|
||||
out, briefs = [], set()
|
||||
|
||||
for i, t in enumerate(instants): # 2. Brief obstruction
|
||||
around = [instants[j] for j in (i - 1, i + 1) if 0 <= j < len(instants)]
|
||||
few = strong.get(t, set())
|
||||
if n(t) >= BRIEF_MIN and n(t) >= SALIENCE * max(map(n, around), default=0):
|
||||
briefs.add(t)
|
||||
few = high[t]
|
||||
elif not (few and len(few) < BRIEF_MIN and n(t) < BRIEF_MIN and len(few & refs) <= 1
|
||||
and not few & set().union(*(rejected.get(u, set()) for u in around))):
|
||||
continue
|
||||
out.append(episode(t, t, "Obstruction", "Brief obstruction", BRIEF.format(targets(len(few))), few, True))
|
||||
|
||||
spans = [] # 3. Poor visibility: runs of each sensor, then the overlapping runs of at least two sensors
|
||||
for s in {s for t in high for s in high[t]}:
|
||||
spans += [(r[0], r[-1], s) for r in runs([t for t in instants if s in high.get(t, ()) and t not in briefs], gap)
|
||||
if len(r) >= RUN_MIN]
|
||||
zones = []
|
||||
for a, b, s in sorted(spans):
|
||||
if zones and a - zones[-1][1] <= gap:
|
||||
zones[-1] = (zones[-1][0], max(zones[-1][1], b), zones[-1][2] | {s})
|
||||
else:
|
||||
zones.append((a, b, {s}))
|
||||
zones = [zone for zone in zones if len(zone[2]) >= 2]
|
||||
lv = _levels(measurements, instants)
|
||||
for a, b, cs in zones:
|
||||
inside = [t for t in instants if a <= t <= b]
|
||||
for k, t in enumerate(inside):
|
||||
seg = [t] if k == 0 or lv[t] != lv[inside[k - 1]] else seg + [t]
|
||||
if k == len(inside) - 1 or lv[inside[k + 1]] != lv[t]:
|
||||
out.append(_poor(seg[0], seg[-1], cs & set().union(*(high.get(u, set()) for u in seg)) or cs, lv[t], refs))
|
||||
|
||||
shimmer = heat_shimmer(measurements) # 4. diffuse nuisance
|
||||
hosts = shimmer + [e for e in out if e["kind"] == "Visibility"]
|
||||
for r in runs([t for t in sorted(z) if statistics.median(z[t].values()) >= K_DIFFUSE], gap):
|
||||
e = _poor(r[0], r[-1], {s for t in r for s, x in z[t].items() if x >= K_DIFFUSE}, "reduced", refs)
|
||||
if len(r) >= RUN_MIN and not any(h["start"] <= e["start"] and e["end"] <= h["end"] for h in hosts):
|
||||
out.append(e)
|
||||
|
||||
seen = lambda t: t in briefs or any(a <= t <= b for a, b, _ in zones) # noqa: E731
|
||||
for s, ts in repeated.items(): # 5. Target obscured
|
||||
for r in runs(ts, TARGET_GAP):
|
||||
if len(r) >= TARGET_MIN and 2 * sum(map(seen, r)) < len(r):
|
||||
out.append(episode(r[0], r[-1], "Obstruction", "Target obscured", OBSCURED.format(s), [s], True))
|
||||
return out + shimmer
|
||||
|
||||
|
||||
# --- Heat shimmer ----------------------------------------------------------------------------------------------------
|
||||
# Atmospheric turbulence (fast noise on the corrected displacements). Method:
|
||||
# 1. Per sensor and per hour H, on the measurements of [H − 1 h, H + 2 h): agitation = median of |Δcorr_x| + |Δcorr_y|
|
||||
# between successive measurements (robust to one-step jumps), and median of the error_value.
|
||||
# 2. Each is divided by the sensor's calm: the lower quartile of its hourly values over the export (often the night).
|
||||
# 3. Turbulent hour: at least SHARE of the sensors at ≥ RATIO times their calm agitation, and median error_value of the
|
||||
# sensors at ≥ RATIO_ERR times the calm (warm air blurs the spots: E-003).
|
||||
# 4. Neighbouring hours merged (a one-hour hole tolerated), episode of at least MIN_DURATION, dropped if more than
|
||||
# REJECT_MAX of the measurements are rejected beyond their sensor's habit (its share of rejections over the export):
|
||||
# it is then poor visibility, another rule. A target that is always rejected (Target issue) does not count (wf-improve
|
||||
# session Visibility, mod. 1, S1: WILDSTRUBEL, R3 rejected all day, the days of heat shimmer were dropped).
|
||||
# Greyed if the median agitation ≥ STRONG.
|
||||
RATIO = 1.5 # tuned on E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night)
|
||||
SHARE = 0.5 # share of the sensors agitated at the same time
|
||||
RATIO_ERR = 1.5 # tuned on E-003 (error_value ×1.7 to ×3.2 the night; ≈ ×1 at noon elsewhere)
|
||||
REJECT_MAX = 0.15 # E-003 ≤ 7 %; cobweb (E-002) and rain (E-005) ≥ 18 %; rejections beyond the habit since wf-improve
|
||||
# session Visibility, mod. 1, S1 (WILDSTRUBEL: 4 to 14 %, against 9 to 22 % raw)
|
||||
STRONG = 2.0 # E-003 (×2.3 to ×2.6) is strong; an episode between ×1.5 and ×2 stays visible
|
||||
MIN_DURATION = timedelta(hours=3) # E-003 lasts 10-11 h; drops the morning of the real movement of E-004 (2 h), which
|
||||
# the classification drops too, whatever the duration
|
||||
# --- Heat shimmer: fast noise of the corrected displacements in warm air.
|
||||
# 1. Per sensor and per hour H, over [H - 1 h, H + 2 h): agitation, the median of |Δcorr_x| + |Δcorr_y| between successive
|
||||
# measurements, and the median error_value; each divided by its calm (lower quartile of the sensor's hourly values).
|
||||
# 2. Turbulent hour: at least SHARE of the sensors agitated at RATIO times their calm, and median error_value at RATIO.
|
||||
# 3. Turbulent hours at most 2 h apart make an episode, dropped if more than REJECT_MAX of its measurements are rejected
|
||||
# beyond each sensor's habit (its share of rejections over the export): that is poor visibility. Greyed if the median
|
||||
# agitation reaches STRONG.
|
||||
RATIO = 1.5 # E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night, error_value ×1.7 to ×3.2)
|
||||
SHARE = 0.5
|
||||
REJECT_MAX = 0.15 # E-003 ≤ 7 %; cobweb (E-002) and rain (E-005) ≥ 18 %; beyond the habit: wf-improve Visibility, mod. 1, S1
|
||||
STRONG = 2.0 # E-003 (×2.3 to ×2.6)
|
||||
HOUR = timedelta(hours=1)
|
||||
|
||||
|
||||
def _hour(t):
|
||||
return t.replace(minute=0, second=0, microsecond=0)
|
||||
SHIMMER = "Warm air made the readings shake ({:.1f}× noisier than in calm hours). The small fast wiggles are not movement."
|
||||
|
||||
|
||||
def _ratios(points):
|
||||
"""[(t, value)] -> {hour H: median over [H − 1 h, H + 2 h) / lower quartile of these medians}."""
|
||||
"""[(t, value)] -> {hour H: median over [H - 1 h, H + 2 h) / lower quartile of these medians}."""
|
||||
by = {}
|
||||
for t, v in points:
|
||||
for k in (-1, 0, 1):
|
||||
by.setdefault(_hour(t) + k * HOUR, []).append(v)
|
||||
by.setdefault(t.replace(minute=0, second=0, microsecond=0) + k * HOUR, []).append(v)
|
||||
med = {h: statistics.median(v) for h, v in by.items() if len(v) >= 3}
|
||||
if len(med) < 8:
|
||||
return {}
|
||||
calm = statistics.quantiles(med.values(), n=4)[0]
|
||||
calm = statistics.quantiles(med.values(), n=4)[0] if len(med) >= 8 else 0
|
||||
return {h: v / calm for h, v in med.items()} if calm > 0 else {}
|
||||
|
||||
|
||||
def hours(measurements):
|
||||
"""-> {hour: {sensor: (agitation / calm, error_value / calm)}}."""
|
||||
out = {}
|
||||
def heat_shimmer(measurements):
|
||||
hours = {} # hour -> {sensor: (agitation / calm, error_value / calm)}
|
||||
for s, ms in measurements.items():
|
||||
ag = _ratios([(b["t"], abs(b["corr_x"] - a["corr_x"]) + abs(b["corr_y"] - a["corr_y"]))
|
||||
for a, b in zip(ms, ms[1:]) if a["corr_x"] is not None and b["corr_x"] is not None])
|
||||
er = _ratios([(m["t"], m["error_value"]) for m in ms if m["error_value"] is not None])
|
||||
for h, r in ag.items():
|
||||
if h in er:
|
||||
out.setdefault(h, {})[s] = (r, er[h])
|
||||
return out
|
||||
|
||||
|
||||
def _turbulent(r):
|
||||
return (sum(a >= RATIO for a, _ in r.values()) >= SHARE * len(r)
|
||||
and statistics.median(e for _, e in r.values()) >= RATIO_ERR)
|
||||
|
||||
|
||||
def heat_shimmer(measurements):
|
||||
H = hours(measurements)
|
||||
blocks = []
|
||||
for h in sorted(h for h, r in H.items() if _turbulent(r)):
|
||||
if blocks and h - blocks[-1][-1] <= 2 * HOUR:
|
||||
blocks[-1].append(h)
|
||||
else:
|
||||
blocks.append([h])
|
||||
habit = {s: sum(m["corr_x"] is None for m in ms) / len(ms) for s, ms in measurements.items()} # usual rejections
|
||||
episodes = []
|
||||
for b in blocks:
|
||||
for h in ag.keys() & er.keys():
|
||||
hours.setdefault(h, {})[s] = (ag[h], er[h])
|
||||
turbulent = [h for h, r in sorted(hours.items()) if sum(a >= RATIO for a, _ in r.values()) >= SHARE * len(r)
|
||||
and statistics.median(e for _, e in r.values()) >= RATIO]
|
||||
habit = {s: sum(m["corr_x"] is None for m in ms) / len(ms) for s, ms in measurements.items()}
|
||||
out = []
|
||||
for b in runs(turbulent, 2 * HOUR):
|
||||
start, end = b[0], b[-1] + HOUR
|
||||
by = {s: [m for m in ms if start <= m["t"] < end] for s, ms in measurements.items()}
|
||||
new = sum(max(0, sum(m["corr_x"] is None for m in d) - habit[s] * len(d)) for s, d in by.items())
|
||||
if end - start < MIN_DURATION or new > REJECT_MAX * sum(map(len, by.values())):
|
||||
if new > REJECT_MAX * sum(map(len, by.values())):
|
||||
continue
|
||||
by_sensor = {}
|
||||
for h in b:
|
||||
for s, (a, _) in H[h].items():
|
||||
by_sensor.setdefault(s, []).append(a)
|
||||
agit = {s: statistics.median(v) for s, v in by_sensor.items()}
|
||||
agit = {s: statistics.median(hours[h][s][0] for h in b if s in hours[h]) for s in {s for h in b for s in hours[h]}}
|
||||
strength = statistics.median(agit.values())
|
||||
episodes.append({
|
||||
"start": iso(start), "end": iso(end), "kind": "Visibility",
|
||||
"label": "Heat shimmer",
|
||||
"text": f"Readings were about {strength:.1f}× noisier than in calm hours because of heat shimmer "
|
||||
"(warm air trembling in the line of sight); the small fast wiggles are not movement.",
|
||||
"ignore": strength >= STRONG,
|
||||
"sensors": sorted(s for s, v in agit.items() if v >= RATIO),
|
||||
})
|
||||
return episodes
|
||||
out.append(episode(start, end, "Visibility", "Heat shimmer", SHIMMER.format(strength),
|
||||
[s for s, v in agit.items() if v >= RATIO], strength >= STRONG))
|
||||
return out
|
||||
Reference in new issue
Block a user