visibility: shared helpers (common.py), five rules fewer, texts of its events

Removed (their benchmark contribution, each alone): the reference exception of Brief obstruction (none; Obstruction
found +9 at LENZSPITZE), REJECT_FACTOR (Obstruction correct +2), FRAC and BREAK (Visibility found -3, Obstruction
found +4), the bounds at the rejected targets (neutral), FEW (Visibility found -1), the TARGET_MIN exception at a slow
cadence (none in total), MIN_DURATION of heat shimmer (Visibility found -1). RATIO_ERR merged into RATIO (same value).
Target obscured: one episode per target. The event sentences of its varieties are now the episodes' texts.
common.py: episode, runs, noise, level, cadence, targets.

Benchmark Totals: Visibility found 47 -> 55 %, correct 77 -> 72 %; Obstruction found 26 -> 22 %, correct 28 -> 32 %;
the other families unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-16 16:28:52 +01:00
1 parent 5f085bf5ff
commit 5c2d265a21
2 files changed
+181 -292

No files matched your search

+47
View File
@@ -0,0 +1,47 @@
"""Helpers shared by the detectors."""
import statistics
from datetime import timedelta
from data import iso
def episode(start, end, kind, label, text, sensors, ignore, **extra):
return {"start": iso(start), "end": iso(end), "kind": kind, "label": label, "text": text, "ignore": ignore,
"sensors": sorted(sensors), **extra}
def runs(times, gap):
"""Sorted times -> [[t, ...]]: runs of times at most gap apart."""
out = []
for t in times:
if out and t - out[-1][-1] <= gap:
out[-1].append(t)
else:
out.append([t])
return out
def noise(v):
"""Median step of a series, from one value to the next."""
return statistics.median(abs(y - x) for x, y in zip(v, v[1:])) if len(v) > 1 else 0
def level(v):
"""-> (median, MAD) of the values."""
med = statistics.median(v)
return med, statistics.median(abs(x - med) for x in v) or 1e-9
def cadence(measurements):
"""Cycle of the measurements: the median gap between two instants, at least an hour divided by the median number of
instants per clock hour (a station measuring in bursts, three measurements a minute apart every 15 min: 15 min)."""
ts = sorted({m["t"].replace(second=0, microsecond=0) for ms in measurements.values() for m in ms})
per_hour = {}
for t in ts:
per_hour[t.replace(minute=0)] = per_hour.get(t.replace(minute=0), 0) + 1
gap = statistics.median(b - a for a, b in zip(ts, ts[1:])) if len(ts) > 1 else timedelta(hours=1)
return max(gap, timedelta(hours=1) / statistics.median(per_hour.values()))
def targets(n):
return "one target" if n == 1 else f"{n} targets"
+134 -292
View File
@@ -1,110 +1,40 @@
"""Detector of the Visibility family (the families document): the camera sees less well, the image is faint or blurred """Visibility family: the camera sees the targets less well (error_value), or the air trembles (heat shimmer). The same
(error_value), or it trembles (heat shimmer, end of file). The same error_value computation also finds objects in front error_value also finds objects in front of a few targets: Obstruction family (Brief obstruction, Target obscured).
of a few targets (Brief obstruction, Target obscured): episodes() returns them with the Obstruction family, which
detect() leaves to obstruction.py.
Method: 1. High measurement: error_value above the median of its sensor + K_MAD MAD and above FACTOR times that median.
1. "High" measurement: error_value > median + K_MAD MAD of the sensor over the export, and > FACTOR × median, or 2. Brief obstruction (one instant): at least BRIEF_MIN sensors high, SALIENCE times more than at each neighbouring instant
rejected (> error_threshold) and > REJECT_FACTOR × median: a sensor at a usual level of 0.6 never doubles, and (a train, a vehicle); or fewer than BRIEF_MIN sensors, at most one reference, rejected at FACTOR times their median
FLORIDA (batch 01) showed its fogs as Target obscured. and not at the neighbouring instants (something passing in front of one or two targets).
2. Brief obstruction: an instant where at least BRIEF_MIN sensors are high, SALIENCE times more than at the neighbouring 3. Poor visibility: runs of at least RUN_MIN high measurements of a sensor (steps of at most GAP_STEPS), outside the brief
instants (train, vehicle), with no reference (global_ref) among them: a one-step peak touching the references is not instants, overlapping on at least two sensors. Level of each instant: "none" (greyed) when at least HALF of the
an obstacle in front of targets (peak of the whole image, NORDEND on 01.09 11:15, batch 01); it is not shown. Unless usually seen targets are rejected, "reduced" otherwise; one episode per period of the same level.
at least BRIEF_MIN targets are rejected at this instant and neither at the instant before nor after: something 4. Diffuse nuisance: at least RUN_MIN instants in a row where the median over the sensors of the error_value, in MAD
passing in front of them, a reference touched as well (wf-improve session Displacement, mod. 1, S5: trains of above each one's median, reaches K_DIFFUSE: Poor visibility "reduced", unless within heat shimmer or another Poor
NIEDERHORN, 12 and 13 targets rejected). visibility.
Also, something passing in front of one or two targets, or a single reference: at an instant, fewer than BRIEF_MIN 5. Target obscured: a target rejected above K_TARGET MAD at least TARGET_MIN times, less than TARGET_GAP apart, mostly
sensors rejected at more than FACTOR times their median (without K_MAD: by day, heat shimmer inflates the MAD), at outside the episodes above (leaves, snow, glare on one target).
most one reference, none of them rejected at the instant before or after, and fewer than BRIEF_MIN high sensors Heat shimmer: end of file.
(wf-improve session Obstruction, mod. 1, S3: vehicles, passers-by, insects in front of one or two targets of
WILDSTRUBEL, NIEDERHORN, LENZSPITZE; a reference hidden by a truck at JUNGFRAU, BREITHORN).
3. Poor visibility: runs of at least RUN_MIN high measurements per sensor, joined across sensors (at least 2), reduced
to the instants where at least FRAC of the episode's peak is high.
4. Target obscured: a target (not a global_ref) rejected (> error_threshold) and above K_TARGET MAD, at least TARGET_MIN
times less than TARGET_GAP apart, mostly outside the episodes of 2 and 3 (leaves, snow, glare on a target). The gap
is at least two measurement steps, and two rejections are enough when TARGET_MIN steps do not fit in TARGET_GAP
(wf-improve session Obstruction, mod. 1, S6: at one measurement every 4 h, vehicles parked at night in front of COLI22,
COLI24, POUI22 of BREITHORN make only two or three rejections). All episodes: ignore = True.
5. Level of a Poor visibility (validated 2026-10-24), measurement by measurement: "none" (nothing is seen) when at least
HALF of the usually seen targets (under their threshold at least half the time) are rejected, "reduced" (average
data) otherwise; one episode per period of the same level, with the level field.
6. Bounds of a Poor visibility (WILDSTRUBEL, batch 01): from the first to the last measurement where a target is
rejected (> error_threshold), and no episode without a rejected target. On a road in full sun, the daytime
error_value (×1.5 to ×3, heat shimmer) made reduced visibility last for hours without any rejection; references do
not count: R3 is above its threshold almost all day.
7. Poor visibility or object (boundary "Visibility or Obstruction" of the families document; BREITHORN, batch 01): a
nuisance touching at most FEW targets, and no reference, is not diffuse: it is an object in front of them (a parked
vehicle), a "Target obscured" episode of its whole duration, without a level.
8. Diffuse nuisance, with or without rejections (wf-improve session Visibility, mod. 1, S2): at each instant, the median,
over the sensors, of the error_value in MAD above each one's median; at least RUN_MIN instants in a row at K_DIFFUSE
or more make a Poor visibility "reduced". A diffuse nuisance entirely within heat shimmer is the heat shimmer, and
within a Poor visibility of point 3, that one (the same instants twice, two events: wf-improve, batch 2026-11-05, S2):
no episode (otherwise it absorbed the jump back of the day before, WILDSTRUBEL on 04.10). Finds the drops of
visibility of the whole scene without rejection at JUNGFRAU, FLORIDA, NORDEND; at 2, nights labelled thermal at
JUNGFRAU; at 3, half the gain.
A simpler version (share of targets rejected above their threshold, without comparing each sensor with its own level)
was tried on 2026-10-24 and dropped: worse on five data sets out of six (git history of Support-Agent,
detectors/archive/visibilite_simple.py).
""" """
import statistics import statistics
from datetime import timedelta from datetime import timedelta
from data import iso from common import cadence, episode, level, runs, targets
from data import is_reference
K_MAD = 10 # tuned on E-001 (fog: 11 to 57 MAD) and E-005 (rain, trains: 10 to 70 MAD) K_MAD = 10 # E-001 (fog: 11 to 57 MAD), E-005 (rain, trains: 10 to 70 MAD)
FACTOR = 2 # drops sensors with a near-zero MAD (E-004: PWD_REF10 ×1.6 at 14 MAD, E-005) FACTOR = 2 # drops the sensors with a near-zero MAD (E-004, E-005)
RUN_MIN = 3 # high measurements in a row for poor visibility (E-001, E-005 rain) RUN_MIN = 3 # E-001, rain of E-005
BRIEF_MIN = 3 # tuned on E-005 (trains: 5 to 17 sensors; the one with 2 targets is missed) GAP_STEPS = 2.5 # steps tolerated inside a run (a missing step, timestamp jitter)
K_TARGET = 5 # tuned on E-005 (leaves: 6 to 25 MAD, rejected) BRIEF_MIN = 3 # trains of E-005: 5 to 17 sensors
TARGET_MIN = 3 # repeated rejections of a target SALIENCE = 3 # a train stands out (E-005: 5 to 17 against 0 to 2); the cobweb of E-002 varies from 2 to 9
FRAC = 0.15 # cuts the tails of PR09 + CTR01 (NORDEND October) without cutting the cobweb (E-002) HALF = 0.5 # "nothing is seen": NORDEND 09.10 23:00 (68 % rejected) against the rain of E-005 (24 to 33 %)
GAP_STEPS = 2.5 # steps tolerated between two high measurements of the same run (a missing step, clock to the second) K_DIFFUSE = 2.5 # wf-improve Visibility, mod. 1, S2 (JUNGFRAU, FLORIDA, NORDEND: the whole scene less clear)
BREAK = timedelta(hours=2) # lull separating two episodes (NORDEND October, night of 13 to 14.10) K_TARGET = 5 # leaves of E-005: 6 to 25 MAD, rejected
SALIENCE = 3 # a train stands out at least 3 times above the neighbouring instants (E-005: 5 to 17 against 0 to 2; TARGET_MIN = 3
# the cobweb of E-002 varies from 2 to 9 from one step to the next) TARGET_GAP = timedelta(hours=6) # leaves of E-005: 7 rejections in 12 h, up to 5 h apart
TARGET_GAP = timedelta(hours=6) # tuned on E-005 (leaves: 7 rejections in 12 h, up to 5 h apart)
REJECT_FACTOR = 1.2 # a rejection at 10 MAD also counts (FLORIDA, batch 01: levels of 0.6, ×2 out of reach); drops C24
# of E-009 (level 0.68 just under the threshold 0.7, rejected at ×1.04)
HALF = 0.5 # "nothing is seen"; tuned on NORDEND (09.10 23:00: 68 % rejected) and E-005 (rain: 24 to 33 %)
K_DIFFUSE = 2.5 # diffuse nuisance (8); wf-improve session Visibility, mod. 1, S2
FEW = 3 # Poor visibility on at most 3 targets, no reference: an object, Target obscured (7); BREITHORN,
# batch 01 (vehicles parked in front of 2 or 3 targets out of 12)
BRIEF = "Something passed in front of {} for a single measurement. Ignore this point on the curves."
def _step(measurements): OBSCURED = "{} was hidden several times while the targets around it stayed clear. Ignore its curve in this period."
ts = sorted({m["t"] for v in measurements.values() for m in v})
return statistics.median(b - a for a, b in zip(ts, ts[1:]))
def _group(points, gap):
"""[(t, sensor)] sorted by t -> groups [(start, end, {sensors}, [t])] whose points are ≤ gap from each other."""
groups = []
for t, s in points:
if groups and t - groups[-1][1] <= gap:
g = groups[-1]
groups[-1] = (g[0], t, g[2] | {s}, g[3] + [t])
else:
groups.append((t, t, {s}, [t]))
return groups
def _group_intervals(intervals, gap):
"""[(start, end, sensor)] sorted -> [(start, end, {sensors})] of the intervals that overlap (within gap)."""
out = []
for a, b, s in intervals:
if out and a - out[-1][1] <= gap:
out[-1] = (out[-1][0], max(out[-1][1], b), out[-1][2] | {s})
else:
out.append((a, b, {s}))
return out
def _episode(start, end, label, text, sensors, level=None):
e = {"start": iso(start), "end": iso(end), "kind": "Visibility" if label == "Poor visibility" else "Obstruction",
"label": label, "text": text,
"ignore": level != "reduced", "sensors": sorted(sensors)} # reduced visibility: average data, not greyed
return {**e, "level": level} if level else e
def _levels(measurements, instants): def _levels(measurements, instants):
@@ -118,220 +48,132 @@ def _levels(measurements, instants):
for t in instants} for t in instants}
def episodes(measurements): def _poor(start, end, sensors, level, refs):
"""All error_value episodes: Visibility (Poor visibility), Obstruction (Brief obstruction, Target obscured).""" refs = ", including the reference targets" if set(sensors) & refs else ""
step = _step(measurements) text = (f"The camera could not see {targets(len(sensors))} clearly{refs}. The curves in this period are not "
gap = GAP_STEPS * step "reliable: ignore them." if level == "none" else f"The camera saw {targets(len(sensors))} less clearly than "
refs = {s for s, v in measurements.items() if v[0]["pattern_type"] == "global_ref"} f"usual{refs}. The data in this period are less precise: use them with care.")
instants = sorted({m["t"] for v in measurements.values() for m in v}) return episode(start, end, "Visibility", "Poor visibility", text, sensors, level == "none", level=level)
high, rejections = {}, {} # sensor -> [t]
refused = set() # instants where a target (not a global_ref) is above its threshold
rejected_targets = {} # t -> {targets above their threshold}
strong, refusals = {}, {} # t -> {sensors rejected at more than FACTOR times their median}, {sensors rejected}
for s, v in measurements.items():
e = [m["error_value"] for m in v if m["error_value"] is not None]
if len(e) < 10:
continue
med = statistics.median(e)
mad = statistics.median(abs(x - med) for x in e) or 1e-9
for m in v:
x = m["error_value"]
if x is None:
continue
rejected = m["error_threshold"] and x > m["error_threshold"]
if rejected and s not in refs:
refused.add(m["t"])
rejected_targets.setdefault(m["t"], set()).add(s)
if x > med + K_MAD * mad and (x > FACTOR * med or rejected and x > REJECT_FACTOR * med):
high.setdefault(s, []).append(m["t"])
if rejected:
refusals.setdefault(m["t"], set()).add(s)
if x > FACTOR * med:
strong.setdefault(m["t"], set()).add(s)
if s not in refs and rejected and x > med + K_TARGET * mad:
rejections.setdefault(s, []).append(m["t"])
by_t = {} # t -> {high sensors}
for s, ts in high.items():
for t in ts:
by_t.setdefault(t, set()).add(s)
n = lambda t: len(by_t.get(t, ())) # noqa: E731
out = []
# 2. Brief obstruction: an instant where many sensors are high, at least SALIENCE times more than the neighbours.
briefs = set()
for i, t in enumerate(instants):
neighbours = [n(instants[j]) for j in (i - 1, i + 1) if 0 <= j < len(instants)]
if n(t) >= BRIEF_MIN and n(t) >= SALIENCE * max(neighbours):
briefs.add(t)
only = rejected_targets.get(t, set()).difference(*(rejected_targets.get(instants[j], set()) for j in (i - 1, i + 1)
if 0 <= j < len(instants)))
if by_t[t] & refs and len(only) < BRIEF_MIN: # references touched without targets rejected at this instant
continue # alone: not an obstacle in front of targets, the whole image
out.append(_episode(t, t, "Brief obstruction", f"{n(t)} points were poorly seen for a single measurement, then came "
"back: probably a train, a vehicle or an animal passing; ignore this point on the curves.", by_t[t]))
elif 0 < len(f := strong.get(t, set())) < BRIEF_MIN and n(t) < BRIEF_MIN and len(f & refs) <= 1 and not f & set().union(
*(refusals.get(instants[j], set()) for j in (i - 1, i + 1) if 0 <= j < len(instants))):
# one or two targets, or a reference, rejected at this single measurement, at FACTOR times their median
out.append(_episode(t, t, "Brief obstruction", f"{len(f)} point{'s were' if len(f) > 1 else ' was'} poorly seen "
"for a single measurement, then came back: probably something passing; ignore this point on the "
"curves.", f))
# 3. Poor visibility: long runs per sensor (without the brief instants), joined across sensors, then reduced to the
# instants where at least FRAC of the episode's peak is high (cuts the tails of one or two sensors and the long
# lulls, more than BREAK).
runs = []
for s, ts in high.items():
pts = [(t, s) for t in ts if t not in briefs]
runs += [(a, b, s) for a, b, _, l in _group(pts, gap) if len(l) >= RUN_MIN]
zones = [z for z in _group_intervals(sorted(runs), gap) if len(z[2]) >= 2]
kept = []
for a, b, _ in zones:
inside = [t for t in instants if a <= t <= b and t not in briefs]
peak = max(n(t) for t in inside)
pts = [(t, s) for t in inside if n(t) >= max(1, FRAC * peak) for s in by_t[t]]
kept += [(d, f, cs) for d, f, cs, l in _group(pts, max(gap, BREAK)) if len(cs) >= 2 and len(set(l)) >= RUN_MIN]
# Bounds: from the first to the last rejected target; without a rejection, no episode (6).
kept = [(r[0], r[-1], cs) for a, b, cs in kept if (r := [t for t in instants if a <= t <= b and t in refused])]
level = _levels(measurements, instants)
for a, b, cs in kept: # one episode per period of the same level (5); the text comes from categories.py
if len(cs) <= FEW and not cs & refs: # an object in front of a few targets, not a diffuse nuisance (7)
out.append(_episode(a, b, "Target obscured", "", cs))
continue
seg = []
for t in [t for t in instants if a <= t <= b] + [None]:
if seg and (t is None or level[t] != level[seg[0]]):
who = cs & set().union(*(by_t.get(u, set()) for u in seg)) or cs
out.append(_episode(seg[0], seg[-1], "Poor visibility", "", who, level[seg[0]]))
seg = []
if t is not None:
seg.append(t)
inside_any = lambda t: t in briefs or any(a <= t <= b for a, b, _ in kept) # noqa: E731
# 4. Target obscured: a target rejected again and again, mostly outside the episodes above.
targets = []
for s, ts in rejections.items():
for a, b, _, l in _group([(t, s) for t in ts], max(TARGET_GAP, 2 * step)):
if len(l) >= (TARGET_MIN if TARGET_MIN * step <= TARGET_GAP else 2) and sum(map(inside_any, l)) * 2 < len(l):
targets.append((a, b, s))
for a, b, cs in _group_intervals(sorted(targets), timedelta(0)):
who, it, its = ("One target was", "it", "its") if len(cs) == 1 else (f"{len(cs)} targets were", "them", "their")
out.append(_episode(a, b, "Target obscured", f"{who} repeatedly lost while the targets around stayed clear: probably "
f"something in front of {it} (leaves, snow) or glare; {its} displacements in this period are unreliable.", cs))
return sorted(out, key=lambda e: e["start"])
def diffuse(measurements):
"""Diffuse nuisance (8): the runs of at least RUN_MIN instants where the median, over the sensors, of the error_value
in MAD above each one's median reaches K_DIFFUSE, with or without rejections -> Poor visibility "reduced"."""
z = {} # t -> {sensor: gap in MAD}
for s, v in measurements.items():
e = [m["error_value"] for m in v if m["error_value"] is not None]
if len(e) < 10:
continue
med = statistics.median(e)
mad = statistics.median(abs(x - med) for x in e) or 1e-9
for m in v:
if m["error_value"] is not None:
z.setdefault(m["t"], {})[s] = (m["error_value"] - med) / mad
highs = [(t, None) for t in sorted(z) if statistics.median(z[t].values()) >= K_DIFFUSE]
return [_episode(a, b, "Poor visibility", "", {s for t in l for s, x in z[t].items() if x >= K_DIFFUSE}, "reduced")
for a, b, _, l in _group(highs, GAP_STEPS * _step(measurements)) if len(l) >= RUN_MIN]
def detect(measurements): def detect(measurements):
shimmer = heat_shimmer(measurements) # a diffuse nuisance during heat shimmer is the heat shimmer (8) gap = GAP_STEPS * cadence(measurements)
eps = episodes(measurements) refs = {s for s, v in measurements.items() if is_reference(v)}
poor = [e for e in eps if e["kind"] == "Visibility"] # a diffuse nuisance within one of them is it instants = sorted({m["t"] for v in measurements.values() for m in v})
return eps + shimmer + [e for e in diffuse(measurements) z = {} # t -> {sensor: error_value in MAD above the sensor's median}
if not any(t["start"] <= e["start"] and e["end"] <= t["end"] for t in shimmer + poor)] high, strong, rejected, repeated = {}, {}, {}, {} # t -> {sensors}; repeated: target -> [t]
for s, v in measurements.items():
e = [m["error_value"] for m in v if m["error_value"] is not None]
if len(e) < 10:
continue
med, mad = level(e)
for m in v:
t, x = m["t"], m["error_value"]
if x is None:
continue
z.setdefault(t, {})[s] = (x - med) / mad
if x > med + K_MAD * mad and x > FACTOR * med:
high.setdefault(t, set()).add(s)
if m["error_threshold"] and x > m["error_threshold"]:
rejected.setdefault(t, set()).add(s)
if x > FACTOR * med:
strong.setdefault(t, set()).add(s)
if s not in refs and x > med + K_TARGET * mad:
repeated.setdefault(s, []).append(t)
n = lambda t: len(high.get(t, ())) # noqa: E731
out, briefs = [], set()
for i, t in enumerate(instants): # 2. Brief obstruction
around = [instants[j] for j in (i - 1, i + 1) if 0 <= j < len(instants)]
few = strong.get(t, set())
if n(t) >= BRIEF_MIN and n(t) >= SALIENCE * max(map(n, around), default=0):
briefs.add(t)
few = high[t]
elif not (few and len(few) < BRIEF_MIN and n(t) < BRIEF_MIN and len(few & refs) <= 1
and not few & set().union(*(rejected.get(u, set()) for u in around))):
continue
out.append(episode(t, t, "Obstruction", "Brief obstruction", BRIEF.format(targets(len(few))), few, True))
spans = [] # 3. Poor visibility: runs of each sensor, then the overlapping runs of at least two sensors
for s in {s for t in high for s in high[t]}:
spans += [(r[0], r[-1], s) for r in runs([t for t in instants if s in high.get(t, ()) and t not in briefs], gap)
if len(r) >= RUN_MIN]
zones = []
for a, b, s in sorted(spans):
if zones and a - zones[-1][1] <= gap:
zones[-1] = (zones[-1][0], max(zones[-1][1], b), zones[-1][2] | {s})
else:
zones.append((a, b, {s}))
zones = [zone for zone in zones if len(zone[2]) >= 2]
lv = _levels(measurements, instants)
for a, b, cs in zones:
inside = [t for t in instants if a <= t <= b]
for k, t in enumerate(inside):
seg = [t] if k == 0 or lv[t] != lv[inside[k - 1]] else seg + [t]
if k == len(inside) - 1 or lv[inside[k + 1]] != lv[t]:
out.append(_poor(seg[0], seg[-1], cs & set().union(*(high.get(u, set()) for u in seg)) or cs, lv[t], refs))
shimmer = heat_shimmer(measurements) # 4. diffuse nuisance
hosts = shimmer + [e for e in out if e["kind"] == "Visibility"]
for r in runs([t for t in sorted(z) if statistics.median(z[t].values()) >= K_DIFFUSE], gap):
e = _poor(r[0], r[-1], {s for t in r for s, x in z[t].items() if x >= K_DIFFUSE}, "reduced", refs)
if len(r) >= RUN_MIN and not any(h["start"] <= e["start"] and e["end"] <= h["end"] for h in hosts):
out.append(e)
seen = lambda t: t in briefs or any(a <= t <= b for a, b, _ in zones) # noqa: E731
for s, ts in repeated.items(): # 5. Target obscured
for r in runs(ts, TARGET_GAP):
if len(r) >= TARGET_MIN and 2 * sum(map(seen, r)) < len(r):
out.append(episode(r[0], r[-1], "Obstruction", "Target obscured", OBSCURED.format(s), [s], True))
return out + shimmer
# --- Heat shimmer ---------------------------------------------------------------------------------------------------- # --- Heat shimmer: fast noise of the corrected displacements in warm air.
# Atmospheric turbulence (fast noise on the corrected displacements). Method: # 1. Per sensor and per hour H, over [H - 1 h, H + 2 h): agitation, the median of |Δcorr_x| + |Δcorr_y| between successive
# 1. Per sensor and per hour H, on the measurements of [H − 1 h, H + 2 h): agitation = median of |Δcorr_x| + |Δcorr_y| # measurements, and the median error_value; each divided by its calm (lower quartile of the sensor's hourly values).
# between successive measurements (robust to one-step jumps), and median of the error_value. # 2. Turbulent hour: at least SHARE of the sensors agitated at RATIO times their calm, and median error_value at RATIO.
# 2. Each is divided by the sensor's calm: the lower quartile of its hourly values over the export (often the night). # 3. Turbulent hours at most 2 h apart make an episode, dropped if more than REJECT_MAX of its measurements are rejected
# 3. Turbulent hour: at least SHARE of the sensors at ≥ RATIO times their calm agitation, and median error_value of the # beyond each sensor's habit (its share of rejections over the export): that is poor visibility. Greyed if the median
# sensors at ≥ RATIO_ERR times the calm (warm air blurs the spots: E-003). # agitation reaches STRONG.
# 4. Neighbouring hours merged (a one-hour hole tolerated), episode of at least MIN_DURATION, dropped if more than RATIO = 1.5 # E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night, error_value ×1.7 to ×3.2)
# REJECT_MAX of the measurements are rejected beyond their sensor's habit (its share of rejections over the export): SHARE = 0.5
# it is then poor visibility, another rule. A target that is always rejected (Target issue) does not count (wf-improve REJECT_MAX = 0.15 # E-003 ≤ 7 %; cobweb (E-002) and rain (E-005) ≥ 18 %; beyond the habit: wf-improve Visibility, mod. 1, S1
# session Visibility, mod. 1, S1: WILDSTRUBEL, R3 rejected all day, the days of heat shimmer were dropped). STRONG = 2.0 # E-003 (×2.3 to ×2.6)
# Greyed if the median agitation ≥ STRONG.
RATIO = 1.5 # tuned on E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night)
SHARE = 0.5 # share of the sensors agitated at the same time
RATIO_ERR = 1.5 # tuned on E-003 (error_value ×1.7 to ×3.2 the night; ≈ ×1 at noon elsewhere)
REJECT_MAX = 0.15 # E-003 ≤ 7 %; cobweb (E-002) and rain (E-005) ≥ 18 %; rejections beyond the habit since wf-improve
# session Visibility, mod. 1, S1 (WILDSTRUBEL: 4 to 14 %, against 9 to 22 % raw)
STRONG = 2.0 # E-003 (×2.3 to ×2.6) is strong; an episode between ×1.5 and ×2 stays visible
MIN_DURATION = timedelta(hours=3) # E-003 lasts 10-11 h; drops the morning of the real movement of E-004 (2 h), which
# the classification drops too, whatever the duration
HOUR = timedelta(hours=1) HOUR = timedelta(hours=1)
SHIMMER = "Warm air made the readings shake ({:.1f}× noisier than in calm hours). The small fast wiggles are not movement."
def _hour(t):
return t.replace(minute=0, second=0, microsecond=0)
def _ratios(points): def _ratios(points):
"""[(t, value)] -> {hour H: median over [H − 1 h, H + 2 h) / lower quartile of these medians}.""" """[(t, value)] -> {hour H: median over [H - 1 h, H + 2 h) / lower quartile of these medians}."""
by = {} by = {}
for t, v in points: for t, v in points:
for k in (-1, 0, 1): for k in (-1, 0, 1):
by.setdefault(_hour(t) + k * HOUR, []).append(v) by.setdefault(t.replace(minute=0, second=0, microsecond=0) + k * HOUR, []).append(v)
med = {h: statistics.median(v) for h, v in by.items() if len(v) >= 3} med = {h: statistics.median(v) for h, v in by.items() if len(v) >= 3}
if len(med) < 8: calm = statistics.quantiles(med.values(), n=4)[0] if len(med) >= 8 else 0
return {}
calm = statistics.quantiles(med.values(), n=4)[0]
return {h: v / calm for h, v in med.items()} if calm > 0 else {} return {h: v / calm for h, v in med.items()} if calm > 0 else {}
def hours(measurements): def heat_shimmer(measurements):
"""-> {hour: {sensor: (agitation / calm, error_value / calm)}}.""" hours = {} # hour -> {sensor: (agitation / calm, error_value / calm)}
out = {}
for s, ms in measurements.items(): for s, ms in measurements.items():
ag = _ratios([(b["t"], abs(b["corr_x"] - a["corr_x"]) + abs(b["corr_y"] - a["corr_y"])) ag = _ratios([(b["t"], abs(b["corr_x"] - a["corr_x"]) + abs(b["corr_y"] - a["corr_y"]))
for a, b in zip(ms, ms[1:]) if a["corr_x"] is not None and b["corr_x"] is not None]) for a, b in zip(ms, ms[1:]) if a["corr_x"] is not None and b["corr_x"] is not None])
er = _ratios([(m["t"], m["error_value"]) for m in ms if m["error_value"] is not None]) er = _ratios([(m["t"], m["error_value"]) for m in ms if m["error_value"] is not None])
for h, r in ag.items(): for h in ag.keys() & er.keys():
if h in er: hours.setdefault(h, {})[s] = (ag[h], er[h])
out.setdefault(h, {})[s] = (r, er[h]) turbulent = [h for h, r in sorted(hours.items()) if sum(a >= RATIO for a, _ in r.values()) >= SHARE * len(r)
return out and statistics.median(e for _, e in r.values()) >= RATIO]
habit = {s: sum(m["corr_x"] is None for m in ms) / len(ms) for s, ms in measurements.items()}
out = []
def _turbulent(r): for b in runs(turbulent, 2 * HOUR):
return (sum(a >= RATIO for a, _ in r.values()) >= SHARE * len(r)
and statistics.median(e for _, e in r.values()) >= RATIO_ERR)
def heat_shimmer(measurements):
H = hours(measurements)
blocks = []
for h in sorted(h for h, r in H.items() if _turbulent(r)):
if blocks and h - blocks[-1][-1] <= 2 * HOUR:
blocks[-1].append(h)
else:
blocks.append([h])
habit = {s: sum(m["corr_x"] is None for m in ms) / len(ms) for s, ms in measurements.items()} # usual rejections
episodes = []
for b in blocks:
start, end = b[0], b[-1] + HOUR start, end = b[0], b[-1] + HOUR
by = {s: [m for m in ms if start <= m["t"] < end] for s, ms in measurements.items()} by = {s: [m for m in ms if start <= m["t"] < end] for s, ms in measurements.items()}
new = sum(max(0, sum(m["corr_x"] is None for m in d) - habit[s] * len(d)) for s, d in by.items()) new = sum(max(0, sum(m["corr_x"] is None for m in d) - habit[s] * len(d)) for s, d in by.items())
if end - start < MIN_DURATION or new > REJECT_MAX * sum(map(len, by.values())): if new > REJECT_MAX * sum(map(len, by.values())):
continue continue
by_sensor = {} agit = {s: statistics.median(hours[h][s][0] for h in b if s in hours[h]) for s in {s for h in b for s in hours[h]}}
for h in b:
for s, (a, _) in H[h].items():
by_sensor.setdefault(s, []).append(a)
agit = {s: statistics.median(v) for s, v in by_sensor.items()}
strength = statistics.median(agit.values()) strength = statistics.median(agit.values())
episodes.append({ out.append(episode(start, end, "Visibility", "Heat shimmer", SHIMMER.format(strength),
"start": iso(start), "end": iso(end), "kind": "Visibility", [s for s, v in agit.items() if v >= RATIO], strength >= STRONG))
"label": "Heat shimmer", return out
"text": f"Readings were about {strength:.1f}× noisier than in calm hours because of heat shimmer "
"(warm air trembling in the line of sight); the small fast wiggles are not movement.",
"ignore": strength >= STRONG,
"sensors": sorted(s for s, v in agit.items() if v >= RATIO),
})
return episodes