"""Detector of the Visibility family (labeling/familles.md): the camera sees less well, the image is faint or blurred (error_value), or it trembles (heat shimmer, end of file). The same error_value computation also finds objects in front of a few targets (Brief obstruction, Target obscured): episodes() returns them with the Obstruction family, which detect() leaves to obstruction.py. Method: 1. "High" measurement: error_value > median + K_MAD MAD of the sensor over the export, and > FACTOR × median, or rejected (> error_threshold) and > REJECT_FACTOR × median: a sensor at a usual level of 0.6 never doubles, and FLORIDA (batch 01) showed its fogs as Target obscured. 2. Brief obstruction: an instant where at least BRIEF_MIN sensors are high, SALIENCE times more than at the neighbouring instants (train, vehicle), with no reference (global_ref) among them: a one-step peak touching the references is not an obstacle in front of targets (peak of the whole image, NORDEND on 01.09 11:15, batch 01); it is not shown. Unless at least BRIEF_MIN targets are rejected at this instant and neither at the instant before nor after: something passing in front of them, a reference touched as well (learner session Displacement, mod. 1, S5: trains of NIEDERHORN, 12 and 13 targets rejected). Also, something passing in front of one or two targets, or a single reference: at an instant, fewer than BRIEF_MIN sensors rejected at more than FACTOR times their median (without K_MAD: by day, heat shimmer inflates the MAD), at most one reference, none of them rejected at the instant before or after, and fewer than BRIEF_MIN high sensors (learner session Obstruction, mod. 1, S3: vehicles, passers-by, insects in front of one or two targets of WILDSTRUBEL, NIEDERHORN, LENZSPITZE; a reference hidden by a truck at JUNGFRAU, BREITHORN). 3. Poor visibility: runs of at least RUN_MIN high measurements per sensor, joined across sensors (at least 2), reduced to the instants where at least FRAC of the episode's peak is high. 4. Target obscured: a target (not a global_ref) rejected (> error_threshold) and above K_TARGET MAD, at least TARGET_MIN times less than TARGET_GAP apart, mostly outside the episodes of 2 and 3 (leaves, snow, glare on a target). The gap is at least two measurement steps, and two rejections are enough when TARGET_MIN steps do not fit in TARGET_GAP (learner session Obstruction, mod. 1, S6: at one measurement every 4 h, vehicles parked at night in front of COLI22, COLI24, POUI22 of BREITHORN make only two or three rejections). All episodes: ignore = True. 5. Level of a Poor visibility (validated 2026-10-24), measurement by measurement: "none" (nothing is seen) when at least HALF of the usually seen targets (under their threshold at least half the time) are rejected, "reduced" (average data) otherwise; one episode per period of the same level, with the level field. 6. Bounds of a Poor visibility (WILDSTRUBEL, batch 01): from the first to the last measurement where a target is rejected (> error_threshold), and no episode without a rejected target. On a road in full sun, the daytime error_value (×1.5 to ×3, heat shimmer) made reduced visibility last for hours without any rejection; references do not count: R3 is above its threshold almost all day. 7. Poor visibility or object (boundary "Visibility or Obstruction" of labeling/familles.md; BREITHORN, batch 01): a nuisance touching at most FEW targets, and no reference, is not diffuse: it is an object in front of them (a parked vehicle), a "Target obscured" episode of its whole duration, without a level. 8. Diffuse nuisance, with or without rejections (learner session Visibility, mod. 1, S2): at each instant, the median, over the sensors, of the error_value in MAD above each one's median; at least RUN_MIN instants in a row at K_DIFFUSE or more make a Poor visibility "reduced". A diffuse nuisance entirely within heat shimmer is the heat shimmer: no episode (otherwise it absorbed the jump back of the day before, WILDSTRUBEL on 04.10). Finds the drops of visibility of the whole scene without rejection at JUNGFRAU, FLORIDA, NORDEND; at 2, nights labelled thermal at JUNGFRAU; at 3, half the gain. A simpler version (share of targets rejected above their threshold, without comparing each sensor with its own level) was tried on 2026-10-24 and dropped: worse on five data sets out of six (git history of Support-Agent, detectors/archive/visibilite_simple.py). """ import os import statistics import sys from datetime import timedelta sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it from data import iso, load # noqa: E402 K_MAD = 10 # tuned on E-001 (fog: 11 to 57 MAD) and E-005 (rain, trains: 10 to 70 MAD) FACTOR = 2 # drops sensors with a near-zero MAD (E-004: PWD_REF10 ×1.6 at 14 MAD, E-005) RUN_MIN = 3 # high measurements in a row for poor visibility (E-001, E-005 rain) BRIEF_MIN = 3 # tuned on E-005 (trains: 5 to 17 sensors; the one with 2 targets is missed) K_TARGET = 5 # tuned on E-005 (leaves: 6 to 25 MAD, rejected) TARGET_MIN = 3 # repeated rejections of a target FRAC = 0.15 # cuts the tails of PR09 + CTR01 (NORDEND October) without cutting the cobweb (E-002) GAP_STEPS = 2.5 # steps tolerated between two high measurements of the same run (a missing step, clock to the second) BREAK = timedelta(hours=2) # lull separating two episodes (NORDEND October, night of 13 to 14.10) SALIENCE = 3 # a train stands out at least 3 times above the neighbouring instants (E-005: 5 to 17 against 0 to 2; # the cobweb of E-002 varies from 2 to 9 from one step to the next) TARGET_GAP = timedelta(hours=6) # tuned on E-005 (leaves: 7 rejections in 12 h, up to 5 h apart) REJECT_FACTOR = 1.2 # a rejection at 10 MAD also counts (FLORIDA, batch 01: levels of 0.6, ×2 out of reach); drops C24 # of E-009 (level 0.68 just under the threshold 0.7, rejected at ×1.04) HALF = 0.5 # "nothing is seen"; tuned on NORDEND (09.10 23:00: 68 % rejected) and E-005 (rain: 24 to 33 %) K_DIFFUSE = 2.5 # diffuse nuisance (8); learner session Visibility, mod. 1, S2 FEW = 3 # Poor visibility on at most 3 targets, no reference: an object, Target obscured (7); BREITHORN, # batch 01 (vehicles parked in front of 2 or 3 targets out of 12) def _step(measurements): ts = sorted({m["t"] for v in measurements.values() for m in v}) return statistics.median(b - a for a, b in zip(ts, ts[1:])) def _group(points, gap): """[(t, sensor)] sorted by t -> groups [(start, end, {sensors}, [t])] whose points are ≤ gap from each other.""" groups = [] for t, s in points: if groups and t - groups[-1][1] <= gap: g = groups[-1] groups[-1] = (g[0], t, g[2] | {s}, g[3] + [t]) else: groups.append((t, t, {s}, [t])) return groups def _group_intervals(intervals, gap): """[(start, end, sensor)] sorted -> [(start, end, {sensors})] of the intervals that overlap (within gap).""" out = [] for a, b, s in intervals: if out and a - out[-1][1] <= gap: out[-1] = (out[-1][0], max(out[-1][1], b), out[-1][2] | {s}) else: out.append((a, b, {s})) return out def _episode(start, end, label, text, sensors, level=None): e = {"start": iso(start), "end": iso(end), "kind": "Visibility" if label == "Poor visibility" else "Obstruction", "label": label, "text": text, "ignore": level != "reduced", "sensors": sorted(sensors)} # reduced visibility: average data, not greyed return {**e, "level": level} if level else e def _levels(measurements, instants): """t -> "none" if at least HALF of the usually seen targets are rejected at t, "reduced" otherwise.""" ratios = [] for v in measurements.values(): r = {m["t"]: m["error_value"] / m["error_threshold"] for m in v if m["error_value"] is not None and m["error_threshold"]} if r and statistics.median(r.values()) < 1: ratios.append(r) return {t: "none" if sum(r.get(t, 0) > 1 for r in ratios) >= HALF * max(1, sum(t in r for r in ratios)) else "reduced" for t in instants} def episodes(measurements): """All error_value episodes: Visibility (Poor visibility), Obstruction (Brief obstruction, Target obscured).""" step = _step(measurements) gap = GAP_STEPS * step refs = {s for s, v in measurements.items() if v[0]["pattern_type"] == "global_ref"} instants = sorted({m["t"] for v in measurements.values() for m in v}) high, rejections = {}, {} # sensor -> [t] refused = set() # instants where a target (not a global_ref) is above its threshold rejected_targets = {} # t -> {targets above their threshold} strong, refusals = {}, {} # t -> {sensors rejected at more than FACTOR times their median}, {sensors rejected} for s, v in measurements.items(): e = [m["error_value"] for m in v if m["error_value"] is not None] if len(e) < 10: continue med = statistics.median(e) mad = statistics.median(abs(x - med) for x in e) or 1e-9 for m in v: x = m["error_value"] if x is None: continue rejected = m["error_threshold"] and x > m["error_threshold"] if rejected and s not in refs: refused.add(m["t"]) rejected_targets.setdefault(m["t"], set()).add(s) if x > med + K_MAD * mad and (x > FACTOR * med or rejected and x > REJECT_FACTOR * med): high.setdefault(s, []).append(m["t"]) if rejected: refusals.setdefault(m["t"], set()).add(s) if x > FACTOR * med: strong.setdefault(m["t"], set()).add(s) if s not in refs and rejected and x > med + K_TARGET * mad: rejections.setdefault(s, []).append(m["t"]) by_t = {} # t -> {high sensors} for s, ts in high.items(): for t in ts: by_t.setdefault(t, set()).add(s) n = lambda t: len(by_t.get(t, ())) # noqa: E731 out = [] # 2. Brief obstruction: an instant where many sensors are high, at least SALIENCE times more than the neighbours. briefs = set() for i, t in enumerate(instants): neighbours = [n(instants[j]) for j in (i - 1, i + 1) if 0 <= j < len(instants)] if n(t) >= BRIEF_MIN and n(t) >= SALIENCE * max(neighbours): briefs.add(t) only = rejected_targets.get(t, set()).difference(*(rejected_targets.get(instants[j], set()) for j in (i - 1, i + 1) if 0 <= j < len(instants))) if by_t[t] & refs and len(only) < BRIEF_MIN: # references touched without targets rejected at this instant continue # alone: not an obstacle in front of targets, the whole image out.append(_episode(t, t, "Brief obstruction", f"{n(t)} points were poorly seen for a single measurement, then came " "back: probably a train, a vehicle or an animal passing; ignore this point on the curves.", by_t[t])) elif 0 < len(f := strong.get(t, set())) < BRIEF_MIN and n(t) < BRIEF_MIN and len(f & refs) <= 1 and not f & set().union( *(refusals.get(instants[j], set()) for j in (i - 1, i + 1) if 0 <= j < len(instants))): # one or two targets, or a reference, rejected at this single measurement, at FACTOR times their median out.append(_episode(t, t, "Brief obstruction", f"{len(f)} point{'s were' if len(f) > 1 else ' was'} poorly seen " "for a single measurement, then came back: probably something passing; ignore this point on the " "curves.", f)) # 3. Poor visibility: long runs per sensor (without the brief instants), joined across sensors, then reduced to the # instants where at least FRAC of the episode's peak is high (cuts the tails of one or two sensors and the long # lulls, more than BREAK). runs = [] for s, ts in high.items(): pts = [(t, s) for t in ts if t not in briefs] runs += [(a, b, s) for a, b, _, l in _group(pts, gap) if len(l) >= RUN_MIN] zones = [z for z in _group_intervals(sorted(runs), gap) if len(z[2]) >= 2] kept = [] for a, b, _ in zones: inside = [t for t in instants if a <= t <= b and t not in briefs] peak = max(n(t) for t in inside) pts = [(t, s) for t in inside if n(t) >= max(1, FRAC * peak) for s in by_t[t]] kept += [(d, f, cs) for d, f, cs, l in _group(pts, max(gap, BREAK)) if len(cs) >= 2 and len(set(l)) >= RUN_MIN] # Bounds: from the first to the last rejected target; without a rejection, no episode (6). kept = [(r[0], r[-1], cs) for a, b, cs in kept if (r := [t for t in instants if a <= t <= b and t in refused])] level = _levels(measurements, instants) for a, b, cs in kept: # one episode per period of the same level (5); the text comes from categories.py if len(cs) <= FEW and not cs & refs: # an object in front of a few targets, not a diffuse nuisance (7) out.append(_episode(a, b, "Target obscured", "", cs)) continue seg = [] for t in [t for t in instants if a <= t <= b] + [None]: if seg and (t is None or level[t] != level[seg[0]]): who = cs & set().union(*(by_t.get(u, set()) for u in seg)) or cs out.append(_episode(seg[0], seg[-1], "Poor visibility", "", who, level[seg[0]])) seg = [] if t is not None: seg.append(t) inside_any = lambda t: t in briefs or any(a <= t <= b for a, b, _ in kept) # noqa: E731 # 4. Target obscured: a target rejected again and again, mostly outside the episodes above. targets = [] for s, ts in rejections.items(): for a, b, _, l in _group([(t, s) for t in ts], max(TARGET_GAP, 2 * step)): if len(l) >= (TARGET_MIN if TARGET_MIN * step <= TARGET_GAP else 2) and sum(map(inside_any, l)) * 2 < len(l): targets.append((a, b, s)) for a, b, cs in _group_intervals(sorted(targets), timedelta(0)): who, it, its = ("One target was", "it", "its") if len(cs) == 1 else (f"{len(cs)} targets were", "them", "their") out.append(_episode(a, b, "Target obscured", f"{who} repeatedly lost while the targets around stayed clear: probably " f"something in front of {it} (leaves, snow) or glare; {its} displacements in this period are unreliable.", cs)) return sorted(out, key=lambda e: e["start"]) def diffuse(measurements): """Diffuse nuisance (8): the runs of at least RUN_MIN instants where the median, over the sensors, of the error_value in MAD above each one's median reaches K_DIFFUSE, with or without rejections -> Poor visibility "reduced".""" z = {} # t -> {sensor: gap in MAD} for s, v in measurements.items(): e = [m["error_value"] for m in v if m["error_value"] is not None] if len(e) < 10: continue med = statistics.median(e) mad = statistics.median(abs(x - med) for x in e) or 1e-9 for m in v: if m["error_value"] is not None: z.setdefault(m["t"], {})[s] = (m["error_value"] - med) / mad highs = [(t, None) for t in sorted(z) if statistics.median(z[t].values()) >= K_DIFFUSE] return [_episode(a, b, "Poor visibility", "", {s for t in l for s, x in z[t].items() if x >= K_DIFFUSE}, "reduced") for a, b, _, l in _group(highs, GAP_STEPS * _step(measurements)) if len(l) >= RUN_MIN] def detect(measurements): shimmer = heat_shimmer(measurements) # a diffuse nuisance during heat shimmer is the heat shimmer (8) return ([e for e in episodes(measurements) if e["kind"] == "Visibility"] + shimmer + [e for e in diffuse(measurements) if not any(t["start"] <= e["start"] and e["end"] <= t["end"] for t in shimmer)]) # --- Heat shimmer ---------------------------------------------------------------------------------------------------- # Atmospheric turbulence (fast noise on the corrected displacements). Method: # 1. Per sensor and per hour H, on the measurements of [H − 1 h, H + 2 h): agitation = median of |Δcorr_x| + |Δcorr_y| # between successive measurements (robust to one-step jumps), and median of the error_value. # 2. Each is divided by the sensor's calm: the lower quartile of its hourly values over the export (often the night). # 3. Turbulent hour: at least SHARE of the sensors at ≥ RATIO times their calm agitation, and median error_value of the # sensors at ≥ RATIO_ERR times the calm (warm air blurs the spots: E-003). # 4. Neighbouring hours merged (a one-hour hole tolerated), episode of at least MIN_DURATION, dropped if more than # REJECT_MAX of the measurements are rejected beyond their sensor's habit (its share of rejections over the export): # it is then poor visibility, another rule. A target that is always rejected (Target issue) does not count (learner # session Visibility, mod. 1, S1: WILDSTRUBEL, R3 rejected all day, the days of heat shimmer were dropped). # Greyed if the median agitation ≥ STRONG. RATIO = 1.5 # tuned on E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night) SHARE = 0.5 # share of the sensors agitated at the same time RATIO_ERR = 1.5 # tuned on E-003 (error_value ×1.7 to ×3.2 the night; ≈ ×1 at noon elsewhere) REJECT_MAX = 0.15 # E-003 ≤ 7 %; cobweb (E-002) and rain (E-005) ≥ 18 %; rejections beyond the habit since learner # session Visibility, mod. 1, S1 (WILDSTRUBEL: 4 to 14 %, against 9 to 22 % raw) STRONG = 2.0 # E-003 (×2.3 to ×2.6) is strong; an episode between ×1.5 and ×2 stays visible MIN_DURATION = timedelta(hours=3) # E-003 lasts 10-11 h; drops the morning of the real movement of E-004 (2 h), which # the classification drops too, whatever the duration HOUR = timedelta(hours=1) def _hour(t): return t.replace(minute=0, second=0, microsecond=0) def _ratios(points): """[(t, value)] -> {hour H: median over [H − 1 h, H + 2 h) / lower quartile of these medians}.""" by = {} for t, v in points: for k in (-1, 0, 1): by.setdefault(_hour(t) + k * HOUR, []).append(v) med = {h: statistics.median(v) for h, v in by.items() if len(v) >= 3} if len(med) < 8: return {} calm = statistics.quantiles(med.values(), n=4)[0] return {h: v / calm for h, v in med.items()} if calm > 0 else {} def hours(measurements): """-> {hour: {sensor: (agitation / calm, error_value / calm)}}.""" out = {} for s, ms in measurements.items(): ag = _ratios([(b["t"], abs(b["corr_x"] - a["corr_x"]) + abs(b["corr_y"] - a["corr_y"])) for a, b in zip(ms, ms[1:]) if a["corr_x"] is not None and b["corr_x"] is not None]) er = _ratios([(m["t"], m["error_value"]) for m in ms if m["error_value"] is not None]) for h, r in ag.items(): if h in er: out.setdefault(h, {})[s] = (r, er[h]) return out def _turbulent(r): return (sum(a >= RATIO for a, _ in r.values()) >= SHARE * len(r) and statistics.median(e for _, e in r.values()) >= RATIO_ERR) def heat_shimmer(measurements): H = hours(measurements) blocks = [] for h in sorted(h for h, r in H.items() if _turbulent(r)): if blocks and h - blocks[-1][-1] <= 2 * HOUR: blocks[-1].append(h) else: blocks.append([h]) habit = {s: sum(m["corr_x"] is None for m in ms) / len(ms) for s, ms in measurements.items()} # usual rejections episodes = [] for b in blocks: start, end = b[0], b[-1] + HOUR by = {s: [m for m in ms if start <= m["t"] < end] for s, ms in measurements.items()} new = sum(max(0, sum(m["corr_x"] is None for m in d) - habit[s] * len(d)) for s, d in by.items()) if end - start < MIN_DURATION or new > REJECT_MAX * sum(map(len, by.values())): continue by_sensor = {} for h in b: for s, (a, _) in H[h].items(): by_sensor.setdefault(s, []).append(a) agit = {s: statistics.median(v) for s, v in by_sensor.items()} strength = statistics.median(agit.values()) episodes.append({ "start": iso(start), "end": iso(end), "kind": "Visibility", "label": "Heat shimmer", "text": f"Readings were about {strength:.1f}× noisier than in calm hours because of heat shimmer " "(warm air trembling in the line of sight); the small fast wiggles are not movement.", "ignore": strength >= STRONG, "sensors": sorted(s for s, v in agit.items() if v >= RATIO), }) return episodes if __name__ == "__main__": for e in detect(load(sys.argv[1])): print(e["start"], "→", e["end"], e["label"], f"| {len(e['sensors'])}:", ",".join(e["sensors"][:8]), "|", e["text"])