diff --git a/scripts/no_data.py b/scripts/no_data.py index 628d672..c33dcf5 100644 --- a/scripts/no_data.py +++ b/scripts/no_data.py @@ -1,8 +1,11 @@ """Detector of the No data family (the families document): moments without a measurement (the cadence of the "advanced" export is broken). -Cadence: median gap between two measurement instants, all sensors together. A hole: more than MISSING missing steps -between two measurements (gap > MISSING + 1 cadences); the episode runs from the last measurement to the next one. +Cadence: median gap between two measurement instants, all sensors together; at least an hour divided by the median +number of instants of the clock hours that have one: a station measuring in bursts (three measurements a minute apart +every 15 min) has a median gap of a minute, but a cycle of 5 min; at the median gap, every gap between two bursts was a +hole, a whole day of No data (wf-improve, batch 2026-11-05, S1). A hole: more than MISSING missing steps between two +measurements (gap > MISSING + 1 cadences); the episode runs from the last measurement to the next one. End of the window read (end, when the caller gives it: the chunked run of categories.py): the same rule between the last measurement and end, the episode running up to end (CLARIDEN, batch 01: 44 h without a measurement up to the end of the batch, not reported). No measurement in the export does not mean the station stopped: the text only says that the @@ -30,7 +33,10 @@ def detect(measurements, end=None): if len(ts) < 3: return [] gaps = [(b - a).total_seconds() for a, b in zip(ts, ts[1:])] - step = statistics.median(gaps) + per_hour = {} + for t in ts: + per_hour[t.replace(minute=0)] = per_hour.get(t.replace(minute=0), 0) + 1 + step = max(statistics.median(gaps), 3600 / statistics.median(per_hour.values())) # bursts: the cycle out = [(a, b, g, "") for a, b, g in zip(ts, ts[1:], gaps)] if end: # end excluded: the last expected measurement is at end − 1 step, the gap counts as between two measurements out.append((ts[-1], end, (end - ts[-1]).total_seconds(), ", up to the end of the period")) diff --git a/thresholds.md b/thresholds.md index 0206241..396ce12 100644 --- a/thresholds.md +++ b/thresholds.md @@ -59,7 +59,7 @@ The detectors must detect everything (the user, 2026-10-26, D-068): a setting ch | | `MIN_WARM_HOURS` | 2 h | WILDSTRUBEL, batch 01: a single warm hour is not a cycle (NIEDERHORN, E-005: noon, trains and holes) | | | `SLIDE_K` | 3 noises | common camera slide without a cycle ("Camera thermal movement"): gap from the median of the day before, raw positions and corrected curves; wf-improve MICHIGAN, mod. 1 (at 2, Displacement instants of BREITHORN taken) | | | `COLD` | 4 one-hour noises | the camera moves with the cold ("Camera thermal movement", without a cycle): the pose (tilt, rotation or scale) moves away from its median of the day before while the temperature falls under its own, the same hour; noise: variations over an hour, one step at 4 h; wf-improve Thermal displacement, mod. 1, S3 (BREITHORN, CLARIDEN, NEBRASKA: targets in opposite directions, or compensated; at 3, Displacement instants of NEBRASKA taken) | -| `no_data` | `MISSING` | 2 steps | user's rule, 2026-10-23; also applied between the last measurement and the end of the window read (CLARIDEN, batch 01) | +| `no_data` | `MISSING` | 2 steps | user's rule, 2026-10-23; also applied between the last measurement and the end of the window read (CLARIDEN, batch 01); the step is at least an hour divided by the median instants per clock hour (bursts of measurements: the cycle, not the minute inside a burst; wf-improve, batch 2026-11-05, S1: SUSTENHORN) | | `target_issue` | `CALM` | 0.25 | (detector added on 2026-10-27, see its docstring) | | | `SHARE` | 0.5 | —; the target hidden at times ("Target obscured") | | | `SHARE_WEEK` | 0.25 | share of the calm instants rejected that makes a degraded day of the problem target ("Degraded target", at least `WEEK` days): regularity rather than the share of the day; wf-improve Target issue, mod. 1, S1 (WILDSTRUBEL R3 rejected from 04:00 to 16:00 every day, 30 to 45 %: seen only on days without night measurements; NORDEND PR04 and CTR01, NIEDERHORN PWD_IMP_B08_024_C09, under half) |