"""Detector of the Thermal displacement family (labeling/familles.md): the daily wave of the corrected displacements (expansion by day, contraction in the evening). Method, per target (global references excluded) and per axis (corr_x, corr_y): 1. Each UTC day of at least MIN_HOURS hours: mean of each hour minus the median of the day (removes jumps and drift). 2. Profile = median, hour by hour, over the days; "repeatability" = median, over the days, of the correlation between the day and the profile of the other days. Axis concerned if repeatability ≥ REPRO. An incomplete day (at least PARTIAL hours, fewer than MIN_HOURS) does not enter the profile, but its likeness is measured on the hours it has: a hole of half a day no longer takes its episode away (wf-improve session Thermal displacement, mod. 1, S4; at MIN_HOURS = 12 or 16, the incomplete days distorted the profile, NIEDERHORN went back). 3. Cycle of the station if at least SHARE of the targets have an axis concerned, **and** (second method, asked by the user) if a 24 h sinusoid, a + b·sin(2πt/24 h) + c·cos(2πt/24 h) by least squares, explains at least R2_MIN of the curve of an axis for at least SHARE of the targets. 4. Warm hours: night level = median of the NIGHT flattest consecutive hours of the profile; an hour is warm if, as a median over the axes concerned, the profile moves away from it by at least WARM times its amplitude. 5. Two episodes per complete day that looks like the others (median correlation ≥ DAY_MATCH over the axes concerned): "Thermal expansion" from the start to the hour of the largest gap from the night level (the peak), then "Thermal contraction" until the return. The warm hours say there is a cycle (at least MIN_WARM_HOURS of them); its bounds are read lower, at EDGE times the amplitude: the start, going back from the peak while the gap stays ≥ EDGE (at most to the calmest hour of the profile); the return, going down from the peak until it goes back under EDGE, after midnight if needed, at most to the next day's start: a slow return at night chains the cycles (wf-improve session Thermal displacement, mod. 1, S1). The direction (a rising or falling curve) depends on the target's orientation: it is not looked at, only the gap from the night level. Reading: the cycle is almost everywhere in the raw positions (common mode) and the compensation removes it, except at PS des Coudres where it stays in corr, with the same shape as in the raw positions. Camera thermal effect (D-074): without a cycle in corr, the same method on the raw positions x, y gives "Camera thermal expansion" and "Camera thermal contraction": the camera moves with the heat of its mount, the compensation removes it; same categories, own sentence. On the raw positions the second method is not asked (WILDSTRUBEL, batch 01: the camera rose by a 3 px block at sunrise, held 3 h and dropped in one step around 09:30, six days in a row, which the sinusoid does not explain); in exchange, a single warm hour is not a cycle (MIN_WARM_HOURS: NIEDERHORN, E-005, hourly, would otherwise have its noon hour as a camera contraction). The camera episode covers all sensors, references included: the whole image moves. Common slide, without a cycle (wf-improve MICHIGAN, mod. 1): lacking a cycle in the corrected curves as in the raw positions, an instant is a camera slide if the raw positions (x or y) of at least SHARE of the sensors, references included, **and** the corrected curves (corr_x or corr_y) of at least SHARE of the targets move away on the same side by at least SLIDE_K noises (median of the steps of the series) from their median of the day before; slides less than a day apart make one episode, "Camera thermal movement", on all sensors, host of the jumps back (categories.HOSTS). The condition on the raw positions leaves to the jumps a flip of the computed pose alone (raw positions still). The camera moves with the cold (wf-improve session Thermal displacement, mod. 1, S3): lacking a cycle, also a "Camera thermal movement" where the camera pose (tilt, rotation or scale) moves away from its median of the day before by at least COLD noises, while the station temperature (detect declares it, D-124) falls under its own by at least COLD noises the same hour; noise: the median of the variations over an hour (over one step, at one measurement every 4 h), so that the threshold does not depend on the cadence. The targets may move in opposite directions or stay almost still (the compensation absorbs the movement): the slide rule does not see it. Consecutive pose instants make one episode. Without a pose or without a temperature, nothing. The heat is not taken: in fine weather, it is the day's cycle. """ import math import os import statistics import sys from datetime import datetime, timedelta, timezone sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it from data import iso, load # noqa: E402 MIN_HOURS = 20 # a UTC day counts in the profile if it has at least 20 measured hours PARTIAL = 12 # an incomplete day of at least 12 hours gets its episode if it looks like the profile REPRO = 0.6 # an axis has a cycle if each day looks like the others (pure noise ≈ 0) SHARE = 0.5 # share of the targets concerned; 2026-10_NORDEND is just under (9/19) WARM = 0.3 # gap from the night level, as a share of the profile's amplitude DAY_MATCH = 0.5 # likeness of a day to the others to give it its episode NIGHT = 8 # flattest consecutive hours of the profile = night level R2_MIN = 0.3 # draft of 2026-10-24: PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48 MIN_WARM_HOURS = 2 # WILDSTRUBEL, batch 01; NIEDERHORN (E-005) has only one warm hour (noon) EDGE = 0.1 # bounds of the wave (start, return), as a share of the amplitude; WARM says whether there is a cycle SLIDE_K = 3 # common slide, in noises (median of the steps) away from the median of the day before COLD = 4 # the camera moves with the cold, in one-hour noises away from the day before (pose, temperature) W = 2 * math.pi / 86400 # angular frequency of a 24 h cycle, in rad/s def _corr(a, b): ma, mb = statistics.mean(a), statistics.mean(b) num = sum((x - ma) * (y - mb) for x, y in zip(a, b)) den = (sum((x - ma) ** 2 for x in a) * sum((y - mb) ** 2 for y in b)) ** 0.5 return num / den if den else 0.0 def _axis(ms, col): """-> (24 h profile, {day: correlation with the profile of the other days}) or None.""" days = {} for m in ms: if m[col] is not None: days.setdefault(m["t"].date(), {}).setdefault(m["t"].hour, []).append(m[col]) full, every = {}, {} for d, hs in days.items(): if len(hs) >= PARTIAL: med = statistics.median(v for l in hs.values() for v in l) every[d] = {h: statistics.mean(l) - med for h, l in hs.items()} if len(hs) >= MIN_HOURS: full[d] = every[d] if len(full) < 2: return None profile = [statistics.median(q[h] for q in full.values() if h in q) for h in range(24) if any(h in q for q in full.values())] if len(profile) < 24: return None likeness = {} for d, p in every.items(): others = [q for e, q in full.items() if e != d] hs = [h for h in p if all(h in q for q in others)] if len(hs) >= 3: likeness[d] = _corr([p[h] for h in hs], [statistics.median(q[h] for q in others) for h in hs]) return profile, likeness def sine_r2(ms, col): """Share of the curve explained by a 24 h sinusoid (least squares, 3 unknowns), or None if too short.""" pts = [(m["t"].timestamp(), m[col]) for m in ms if m[col] is not None] if len(pts) <= 48: return None rows = [(1.0, math.sin(W * t), math.cos(W * t)) for t, _ in pts] A = [[sum(r[i] * r[j] for r in rows) for j in range(3)] for i in range(3)] B = [sum(r[i] * y for r, (_, y) in zip(rows, pts)) for i in range(3)] for i in range(3): # Gaussian elimination for k in range(i + 1, 3): f = A[k][i] / A[i][i] A[k] = [x - f * y for x, y in zip(A[k], A[i])] B[k] -= f * B[i] x = [0.0] * 3 for i in (2, 1, 0): x[i] = (B[i] - sum(A[i][j] * x[j] for j in range(i + 1, 3))) / A[i][i] m = statistics.mean(y for _, y in pts) sst = sum((y - m) ** 2 for _, y in pts) sse = sum((y - sum(c * v for c, v in zip(x, r))) ** 2 for r, (_, y) in zip(rows, pts)) return 1 - sse / sst if sst else None def sine(measurements, cols=("corr_x", "corr_y")): """-> (targets with an axis following a 24 h sinusoid, targets evaluated).""" yes, evaluated = set(), set() for s, ms in measurements.items(): if ms[0]["pattern_type"] == "global_ref": continue for col in cols: r2 = sine_r2(ms, col) if r2 is not None: evaluated.add(s) if r2 >= R2_MIN: yes.add(s) return yes, evaluated def cycle(measurements, cols=("corr_x", "corr_y")): """-> (axes concerned [(sensor, col, profile, likeness)], number of targets evaluated).""" concerned, targets = [], set() for s, ms in measurements.items(): if ms[0]["pattern_type"] == "global_ref": continue for col in cols: r = _axis(ms, col) if r is None: continue targets.add(s) if statistics.median(r[1].values()) >= REPRO: concerned.append((s, col) + r) return concerned, len(targets) def gaps(concerned): """Median gap (over the axes concerned) from the night level, hour by hour, as a share of the profile's amplitude.""" def score(p, h): start = min(range(24), key=lambda i: max(p[(i + k) % 24] for k in range(NIGHT)) - min(p[(i + k) % 24] for k in range(NIGHT))) night = statistics.median(p[(start + k) % 24] for k in range(NIGHT)) return abs(p[h] - night) / (max(p) - min(p)) return [statistics.median(score(c[2], h) for c in concerned) for h in range(24)] def warm_hours(concerned): return [h for h, e in enumerate(gaps(concerned)) if e >= WARM] def detect(measurements, temperature=(), camera=()): """Cycle of the corrected curves (the structure); otherwise, that of the raw positions (the camera moving with the expansion of its mount, compensated: D-074); otherwise, the camera slides (with the cold).""" return (_episodes(measurements, ("corr_x", "corr_y"), "Thermal", False) or _episodes(measurements, ("x", "y"), "Camera thermal", True) or slides(measurements, temperature, camera)) def _signs(ms, col, hourly=False, k=None): """{instant: +1 or -1} where the measurement moves away by at least k (default SLIDE_K) noises from the median of the day before; noise: the median of the steps from one measurement to the next or, hourly, of the variations over an hour (one step if the cadence is slow).""" pts = [(m["t"], m[col]) for m in ms if m[col] is not None] if len(pts) < 3: return {} step = statistics.median((b - a).total_seconds() for (a, _), (b, _) in zip(pts, pts[1:])) if hourly else 0 n = max(1, round(3600 / step)) if step else 1 noise = statistics.median(abs(pts[i + n][1] - pts[i][1]) for i in range(len(pts) - n)) if len(pts) > n else 0 out, j = {}, 0 for i, (t, v) in enumerate(pts): while pts[j][0] < t - timedelta(days=1): j += 1 before = [w for _, w in pts[j:i]] if noise and len(before) >= 3 and abs(v - statistics.median(before)) >= (k or SLIDE_K) * noise: out[t] = 1 if v > statistics.median(before) else -1 return out def _common(series): """[{instant: sign}] of a group of sensors -> instants where at least SHARE of them move away on the same side.""" n, count = len(series), {} for sg in series: for t, x in sg.items(): count[(t, x)] = count.get((t, x), 0) + 1 return {t for (t, x), c in count.items() if n and c >= SHARE * n} def _cold(measurements, temperature, camera): """[(start, end)] of the runs of instants where the camera pose (tilt, rotation or scale) moves away from its day before while the temperature falls under its own, by at least COLD one-hour noises; empty without one of them.""" ts = [m["t"] for ms in measurements.values() for m in ms] hour = lambda t: t.replace(minute=0, second=0, microsecond=0) # noqa: E731 cold = {hour(t) for t, x in _signs([{"t": t, "T": v} for t, v in temperature], "T", True, COLD).items() if x < 0} pose = [p for p in camera if min(ts) <= p["t"] <= max(ts)] if ts else [] seen = {t for c in ("tilt", "rotation", "scale") for t in _signs(pose, c, True, COLD) if hour(t) in cold} episodes = [] # consecutive pose instants make one episode for i, p in enumerate(pose): if p["t"] in seen: if episodes and episodes[-1][2] == i - 1: episodes[-1][1:] = [p["t"], i] else: episodes.append([p["t"], p["t"], i]) return [(a, b) for a, b, _ in episodes] def slides(measurements, temperature=(), camera=()): """Common camera slide, without a 24 h cycle: the raw positions of most sensors, references included, and the corrected curves of most targets move away together, on the same side, from their day before; or the camera pose moves away from it with the cold (_cold).""" targets = [ms for ms in measurements.values() if ms[0]["pattern_type"] != "global_ref"] raw = set().union(*(_common([_signs(ms, c) for ms in measurements.values()]) for c in ("x", "y"))) corr = set().union(*(_common([_signs(ms, c) for ms in targets]) for c in ("corr_x", "corr_y"))) episodes = [] for t in sorted(raw & corr): if episodes and t - episodes[-1][1] <= timedelta(days=1): # less than a day between two: the same slide episodes[-1][1] = t else: episodes.append([t, t]) return [{"start": iso(a), "end": iso(b), "kind": "Thermal displacement", "label": "Camera thermal movement", "text": "The camera moved with a change of temperature and all the curves moved with it; this is not a " "movement of the structure, nothing to do.", "ignore": False, "sensors": sorted(measurements)} for a, b in episodes + _cold(measurements, temperature, camera)] def _episodes(measurements, cols, name, camera): concerned, n = cycle(measurements, cols) sensors = sorted({c[0] for c in concerned}) if not n or len(sensors) < SHARE * n: return [] if not camera: # second method, for the structure only: the sinusoid must see it too yes, evaluated = sine(measurements, cols) if not evaluated or len(yes) < SHARE * len(evaluated): return [] warm = warm_hours(concerned) if len(warm) < MIN_WARM_HOURS: return [] e = gaps(concerned) peak = max(warm, key=lambda h: e[h]) # the largest gap from the night level: end of the expansion rest = min(range(24), key=lambda h: e[h]) a = peak # start: going back from the peak while the profile stays above EDGE, up to the calmest hour while (a - 1) % 24 != rest and e[(a - 1) % 24] >= EDGE: a -= 1 b = peak + 1 # return: until the profile goes back under EDGE, at most to the next start while b < a + 24 and e[b % 24] >= EDGE: b += 1 last = max(m["t"] for ms in measurements.values() for m in ms) days = sorted({d for c in concerned for d in c[3]}) if camera: # the whole image moves, references included sensors = sorted(measurements) episodes = [] for d in days: # ponytail: days of at least PARTIAL hours, the export's ends (22:00 → midnight) get no episode if statistics.median(c[3][d] for c in concerned if d in c[3]) < DAY_MATCH: continue midnight = datetime(d.year, d.month, d.day, tzinfo=timezone.utc) for label, u, v in ((f"{name} expansion", a, peak), (f"{name} contraction", peak, b)): start, end = midnight + timedelta(hours=u), min(midnight + timedelta(hours=v), last) if end > start: episodes.append({"start": iso(start), "end": iso(end), "kind": "Thermal displacement", "label": label, "text": "", "ignore": False, "sensors": sensors}) return episodes if __name__ == "__main__": measurements = load(sys.argv[1]) for e in detect(measurements): print(e["start"], "→", e["end"], e["label"], len(e["sensors"]), "targets") for cols in (("corr_x", "corr_y"), ("x", "y")): concerned, n = cycle(measurements, cols) k = len({c[0] for c in concerned}) amp = statistics.median(max(c[2]) - min(c[2]) for c in concerned) if concerned else 0 yes, evaluated = sine(measurements, cols) print(f"{cols[0][:-2] or 'raw'}: targets {k}/{n}, median amplitude {amp:.3f} px, warm hours (UTC) " f"{warm_hours(concerned) if concerned else '-'}; sinusoid: targets {len(yes)}/{len(evaluated)}")