293 lines
17 KiB
Python
293 lines
17 KiB
Python
"""Detector of the Thermal displacement family (labeling/familles.md): the daily wave of the corrected displacements
|
|
(expansion by day, contraction in the evening).
|
|
|
|
Method, per target (global references excluded) and per axis (corr_x, corr_y):
|
|
1. Each UTC day of at least MIN_HOURS hours: mean of each hour minus the median of the day (removes jumps and drift).
|
|
2. Profile = median, hour by hour, over the days; "repeatability" = median, over the days, of the correlation between
|
|
the day and the profile of the other days. Axis concerned if repeatability ≥ REPRO. An incomplete day (at least
|
|
PARTIAL hours, fewer than MIN_HOURS) does not enter the profile, but its likeness is measured on the hours it has: a
|
|
hole of half a day no longer takes its episode away (wf-improve session Thermal displacement, mod. 1, S4; at
|
|
MIN_HOURS = 12 or 16, the incomplete days distorted the profile, NIEDERHORN went back).
|
|
3. Cycle of the station if at least SHARE of the targets have an axis concerned, **and** (second method, asked by the
|
|
user) if a 24 h sinusoid, a + b·sin(2πt/24 h) + c·cos(2πt/24 h) by least squares, explains at least R2_MIN of the
|
|
curve of an axis for at least SHARE of the targets.
|
|
4. Warm hours: night level = median of the NIGHT flattest consecutive hours of the profile; an hour is warm if, as a
|
|
median over the axes concerned, the profile moves away from it by at least WARM times its amplitude.
|
|
5. Two episodes per complete day that looks like the others (median correlation ≥ DAY_MATCH over the axes concerned):
|
|
"Thermal expansion" from the start to the hour of the largest gap from the night level (the peak), then "Thermal
|
|
contraction" until the return. The warm hours say there is a cycle (at least MIN_WARM_HOURS of them); its bounds
|
|
are read lower, at EDGE times the amplitude: the start, going back from the peak while the gap stays ≥ EDGE (at most
|
|
to the calmest hour of the profile); the return, going down from the peak until it goes back under EDGE, after
|
|
midnight if needed, at most to the next day's start: a slow return at night chains the cycles (wf-improve session
|
|
Thermal displacement, mod. 1, S1). The direction (a rising or falling curve) depends on the target's orientation:
|
|
it is not looked at, only the gap from the night level.
|
|
Reading: the cycle is almost everywhere in the raw positions (common mode) and the compensation removes it, except at
|
|
PS des Coudres where it stays in corr, with the same shape as in the raw positions.
|
|
Camera thermal effect (D-074): without a cycle in corr, the same method on the raw positions x, y gives "Camera thermal
|
|
expansion" and "Camera thermal contraction": the camera moves with the heat of its mount, the compensation removes it;
|
|
same categories, own sentence. On the raw positions the second method is not asked (WILDSTRUBEL, batch 01: the camera
|
|
rose by a 3 px block at sunrise, held 3 h and dropped in one step around 09:30, six days in a row, which the sinusoid
|
|
does not explain); in exchange, a single warm hour is not a cycle (MIN_WARM_HOURS: NIEDERHORN, E-005, hourly, would
|
|
otherwise have its noon hour as a camera contraction). The camera episode covers all sensors, references included: the
|
|
whole image moves.
|
|
Common slide, without a cycle (wf-improve MICHIGAN, mod. 1): lacking a cycle in the corrected curves as in the raw
|
|
positions, an instant is a camera slide if the raw positions (x or y) of at least SHARE of the sensors, references
|
|
included, **and** the corrected curves (corr_x or corr_y) of at least SHARE of the targets move away on the same side by
|
|
at least SLIDE_K noises (median of the steps of the series) from their median of the day before; slides less than a
|
|
day apart make one episode, "Camera thermal movement", on all sensors, host of the jumps back (categories.HOSTS). The
|
|
condition on the raw positions leaves to the jumps a flip of the computed pose alone (raw positions still).
|
|
The camera moves with the cold (wf-improve session Thermal displacement, mod. 1, S3): lacking a cycle, also a "Camera
|
|
thermal movement" where the camera pose (tilt, rotation or scale) moves away from its median of the day before by at
|
|
least COLD noises, while the station temperature (detect declares it, D-124) falls under its own by at least COLD
|
|
noises the same hour; noise: the median of the variations over an hour (over one step, at one measurement every 4 h),
|
|
so that the threshold does not depend on the cadence. The targets may move in opposite directions or stay almost still
|
|
(the compensation absorbs the movement): the slide rule does not see it. Consecutive pose instants make one episode.
|
|
Without a pose or without a temperature, nothing. The heat is not taken: in fine weather, it is the day's cycle.
|
|
"""
|
|
import math
|
|
import os
|
|
import statistics
|
|
import sys
|
|
from datetime import datetime, timedelta, timezone
|
|
|
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
|
|
from data import iso, load # noqa: E402
|
|
|
|
MIN_HOURS = 20 # a UTC day counts in the profile if it has at least 20 measured hours
|
|
PARTIAL = 12 # an incomplete day of at least 12 hours gets its episode if it looks like the profile
|
|
REPRO = 0.6 # an axis has a cycle if each day looks like the others (pure noise ≈ 0)
|
|
SHARE = 0.5 # share of the targets concerned; 2026-10_NORDEND is just under (9/19)
|
|
WARM = 0.3 # gap from the night level, as a share of the profile's amplitude
|
|
DAY_MATCH = 0.5 # likeness of a day to the others to give it its episode
|
|
NIGHT = 8 # flattest consecutive hours of the profile = night level
|
|
R2_MIN = 0.3 # draft of 2026-10-24: PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48
|
|
MIN_WARM_HOURS = 2 # WILDSTRUBEL, batch 01; NIEDERHORN (E-005) has only one warm hour (noon)
|
|
EDGE = 0.1 # bounds of the wave (start, return), as a share of the amplitude; WARM says whether there is a cycle
|
|
SLIDE_K = 3 # common slide, in noises (median of the steps) away from the median of the day before
|
|
COLD = 4 # the camera moves with the cold, in one-hour noises away from the day before (pose, temperature)
|
|
W = 2 * math.pi / 86400 # angular frequency of a 24 h cycle, in rad/s
|
|
|
|
|
|
def _corr(a, b):
|
|
ma, mb = statistics.mean(a), statistics.mean(b)
|
|
num = sum((x - ma) * (y - mb) for x, y in zip(a, b))
|
|
den = (sum((x - ma) ** 2 for x in a) * sum((y - mb) ** 2 for y in b)) ** 0.5
|
|
return num / den if den else 0.0
|
|
|
|
|
|
def _axis(ms, col):
|
|
"""-> (24 h profile, {day: correlation with the profile of the other days}) or None."""
|
|
days = {}
|
|
for m in ms:
|
|
if m[col] is not None:
|
|
days.setdefault(m["t"].date(), {}).setdefault(m["t"].hour, []).append(m[col])
|
|
full, every = {}, {}
|
|
for d, hs in days.items():
|
|
if len(hs) >= PARTIAL:
|
|
med = statistics.median(v for l in hs.values() for v in l)
|
|
every[d] = {h: statistics.mean(l) - med for h, l in hs.items()}
|
|
if len(hs) >= MIN_HOURS:
|
|
full[d] = every[d]
|
|
if len(full) < 2:
|
|
return None
|
|
profile = [statistics.median(q[h] for q in full.values() if h in q) for h in range(24) if any(h in q for q in full.values())]
|
|
if len(profile) < 24:
|
|
return None
|
|
likeness = {}
|
|
for d, p in every.items():
|
|
others = [q for e, q in full.items() if e != d]
|
|
hs = [h for h in p if all(h in q for q in others)]
|
|
if len(hs) >= 3:
|
|
likeness[d] = _corr([p[h] for h in hs], [statistics.median(q[h] for q in others) for h in hs])
|
|
return profile, likeness
|
|
|
|
|
|
def sine_r2(ms, col):
|
|
"""Share of the curve explained by a 24 h sinusoid (least squares, 3 unknowns), or None if too short."""
|
|
pts = [(m["t"].timestamp(), m[col]) for m in ms if m[col] is not None]
|
|
if len(pts) <= 48:
|
|
return None
|
|
rows = [(1.0, math.sin(W * t), math.cos(W * t)) for t, _ in pts]
|
|
A = [[sum(r[i] * r[j] for r in rows) for j in range(3)] for i in range(3)]
|
|
B = [sum(r[i] * y for r, (_, y) in zip(rows, pts)) for i in range(3)]
|
|
for i in range(3): # Gaussian elimination
|
|
for k in range(i + 1, 3):
|
|
f = A[k][i] / A[i][i]
|
|
A[k] = [x - f * y for x, y in zip(A[k], A[i])]
|
|
B[k] -= f * B[i]
|
|
x = [0.0] * 3
|
|
for i in (2, 1, 0):
|
|
x[i] = (B[i] - sum(A[i][j] * x[j] for j in range(i + 1, 3))) / A[i][i]
|
|
m = statistics.mean(y for _, y in pts)
|
|
sst = sum((y - m) ** 2 for _, y in pts)
|
|
sse = sum((y - sum(c * v for c, v in zip(x, r))) ** 2 for r, (_, y) in zip(rows, pts))
|
|
return 1 - sse / sst if sst else None
|
|
|
|
|
|
def sine(measurements, cols=("corr_x", "corr_y")):
|
|
"""-> (targets with an axis following a 24 h sinusoid, targets evaluated)."""
|
|
yes, evaluated = set(), set()
|
|
for s, ms in measurements.items():
|
|
if ms[0]["pattern_type"] == "global_ref":
|
|
continue
|
|
for col in cols:
|
|
r2 = sine_r2(ms, col)
|
|
if r2 is not None:
|
|
evaluated.add(s)
|
|
if r2 >= R2_MIN:
|
|
yes.add(s)
|
|
return yes, evaluated
|
|
|
|
|
|
def cycle(measurements, cols=("corr_x", "corr_y")):
|
|
"""-> (axes concerned [(sensor, col, profile, likeness)], number of targets evaluated)."""
|
|
concerned, targets = [], set()
|
|
for s, ms in measurements.items():
|
|
if ms[0]["pattern_type"] == "global_ref":
|
|
continue
|
|
for col in cols:
|
|
r = _axis(ms, col)
|
|
if r is None:
|
|
continue
|
|
targets.add(s)
|
|
if statistics.median(r[1].values()) >= REPRO:
|
|
concerned.append((s, col) + r)
|
|
return concerned, len(targets)
|
|
|
|
|
|
def gaps(concerned):
|
|
"""Median gap (over the axes concerned) from the night level, hour by hour, as a share of the profile's amplitude."""
|
|
def score(p, h):
|
|
start = min(range(24), key=lambda i: max(p[(i + k) % 24] for k in range(NIGHT)) - min(p[(i + k) % 24] for k in range(NIGHT)))
|
|
night = statistics.median(p[(start + k) % 24] for k in range(NIGHT))
|
|
return abs(p[h] - night) / (max(p) - min(p))
|
|
return [statistics.median(score(c[2], h) for c in concerned) for h in range(24)]
|
|
|
|
|
|
def warm_hours(concerned):
|
|
return [h for h, e in enumerate(gaps(concerned)) if e >= WARM]
|
|
|
|
|
|
def detect(measurements, temperature=(), camera=()):
|
|
"""Cycle of the corrected curves (the structure); otherwise, that of the raw positions (the camera moving with the
|
|
expansion of its mount, compensated: D-074); otherwise, the camera slides (with the cold)."""
|
|
return (_episodes(measurements, ("corr_x", "corr_y"), "Thermal", False) or
|
|
_episodes(measurements, ("x", "y"), "Camera thermal", True) or slides(measurements, temperature, camera))
|
|
|
|
|
|
def _signs(ms, col, hourly=False, k=None):
|
|
"""{instant: +1 or -1} where the measurement moves away by at least k (default SLIDE_K) noises from the median of the
|
|
day before; noise: the median of the steps from one measurement to the next or, hourly, of the variations over an
|
|
hour (one step if the cadence is slow)."""
|
|
pts = [(m["t"], m[col]) for m in ms if m[col] is not None]
|
|
if len(pts) < 3:
|
|
return {}
|
|
step = statistics.median((b - a).total_seconds() for (a, _), (b, _) in zip(pts, pts[1:])) if hourly else 0
|
|
n = max(1, round(3600 / step)) if step else 1
|
|
noise = statistics.median(abs(pts[i + n][1] - pts[i][1]) for i in range(len(pts) - n)) if len(pts) > n else 0
|
|
out, j = {}, 0
|
|
for i, (t, v) in enumerate(pts):
|
|
while pts[j][0] < t - timedelta(days=1):
|
|
j += 1
|
|
before = [w for _, w in pts[j:i]]
|
|
if noise and len(before) >= 3 and abs(v - statistics.median(before)) >= (k or SLIDE_K) * noise:
|
|
out[t] = 1 if v > statistics.median(before) else -1
|
|
return out
|
|
|
|
|
|
def _common(series):
|
|
"""[{instant: sign}] of a group of sensors -> instants where at least SHARE of them move away on the same side."""
|
|
n, count = len(series), {}
|
|
for sg in series:
|
|
for t, x in sg.items():
|
|
count[(t, x)] = count.get((t, x), 0) + 1
|
|
return {t for (t, x), c in count.items() if n and c >= SHARE * n}
|
|
|
|
|
|
def _cold(measurements, temperature, camera):
|
|
"""[(start, end)] of the runs of instants where the camera pose (tilt, rotation or scale) moves away from its day
|
|
before while the temperature falls under its own, by at least COLD one-hour noises; empty without one of them."""
|
|
ts = [m["t"] for ms in measurements.values() for m in ms]
|
|
hour = lambda t: t.replace(minute=0, second=0, microsecond=0) # noqa: E731
|
|
cold = {hour(t) for t, x in _signs([{"t": t, "T": v} for t, v in temperature], "T", True, COLD).items() if x < 0}
|
|
pose = [p for p in camera if min(ts) <= p["t"] <= max(ts)] if ts else []
|
|
seen = {t for c in ("tilt", "rotation", "scale") for t in _signs(pose, c, True, COLD) if hour(t) in cold}
|
|
episodes = [] # consecutive pose instants make one episode
|
|
for i, p in enumerate(pose):
|
|
if p["t"] in seen:
|
|
if episodes and episodes[-1][2] == i - 1:
|
|
episodes[-1][1:] = [p["t"], i]
|
|
else:
|
|
episodes.append([p["t"], p["t"], i])
|
|
return [(a, b) for a, b, _ in episodes]
|
|
|
|
|
|
def slides(measurements, temperature=(), camera=()):
|
|
"""Common camera slide, without a 24 h cycle: the raw positions of most sensors, references included, and the
|
|
corrected curves of most targets move away together, on the same side, from their day before; or the camera pose
|
|
moves away from it with the cold (_cold)."""
|
|
targets = [ms for ms in measurements.values() if ms[0]["pattern_type"] != "global_ref"]
|
|
raw = set().union(*(_common([_signs(ms, c) for ms in measurements.values()]) for c in ("x", "y")))
|
|
corr = set().union(*(_common([_signs(ms, c) for ms in targets]) for c in ("corr_x", "corr_y")))
|
|
episodes = []
|
|
for t in sorted(raw & corr):
|
|
if episodes and t - episodes[-1][1] <= timedelta(days=1): # less than a day between two: the same slide
|
|
episodes[-1][1] = t
|
|
else:
|
|
episodes.append([t, t])
|
|
return [{"start": iso(a), "end": iso(b), "kind": "Thermal displacement", "label": "Camera thermal movement",
|
|
"text": "The camera moved with a change of temperature and all the curves moved with it; this is not a "
|
|
"movement of the structure, nothing to do.", "ignore": False, "sensors": sorted(measurements)}
|
|
for a, b in episodes + _cold(measurements, temperature, camera)]
|
|
|
|
|
|
def _episodes(measurements, cols, name, camera):
|
|
concerned, n = cycle(measurements, cols)
|
|
sensors = sorted({c[0] for c in concerned})
|
|
if not n or len(sensors) < SHARE * n:
|
|
return []
|
|
if not camera: # second method, for the structure only: the sinusoid must see it too
|
|
yes, evaluated = sine(measurements, cols)
|
|
if not evaluated or len(yes) < SHARE * len(evaluated):
|
|
return []
|
|
warm = warm_hours(concerned)
|
|
if len(warm) < MIN_WARM_HOURS:
|
|
return []
|
|
e = gaps(concerned)
|
|
peak = max(warm, key=lambda h: e[h]) # the largest gap from the night level: end of the expansion
|
|
rest = min(range(24), key=lambda h: e[h])
|
|
a = peak # start: going back from the peak while the profile stays above EDGE, up to the calmest hour
|
|
while (a - 1) % 24 != rest and e[(a - 1) % 24] >= EDGE:
|
|
a -= 1
|
|
b = peak + 1 # return: until the profile goes back under EDGE, at most to the next start
|
|
while b < a + 24 and e[b % 24] >= EDGE:
|
|
b += 1
|
|
last = max(m["t"] for ms in measurements.values() for m in ms)
|
|
days = sorted({d for c in concerned for d in c[3]})
|
|
if camera: # the whole image moves, references included
|
|
sensors = sorted(measurements)
|
|
episodes = []
|
|
for d in days: # ponytail: days of at least PARTIAL hours, the export's ends (22:00 → midnight) get no episode
|
|
if statistics.median(c[3][d] for c in concerned if d in c[3]) < DAY_MATCH:
|
|
continue
|
|
midnight = datetime(d.year, d.month, d.day, tzinfo=timezone.utc)
|
|
for label, u, v in ((f"{name} expansion", a, peak), (f"{name} contraction", peak, b)):
|
|
start, end = midnight + timedelta(hours=u), min(midnight + timedelta(hours=v), last)
|
|
if end > start:
|
|
episodes.append({"start": iso(start), "end": iso(end),
|
|
"kind": "Thermal displacement", "label": label, "text": "", "ignore": False, "sensors": sensors})
|
|
return episodes
|
|
|
|
|
|
if __name__ == "__main__":
|
|
measurements = load(sys.argv[1])
|
|
for e in detect(measurements):
|
|
print(e["start"], "→", e["end"], e["label"], len(e["sensors"]), "targets")
|
|
for cols in (("corr_x", "corr_y"), ("x", "y")):
|
|
concerned, n = cycle(measurements, cols)
|
|
k = len({c[0] for c in concerned})
|
|
amp = statistics.median(max(c[2]) - min(c[2]) for c in concerned) if concerned else 0
|
|
yes, evaluated = sine(measurements, cols)
|
|
print(f"{cols[0][:-2] or 'raw'}: targets {k}/{n}, median amplitude {amp:.3f} px, warm hours (UTC) "
|
|
f"{warm_hours(concerned) if concerned else '-'}; sinusoid: targets {len(yes)}/{len(evaluated)}")
|