Files
aimsight-detectors/scripts/thermal_displacement.py
T

295 lines
17 KiB
Python
Raw Normal View History

"""Detector of the Thermal displacement family (the families document): the daily wave of the corrected displacements
(expansion by day, contraction in the evening).
Method, per target (global references excluded) and per axis (corr_x, corr_y):
1. Each UTC day of at least MIN_HOURS hours: mean of each hour minus the median of the day (removes jumps and drift).
2. Profile = median, hour by hour, over the days; "repeatability" = median, over the days, of the correlation between
the day and the profile of the other days. Axis concerned if repeatability ≥ REPRO. An incomplete day (at least
PARTIAL hours, fewer than MIN_HOURS) does not enter the profile, but its likeness is measured on the hours it has: a
hole of half a day no longer takes its episode away (wf-improve session Thermal displacement, mod. 1, S4; at
MIN_HOURS = 12 or 16, the incomplete days distorted the profile, NIEDERHORN went back).
3. Cycle of the station if at least SHARE of the targets have an axis concerned, **and** (second method, asked by the
user) if a 24 h sinusoid, a + b·sin(2πt/24 h) + c·cos(2πt/24 h) by least squares, explains at least R2_MIN of the
curve of an axis for at least SHARE of the targets.
4. Warm hours: night level = median of the NIGHT flattest consecutive hours of the profile; an hour is warm if, as a
median over the axes concerned, the profile moves away from it by at least WARM times its amplitude.
5. Two episodes per complete day that looks like the others (median correlation ≥ DAY_MATCH over the axes concerned):
"Thermal expansion" from the start to the hour of the largest gap from the night level (the peak), then "Thermal
contraction" until the return. The warm hours say there is a cycle (at least MIN_WARM_HOURS of them); its bounds
are read lower, at EDGE times the amplitude: the start, going back from the peak while the gap stays ≥ EDGE (at most
2026-11-16 11:56:56 +01:00
to the calmest hour of the profile); the return, going down from the peak until it goes back under RETURN, after
midnight if needed, at most to the next day's start: a slow return at night chains the cycles (wf-improve session
2026-11-16 11:56:56 +01:00
Thermal displacement, mod. 1, S1). RETURN is half of EDGE: the warm-up is quick, the cooling tail slow, and the
labelled nights went on after the return reached EDGE (wf-improve NORDEND 2026-11-05, S1: NORDEND, JUNGFRAU; EDGE
itself at 0.05 made FLORIDA and SNCF01 overflow in the morning). The direction (a rising or falling curve) depends on the target's orientation:
it is not looked at, only the gap from the night level.
Reading: the cycle is almost everywhere in the raw positions (common mode) and the compensation removes it, except at
PS des Coudres where it stays in corr, with the same shape as in the raw positions.
Camera thermal effect (D-074): without a cycle in corr, the same method on the raw positions x, y gives "Camera thermal
expansion" and "Camera thermal contraction": the camera moves with the heat of its mount, the compensation removes it;
same categories, own sentence. On the raw positions the second method is not asked (WILDSTRUBEL, batch 01: the camera
rose by a 3 px block at sunrise, held 3 h and dropped in one step around 09:30, six days in a row, which the sinusoid
does not explain); in exchange, a single warm hour is not a cycle (MIN_WARM_HOURS: NIEDERHORN, E-005, hourly, would
otherwise have its noon hour as a camera contraction). The camera episode covers all sensors, references included: the
whole image moves.
Common slide, without a cycle (wf-improve MICHIGAN, mod. 1): lacking a cycle in the corrected curves as in the raw
positions, an instant is a camera slide if the raw positions (x or y) of at least SHARE of the sensors, references
included, **and** the corrected curves (corr_x or corr_y) of at least SHARE of the targets move away on the same side by
at least SLIDE_K noises (median of the steps of the series) from their median of the day before; slides less than a
day apart make one episode, "Camera thermal movement", on all sensors, host of the jumps back (categories.HOSTS). The
condition on the raw positions leaves to the jumps a flip of the computed pose alone (raw positions still).
The camera moves with the cold (wf-improve session Thermal displacement, mod. 1, S3): lacking a cycle, also a "Camera
thermal movement" where the camera pose (tilt, rotation or scale) moves away from its median of the day before by at
least COLD noises, while the station temperature (detect declares it, D-124) falls under its own by at least COLD
noises the same hour; noise: the median of the variations over an hour (over one step, at one measurement every 4 h),
so that the threshold does not depend on the cadence. The targets may move in opposite directions or stay almost still
(the compensation absorbs the movement): the slide rule does not see it. Consecutive pose instants make one episode.
Without a pose or without a temperature, nothing. The heat is not taken: in fine weather, it is the day's cycle.
"""
import math
import os
import statistics
import sys
from datetime import datetime, timedelta, timezone
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
from data import iso, load # noqa: E402
MIN_HOURS = 20 # a UTC day counts in the profile if it has at least 20 measured hours
PARTIAL = 12 # an incomplete day of at least 12 hours gets its episode if it looks like the profile
REPRO = 0.6 # an axis has a cycle if each day looks like the others (pure noise ≈ 0)
SHARE = 0.5 # share of the targets concerned; 2026-10_NORDEND is just under (9/19)
WARM = 0.3 # gap from the night level, as a share of the profile's amplitude
DAY_MATCH = 0.5 # likeness of a day to the others to give it its episode
NIGHT = 8 # flattest consecutive hours of the profile = night level
R2_MIN = 0.3 # draft of 2026-10-24: PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48
MIN_WARM_HOURS = 2 # WILDSTRUBEL, batch 01; NIEDERHORN (E-005) has only one warm hour (noon)
2026-11-16 11:56:56 +01:00
EDGE = 0.1 # start of the wave, as a share of the amplitude; WARM says whether there is a cycle
RETURN = 0.05 # return of the wave, lower: the cooling tail is slow (wf-improve NORDEND 2026-11-05, S1)
SLIDE_K = 3 # common slide, in noises (median of the steps) away from the median of the day before
COLD = 4 # the camera moves with the cold, in one-hour noises away from the day before (pose, temperature)
W = 2 * math.pi / 86400 # angular frequency of a 24 h cycle, in rad/s
def _corr(a, b):
ma, mb = statistics.mean(a), statistics.mean(b)
num = sum((x - ma) * (y - mb) for x, y in zip(a, b))
den = (sum((x - ma) ** 2 for x in a) * sum((y - mb) ** 2 for y in b)) ** 0.5
return num / den if den else 0.0
def _axis(ms, col):
"""-> (24 h profile, {day: correlation with the profile of the other days}) or None."""
days = {}
for m in ms:
if m[col] is not None:
days.setdefault(m["t"].date(), {}).setdefault(m["t"].hour, []).append(m[col])
full, every = {}, {}
for d, hs in days.items():
if len(hs) >= PARTIAL:
med = statistics.median(v for l in hs.values() for v in l)
every[d] = {h: statistics.mean(l) - med for h, l in hs.items()}
if len(hs) >= MIN_HOURS:
full[d] = every[d]
if len(full) < 2:
return None
profile = [statistics.median(q[h] for q in full.values() if h in q) for h in range(24) if any(h in q for q in full.values())]
if len(profile) < 24:
return None
likeness = {}
for d, p in every.items():
others = [q for e, q in full.items() if e != d]
hs = [h for h in p if all(h in q for q in others)]
if len(hs) >= 3:
likeness[d] = _corr([p[h] for h in hs], [statistics.median(q[h] for q in others) for h in hs])
return profile, likeness
def sine_r2(ms, col):
"""Share of the curve explained by a 24 h sinusoid (least squares, 3 unknowns), or None if too short."""
pts = [(m["t"].timestamp(), m[col]) for m in ms if m[col] is not None]
if len(pts) <= 48:
return None
rows = [(1.0, math.sin(W * t), math.cos(W * t)) for t, _ in pts]
A = [[sum(r[i] * r[j] for r in rows) for j in range(3)] for i in range(3)]
B = [sum(r[i] * y for r, (_, y) in zip(rows, pts)) for i in range(3)]
for i in range(3): # Gaussian elimination
for k in range(i + 1, 3):
f = A[k][i] / A[i][i]
A[k] = [x - f * y for x, y in zip(A[k], A[i])]
B[k] -= f * B[i]
x = [0.0] * 3
for i in (2, 1, 0):
x[i] = (B[i] - sum(A[i][j] * x[j] for j in range(i + 1, 3))) / A[i][i]
m = statistics.mean(y for _, y in pts)
sst = sum((y - m) ** 2 for _, y in pts)
sse = sum((y - sum(c * v for c, v in zip(x, r))) ** 2 for r, (_, y) in zip(rows, pts))
return 1 - sse / sst if sst else None
def sine(measurements, cols=("corr_x", "corr_y")):
"""-> (targets with an axis following a 24 h sinusoid, targets evaluated)."""
yes, evaluated = set(), set()
for s, ms in measurements.items():
if ms[0]["pattern_type"] == "global_ref":
continue
for col in cols:
r2 = sine_r2(ms, col)
if r2 is not None:
evaluated.add(s)
if r2 >= R2_MIN:
yes.add(s)
return yes, evaluated
def cycle(measurements, cols=("corr_x", "corr_y")):
"""-> (axes concerned [(sensor, col, profile, likeness)], number of targets evaluated)."""
concerned, targets = [], set()
for s, ms in measurements.items():
if ms[0]["pattern_type"] == "global_ref":
continue
for col in cols:
r = _axis(ms, col)
if r is None:
continue
targets.add(s)
if statistics.median(r[1].values()) >= REPRO:
concerned.append((s, col) + r)
return concerned, len(targets)
def gaps(concerned):
"""Median gap (over the axes concerned) from the night level, hour by hour, as a share of the profile's amplitude."""
def score(p, h):
start = min(range(24), key=lambda i: max(p[(i + k) % 24] for k in range(NIGHT)) - min(p[(i + k) % 24] for k in range(NIGHT)))
night = statistics.median(p[(start + k) % 24] for k in range(NIGHT))
return abs(p[h] - night) / (max(p) - min(p))
return [statistics.median(score(c[2], h) for c in concerned) for h in range(24)]
def warm_hours(concerned):
return [h for h, e in enumerate(gaps(concerned)) if e >= WARM]
def detect(measurements, temperature=(), camera=()):
"""Cycle of the corrected curves (the structure); otherwise, that of the raw positions (the camera moving with the
expansion of its mount, compensated: D-074); otherwise, the camera slides (with the cold)."""
return (_episodes(measurements, ("corr_x", "corr_y"), "Thermal", False) or
_episodes(measurements, ("x", "y"), "Camera thermal", True) or slides(measurements, temperature, camera))
def _signs(ms, col, hourly=False, k=None):
"""{instant: +1 or -1} where the measurement moves away by at least k (default SLIDE_K) noises from the median of the
day before; noise: the median of the steps from one measurement to the next or, hourly, of the variations over an
hour (one step if the cadence is slow)."""
pts = [(m["t"], m[col]) for m in ms if m[col] is not None]
if len(pts) < 3:
return {}
step = statistics.median((b - a).total_seconds() for (a, _), (b, _) in zip(pts, pts[1:])) if hourly else 0
n = max(1, round(3600 / step)) if step else 1
noise = statistics.median(abs(pts[i + n][1] - pts[i][1]) for i in range(len(pts) - n)) if len(pts) > n else 0
out, j = {}, 0
for i, (t, v) in enumerate(pts):
while pts[j][0] < t - timedelta(days=1):
j += 1
before = [w for _, w in pts[j:i]]
if noise and len(before) >= 3 and abs(v - statistics.median(before)) >= (k or SLIDE_K) * noise:
out[t] = 1 if v > statistics.median(before) else -1
return out
def _common(series):
"""[{instant: sign}] of a group of sensors -> instants where at least SHARE of them move away on the same side."""
n, count = len(series), {}
for sg in series:
for t, x in sg.items():
count[(t, x)] = count.get((t, x), 0) + 1
return {t for (t, x), c in count.items() if n and c >= SHARE * n}
def _cold(measurements, temperature, camera):
"""[(start, end)] of the runs of instants where the camera pose (tilt, rotation or scale) moves away from its day
before while the temperature falls under its own, by at least COLD one-hour noises; empty without one of them."""
ts = [m["t"] for ms in measurements.values() for m in ms]
hour = lambda t: t.replace(minute=0, second=0, microsecond=0) # noqa: E731
cold = {hour(t) for t, x in _signs([{"t": t, "T": v} for t, v in temperature], "T", True, COLD).items() if x < 0}
pose = [p for p in camera if min(ts) <= p["t"] <= max(ts)] if ts else []
seen = {t for c in ("tilt", "rotation", "scale") for t in _signs(pose, c, True, COLD) if hour(t) in cold}
episodes = [] # consecutive pose instants make one episode
for i, p in enumerate(pose):
if p["t"] in seen:
if episodes and episodes[-1][2] == i - 1:
episodes[-1][1:] = [p["t"], i]
else:
episodes.append([p["t"], p["t"], i])
return [(a, b) for a, b, _ in episodes]
def slides(measurements, temperature=(), camera=()):
"""Common camera slide, without a 24 h cycle: the raw positions of most sensors, references included, and the
corrected curves of most targets move away together, on the same side, from their day before; or the camera pose
moves away from it with the cold (_cold)."""
targets = [ms for ms in measurements.values() if ms[0]["pattern_type"] != "global_ref"]
raw = set().union(*(_common([_signs(ms, c) for ms in measurements.values()]) for c in ("x", "y")))
corr = set().union(*(_common([_signs(ms, c) for ms in targets]) for c in ("corr_x", "corr_y")))
episodes = []
for t in sorted(raw & corr):
if episodes and t - episodes[-1][1] <= timedelta(days=1): # less than a day between two: the same slide
episodes[-1][1] = t
else:
episodes.append([t, t])
return [{"start": iso(a), "end": iso(b), "kind": "Thermal displacement", "label": "Camera thermal movement",
"text": "The camera moved with a change of temperature and all the curves moved with it; this is not a "
"movement of the structure, nothing to do.", "ignore": False, "sensors": sorted(measurements)}
for a, b in episodes + _cold(measurements, temperature, camera)]
def _episodes(measurements, cols, name, camera):
concerned, n = cycle(measurements, cols)
sensors = sorted({c[0] for c in concerned})
if not n or len(sensors) < SHARE * n:
return []
if not camera: # second method, for the structure only: the sinusoid must see it too
yes, evaluated = sine(measurements, cols)
if not evaluated or len(yes) < SHARE * len(evaluated):
return []
warm = warm_hours(concerned)
if len(warm) < MIN_WARM_HOURS:
return []
e = gaps(concerned)
peak = max(warm, key=lambda h: e[h]) # the largest gap from the night level: end of the expansion
rest = min(range(24), key=lambda h: e[h])
a = peak # start: going back from the peak while the profile stays above EDGE, up to the calmest hour
while (a - 1) % 24 != rest and e[(a - 1) % 24] >= EDGE:
a -= 1
2026-11-16 11:56:56 +01:00
b = peak + 1 # return: until the profile goes back under RETURN, at most to the next start
while b < a + 24 and e[b % 24] >= RETURN:
b += 1
last = max(m["t"] for ms in measurements.values() for m in ms)
days = sorted({d for c in concerned for d in c[3]})
if camera: # the whole image moves, references included
sensors = sorted(measurements)
episodes = []
for d in days: # ponytail: days of at least PARTIAL hours, the export's ends (22:00 → midnight) get no episode
if statistics.median(c[3][d] for c in concerned if d in c[3]) < DAY_MATCH:
continue
midnight = datetime(d.year, d.month, d.day, tzinfo=timezone.utc)
for label, u, v in ((f"{name} expansion", a, peak), (f"{name} contraction", peak, b)):
start, end = midnight + timedelta(hours=u), min(midnight + timedelta(hours=v), last)
if end > start:
episodes.append({"start": iso(start), "end": iso(end),
"kind": "Thermal displacement", "label": label, "text": "", "ignore": False, "sensors": sensors})
return episodes
if __name__ == "__main__":
measurements = load(sys.argv[1])
for e in detect(measurements):
print(e["start"], "→", e["end"], e["label"], len(e["sensors"]), "targets")
for cols in (("corr_x", "corr_y"), ("x", "y")):
concerned, n = cycle(measurements, cols)
k = len({c[0] for c in concerned})
amp = statistics.median(max(c[2]) - min(c[2]) for c in concerned) if concerned else 0
yes, evaluated = sine(measurements, cols)
print(f"{cols[0][:-2] or 'raw'}: targets {k}/{n}, median amplitude {amp:.3f} px, warm hours (UTC) "
f"{warm_hours(concerned) if concerned else '-'}; sinusoid: targets {len(yes)}/{len(evaluated)}")