Detectors in English, under scripts/; learner tools under scripts/learner/
- data.py (was donnees.py): unused filtrer, heure_image and image sorting removed - score.py: dead functions (validite, marquer, detectes, score, familles) and the validity levels removed; --from/--to - regression.py: history.csv and --seance/--etape removed - categories.py: --from/--to/--chunk/--history; process values images/none/check - benchmark on all stations unchanged (same 132 figures) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9b3ac0ecfe
commit
969637499f
23 files changed
+1877
-1971
No files matched your search
@@ -0,0 +1,292 @@
|
||||
"""Detector of the Thermal displacement family (labeling/familles.md): the daily wave of the corrected displacements
|
||||
(expansion by day, contraction in the evening).
|
||||
|
||||
Method, per target (global references excluded) and per axis (corr_x, corr_y):
|
||||
1. Each UTC day of at least MIN_HOURS hours: mean of each hour minus the median of the day (removes jumps and drift).
|
||||
2. Profile = median, hour by hour, over the days; "repeatability" = median, over the days, of the correlation between
|
||||
the day and the profile of the other days. Axis concerned if repeatability ≥ REPRO. An incomplete day (at least
|
||||
PARTIAL hours, fewer than MIN_HOURS) does not enter the profile, but its likeness is measured on the hours it has: a
|
||||
hole of half a day no longer takes its episode away (learner session Thermal displacement, mod. 1, S4; at
|
||||
MIN_HOURS = 12 or 16, the incomplete days distorted the profile, NIEDERHORN went back).
|
||||
3. Cycle of the station if at least SHARE of the targets have an axis concerned, **and** (second method, asked by the
|
||||
user) if a 24 h sinusoid, a + b·sin(2πt/24 h) + c·cos(2πt/24 h) by least squares, explains at least R2_MIN of the
|
||||
curve of an axis for at least SHARE of the targets.
|
||||
4. Warm hours: night level = median of the NIGHT flattest consecutive hours of the profile; an hour is warm if, as a
|
||||
median over the axes concerned, the profile moves away from it by at least WARM times its amplitude.
|
||||
5. Two episodes per complete day that looks like the others (median correlation ≥ DAY_MATCH over the axes concerned):
|
||||
"Thermal expansion" from the start to the hour of the largest gap from the night level (the peak), then "Thermal
|
||||
contraction" until the return. The warm hours say there is a cycle (at least MIN_WARM_HOURS of them); its bounds
|
||||
are read lower, at EDGE times the amplitude: the start, going back from the peak while the gap stays ≥ EDGE (at most
|
||||
to the calmest hour of the profile); the return, going down from the peak until it goes back under EDGE, after
|
||||
midnight if needed, at most to the next day's start: a slow return at night chains the cycles (learner session
|
||||
Thermal displacement, mod. 1, S1). The direction (a rising or falling curve) depends on the target's orientation:
|
||||
it is not looked at, only the gap from the night level.
|
||||
Reading: the cycle is almost everywhere in the raw positions (common mode) and the compensation removes it, except at
|
||||
PS des Coudres where it stays in corr, with the same shape as in the raw positions.
|
||||
Camera thermal effect (D-074): without a cycle in corr, the same method on the raw positions x, y gives "Camera thermal
|
||||
expansion" and "Camera thermal contraction": the camera moves with the heat of its mount, the compensation removes it;
|
||||
same categories, own sentence. On the raw positions the second method is not asked (WILDSTRUBEL, batch 01: the camera
|
||||
rose by a 3 px block at sunrise, held 3 h and dropped in one step around 09:30, six days in a row, which the sinusoid
|
||||
does not explain); in exchange, a single warm hour is not a cycle (MIN_WARM_HOURS: NIEDERHORN, E-005, hourly, would
|
||||
otherwise have its noon hour as a camera contraction). The camera episode covers all sensors, references included: the
|
||||
whole image moves.
|
||||
Common slide, without a cycle (learner MICHIGAN, mod. 1): lacking a cycle in the corrected curves as in the raw
|
||||
positions, an instant is a camera slide if the raw positions (x or y) of at least SHARE of the sensors, references
|
||||
included, **and** the corrected curves (corr_x or corr_y) of at least SHARE of the targets move away on the same side by
|
||||
at least SLIDE_K noises (median of the steps of the series) from their median of the day before; slides less than a
|
||||
day apart make one episode, "Camera thermal movement", on all sensors, host of the jumps back (categories.HOSTS). The
|
||||
condition on the raw positions leaves to the jumps a flip of the computed pose alone (raw positions still).
|
||||
The camera moves with the cold (learner session Thermal displacement, mod. 1, S3): lacking a cycle, also a "Camera
|
||||
thermal movement" where the camera pose (tilt, rotation or scale) moves away from its median of the day before by at
|
||||
least COLD noises, while the station temperature (detect declares it, D-124) falls under its own by at least COLD
|
||||
noises the same hour; noise: the median of the variations over an hour (over one step, at one measurement every 4 h),
|
||||
so that the threshold does not depend on the cadence. The targets may move in opposite directions or stay almost still
|
||||
(the compensation absorbs the movement): the slide rule does not see it. Consecutive pose instants make one episode.
|
||||
Without a pose or without a temperature, nothing. The heat is not taken: in fine weather, it is the day's cycle.
|
||||
"""
|
||||
import math
|
||||
import os
|
||||
import statistics
|
||||
import sys
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # data.py, next to it
|
||||
from data import iso, load # noqa: E402
|
||||
|
||||
MIN_HOURS = 20 # a UTC day counts in the profile if it has at least 20 measured hours
|
||||
PARTIAL = 12 # an incomplete day of at least 12 hours gets its episode if it looks like the profile
|
||||
REPRO = 0.6 # an axis has a cycle if each day looks like the others (pure noise ≈ 0)
|
||||
SHARE = 0.5 # share of the targets concerned; 2026-10_NORDEND is just under (9/19)
|
||||
WARM = 0.3 # gap from the night level, as a share of the profile's amplitude
|
||||
DAY_MATCH = 0.5 # likeness of a day to the others to give it its episode
|
||||
NIGHT = 8 # flattest consecutive hours of the profile = night level
|
||||
R2_MIN = 0.3 # draft of 2026-10-24: PS des Coudres 14 axes out of 22 above, CLARIDEN 0 out of 48
|
||||
MIN_WARM_HOURS = 2 # WILDSTRUBEL, batch 01; NIEDERHORN (E-005) has only one warm hour (noon)
|
||||
EDGE = 0.1 # bounds of the wave (start, return), as a share of the amplitude; WARM says whether there is a cycle
|
||||
SLIDE_K = 3 # common slide, in noises (median of the steps) away from the median of the day before
|
||||
COLD = 4 # the camera moves with the cold, in one-hour noises away from the day before (pose, temperature)
|
||||
W = 2 * math.pi / 86400 # angular frequency of a 24 h cycle, in rad/s
|
||||
|
||||
|
||||
def _corr(a, b):
|
||||
ma, mb = statistics.mean(a), statistics.mean(b)
|
||||
num = sum((x - ma) * (y - mb) for x, y in zip(a, b))
|
||||
den = (sum((x - ma) ** 2 for x in a) * sum((y - mb) ** 2 for y in b)) ** 0.5
|
||||
return num / den if den else 0.0
|
||||
|
||||
|
||||
def _axis(ms, col):
|
||||
"""-> (24 h profile, {day: correlation with the profile of the other days}) or None."""
|
||||
days = {}
|
||||
for m in ms:
|
||||
if m[col] is not None:
|
||||
days.setdefault(m["t"].date(), {}).setdefault(m["t"].hour, []).append(m[col])
|
||||
full, every = {}, {}
|
||||
for d, hs in days.items():
|
||||
if len(hs) >= PARTIAL:
|
||||
med = statistics.median(v for l in hs.values() for v in l)
|
||||
every[d] = {h: statistics.mean(l) - med for h, l in hs.items()}
|
||||
if len(hs) >= MIN_HOURS:
|
||||
full[d] = every[d]
|
||||
if len(full) < 2:
|
||||
return None
|
||||
profile = [statistics.median(q[h] for q in full.values() if h in q) for h in range(24) if any(h in q for q in full.values())]
|
||||
if len(profile) < 24:
|
||||
return None
|
||||
likeness = {}
|
||||
for d, p in every.items():
|
||||
others = [q for e, q in full.items() if e != d]
|
||||
hs = [h for h in p if all(h in q for q in others)]
|
||||
if len(hs) >= 3:
|
||||
likeness[d] = _corr([p[h] for h in hs], [statistics.median(q[h] for q in others) for h in hs])
|
||||
return profile, likeness
|
||||
|
||||
|
||||
def sine_r2(ms, col):
|
||||
"""Share of the curve explained by a 24 h sinusoid (least squares, 3 unknowns), or None if too short."""
|
||||
pts = [(m["t"].timestamp(), m[col]) for m in ms if m[col] is not None]
|
||||
if len(pts) <= 48:
|
||||
return None
|
||||
rows = [(1.0, math.sin(W * t), math.cos(W * t)) for t, _ in pts]
|
||||
A = [[sum(r[i] * r[j] for r in rows) for j in range(3)] for i in range(3)]
|
||||
B = [sum(r[i] * y for r, (_, y) in zip(rows, pts)) for i in range(3)]
|
||||
for i in range(3): # Gaussian elimination
|
||||
for k in range(i + 1, 3):
|
||||
f = A[k][i] / A[i][i]
|
||||
A[k] = [x - f * y for x, y in zip(A[k], A[i])]
|
||||
B[k] -= f * B[i]
|
||||
x = [0.0] * 3
|
||||
for i in (2, 1, 0):
|
||||
x[i] = (B[i] - sum(A[i][j] * x[j] for j in range(i + 1, 3))) / A[i][i]
|
||||
m = statistics.mean(y for _, y in pts)
|
||||
sst = sum((y - m) ** 2 for _, y in pts)
|
||||
sse = sum((y - sum(c * v for c, v in zip(x, r))) ** 2 for r, (_, y) in zip(rows, pts))
|
||||
return 1 - sse / sst if sst else None
|
||||
|
||||
|
||||
def sine(measurements, cols=("corr_x", "corr_y")):
|
||||
"""-> (targets with an axis following a 24 h sinusoid, targets evaluated)."""
|
||||
yes, evaluated = set(), set()
|
||||
for s, ms in measurements.items():
|
||||
if ms[0]["pattern_type"] == "global_ref":
|
||||
continue
|
||||
for col in cols:
|
||||
r2 = sine_r2(ms, col)
|
||||
if r2 is not None:
|
||||
evaluated.add(s)
|
||||
if r2 >= R2_MIN:
|
||||
yes.add(s)
|
||||
return yes, evaluated
|
||||
|
||||
|
||||
def cycle(measurements, cols=("corr_x", "corr_y")):
|
||||
"""-> (axes concerned [(sensor, col, profile, likeness)], number of targets evaluated)."""
|
||||
concerned, targets = [], set()
|
||||
for s, ms in measurements.items():
|
||||
if ms[0]["pattern_type"] == "global_ref":
|
||||
continue
|
||||
for col in cols:
|
||||
r = _axis(ms, col)
|
||||
if r is None:
|
||||
continue
|
||||
targets.add(s)
|
||||
if statistics.median(r[1].values()) >= REPRO:
|
||||
concerned.append((s, col) + r)
|
||||
return concerned, len(targets)
|
||||
|
||||
|
||||
def gaps(concerned):
|
||||
"""Median gap (over the axes concerned) from the night level, hour by hour, as a share of the profile's amplitude."""
|
||||
def score(p, h):
|
||||
start = min(range(24), key=lambda i: max(p[(i + k) % 24] for k in range(NIGHT)) - min(p[(i + k) % 24] for k in range(NIGHT)))
|
||||
night = statistics.median(p[(start + k) % 24] for k in range(NIGHT))
|
||||
return abs(p[h] - night) / (max(p) - min(p))
|
||||
return [statistics.median(score(c[2], h) for c in concerned) for h in range(24)]
|
||||
|
||||
|
||||
def warm_hours(concerned):
|
||||
return [h for h, e in enumerate(gaps(concerned)) if e >= WARM]
|
||||
|
||||
|
||||
def detect(measurements, temperature=(), camera=()):
|
||||
"""Cycle of the corrected curves (the structure); otherwise, that of the raw positions (the camera moving with the
|
||||
expansion of its mount, compensated: D-074); otherwise, the camera slides (with the cold)."""
|
||||
return (_episodes(measurements, ("corr_x", "corr_y"), "Thermal", False) or
|
||||
_episodes(measurements, ("x", "y"), "Camera thermal", True) or slides(measurements, temperature, camera))
|
||||
|
||||
|
||||
def _signs(ms, col, hourly=False, k=None):
|
||||
"""{instant: +1 or -1} where the measurement moves away by at least k (default SLIDE_K) noises from the median of the
|
||||
day before; noise: the median of the steps from one measurement to the next or, hourly, of the variations over an
|
||||
hour (one step if the cadence is slow)."""
|
||||
pts = [(m["t"], m[col]) for m in ms if m[col] is not None]
|
||||
if len(pts) < 3:
|
||||
return {}
|
||||
step = statistics.median((b - a).total_seconds() for (a, _), (b, _) in zip(pts, pts[1:])) if hourly else 0
|
||||
n = max(1, round(3600 / step)) if step else 1
|
||||
noise = statistics.median(abs(pts[i + n][1] - pts[i][1]) for i in range(len(pts) - n)) if len(pts) > n else 0
|
||||
out, j = {}, 0
|
||||
for i, (t, v) in enumerate(pts):
|
||||
while pts[j][0] < t - timedelta(days=1):
|
||||
j += 1
|
||||
before = [w for _, w in pts[j:i]]
|
||||
if noise and len(before) >= 3 and abs(v - statistics.median(before)) >= (k or SLIDE_K) * noise:
|
||||
out[t] = 1 if v > statistics.median(before) else -1
|
||||
return out
|
||||
|
||||
|
||||
def _common(series):
|
||||
"""[{instant: sign}] of a group of sensors -> instants where at least SHARE of them move away on the same side."""
|
||||
n, count = len(series), {}
|
||||
for sg in series:
|
||||
for t, x in sg.items():
|
||||
count[(t, x)] = count.get((t, x), 0) + 1
|
||||
return {t for (t, x), c in count.items() if n and c >= SHARE * n}
|
||||
|
||||
|
||||
def _cold(measurements, temperature, camera):
|
||||
"""[(start, end)] of the runs of instants where the camera pose (tilt, rotation or scale) moves away from its day
|
||||
before while the temperature falls under its own, by at least COLD one-hour noises; empty without one of them."""
|
||||
ts = [m["t"] for ms in measurements.values() for m in ms]
|
||||
hour = lambda t: t.replace(minute=0, second=0, microsecond=0) # noqa: E731
|
||||
cold = {hour(t) for t, x in _signs([{"t": t, "T": v} for t, v in temperature], "T", True, COLD).items() if x < 0}
|
||||
pose = [p for p in camera if min(ts) <= p["t"] <= max(ts)] if ts else []
|
||||
seen = {t for c in ("tilt", "rotation", "scale") for t in _signs(pose, c, True, COLD) if hour(t) in cold}
|
||||
episodes = [] # consecutive pose instants make one episode
|
||||
for i, p in enumerate(pose):
|
||||
if p["t"] in seen:
|
||||
if episodes and episodes[-1][2] == i - 1:
|
||||
episodes[-1][1:] = [p["t"], i]
|
||||
else:
|
||||
episodes.append([p["t"], p["t"], i])
|
||||
return [(a, b) for a, b, _ in episodes]
|
||||
|
||||
|
||||
def slides(measurements, temperature=(), camera=()):
|
||||
"""Common camera slide, without a 24 h cycle: the raw positions of most sensors, references included, and the
|
||||
corrected curves of most targets move away together, on the same side, from their day before; or the camera pose
|
||||
moves away from it with the cold (_cold)."""
|
||||
targets = [ms for ms in measurements.values() if ms[0]["pattern_type"] != "global_ref"]
|
||||
raw = set().union(*(_common([_signs(ms, c) for ms in measurements.values()]) for c in ("x", "y")))
|
||||
corr = set().union(*(_common([_signs(ms, c) for ms in targets]) for c in ("corr_x", "corr_y")))
|
||||
episodes = []
|
||||
for t in sorted(raw & corr):
|
||||
if episodes and t - episodes[-1][1] <= timedelta(days=1): # less than a day between two: the same slide
|
||||
episodes[-1][1] = t
|
||||
else:
|
||||
episodes.append([t, t])
|
||||
return [{"start": iso(a), "end": iso(b), "kind": "Thermal displacement", "label": "Camera thermal movement",
|
||||
"text": "The camera moved with a change of temperature and all the curves moved with it; this is not a "
|
||||
"movement of the structure, nothing to do.", "ignore": False, "sensors": sorted(measurements)}
|
||||
for a, b in episodes + _cold(measurements, temperature, camera)]
|
||||
|
||||
|
||||
def _episodes(measurements, cols, name, camera):
|
||||
concerned, n = cycle(measurements, cols)
|
||||
sensors = sorted({c[0] for c in concerned})
|
||||
if not n or len(sensors) < SHARE * n:
|
||||
return []
|
||||
if not camera: # second method, for the structure only: the sinusoid must see it too
|
||||
yes, evaluated = sine(measurements, cols)
|
||||
if not evaluated or len(yes) < SHARE * len(evaluated):
|
||||
return []
|
||||
warm = warm_hours(concerned)
|
||||
if len(warm) < MIN_WARM_HOURS:
|
||||
return []
|
||||
e = gaps(concerned)
|
||||
peak = max(warm, key=lambda h: e[h]) # the largest gap from the night level: end of the expansion
|
||||
rest = min(range(24), key=lambda h: e[h])
|
||||
a = peak # start: going back from the peak while the profile stays above EDGE, up to the calmest hour
|
||||
while (a - 1) % 24 != rest and e[(a - 1) % 24] >= EDGE:
|
||||
a -= 1
|
||||
b = peak + 1 # return: until the profile goes back under EDGE, at most to the next start
|
||||
while b < a + 24 and e[b % 24] >= EDGE:
|
||||
b += 1
|
||||
last = max(m["t"] for ms in measurements.values() for m in ms)
|
||||
days = sorted({d for c in concerned for d in c[3]})
|
||||
if camera: # the whole image moves, references included
|
||||
sensors = sorted(measurements)
|
||||
episodes = []
|
||||
for d in days: # ponytail: days of at least PARTIAL hours, the export's ends (22:00 → midnight) get no episode
|
||||
if statistics.median(c[3][d] for c in concerned if d in c[3]) < DAY_MATCH:
|
||||
continue
|
||||
midnight = datetime(d.year, d.month, d.day, tzinfo=timezone.utc)
|
||||
for label, u, v in ((f"{name} expansion", a, peak), (f"{name} contraction", peak, b)):
|
||||
start, end = midnight + timedelta(hours=u), min(midnight + timedelta(hours=v), last)
|
||||
if end > start:
|
||||
episodes.append({"start": iso(start), "end": iso(end),
|
||||
"kind": "Thermal displacement", "label": label, "text": "", "ignore": False, "sensors": sensors})
|
||||
return episodes
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
measurements = load(sys.argv[1])
|
||||
for e in detect(measurements):
|
||||
print(e["start"], "→", e["end"], e["label"], len(e["sensors"]), "targets")
|
||||
for cols in (("corr_x", "corr_y"), ("x", "y")):
|
||||
concerned, n = cycle(measurements, cols)
|
||||
k = len({c[0] for c in concerned})
|
||||
amp = statistics.median(max(c[2]) - min(c[2]) for c in concerned) if concerned else 0
|
||||
yes, evaluated = sine(measurements, cols)
|
||||
print(f"{cols[0][:-2] or 'raw'}: targets {k}/{n}, median amplitude {amp:.3f} px, warm hours (UTC) "
|
||||
f"{warm_hours(concerned) if concerned else '-'}; sinusoid: targets {len(yes)}/{len(evaluated)}")
|
||||
Reference in new issue
Block a user