Skill learner -> wf-improve (/detectors:wf-improve); scripts/wf-improve/, sessions in wf-improve/

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-13 12:26:02 +01:00
1 parent b9595d6011
commit 6df274f580
16 files changed
+93 -93

No files matched your search

+6 -6
View File
@@ -7,18 +7,18 @@ episodes; their label gives the variety (Poor visibility, Heat shimmer, Brief ob
shimmer, and last the thermal effect, if it touches several targets) is absorbed into it: it takes its cause and
family; otherwise it stays Displacement, cause not established, to be checked on the images (familles.md, "A jump then
the return"); a Brief obstruction (one instant) only absorbs the jumps starting at its instant, not those of several
days it crosses (learner session Displacement, mod. 1, S5);
days it crosses (wf-improve session Displacement, mod. 1, S5);
- a slow drift or a jump back whose targets are all in an overlapping Target issue episode is absorbed into it (the
curve of a problem target is to be ignored, familles.md);
- a shift that lasts, or too recent to know (SHIFTS), is never absorbed;
- heat shimmer overlapping a shift of several targets is not kept: a real movement also shakes the curves; the shift of
a single target does not remove the heat shimmer of the whole scene, and heat shimmer only absorbs the jumps back
starting during it, not the one of the day before coming back within its day (learner session Visibility, mod. 1,
starting during it, not the one of the day before coming back within its day (wf-improve session Visibility, mod. 1,
S1: WILDSTRUBEL);
- two events of the same variety less than MERGE apart make one; two Obstruction (Target obscured, Brief obstruction)
only if they share a sensor (learner session Obstruction, mod. 1: S1, different targets chained into an event of
only if they share a sensor (wf-improve session Obstruction, mod. 1: S1, different targets chained into an event of
several days; S3, passings of a few minutes into a whole hour); two Target issue too: each problem target keeps its
bounds (learner session Target issue, mod. 1, S3); a Brief obstruction only absorbs the jumps back within MERGE after
bounds (wf-improve session Target issue, mod. 1, S3); a Brief obstruction only absorbs the jumps back within MERGE after
its instant (S3: not the 30 h jump of another target).
Each event gets its sentence, in English (the site): what happened and what to do about it, as the "For the client"
column of familles.md says; the cause is only given if the variety establishes it, otherwise the images show it.
@@ -173,7 +173,7 @@ def classify(episodes, references=frozenset()):
and datetime.fromisoformat(e["end"]) - datetime.fromisoformat(h["start"]) <= MERGE)), None)
for e in eps if e["label"] == JUMP_BACK}
# a drift or a jump back whose targets are all in an overlapping Target issue episode: its curve is to be ignored,
# it takes its family (learner session Displacement, mod. 1, S6)
# it takes its family (wf-improve session Displacement, mod. 1, S6)
host.update({id(e): h for e in eps if e["label"] in (JUMP_BACK, "Slow drift") and not host.get(id(e))
and (h := next((h for h in eps if h["kind"] == "Target issue" and overlap(e, h)
and set(e["sensors"]) <= set(h["sensors"])), None))})
@@ -219,7 +219,7 @@ if __name__ == "__main__":
p.add_argument("--chunk", type=int, default=0, help="days per chunk (default: a single pass)")
# without history, a chunk starting just before a real jump takes it for a jump back (PR17 at NORDEND, 08/10 02:30:
# absorbed into the fog); 3 days give it back; user's rule (2026-10-24): 3 days of history by default, in the
# learner as in the real application
# workflow as in the real application
p.add_argument("--history", type=int, default=3, help="days of earlier measurements given to the detectors (default 3)")
a = p.parse_args()
measurements = load(a.folder)
+4 -4
View File
@@ -5,7 +5,7 @@ belongs to the Target issue family: episodes() returns it, detect() leaves it to
1. Noise of the series = median of the gaps from one measurement to the next; jump = measurement K noises or more away
from the level before (median of the BASE previous measurements); the first BASE measurements, lacking a level
before, are compared with the level after (median of the BASE next ones), a gap there is a jump back at once.
2. Back (an hour of measurements in a row within half the jump: BASE at 15 min, a single one at 4 h; learner
2. Back (an hour of measurements in a row within half the jump: BASE at 15 min, a single one at 4 h; wf-improve
MICHIGAN, mod. 1) within a day, or one day later back at its usual place (first day of the export) or at the day
before's: "Sudden jump, back to normal", greyed; its cause is not established here (categories.py gives it the
cause of the category that overlaps it, otherwise: to be checked on the images).
@@ -22,7 +22,7 @@ belongs to the Target issue family: episodes() returns it, detect() leaves it to
(REPEAT jumps back or more, each within a day of the previous one) makes a "back to normal" episode on its own, from
the first to the last flip, outside the common episodes. A target that flips for at least FLIP, with at most
MAX_FLIPPING targets at the same time, becomes "Target flipping" (Target issue, D-074); a lasting jump of at least
LARGE of the targets, "Large shift" (everything moved), always to be checked. Learner session Target issue, mod. 1,
LARGE of the targets, "Large shift" (everything moved), always to be checked. wf-improve session Target issue, mod. 1,
S2: FLIP from 6 h to 36 h; a target flipping for a day during a nuisance of the whole scene (rain, fog, night of poor
visibility) is not a problem target; its jumps back go to the nuisance overlapping them, otherwise stay
Displacement.
@@ -206,14 +206,14 @@ def detect(measurements):
# A target that moves little by little, day after day. Takes up the trend of the Displacements tab (D-055) on the
# "advanced" export (corr_x, corr_y, px). Method, per target (global references excluded) and per axis:
# 1. Level of the day: median of the measurements of each UTC day with at least HOURS measured hours out of 24, as a
# share of the hours the station measures (learner MICHIGAN, mod. 1: at one measurement every 4 h, 5 of the 6 hours;
# share of the hours the station measures (wf-improve MICHIGAN, mod. 1: at one measurement every 4 h, 5 of the 6 hours;
# before, no day counted); at least DAYS days.
# 2. Theil-Sen slope and Kendall tau of the daily levels; amplitude of the day: gap between its 9th and 1st deciles.
# 3. Drift: |slope| × duration ≥ SIZE times the median daily amplitude, |tau| ≥ STEADY, and no step from one day to the
# next of more than half the drift (a jump that lasts is not a drift: the jump rule takes it).
# 4. One episode per target (one axis is enough), from the first measurement of the first day to the last of the last
# day; not greyed.
# 4b. Piecewise (learner session Displacement, mod. 1, S1): a target without a drift over the whole window may drift for
# 4b. Piecewise (wf-improve session Displacement, mod. 1, S1): a target without a drift over the whole window may drift for
# a few days, then settle or go back. On each run of at least DAYS kept days, drift = level of the last day minus
# that of the first, at least PIECE daily amplitudes (median of the series), |tau| ≥ STEADY with the same sign, no
# step from one kept day to the next of more than half the drift; pieces taken from the largest drift to the
+1 -1
View File
@@ -5,7 +5,7 @@ for long. It reuses two computations shared with other families:
- the day-by-day rejections of target_issue.py: a target rejected for less than a week, Target obscured.
A sensor already in an overlapping Target issue episode (Degraded target, Target stuck) is not returned as Obstruction:
its curve is already to be ignored, the family is Target issue (boundary "Target issue or an episode of another family"
of labeling/familles.md; learner session Obstruction, mod. 1: S1, problem targets of FLORIDA, NORDEND, SNCF01,
of labeling/familles.md; wf-improve session Obstruction, mod. 1: S1, problem targets of FLORIDA, NORDEND, SNCF01,
NIEDERHORN, JUNGFRAU, CLARIDEN cut into Target obscured; S3, their isolated rejections as Brief obstruction).
Usage: python3 detectors/obstruction.py <station folder> (prints the episodes)
"""
+5 -5
View File
@@ -10,11 +10,11 @@ Method:
fog, which touch everyone, are thus set aside.
2. Per UTC day and per sensor, on its calm instants (at least MIN_HOURS hours of instants at the station's cadence,
median gap between two measurement instants; otherwise the day does not count): degraded day if at least SHARE of
them are rejected. Cadence (learner NEBRASKA, batch 01): 12 fixed instants, "3 h at 15 min", counted no day at one
them are rejected. Cadence (wf-improve NEBRASKA, batch 01): 12 fixed instants, "3 h at 15 min", counted no day at one
measurement every 4 h; REFCOLH22, rejected all the time since it was lost, was not seen.
3. Episode: at least MIN_DAYS degraded days in a row (a day that does not count does not break the run), from the first
rejected measurement of these days to the last; ignore = True: the target's curve is not reliable.
4. Hindered all the time or at times (boundary "Target issue or Obstruction" of labeling/familles.md; learner
4. Hindered all the time or at times (boundary "Target issue or Obstruction" of labeling/familles.md; wf-improve
NEBRASKA, batch 01): a run of at least WEEK days is a problem target, "Degraded target"; shorter, a target hidden at
times, "Target obscured" (Obstruction family), one episode per run of rejected measurements (the hours when it is
seen are not part of it). BREITHORN: vehicles parked in front of COLI24 and POUI22, 2 to 5 days; NEBRASKA and
@@ -22,11 +22,11 @@ Method:
For the problem target, the run is bridged: at most BRIDGE non-degraded days in a row do not break it, and it then
needs WEEK days from its first to its last degraded day, at SHARE_WEEK of rejections over all its days ("rejections
specific to the target almost every day of the range"); the Target obscured episodes it covers no longer exist
(learner session Obstruction, mod. 1, S1: FLORIDA Sellette 6 and Massif 2, SNCF01 F4.3 and F4.4, NORDEND CTR01,
(wf-improve session Obstruction, mod. 1, S1: FLORIDA Sellette 6 and Massif 2, SNCF01 F4.3 and F4.4, NORDEND CTR01,
NIEDERHORN e01, cut into Target obscured by a day under SHARE; BREITHORN, parked vehicles, unchanged).
The problem target judges regularity rather than the share of the day: a day is degraded from SHARE_WEEK (a quarter)
of its calm instants rejected, and BRIDGE is one day; the target hidden at times keeps SHARE. A target rejected every
day at the same hours, by day only or by night only, stays under half and was not seen (learner session Target
day at the same hours, by day only or by night only, stays under half and was not seen (wf-improve session Target
issue, mod. 1, S1: WILDSTRUBEL R3, 04:00 to 16:00 every day; NORDEND PR04 and CTR01, NIEDERHORN PWD_IMP_B08_024_C09
and e01, SNCF01 F4.3).
The cause (dirt, vegetation, setting) can only be read on the images.
@@ -47,7 +47,7 @@ from data import is_reference, iso, load # noqa: E402
CALM = 0.25 # share of the other sensors rejected, at most, for a calm instant
SHARE = 0.5 # half of the calm instants rejected: "most of the time"
SHARE_WEEK = 0.25 # a quarter of the calm instants rejected, each day of a week or more: a problem target, even when
# rejected by day only or by night only (learner session Target issue, mod. 1, S1)
# rejected by day only or by night only (wf-improve session Target issue, mod. 1, S1)
MIN_HOURS = 3 # hours of calm instants, at the station's cadence: below that, the day does not count
MIN_DAYS = 2 # a single day is too often an accident (NORDEND: isolated days on 16 sensors)
BRIDGE = 1 # non-degraded days that do not break a run of degraded days (4); at 2, the parked vehicles of BREITHORN
+4 -4
View File
@@ -6,7 +6,7 @@ Method, per target (global references excluded) and per axis (corr_x, corr_y):
2. Profile = median, hour by hour, over the days; "repeatability" = median, over the days, of the correlation between
the day and the profile of the other days. Axis concerned if repeatability ≥ REPRO. An incomplete day (at least
PARTIAL hours, fewer than MIN_HOURS) does not enter the profile, but its likeness is measured on the hours it has: a
hole of half a day no longer takes its episode away (learner session Thermal displacement, mod. 1, S4; at
hole of half a day no longer takes its episode away (wf-improve session Thermal displacement, mod. 1, S4; at
MIN_HOURS = 12 or 16, the incomplete days distorted the profile, NIEDERHORN went back).
3. Cycle of the station if at least SHARE of the targets have an axis concerned, **and** (second method, asked by the
user) if a 24 h sinusoid, a + b·sin(2πt/24 h) + c·cos(2πt/24 h) by least squares, explains at least R2_MIN of the
@@ -18,7 +18,7 @@ Method, per target (global references excluded) and per axis (corr_x, corr_y):
contraction" until the return. The warm hours say there is a cycle (at least MIN_WARM_HOURS of them); its bounds
are read lower, at EDGE times the amplitude: the start, going back from the peak while the gap stays ≥ EDGE (at most
to the calmest hour of the profile); the return, going down from the peak until it goes back under EDGE, after
midnight if needed, at most to the next day's start: a slow return at night chains the cycles (learner session
midnight if needed, at most to the next day's start: a slow return at night chains the cycles (wf-improve session
Thermal displacement, mod. 1, S1). The direction (a rising or falling curve) depends on the target's orientation:
it is not looked at, only the gap from the night level.
Reading: the cycle is almost everywhere in the raw positions (common mode) and the compensation removes it, except at
@@ -30,13 +30,13 @@ rose by a 3 px block at sunrise, held 3 h and dropped in one step around 09:30,
does not explain); in exchange, a single warm hour is not a cycle (MIN_WARM_HOURS: NIEDERHORN, E-005, hourly, would
otherwise have its noon hour as a camera contraction). The camera episode covers all sensors, references included: the
whole image moves.
Common slide, without a cycle (learner MICHIGAN, mod. 1): lacking a cycle in the corrected curves as in the raw
Common slide, without a cycle (wf-improve MICHIGAN, mod. 1): lacking a cycle in the corrected curves as in the raw
positions, an instant is a camera slide if the raw positions (x or y) of at least SHARE of the sensors, references
included, **and** the corrected curves (corr_x or corr_y) of at least SHARE of the targets move away on the same side by
at least SLIDE_K noises (median of the steps of the series) from their median of the day before; slides less than a
day apart make one episode, "Camera thermal movement", on all sensors, host of the jumps back (categories.HOSTS). The
condition on the raw positions leaves to the jumps a flip of the computed pose alone (raw positions still).
The camera moves with the cold (learner session Thermal displacement, mod. 1, S3): lacking a cycle, also a "Camera
The camera moves with the cold (wf-improve session Thermal displacement, mod. 1, S3): lacking a cycle, also a "Camera
thermal movement" where the camera pose (tilt, rotation or scale) moves away from its median of the day before by at
least COLD noises, while the station temperature (detect declares it, D-124) falls under its own by at least COLD
noises the same hour; noise: the median of the variations over an hour (over one step, at one measurement every 4 h),
+7 -7
View File
@@ -11,19 +11,19 @@ Method:
instants (train, vehicle), with no reference (global_ref) among them: a one-step peak touching the references is not
an obstacle in front of targets (peak of the whole image, NORDEND on 01.09 11:15, batch 01); it is not shown. Unless
at least BRIEF_MIN targets are rejected at this instant and neither at the instant before nor after: something
passing in front of them, a reference touched as well (learner session Displacement, mod. 1, S5: trains of
passing in front of them, a reference touched as well (wf-improve session Displacement, mod. 1, S5: trains of
NIEDERHORN, 12 and 13 targets rejected).
Also, something passing in front of one or two targets, or a single reference: at an instant, fewer than BRIEF_MIN
sensors rejected at more than FACTOR times their median (without K_MAD: by day, heat shimmer inflates the MAD), at
most one reference, none of them rejected at the instant before or after, and fewer than BRIEF_MIN high sensors
(learner session Obstruction, mod. 1, S3: vehicles, passers-by, insects in front of one or two targets of
(wf-improve session Obstruction, mod. 1, S3: vehicles, passers-by, insects in front of one or two targets of
WILDSTRUBEL, NIEDERHORN, LENZSPITZE; a reference hidden by a truck at JUNGFRAU, BREITHORN).
3. Poor visibility: runs of at least RUN_MIN high measurements per sensor, joined across sensors (at least 2), reduced
to the instants where at least FRAC of the episode's peak is high.
4. Target obscured: a target (not a global_ref) rejected (> error_threshold) and above K_TARGET MAD, at least TARGET_MIN
times less than TARGET_GAP apart, mostly outside the episodes of 2 and 3 (leaves, snow, glare on a target). The gap
is at least two measurement steps, and two rejections are enough when TARGET_MIN steps do not fit in TARGET_GAP
(learner session Obstruction, mod. 1, S6: at one measurement every 4 h, vehicles parked at night in front of COLI22,
(wf-improve session Obstruction, mod. 1, S6: at one measurement every 4 h, vehicles parked at night in front of COLI22,
COLI24, POUI22 of BREITHORN make only two or three rejections). All episodes: ignore = True.
5. Level of a Poor visibility (validated 2026-10-24), measurement by measurement: "none" (nothing is seen) when at least
HALF of the usually seen targets (under their threshold at least half the time) are rejected, "reduced" (average
@@ -35,7 +35,7 @@ Method:
7. Poor visibility or object (boundary "Visibility or Obstruction" of labeling/familles.md; BREITHORN, batch 01): a
nuisance touching at most FEW targets, and no reference, is not diffuse: it is an object in front of them (a parked
vehicle), a "Target obscured" episode of its whole duration, without a level.
8. Diffuse nuisance, with or without rejections (learner session Visibility, mod. 1, S2): at each instant, the median,
8. Diffuse nuisance, with or without rejections (wf-improve session Visibility, mod. 1, S2): at each instant, the median,
over the sensors, of the error_value in MAD above each one's median; at least RUN_MIN instants in a row at K_DIFFUSE
or more make a Poor visibility "reduced". A diffuse nuisance entirely within heat shimmer is the heat shimmer: no
episode (otherwise it absorbed the jump back of the day before, WILDSTRUBEL on 04.10). Finds the drops of
@@ -69,7 +69,7 @@ TARGET_GAP = timedelta(hours=6) # tuned on E-005 (leaves: 7 rejections in 12 h,
REJECT_FACTOR = 1.2 # a rejection at 10 MAD also counts (FLORIDA, batch 01: levels of 0.6, ×2 out of reach); drops C24
# of E-009 (level 0.68 just under the threshold 0.7, rejected at ×1.04)
HALF = 0.5 # "nothing is seen"; tuned on NORDEND (09.10 23:00: 68 % rejected) and E-005 (rain: 24 to 33 %)
K_DIFFUSE = 2.5 # diffuse nuisance (8); learner session Visibility, mod. 1, S2
K_DIFFUSE = 2.5 # diffuse nuisance (8); wf-improve session Visibility, mod. 1, S2
FEW = 3 # Poor visibility on at most 3 targets, no reference: an object, Target obscured (7); BREITHORN,
# batch 01 (vehicles parked in front of 2 or 3 targets out of 12)
@@ -255,13 +255,13 @@ def detect(measurements):
# sensors at ≥ RATIO_ERR times the calm (warm air blurs the spots: E-003).
# 4. Neighbouring hours merged (a one-hour hole tolerated), episode of at least MIN_DURATION, dropped if more than
# REJECT_MAX of the measurements are rejected beyond their sensor's habit (its share of rejections over the export):
# it is then poor visibility, another rule. A target that is always rejected (Target issue) does not count (learner
# it is then poor visibility, another rule. A target that is always rejected (Target issue) does not count (wf-improve
# session Visibility, mod. 1, S1: WILDSTRUBEL, R3 rejected all day, the days of heat shimmer were dropped).
# Greyed if the median agitation ≥ STRONG.
RATIO = 1.5 # tuned on E-003 (WILDSTRUBEL: by day, corr ×1.7 to ×3 the night)
SHARE = 0.5 # share of the sensors agitated at the same time
RATIO_ERR = 1.5 # tuned on E-003 (error_value ×1.7 to ×3.2 the night; ≈ ×1 at noon elsewhere)
REJECT_MAX = 0.15 # E-003 ≤ 7 %; cobweb (E-002) and rain (E-005) ≥ 18 %; rejections beyond the habit since learner
REJECT_MAX = 0.15 # E-003 ≤ 7 %; cobweb (E-002) and rain (E-005) ≥ 18 %; rejections beyond the habit since wf-improve
# session Visibility, mod. 1, S1 (WILDSTRUBEL: 4 to 14 %, against 9 to 22 % raw)
STRONG = 2.0 # E-003 (×2.3 to ×2.6) is strong; an episode between ×1.5 and ×2 stays visible
MIN_DURATION = timedelta(hours=3) # E-003 lasts 10-11 h; drops the morning of the real movement of E-004 (2 h), which
@@ -1,11 +1,11 @@
"""Benchmark of the learner (D-116): the detectors against all the labeller's labels (labeling/auto/).
"""Benchmark of the wf-improve workflow (D-116): the detectors against all the labeller's labels (labeling/auto/).
For each station of labeling/auto/ that has its folder in data/, the score per family and per instant of score.py
(D-112), over the range of its labels: from the day of the first to the midnight after the last. Then the total, family
by family, which says whether the detectors do better overall from one session to the next. The block of the session's
station is its score; the other blocks, the non-regression: a family that goes back on a station is a regression, even
if the total rises.
Usage: python3 detectors/scripts/learner/regression.py [--stations A,B]
Usage: python3 detectors/scripts/wf-improve/regression.py [--stations A,B]
"""
import argparse
import json
@@ -8,7 +8,7 @@ range classified with its 3 days of history). The category of an event is its fa
The score is a fact, not a verdict: the labels can be wrong too. No label on the range: exits with an error.
Label fields (written by Diagnostic-app): debut, fin, categorie (the family), description.
Run from the working repository, where detectors/ is mounted.
Usage: python3 detectors/scripts/learner/score.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD
Usage: python3 detectors/scripts/wf-improve/score.py <station folder> --from YYYY-MM-DD --to YYYY-MM-DD
"""
import argparse
import json
@@ -1,7 +1,7 @@
"""For the upgrader: the cadence of the measurements of the range, the number of targets and references, then each event
of the range with its sensors (categories.py only gives their number). Same computation as categories.py without
--chunk (3 days of history).
Usage: python3 detectors/scripts/learner/sensors.py <folder> --from YYYY-MM-DD --to YYYY-MM-DD
Usage: python3 detectors/scripts/wf-improve/sensors.py <folder> --from YYYY-MM-DD --to YYYY-MM-DD
"""
import argparse
import statistics