Files
aimsight-labeling/auto/analyses/NIEDERHORN_2026-10-01T0000_2026-10-11T0500/scripts/edit_analysis.py
T
maximeandClaude Opus 5.5 ddb17c2189 auto: labels requalified under the 5-noise rules (only what stands out from the noise): 32 analyses refined, the 6 paused periods verified, the long labels requalified with the fixes the period analysts found; 1153 -> 961 labels
- 32 analyses: each label kept, rewritten or removed by its analyst (section Requalification of the report; mm magnitudes where the targets have them)
- SNCF01 x3, WILDSTRUBEL 2026-09-01 -> 2026-10-06 x3: verified from their existing labels under the new rules
- long labels (no analysis): auto/analyses/long-labels/<STATION>/; NORDEND-169 and -195 run to 07/11, so NORDEND-283 and -284 (their continuation) are removed
- reading-rules.md, stations.md: the citations of removed or rewritten labels point to their successor, or to [expert]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 23:58:00 +01:00

67 lines
8.9 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Insert the requalification of 2026-11-17 into analysis.md (run once, from the repository root)."""
D = "labeling/auto/analyses/NIEDERHORN_2026-10-01T0000_2026-10-11T0500"
p = f"{D}/analysis.md"
t = open(p).read()
note = """> **Requalification of 2026-11-17** (new rules, `labeling-rules.md` 1): 10 of the labels below are removed, NIEDERHORN-155 is rewritten; see section 2 and "## Requalification". The subsections of the removed labels (01/10 23:00 → 02/10 01:00, 03/10 09:00 → 11:00, the 05/10 sunrise in the grouped sun subsection, 06/10 10:00 → 12:00) stay as a record of the reading, but those labels are no longer in `labels.jsonl`.
"""
removed = """### Removed at the requalification (2026-11-17)
Deviations at the touched measurements, all targets, corr_x and corr_y, in each target's own noise: "global" against the median of the range (as `ranges.py --k 5`), "rolling" against the median of the 6 measurements on each side (`--rolling 6`), the reference under a thermal cycle or a slow level. "Rejected": filtered by the station (missing point).
- **NIEDERHORN-15**, 2026-10-01T20:00Z → 22:00Z, Obstruction (train, 21:00): missing points only (e01, e02, e06, e09, PWD_RMP_B17_026_C09 rejected). Global: e10 corr_x −8.0, PWD_RMP_B16_026_C09 corr_x −6.7, but both are the slow post-rain evening level (e10 already −10.9 at 20:00 and −11.2 at 23:00); rolling max 3.8 (PWD_RMP_B16_026_C09 corr_y). Nothing jumps with the train.
- **NIEDERHORN-145**, 2026-10-01T23:00Z → 02T01:00Z, Displacement (PWD_IMP_B08_025_C09 dip, 00:00): PWD_IMP_B08_025_C09 corr_x −2.4 noises rolling, under 5 global; max of all targets rolling 3.7 (e10 corr_y); global e10 corr_x −9.5 is the slow level of that night. Under the noise.
- **NIEDERHORN-21**, 2026-10-03T06:00Z → 08:00Z, Obstruction (plant, 07:00): missing point only (PWD_IMP_B07_016_C09 rejected alone; a target, no effect on the compensation). Rolling max 1.9; global 5.2 on PWD_RMP_B12_019_C09 corr_x, its ordinary night level, not a jump.
- **NIEDERHORN-149**, 2026-10-03T09:00Z → 11:00Z, Obstruction (e01, e02 hidden, 10:00): missing points only. Rolling max 3.8 (e06 corr_y); global 6.7 / 6.2 on PWD_RMP_B15_018_C09 corr_y and PWD_RMP_B12_019_C09 corr_x, the night level, not a jump.
- **NIEDERHORN-153**, 2026-10-05T12:00Z → 14:00Z, Visibility (sun at sunrise, 13:00): at the start of the thermal cycle (NIEDERHORN-28), rolling max 4.7 (PWD_RMP_B15_018_C09 corr_y); the global 6.4 on PWD_IMP_B07_018_C09 corr_x is the onset of the thermal drift, already in NIEDERHORN-28. The label itself said "the other targets hardly". Under the noise.
- **NIEDERHORN-30**, 2026-10-06T03:00Z → 05:00Z, Obstruction (plant, 04:00): missing point only (PWD_IMP_B07_016_C09 rejected). Rolling max 1.7; global 6.5 on PWD_RMP_B12_019_C09 corr_x, its night level.
- **NIEDERHORN-154**, 2026-10-06T10:00Z → 12:00Z, Visibility (PWD_RMP_B12_019_C09 rejected at dawn, 11:00): missing point only. At 11:00, rolling +7.5 on PWD_IMP_B07_018_C09 corr_x and +5.5 on e09 corr_y, global +8.6 on e09 corr_y; but the raw values at 11:00 continue those of 09:00-10:00 (PWD_IMP_B07_018_C09 0.438, 0.467, 0.456 px; e09 −0.025, −0.062, −0.013 px): no jump; the rolling deviation comes from the thermal drop of the following hours inside its window, the global one from the slow pre-dawn level. Not an effect of the rejection.
- **NIEDERHORN-35**, 2026-10-06T22:00Z → 07T00:00Z, Obstruction (plant, 23:00): missing point only (PWD_IMP_B07_016_C09 rejected, a target). At 23:00 PWD_RMP_B15_018_C09 and PWD_RMP_B15_019_C09 corr_x make a one-measurement spike (rolling +7.6 and +6.5) while PWD_IMP_B07_018_C09 regains its night level in one step: the return of the thermal cycle, inside NIEDERHORN-33 (→ 07/10 00:00), not caused by the plant.
- **NIEDERHORN-43**, 2026-10-08T16:00Z → 18:00Z, Obstruction (plant, 17:00): missing points only (PWD_RMP_B12_017_C09 and PWD_IMP_B07_016_C09 rejected, both targets). The deviations at 17:00 (global −10.3 PWD_IMP_B07_017_C09, rolling +6.8 e06) are the thermal cycle and its hot-hour agitation, inside NIEDERHORN-42.
- **NIEDERHORN-48**, 2026-10-09T19:00Z → 21:00Z, Obstruction (train, 20:00): missing points only (e01, e02, e06, e09, PWD_RMP_B17_026_C09 rejected). Rolling max 4.6 (e13 corr_y); global 6.4 on PWD_RMP_B12_018_C09 corr_x, the evening thermal level (NIEDERHORN-46).
### Earlier notes (first analysis)
"""
req = """## Requalification
Rules of 2026-11-17 (`labeling-rules.md` 1): a movement is kept if at least one target deviates by more than 5 of its own noises over the label's range; a thermal label if the corrected curves keep it; a label of missing points is removed unless it moves the other curves; No data only when nothing is delivered.
Commands (outputs in `work/requalify/`):
- `python3 tools/scripts/ranges.py data/NIEDERHORN/ --columns corr_x,corr_y --start 2026-10-01T00:00Z --end 2026-10-11T05:00Z` → `ranges_k5.txt`
- `python3 tools/scripts/ranges.py data/NIEDERHORN/ --columns corr_x,corr_y --rolling 6 --start 2026-09-30T12:00Z --end 2026-10-11T05:00Z` → `ranges_k5_rolling6.txt`
- `python3 scripts/requalify.py work/requalify_input.jsonl` + both outputs → `per_label.txt`: the k5 ranges of the targets overlapping the touched measurements of each label
- `python3 scripts/maxdev.py data/NIEDERHORN work/requalify_input.jsonl` + ids → `maxdev.txt`: the largest deviation in noises at the touched measurements, all targets, global and rolling (same level and noise as `ranges.py`)
- No data: number of measurements of all sensors at 01/10 01:00, 01/10 13:00, 02/10 12:00, 08/10 20:00, 11/10 00:00: 0 each (35 at the neighbouring hours).
Result: 52 in, 41 kept as they are, 1 rewritten, 10 removed (section 2), 42 out.
Values of the kept labels (largest target deviation at the touched measurements, global / rolling, in noises):
- No data NIEDERHORN-1, -14, -18, -44, -51: no measurement of any sensor; kept.
- Plant on PWD_IMP_B07_016_C09: -140 corr_x −7.1 / −7.5; -142 corr_x +8.9 global (11:00), −9.6 rolling (14:00); -146 corr_y +7.0 / +7.2; -148 corr_x +20.5 / +19.2 (02/10 08:00); -152 corr_x +5.5 / +6.7; -29 corr_y +7.8 / +7.2 (16:00); -156 corr_y −6.3 / −6.2 (22:00); -40 corr_x +9.8 / +9.9 (02:00), corr_y −8.2 / −7.9 (06:00); -159 corr_y +7.4 / +7.0; -161 corr_y −7.6 / −8.0 (00:00).
- Rain: -141 e10 corr_x −31.4 / −23.1; -147 e15 corr_x −32.8 / −29.8; -39 PWD_RMP_B15_019_C09 corr_y +26.1 / +27.3.
- Trains: -143 PWD_RMP_B12_019_C09 corr_y +26.1 / +26.4; -20 PWD_RMP_B17_026_C09 corr_x +7.9 global, corr_y +5.6 rolling; -155 PWD_RMP_B12_019_C09 corr_y +31.5 / +29.9; -34 +38.3 / +40.7; -36 PWD_RMP_B15_019_C09 corr_y +40.7 / +41.4; -52 PWD_RMP_B12_020_C09 corr_y +22.0 / +21.2.
- Sun at sunrise: -150 PWD_IMP_B07_018_C09 corr_x −9.8 global, e02 +7.9 rolling; -151 PWD_IMP_B07_018_C09 −12.5 / −13.6; -32 PWD_IMP_B08_024_C09 corr_x −15.7 / −16.8, e10 +7.8 rolling; -158 e02 +9.4 / +7.0; -160 e02 +10.8 / +10.2; -162 PWD_IMP_B08_025_C09 corr_x +4.7 / +5.5 (kept: beyond 5 against its local level, a spike against the start of the thermal drift; the weakest kept label).
- One-measurement jumps without cause: -144 e10 corr_y +14.1 / +11.8; -157 PWD_RMP_B16_026_C09 corr_y +9.8 / +9.0.
- Bird in front of PWD_REF08_C09: -47 PWD_IMP_B08_025_C09 corr_y +21.4 / +20.1, almost all targets beyond 5.
- Thermal (corrected curves; the global level is close to the night level, the rolling one at 6 h): -19 PWD_IMP_B07_018_C09 corr_x −12.4 / −9.0; -24 PWD_IMP_B07_017_C09 −19.5 global, e01 +18.2 rolling; -26 PWD_IMP_B07_018_C09 −12.5 / −13.6; -28 PWD_IMP_B07_017_C09 −16.6 / −11.6; -33 e01 +21.9 / +23.2; -38 e01 +12.3 global, e06 +9.8 rolling; -42 PWD_IMP_B07_018_C09 −15.2 / −9.4; -46 PWD_IMP_B07_018_C09 corr_x and many others beyond 5 outside the bird hour; -50 PWD_IMP_B07_018_C09 −9.2 global, e06 +7.0 rolling. All keep the cycle on the corrected curves.
Rewritten: NIEDERHORN-155 (train at dawn, 06/10 12:00; PWD_RMP_B12_019_C09 corr_y +31.5): its parenthesis "the rejection of PWD_RMP_B12_019_C09 at the measurement before is labelled apart" pointed to NIEDERHORN-154, removed; dropped, the rest unchanged (it had no review field).
Kept with descriptions that also list rejected targets (trains, rain, plant): their substance is the movement measured above; the lists of rejections stay as context and the user's validation is kept.
"""
src = "- requalification: `instructions/analyst.md`, `labeling/labeling-rules.md` 1-3, `tools/GUIDE.md` (ranges), `work/requalify_input.jsonl`, this analysis, `report.md`, `tools/scripts/inventory.py` (distances, mm/px)\n"
for old, new in (("## 1. My labels\n", "## 1. My labels\n\n" + note),
("## 2. What I left without a label\n\n", "## 2. What I left without a label\n\n" + removed),
("## Difficulties\n", req + "## Difficulties\n"),
("## Sources read\n", "## Sources read\n" + src)):
assert t.count(old) == 1, old
t = t.replace(old, new)
open(p, "w").write(t)