From df3a7120d392fd494f72a7e9b4ddee754385f9b9 Mon Sep 17 00:00:00 2001 From: maxime Date: Wed, 18 Nov 2026 10:19:02 +0100 Subject: [PATCH] Families: Displacement is something that moved and never came back; Perturbation, new, what comes back without another family's cause, nothing to do (spec 2026-11-18-displacement-split-design) Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-11-18-displacement-split.md | 279 ++++++++++++++++++ .../2026-11-18-displacement-split-design.md | 99 +++++++ families.md | 16 +- labeling-rules.md | 5 +- scripts/client.py | 3 + scripts/validate.py | 2 +- 6 files changed, 394 insertions(+), 10 deletions(-) create mode 100644 docs/superpowers/plans/2026-11-18-displacement-split.md create mode 100644 docs/superpowers/specs/2026-11-18-displacement-split-design.md diff --git a/docs/superpowers/plans/2026-11-18-displacement-split.md b/docs/superpowers/plans/2026-11-18-displacement-split.md new file mode 100644 index 0000000..59a47fd --- /dev/null +++ b/docs/superpowers/plans/2026-11-18-displacement-split.md @@ -0,0 +1,279 @@ +# Displacement split Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** `Displacement` means something moved and never came back; a new family `Perturbation` takes what comes back without another family's cause, with nothing to do. + +**Architecture:** The knowledge (`families.md`, `labeling-rules.md`) and the key list (`validate.py`) change first, then the 36 labels and their analysis copies, then the consumers: the detectors (a jump back gets kind `Perturbation`), aimsight-app (a colour), the agents' instructions (the key list). + +**Tech Stack:** Python 3 standard library; React (Vite) for one line of aimsight-app; markdown. + +**Spec:** `docs/superpowers/specs/2026-11-18-displacement-split-design.md` + +## Global Constraints + +- **Before Task 1**: `git -C ~/aimsight-labeling status --short families.md labeling-rules.md reading-rules.md README.md CLAUDE.md scripts/` prints nothing. Another session (the cases work, `docs/superpowers/plans/2026-11-18-cases.md`) is editing these files; a commit by path here would carry its half-done work. If not empty: stop and tell the user. +- Commit by path, never `git add -A` (aimsight-app writes in `auto/` at the same time); each repository commits its own files; messages end with `Co-Authored-By: Claude Opus 5.5 `. +- `manual/` is never read nor written. +- An id never changes. Descriptions in English, in words, without numeric values (`validate.py` checks it). +- `review`: kept on a change of family only; removed on the 6 reworded labels (spec, "Open, for the user", point 1, approved). +- `aimsight-detectors/bench/` is never touched (it is the measure). +- The exact keys: `Displacement`, `Perturbation`. The variety `Sudden jump, back to normal` belongs to `Perturbation`; `Lasting shift`, `Jump, not back yet`, `Slow drift` to `Displacement`. + +## Review Focus + +1. aimsight-app writes a label in `auto/.jsonl` while the relabel runs: the app's write must not be lost (Task 2 re-reads each file right before writing it and changes only the target lines). +2. A label the plan rewords was edited since (its closing sentence no longer matches): the relabel must stop with the id, not skip it (Task 2, Step 1). +3. An analysis copy missing, or matched twice: stop with the id (Task 2, Step 1). +4. aimsight-app proposes a `Perturbation` label: `validate.check(label, proposed=True)` must accept it (Task 1, Step 1). +5. A detectors' event of variety `Sudden jump, back to normal` asks the client: `client.action("Perturbation", "Sudden jump, back to normal")` gives the variety's text, not the family's (Task 1, Step 1). + +--- + +### Task 1: The families, the rules, the key list + +**Files:** +- Modify: `families.md:10`, `:22-24`, `:31-33`, new boundary after `:33` +- Modify: `labeling-rules.md:63`, `:86` +- Modify: `scripts/validate.py:23` (`FAMILIES`), `self_check()` +- Modify: `scripts/client.py` (`__main__` asserts) +- Add: `docs/superpowers/specs/2026-11-18-displacement-split-design.md`, this plan + +**Interfaces:** +- Produces: `validate.FAMILIES` contains `"Perturbation"`; `client.advice()` has keys `Displacement`, `Perturbation` and the four varieties with the texts below. + +- [ ] **Step 1: Write the failing checks** + +In `scripts/client.py`, `__main__`, replace the `Displacement` assert and add: + +```python + assert action("Displacement", "Slow drift", None, t) == "Keep watching it." + assert action("Displacement", "Lasting shift", None, t) == "Check what moved: look at the images." + assert action("Perturbation", "Sudden jump, back to normal", None, t) == "Nothing to do: it came back; ignore these points." + assert action("Perturbation", "x", None, t) == "Nothing to do: the curves came back; ignore these points." +``` + +In `scripts/validate.py`, `self_check()`, before `print("ok")`: + +```python + label = {"start": "2026-11-04T00:00Z", "end": "2026-11-04T01:00Z", "family": "Perturbation", "description": "x"} + assert not any("unknown family" in e for e in check(label, proposed=True)) +``` + +- [ ] **Step 2: Run them, see them fail** + +Run: `python3 scripts/client.py; python3 scripts/validate.py --check` +Expected: `AssertionError` in both. + +- [ ] **Step 3: `families.md`** + +Line 10 becomes the two rows: + +``` +| `Displacement` | `displacement.py` (lasting shift, jump not back yet, slow drift) | something moved, by a jump or a drift, and has not come back by the end of the range: one or more targets, a reference, or the camera outside a thermal effect; the description says what moved | check what moved: the structure (look at the images, check on site) or the camera (check its mount) | +| `Perturbation` | `displacement.py` (jump back to normal) | the corrected curves move beyond their noise and come back, and no other family explains it: a jolt of the camera, a reference or the base shifted for a while, a single point, targets that wander and come back; the cause is not established or several factors mix | nothing to do: the curves came back; ignore these points | +``` + +Varieties, lines 22-24: + +``` +| `Sudden jump, back to normal` | nothing to do: it came back; ignore these points | +| `Lasting shift` | check what moved: look at the images | +| `Jump, not back yet` | check what moved: look at the images | +``` + +Boundaries: line 31, "Visibility, or Displacement without a cause." → "Visibility, or Perturbation without a cause; Displacement if the base stays changed."; line 32, "Without an established cause: Displacement." → "Without an established cause: Perturbation."; line 33, append "What does not follow it and comes back is Perturbation." After line 33, add: + +``` +- **Displacement or Perturbation**: decided by the return, not by the cause. What moved and has not come back by the end of the range (a target, a reference, the camera outside a thermal effect) is Displacement; what comes back, even after days, is Perturbation. A camera that moves and stays is Displacement even when the compensation absorbs it afterwards; the description says it is the camera. A partial return is not a return. +``` + +- [ ] **Step 4: `labeling-rules.md`** + +Line 63: add `` `Perturbation` `` after `` `Displacement` `` in the list of keys. Line 86 becomes the two bullets: + +``` +- **A Displacement says what moved, checked target by target on the raw position, and that it has not come back by the end of the range**: targets moved: "a movement of these targets is not excluded: to check (what to look at), monitor"; the camera moved and keeps its new position: "it is not a movement of the structure; the camera mount is to be checked"; the base stays changed, nothing moving in the image: "it is not a movement of the structure; to check with the operator"; the whole scene deforms while the camera moves: "movement of the structure not established: to monitor". [cross Displacement, NEBRASKA] +- **A Perturbation says what moved, that it came back and when, and concludes "nothing to do: points to ignore"**: its cause is stated in the three ways of section 6; "nothing to do" rests on the return seen, not on a cause. [the user, 2026-11-18] +``` + +- [ ] **Step 5: `scripts/validate.py:23`**: `FAMILIES` gains `"Perturbation"`. + +- [ ] **Step 6: Run the checks** + +Run: `python3 scripts/client.py && python3 scripts/validate.py --check && python3 scripts/validate.py | tail -1` +Expected: `ok`, `ok`, and `0 errors` on the labels' line. + +- [ ] **Step 7: Commit** + +```bash +git add families.md labeling-rules.md scripts/validate.py scripts/client.py docs/superpowers/specs/2026-11-18-displacement-split-design.md docs/superpowers/plans/2026-11-18-displacement-split.md +git commit -m "Families: Displacement is something that moved and never came back; Perturbation, new, what comes back without another family's cause, nothing to do (spec 2026-11-18-displacement-split-design)" +``` + +### Task 2: The 36 labels and the 6 rewordings + +**Files:** +- Modify: `auto/.jsonl` (ALASKA, FLORIDA, JUNGFRAU, LENZSPITZE, MICHIGAN, NEBRASKA, NIEDERHORN, NORDEND, WILDSTRUBEL, WYOMING) +- Modify: `auto/analyses//labels.jsonl`, the folder named by each label's `analysis` +- Create (not committed): `/relabel.py` + +**Interfaces:** +- Consumes: `Perturbation` known to `validate.py` (Task 1). + +- [ ] **Step 1: Write `relabel.py`** + +Data, from the spec: + +```python +PERTURBATION = """ALASKA-1 ALASKA-2 ALASKA-3 ALASKA-5 ALASKA-6 ALASKA-9 FLORIDA-18 FLORIDA-72 JUNGFRAU-110 JUNGFRAU-112 +LENZSPITZE-84 LENZSPITZE-98 LENZSPITZE-105 LENZSPITZE-106 LENZSPITZE-108 MICHIGAN-8 NEBRASKA-3 NIEDERHORN-126 +NIEDERHORN-144 NIEDERHORN-157 NORDEND-80 NORDEND-112 NORDEND-243 NORDEND-251 NORDEND-257 NORDEND-271 NORDEND-278 +NORDEND-279 NORDEND-310 WILDSTRUBEL-136 WILDSTRUBEL-153 WYOMING-9 WYOMING-11 WYOMING-13 WYOMING-16 WYOMING-17""".split() +REWORD = { # id: (closing sentence now, closing sentence after) + "NEBRASKA-3": ("For the customer: the camera straightens and the scene deforms with it: a movement of the structure is not established; the camera tilt is to be monitored, and restoring a third reference will make the curves reliable again.", + "For the customer: the camera straightens and the scene deforms with it, then both come back on 17 October: nothing to do over these days; restoring a third reference will make the curves reliable again."), + "NORDEND-257": ("Cause not established: a movement of PR07 itself is not excluded, but it comes back by itself; to monitor.", + "Cause not established; the curve comes back by itself the same morning: nothing to do, points to ignore."), + "NORDEND-310": ("Cause not established. It is not a movement of the structure; the camera is to be monitored.", + "Cause not established. The camera comes back after about an hour: it is not a movement of the structure, nothing to do; points to ignore."), + "FLORIDA-18": ("Small, without established movement of the structure: to be monitored.", + "The shift fades on the morning of the 15th (return labelled separately): nothing to do, points to ignore."), + "WILDSTRUBEL-136": ("This is not an established movement of the road: do not read these days as a movement, to be checked with the following days.", + "The shift fades on 04/10 (return labelled separately): nothing to do, do not read these days as a movement."), + "NORDEND-250": ("Cause not established: this is not a movement of the structure, the camera moved; point to ignore, the camera is to be watched if it happens again.", + "Cause not established: this is not a movement of the structure; the camera moved and keeps its new position: its mount is to be checked."), +} +``` + +Behaviour: for each station file, read it, and for each line whose `id` is in `PERTURBATION` or `REWORD`: its `family` must be `Displacement` (else stop with the id); set `family` to `Perturbation` if in `PERTURBATION`; if in `REWORD`, the description must end with the "now" sentence (else stop with the id), replace it, delete `review`. In `auto/analyses//labels.jsonl`, the copy is the one line with the label's `start`, `end` and old `description` (zero or two: stop with the id); apply the same change (and delete `review` if the copy has one). Write each file right after reading it, every other line byte for byte; a changed line is `json.dumps(d, ensure_ascii=False)` (the files' format: key order kept, no `\u` escape). Print the count of labels and copies changed. + +- [ ] **Step 2: Run it** + +Run (when the user is not validating in aimsight-app): `python3 /relabel.py` +Expected: `37 labels, 37 copies` (the 36 family changes, 5 of them also reworded, plus NORDEND-250, reworded only). + +- [ ] **Step 3: Check** + +Run: + +```bash +python3 - <<'EOF' +import json, glob +lab = {d["id"]: d for f in glob.glob("auto/*.jsonl") for d in map(json.loads, filter(str.strip, open(f)))} +import sys; sys.path.insert(0, ""); from relabel import PERTURBATION, REWORD +assert all(lab[i]["family"] == "Perturbation" for i in PERTURBATION) +assert all("review" not in lab[i] and lab[i]["description"].endswith(REWORD[i][1]) for i in REWORD) +assert all("review" in lab[i] for i in set(PERTURBATION) - set(REWORD)) +assert lab["NORDEND-250"]["family"] == "Displacement" +print("ok") +EOF +python3 scripts/validate.py | tail -2 +git diff --numstat -- auto/ | awk '{a+=$1} END {print a}' +``` + +Expected: `ok`; `0 errors`; the diff counts the 74 changed lines plus whatever aimsight-app had already left uncommitted in those files (compare with `git diff --numstat` taken before Step 2). + +- [ ] **Step 4: Commit** + +```bash +git add auto/ALASKA.jsonl auto/FLORIDA.jsonl auto/JUNGFRAU.jsonl auto/LENZSPITZE.jsonl auto/MICHIGAN.jsonl auto/NEBRASKA.jsonl auto/NIEDERHORN.jsonl auto/NORDEND.jsonl auto/WILDSTRUBEL.jsonl auto/WYOMING.jsonl auto/analyses/*/labels.jsonl +git commit -m "auto: 36 Displacement labels that came back requalified Perturbation (review kept, a change of family the user asked for); 6 closing sentences reworded to their family, review removed (NEBRASKA-3, NORDEND-257, NORDEND-310, FLORIDA-18, WILDSTRUBEL-136, NORDEND-250)" +``` + +### Task 3: The cases of family Displacement + +**Files:** +- Modify: `auto/cases/.json`, only if `auto/cases/` exists + +- [ ] **Step 1: List them** + +Run: `python3 scripts/cases.py --list | grep "| Displacement |"` +If `auto/cases/` does not exist yet: skip this task and say so; the cases build reads the labels, which carry the new families. + +- [ ] **Step 2: Requalify** + +A case whose signature is something that comes back (the seed's "the base changes without a reference lost" and "targets move slowly, alone, then come back") gets `"family": "Perturbation"` and loses `review` if it has one (the cases rule: a change of family removes it). The others stay. + +- [ ] **Step 3: Check and commit** + +Run: `python3 scripts/validate.py | tail -1 && python3 scripts/cases.py --index` +Expected: `N cases: 0 errors`, the index updated. + +```bash +git add auto/cases/ +git commit -m "cases: the Displacement cases that come back are Perturbation" +``` + +### Task 4: The detectors + +**Files (repository `~/aimsight-detectors`):** +- Modify: `scripts/displacement.py` (`jumps()`, the docstring point 2) +- Modify: `scripts/categories.py` (`CATEGORIES`, docstring line 8) +- Modify: `README.md:7`, `:15`; `CLAUDE.md` ("the six families" → "the seven families") + +**Interfaces:** +- Produces: an episode of state "back" has `kind == "Perturbation"`, `label == "Sudden jump, back to normal"`; `categories.CATEGORIES["Perturbation"] == {"ignore": True, "process": "none"}`, placed right after `"Displacement"`. + +- [ ] **Step 1: The benchmark before** + +Run: `python3 bench/regression.py --labels ~/aimsight-labeling/auto --data ~/aimsight-data/data > /before.txt` +Expected: no `skipped` line; no `Perturbation` row (no detector gives it yet). + +- [ ] **Step 2: Change** + +`jumps()`: the episode's kind is `"Perturbation"` when `state == "back"`, else `"Displacement"`. Docstring point 2: "greyed, its cause not established here" → "a Perturbation, greyed". `categories.py` docstring line 8: "otherwise it stays Displacement, its cause to be checked on the images" → "otherwise it is a Perturbation: nothing to do". `README.md`: "six families" → "seven families"; the `displacement.py` row: "Displacement (Lasting shift; Jump, not back yet; Slow drift), Perturbation (Sudden jump, back to normal)". `CLAUDE.md`: "six families" → "seven families". + +- [ ] **Step 3: The benchmark after** + +Run: `python3 bench/regression.py --labels ~/aimsight-labeling/auto --data ~/aimsight-data/data > /after.txt && python3 bench/compare.py /before.txt /after.txt` +Expected: no `skipped` line; a `Perturbation` row with "found" above zero; the `Displacement` "correct" rises. A regression that compare.py reports is told to the user with the labels concerned (ids), per this repository's rule, not hidden. + +- [ ] **Step 4: Commit** + +```bash +git -C ~/aimsight-detectors add scripts/displacement.py scripts/categories.py README.md CLAUDE.md +git -C ~/aimsight-detectors commit -m "Perturbation, the seventh family: a jump back to normal is a Perturbation (nothing to do), Displacement keeps the lasting shift, the jump not back yet and the slow drift (labeling spec 2026-11-18-displacement-split-design)" +``` + +### Task 5: aimsight-app and the agents' instructions + +**Files:** +- Modify: `~/aimsight-app/app/web/src/pages/Data.jsx:16` (`FAMILIES`), `:19` (comment) +- Modify: `~/aimsight-agents/agents/analyst.md:56` +- Modify: `~/Support-Agent/.claude/skills/support/rapport.md:20` + +- [ ] **Step 1: Data.jsx** + +`FAMILIES` gains `"Perturbation": "#9fd8d0"` right after `"Displacement"`; the comment "one of the six families" → "one of the seven families". + +- [ ] **Step 2: Build and restart** + +Run: `cd ~/aimsight-app/app/web && npm run build && grep -l Perturbation dist/assets/*.js && systemctl --user restart aimsight-app` +Expected: one bundle file named; the service active (`systemctl --user is-active aimsight-app` → `active`). + +- [ ] **Step 3: The instructions** + +`analyst.md:56`: add `` `Perturbation` `` after `` `Displacement` `` in the list of keys. `rapport.md:20`: `` `Displacement` (à vérifier) `` → `` `Displacement` (quelque chose a bougé : à vérifier) ``, and `` `Perturbation` (rien à faire) `` added last in the order. + +- [ ] **Step 4: Commit, one per repository** + +```bash +git -C ~/aimsight-app add app/web/src/pages/Data.jsx +git -C ~/aimsight-app commit -m "Data: the family Perturbation and its colour" +git -C ~/aimsight-agents add agents/analyst.md +git -C ~/aimsight-agents commit -m "analyst: the family Perturbation" +git -C ~/Support-Agent add .claude/skills/support/rapport.md +git -C ~/Support-Agent commit -m "support: la famille Perturbation" +``` + +### Task 6: The whole + +- [ ] **Step 1: Run every check** + +Run: `cd ~/aimsight-labeling && python3 scripts/validate.py | tail -2 && python3 scripts/validate.py --check && python3 scripts/client.py && python3 scripts/cases.py --check` +Expected: `0 errors` (labels and cases), `ok`, `ok`, `ok`. + +- [ ] **Step 2: Look at it in the app** + +Open a range with a Perturbation label (ALASKA, 2026-11-05) on the Data page: the label has the new colour, the Category menu offers `Perturbation`, a detectors' jump back shows as Perturbation, greyed. diff --git a/docs/superpowers/specs/2026-11-18-displacement-split-design.md b/docs/superpowers/specs/2026-11-18-displacement-split-design.md new file mode 100644 index 0000000..7f37823 --- /dev/null +++ b/docs/superpowers/specs/2026-11-18-displacement-split-design.md @@ -0,0 +1,99 @@ +# Displacement split: Displacement and Perturbation + +Status: draft, to be approved by the user. Nothing applied yet (`families.md` is read live by aimsight-app). + +## Why + +`Displacement` today is the bin of D-074 ("À vérifier", 2026-10-27): "a lasting jump, a slow drift, and anything that changes the corrected curves without an established cause". Of its 63 labels in `auto/`, 50 say in their own text that the structure did not move, or that a movement is not established. The user: a displacement is a movement, a real one; families must stay simple. + +## Goals + +- `Displacement` = something moved (a jump or a drift) and never came back. +- One new family, `Perturbation`, for what changes the corrected curves, comes back, and no other family explains: cause unclear or several factors; nothing to do. +- Simple families: two keys where there was one, no family per mechanism. + +Non-goals: changing the other families; reworking `Thermal displacement` (its camera labels keep it); touching `manual/`. + +Anti-goal: a `Perturbation` that hides a movement. What has not come back stays `Displacement`. + +## Approaches + +- **A. One catch-all** (`Unexplained`) for the 50 non-movement labels. Dropped: the old bin under a new name. +- **B. By what moves** (`Displacement` targets, `Camera movement`, `Artifact` base and spikes). Dropped: three keys; the user wants simple families. +- **C. By permanence (chosen by the user).** A camera movement is usually thermal: `Thermal displacement`, as today. Otherwise, the family follows whether the movement is permanent: it stays, `Displacement`; it comes back, `Perturbation`. The same rule for a target, a reference or the camera. It is also the line the detector already draws (`displacement.py`: "back" against "lasting" / "recent"). + +## The families + +Proposed rows for `families.md` (main table): + +| Key | Detector | What happens | For the client | +|---|---|---|---| +| `Displacement` | `displacement.py` (lasting shift, jump not back yet, slow drift) | something moved, by a jump or a drift, and has not come back by the end of the range: one or more targets, a reference, or the camera outside a thermal effect; the description says what moved | check what moved: the structure (look at the images, check on site) or the camera (check its mount) | +| `Perturbation` | `displacement.py` (jump back to normal) | the corrected curves move beyond their noise and come back, and no other family explains it: a jolt of the camera, a reference or the base shifted for a while, a single point, targets that wander and come back; the cause is not established or several factors mix | nothing to do: the curves came back; ignore these points | + +Varieties: + +| Variety | Family | For the client | +|---|---|---| +| `Sudden jump, back to normal` | Perturbation (was Displacement) | nothing to do: it came back; ignore these points | +| `Lasting shift` | Displacement | check what moved: look at the images | +| `Jump, not back yet` | Displacement | check what moved: look at the images | +| `Slow drift` | Displacement | keep watching it | + +Boundaries, in `families.md`: + +- **New, Displacement or Perturbation**: decided by the return, not by the cause. What moved and has not come back by the end of the range (a target, a reference, the camera outside a thermal effect) is Displacement; what comes back, even after days, is Perturbation. A camera that moves and stays is Displacement even when the compensation absorbs it afterwards; the description says it is the camera. A partial return is not a return. +- **Thermal displacement or Displacement** (line 33): unchanged; add that what does not follow the temperature and comes back is Perturbation. +- **A jump then the return** (line 32): "Without an established cause: Displacement" becomes "Perturbation". +- **A hindered reference** (line 31): "Visibility, or Displacement without a cause" becomes "Visibility, or Perturbation without a cause; Displacement if the base stays changed". +- **Target issue or Displacement** (line 37): unchanged. + +Phenomena: since the cases work of 2026-11-18 (`2026-11-18-cases-design.md`), they are the cases of `auto/cases/`, no longer a section of `families.md`. The cases of family `Displacement` that come back move to `Perturbation`; the rest is as first drafted here: the Displacement section keeps the movements that stay ("a real movement of the structure", "a target or a small group jumps and stays shifted", "a step taken during a measurement outage", "the camera tilts little by little", "a step of the camera that the compensation does not erase", when it stays); a new Perturbation section takes "targets move slowly, alone, then come back", "the base changes without a reference lost" (with its return), and adds "a jolt of the camera, back at the next measurement", "a reference measured shifted alone, then back", "a single point of one or a few targets, back at once". + +`labeling-rules.md`: line 63, the list of keys gains `Perturbation`; line 86 is rewritten: a Displacement says what moved and that it has not come back, and concludes "the targets moved: to check on site, monitor" or "the camera moved and stays: check its mount"; a Perturbation says what moved, that it came back and when, and concludes "nothing to do: points to ignore". A Perturbation states its cause in the three ways of line 99, like any label; "nothing to do" rests on the return seen, not on a cause (line 102). + +## The 63 labels + +By the rule above (return judged on the whole range of the analysis): + +**Displacement, 27** (family unchanged, description unchanged but NORDEND-250): +- targets that moved and stay: CLARIDEN-9, CLARIDEN-23, FLORIDA-46, NEBRASKA-5, NIEDERHORN-131, NIEDERHORN-167, NIEDERHORN-168, NORDEND-94, WILDSTRUBEL-89, WILDSTRUBEL-120, WILDSTRUBEL-173, WILDSTRUBEL-183; +- camera that moved and stays: BREITHORN-43, BREITHORN-39, BREITHORN-44, CLARIDEN-74, MICHIGAN-19, NEBRASKA-1, NEBRASKA-6, NORDEND-250, NORDEND-309, SNCF01-7, SNCF01-42, SNCF01-102, SNCF01-132, WILDSTRUBEL-200; +- base that stays changed: NORDEND-276. + +**Perturbation, 36** (family changes): +- base or a reference shifted, then back: ALASKA-1, ALASKA-3, ALASKA-5, ALASKA-6, ALASKA-9, FLORIDA-18, FLORIDA-72, JUNGFRAU-110, JUNGFRAU-112, LENZSPITZE-105, LENZSPITZE-106, LENZSPITZE-108, MICHIGAN-8, NIEDERHORN-126, NORDEND-112, NORDEND-279, WILDSTRUBEL-136, WILDSTRUBEL-153, WYOMING-9, WYOMING-13, WYOMING-17; +- camera jolt or shift, then back: LENZSPITZE-84, NEBRASKA-3, NORDEND-80, NORDEND-243, NORDEND-278, NORDEND-310, WYOMING-11, WYOMING-16; +- single points, back at once: ALASKA-2, LENZSPITZE-98, NIEDERHORN-144, NIEDERHORN-157, NORDEND-251, NORDEND-271; +- a target that moved, then came back: NORDEND-257. + +Calls made by the drafter (to check): +- A partial return is not a return: BREITHORN-39 (inside the camera drift of BREITHORN-43 and -44), NEBRASKA-5 (COLJ19 comes back only in part), NIEDERHORN-131 (three targets come back in part, two stay): Displacement. +- NORDEND-94: its text says the curves of PR17 came back neither to their level before the jump nor to the plateau, to the end of the batch: Displacement (the first draft had it as Perturbation). +- NORDEND-250: the camera keeps its step, the curves come back at the next measurement: Displacement, following "the permanent movement of the camera". +- NORDEND-276: the base stays changed to the end of the range, the targets do not move in the image: Displacement (something stays moved, possibly a reference). +- WILDSTRUBEL-120 mixes a return (PB_5) and a lasting shift (PA3_2): Displacement for the shift that stays. +- No Displacement label becomes Thermal displacement: none comes back with the housing temperature (families.md, line 33). + +Descriptions to reword, their conclusion contradicting their new family: NEBRASKA-3, NORDEND-257, NORDEND-310, FLORIDA-18, WILDSTRUBEL-136 (Perturbation, but "to be monitored", "to be checked"), and NORDEND-250 (Displacement of the camera, but "point to ignore"). Only the closing sentence changes; the plan gives each. + +The copies in `auto/analyses/*/labels.jsonl` follow the same change. + +## What it touches + +- **aimsight-labeling**: `families.md`, `labeling-rules.md`, `scripts/validate.py` (`FAMILIES`), `scripts/client.py` (its check), the 38 labels in `auto/` and their copies in the analyses; `ids.py` untouched, ids never change. +- **aimsight-detectors**: `scripts/displacement.py`, a "back" episode gets kind `Perturbation` (one line); `scripts/categories.py`, `CATEGORIES` gains `Perturbation` (greyed, nothing to do), and `host()` keeps joining a jump back to the Visibility or Obstruction around it. +- **aimsight-app**: `app/web/src/pages/Data.jsx`, `FAMILIES` gains `Perturbation` and its colour (the Category menu and the timeline follow). +- **The agents' instructions**: `aimsight-agents/agents/analyst.md` (the list of family keys); `Support-Agent/.claude/skills/support/rapport.md` (the order of the families for the client). The `etiqueteur` and `detectors-critic` of Support-Agent still speak the old format (`familles.md`, `categorie`): left as they are. +- **The detectors' docs**: "the six families" in `aimsight-detectors/CLAUDE.md` and `README.md`, and in the comment of `Data.jsx`, become seven. +- **aimsight-optimizer**: nothing in code; its benchmark reads `auto/`, so the Displacement scores change meaning from the relabel on. +- **manual/**: the expert's labels keep the old meaning of Displacement; to tell the expert, not to change. + +## Open, for the user + +1. **The review dates.** All 63 labels carry a `review` (2026-11-17, 2026-11-18), and any change removes it (`README.md`). Approved: keep `review` on a change of family made at the user's request, remove it on the 6 reworded descriptions. +2. **The return window of the detector.** The labels judge the return on the whole range; the detector on 24 hours (`BACK_H`): a shift back after two to six days (ALASKA-9, FLORIDA-18, NEBRASKA-3, WILDSTRUBEL-136, WILDSTRUBEL-153) is Perturbation in the labels and Displacement for the detector. Approved: leave `BACK_H` to the optimizer, which learns from the labels. + +## Left to the builder + +The exact wording of the phenomena lines; the colour of Perturbation in aimsight-app; the order of the commits across the three repositories (labeling first, then detectors and app, the same day: aimsight-app rejects a label whose family `validate.py` does not know). diff --git a/families.md b/families.md index 81ff7d7..507fb35 100644 --- a/families.md +++ b/families.md @@ -7,7 +7,8 @@ One label, one anomaly, one family (D-109 to D-111, D-119). An anomaly is what s | `Visibility` | `visibility.py` | things are harder to see: a diffuse nuisance between the camera and the scene. The image is faint or blurred (rain, fog, condensation, light, dirty lens: the spots are fainter or blurrier), or it shakes (turbulence: hot air, above a road or in the sun, makes the spots shake, a fast noise on the curves). The description says which is seen | do not trust the curves over the range; it is not a movement | | `Obstruction` | `visibility.py` (brief), `target_issue.py` (target obscured) | an object hides one or a few targets, for a moment or for a long time (train, vehicle, plant, snow on the target) | ignore the curve of the hidden targets over the range | | `Thermal displacement` | `thermal_displacement.py` | a component (the structure, the camera and its mount) moves with temperature; it is labeled only when the corrected curves keep it, not when the camera compensates it (`labeling-rules.md`, section 1); the direction does not matter | normal, nothing to do | -| `Displacement` | `displacement.py` | a lasting jump, a slow drift, and anything that changes the corrected curves without an established cause (rule D-074: do not alarm the client, hide nothing) | to check: look at the images, monitor | +| `Displacement` | `displacement.py` (lasting shift, jump not back yet, slow drift) | something moved, by a jump or a drift, and has not come back by the end of the range: one or more targets, a reference, or the camera outside a thermal effect; the description says what moved | check what moved: the structure (look at the images, check on site) or the camera (check its mount) | +| `Perturbation` | `displacement.py` (jump back to normal) | the corrected curves move beyond their noise and come back, and no other family explains it: a jolt of the camera, a reference or the base shifted for a while, a single point, targets that wander and come back; the cause is not established or several factors mix | nothing to do: the curves came back; ignore these points | | `Target issue` | `target_issue.py` | a constantly problematic target: rejected almost always, badly oriented, badly installed, badly configured, dirty | act on the target; ignore its curve | | `No data` | `no_data.py` | no measurement for any sensor, all curves missing (acquisition, power, transmission); the cause, if the metrics or the images show it, otherwise not established | the curves are missing over the range; the cause says whether to act | @@ -19,18 +20,19 @@ For the client, some varieties of a family (the `label` of the detectors' episod | `Heat shimmer` | nothing to do: the small fast wiggles are not movement | | `Camera thermal movement` | nothing to do: this is not a movement of the structure | | `Brief obstruction` | ignore this single point on the curves | -| `Sudden jump, back to normal` | look at the images: the cause is not established | -| `Lasting shift` | check whether the structure moved: look at the images | -| `Jump, not back yet` | check whether the structure moved: look at the images | +| `Sudden jump, back to normal` | nothing to do: it came back; ignore these points | +| `Lasting shift` | check what moved: look at the images | +| `Jump, not back yet` | check what moved: look at the images | | `Slow drift` | keep watching it | ## The boundaries - **Visibility or Obstruction**: the nuisance is judged by its extent, not by the light. The whole scene hindered (rain, fog, underexposed image, milky dawn, sun in the field) is Visibility; a few targets hidden by an object, the rest intact, are Obstruction, even if the object is lit (machine, headlights, light of a parked vehicle). A spider web goes by what it does: a thread that hides a target, Obstruction; the web that veils the image, Visibility. - **Obstruction, whatever the duration**: an obstacle that stays (parked vehicle, machine, plant, animal or insects against the lens, mark on the glass in front of a single spot) is Obstruction. -- **A hindered reference** (phenomena: "A reference dropped", "Two references lost"): the family of its cause. A passage in front of a reference that makes the base jump is Obstruction when the object is seen, or deduced from a passage signature; a reference right at its threshold, with nothing in front of it, is not a passage: Visibility, or Displacement without a cause. The description says it is a reference, and that the targets move together because of it. -- **A jump then the return**: the family of its cause, seen on the images. A passing vehicle, and its reflection, are Obstruction (a reflection on the roadway lasting only one measurement is a vehicle: read the measurement before and the one after); the reflection of the sun on the fixed scene is Visibility. Without an established cause: Displacement. -- **Thermal displacement or Displacement**: what follows the temperature of the housing **and comes back with it** (same position at equal temperature), even over several days, or whose thermal cause the expert gives, is Thermal displacement; what follows it only as a trend, without coming back at equal temperature, and what the temperature does not explain, are Displacement. +- **A hindered reference** (phenomena: "A reference dropped", "Two references lost"): the family of its cause. A passage in front of a reference that makes the base jump is Obstruction when the object is seen, or deduced from a passage signature; a reference right at its threshold, with nothing in front of it, is not a passage: Visibility, or Perturbation without a cause; Displacement if the base stays changed. The description says it is a reference, and that the targets move together because of it. +- **A jump then the return**: the family of its cause, seen on the images. A passing vehicle, and its reflection, are Obstruction (a reflection on the roadway lasting only one measurement is a vehicle: read the measurement before and the one after); the reflection of the sun on the fixed scene is Visibility. Without an established cause: Perturbation. +- **Thermal displacement or Displacement**: what follows the temperature of the housing **and comes back with it** (same position at equal temperature), even over several days, or whose thermal cause the expert gives, is Thermal displacement; what follows it only as a trend, without coming back at equal temperature, and what the temperature does not explain, are Displacement. What does not follow it and comes back is Perturbation. +- **Displacement or Perturbation**: decided by the return, not by the cause. What moved and has not come back by the end of the range (a target, a reference, the camera outside a thermal effect) is Displacement; what comes back, even after days, is Perturbation. A camera that moves and stays is Displacement even when the compensation absorbs it afterwards; the description says it is the camera. A partial return is not a return. - **Thermal displacement or Visibility**: curves that follow the scale and rotation of the camera with the housing temperature, the `error_value` at its level or rising only a little, nothing on the images, several targets moving apart in opposite directions through the rest of the compensation, are Thermal displacement. References moving apart from each other and from the targets, an `error_value` rising on most sensors, an episode with sharp edges, or a nuisance seen on the images, are Visibility, even at the temperature trough. The two can coexist: two labels. A daytime agitation, the pose shaking without the `error_value` rising, is not turbulence. - **A gap**: curves missing because targets are rejected are not labeled for themselves, the station filtered them; the episode that rejects them is labeled only if it moves the other curves beyond the noise, in the family of its cause (Visibility, Obstruction); a target rejected almost always is a Target issue; all curves missing, no measurement: No data. The cause of a No data can be outside the station: a gap at the same minutes on several stations is a common cause (transmission or reception), not an outage. The station is running if a synchronization falls at the missing instant, if the image of that instant exists, or if `DB_not_uploaded` rises then falls. - **Target issue or episode of another family**: a Target issue requires rejections specific to the target (outside the episodes shared with its neighbors) almost every day of the range, also on covered or rainy days, under the station's normal regime (the floodlight at night, daylight). A reference rejected like this every day is one, even if its entries and exits do not make the targets jump: the compensation loses it every day. Rejections that come only with a condition seen on the images (sun, rain, night of poor visibility) are not a Target issue: missing points, labeled only when the curves move beyond the noise, in the family of that condition. An `error_value` right at the threshold, with few own rejections, is not a Target issue: first count the own rejections of each target and each reference, day by day. diff --git a/labeling-rules.md b/labeling-rules.md index 9732d66..6e96861 100644 --- a/labeling-rules.md +++ b/labeling-rules.md @@ -60,7 +60,7 @@ Sources in brackets: [instruction] the section "Les étiquettes" of `.claude/age ## 4. The family -- **A single family per label, its key in `families.md`, following its boundaries**: `Visibility`, `Obstruction`, `Thermal displacement`, `Displacement`, `Target issue`, `No data`; no other key, nor old names. [instruction, MICHIGAN, WILDSTRUBEL] +- **A single family per label, its key in `families.md`, following its boundaries**: `Visibility`, `Obstruction`, `Thermal displacement`, `Displacement`, `Perturbation`, `Target issue`, `No data`; no other key, nor old names. [instruction, MICHIGAN, WILDSTRUBEL] - **We decide by the signature, not by coincidence**: a temperature drop at the same moment does not make a thermal, an `error_value` that moves a little does not make a Visibility. [cross Thermal displacement, cross Visibility] - **A same target keeps the same family from one batch to the next, and a same phenomenon on neighboring stations the same family**: otherwise the defect of a target or a site reads as two causes. [NORDEND, consistency] - **The sun at sunrise in the field and the lit background behind a spot are the same nuisance**: same family, same way of stating the cause. [cross Visibility] @@ -83,7 +83,8 @@ Sources in brackets: [instruction] the section "Les étiquettes" of `.claude/age - **The agitation from one measurement to the next is written in the thermal label**: when the pose step exceeds several times that of the night and the `error_value` does not rise; it does not make a Visibility. [finishing niederhorn] - **A thermal movement partly compensated is stated as such**: the description says the pose moves, what the compensation absorbs and what remains on the curves. [cross Thermal displacement] - **A touched reference is stated, with what remains**: the targets move together; two remaining references, "a very bad sign"; none, "the points make no sense". [cross Obstruction] -- **A Displacement says what moves in the image, checked target by target on the raw position, and concludes accordingly**: the camera moves and the targets do not: "it is not a movement of the structure; the camera is to be monitored"; targets move alone: "a movement of these targets is not excluded: to check (what to look at), monitor"; nothing moves in the image, the base or the configuration changes: "it is not a movement of the structure; points to ignore if there is a return, to check with the operator otherwise"; the whole scene deforms while the camera moves: "movement of the structure not established: to monitor". [cross Displacement, NEBRASKA] +- **A Displacement says what moved, checked target by target on the raw position, and that it has not come back by the end of the range**: targets moved: "a movement of these targets is not excluded: to check (what to look at), monitor"; the camera moved and keeps its new position: "it is not a movement of the structure; the camera mount is to be checked"; the base stays changed, nothing moving in the image: "it is not a movement of the structure; to check with the operator"; the whole scene deforms while the camera moves: "movement of the structure not established: to monitor". [cross Displacement, NEBRASKA] +- **A Perturbation says what moved, that it came back and when, and concludes "nothing to do: points to ignore"**: its cause is stated in the three ways of section 6; "nothing to do" rests on the return seen, not on a cause. [the user, 2026-11-18] - **A drift of several days says whether it follows the housing temperature and whether it comes back with it, at equal temperature**: otherwise, "the temperature does not explain it alone, link not established". [cross Displacement] - **A lasting jump states its step**: its duration, its return or not. [cross Displacement] - **Each label of a sequence (outward, return, resumption) names the other**: "return of the drift of…, labeled apart", "without return until the end of the batch". [cross Displacement] diff --git a/scripts/client.py b/scripts/client.py index 3d8c94f..5ddcbf9 100644 --- a/scripts/client.py +++ b/scripts/client.py @@ -37,6 +37,9 @@ if __name__ == "__main__": assert action("Visibility", "Poor visibility", "severe", t) == "Do not trust the curves over the range; it is not a movement.", t assert action("Visibility", "Poor visibility", "reduced", t).startswith("Use the data over the range with care") assert action("Displacement", "Slow drift", None, t) == "Keep watching it." + assert action("Displacement", "Lasting shift", None, t) == "Check what moved: look at the images." + assert action("Perturbation", "Sudden jump, back to normal", None, t) == "Nothing to do: it came back; ignore these points." + assert action("Perturbation", "x", None, t) == "Nothing to do: the curves came back; ignore these points." assert action("Unknown", "x", None, t) == "" assert message("A moved; nothing to do. The jump of 2 targets in this period has the same cause.") \ == "A moved. The jump of 2 targets in this period has the same cause." diff --git a/scripts/validate.py b/scripts/validate.py index 24ae7ce..508f9e8 100644 --- a/scripts/validate.py +++ b/scripts/validate.py @@ -20,7 +20,7 @@ import sys AUTO = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "auto") # ~/aimsight-labeling/auto, or through the labeling/ link -FAMILIES = {"Visibility", "Obstruction", "Thermal displacement", "Displacement", "Target issue", "No data"} +FAMILIES = {"Visibility", "Obstruction", "Thermal displacement", "Displacement", "Perturbation", "Target issue", "No data"} DATE = re.compile(r"^\d{4}-\d\d-\d\dT\d\d:\d\dZ$") # A decimal comma (names such as F4.2 and dates such as 05.10 carry a dot), or a number followed by a unit. NUMBER = re.compile(r"(?