Families: Displacement is something that moved and never came back; Perturbation, new, what comes back without another family's cause, nothing to do (spec 2026-11-18-displacement-split-design)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-18 10:19:02 +01:00
1 parent c89c1b9976
commit df3a7120d3
6 files changed
+394 -10

No files matched your search

@@ -0,0 +1,279 @@
# Displacement split Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** `Displacement` means something moved and never came back; a new family `Perturbation` takes what comes back without another family's cause, with nothing to do.
**Architecture:** The knowledge (`families.md`, `labeling-rules.md`) and the key list (`validate.py`) change first, then the 36 labels and their analysis copies, then the consumers: the detectors (a jump back gets kind `Perturbation`), aimsight-app (a colour), the agents' instructions (the key list).
**Tech Stack:** Python 3 standard library; React (Vite) for one line of aimsight-app; markdown.
**Spec:** `docs/superpowers/specs/2026-11-18-displacement-split-design.md`
## Global Constraints
- **Before Task 1**: `git -C ~/aimsight-labeling status --short families.md labeling-rules.md reading-rules.md README.md CLAUDE.md scripts/` prints nothing. Another session (the cases work, `docs/superpowers/plans/2026-11-18-cases.md`) is editing these files; a commit by path here would carry its half-done work. If not empty: stop and tell the user.
- Commit by path, never `git add -A` (aimsight-app writes in `auto/` at the same time); each repository commits its own files; messages end with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
- `manual/` is never read nor written.
- An id never changes. Descriptions in English, in words, without numeric values (`validate.py` checks it).
- `review`: kept on a change of family only; removed on the 6 reworded labels (spec, "Open, for the user", point 1, approved).
- `aimsight-detectors/bench/` is never touched (it is the measure).
- The exact keys: `Displacement`, `Perturbation`. The variety `Sudden jump, back to normal` belongs to `Perturbation`; `Lasting shift`, `Jump, not back yet`, `Slow drift` to `Displacement`.
## Review Focus
1. aimsight-app writes a label in `auto/<STATION>.jsonl` while the relabel runs: the app's write must not be lost (Task 2 re-reads each file right before writing it and changes only the target lines).
2. A label the plan rewords was edited since (its closing sentence no longer matches): the relabel must stop with the id, not skip it (Task 2, Step 1).
3. An analysis copy missing, or matched twice: stop with the id (Task 2, Step 1).
4. aimsight-app proposes a `Perturbation` label: `validate.check(label, proposed=True)` must accept it (Task 1, Step 1).
5. A detectors' event of variety `Sudden jump, back to normal` asks the client: `client.action("Perturbation", "Sudden jump, back to normal")` gives the variety's text, not the family's (Task 1, Step 1).
---
### Task 1: The families, the rules, the key list
**Files:**
- Modify: `families.md:10`, `:22-24`, `:31-33`, new boundary after `:33`
- Modify: `labeling-rules.md:63`, `:86`
- Modify: `scripts/validate.py:23` (`FAMILIES`), `self_check()`
- Modify: `scripts/client.py` (`__main__` asserts)
- Add: `docs/superpowers/specs/2026-11-18-displacement-split-design.md`, this plan
**Interfaces:**
- Produces: `validate.FAMILIES` contains `"Perturbation"`; `client.advice()` has keys `Displacement`, `Perturbation` and the four varieties with the texts below.
- [ ] **Step 1: Write the failing checks**
In `scripts/client.py`, `__main__`, replace the `Displacement` assert and add:
```python
assert action("Displacement", "Slow drift", None, t) == "Keep watching it."
assert action("Displacement", "Lasting shift", None, t) == "Check what moved: look at the images."
assert action("Perturbation", "Sudden jump, back to normal", None, t) == "Nothing to do: it came back; ignore these points."
assert action("Perturbation", "x", None, t) == "Nothing to do: the curves came back; ignore these points."
```
In `scripts/validate.py`, `self_check()`, before `print("ok")`:
```python
label = {"start": "2026-11-04T00:00Z", "end": "2026-11-04T01:00Z", "family": "Perturbation", "description": "x"}
assert not any("unknown family" in e for e in check(label, proposed=True))
```
- [ ] **Step 2: Run them, see them fail**
Run: `python3 scripts/client.py; python3 scripts/validate.py --check`
Expected: `AssertionError` in both.
- [ ] **Step 3: `families.md`**
Line 10 becomes the two rows:
```
| `Displacement` | `displacement.py` (lasting shift, jump not back yet, slow drift) | something moved, by a jump or a drift, and has not come back by the end of the range: one or more targets, a reference, or the camera outside a thermal effect; the description says what moved | check what moved: the structure (look at the images, check on site) or the camera (check its mount) |
| `Perturbation` | `displacement.py` (jump back to normal) | the corrected curves move beyond their noise and come back, and no other family explains it: a jolt of the camera, a reference or the base shifted for a while, a single point, targets that wander and come back; the cause is not established or several factors mix | nothing to do: the curves came back; ignore these points |
```
Varieties, lines 22-24:
```
| `Sudden jump, back to normal` | nothing to do: it came back; ignore these points |
| `Lasting shift` | check what moved: look at the images |
| `Jump, not back yet` | check what moved: look at the images |
```
Boundaries: line 31, "Visibility, or Displacement without a cause." → "Visibility, or Perturbation without a cause; Displacement if the base stays changed."; line 32, "Without an established cause: Displacement." → "Without an established cause: Perturbation."; line 33, append "What does not follow it and comes back is Perturbation." After line 33, add:
```
- **Displacement or Perturbation**: decided by the return, not by the cause. What moved and has not come back by the end of the range (a target, a reference, the camera outside a thermal effect) is Displacement; what comes back, even after days, is Perturbation. A camera that moves and stays is Displacement even when the compensation absorbs it afterwards; the description says it is the camera. A partial return is not a return.
```
- [ ] **Step 4: `labeling-rules.md`**
Line 63: add `` `Perturbation` `` after `` `Displacement` `` in the list of keys. Line 86 becomes the two bullets:
```
- **A Displacement says what moved, checked target by target on the raw position, and that it has not come back by the end of the range**: targets moved: "a movement of these targets is not excluded: to check (what to look at), monitor"; the camera moved and keeps its new position: "it is not a movement of the structure; the camera mount is to be checked"; the base stays changed, nothing moving in the image: "it is not a movement of the structure; to check with the operator"; the whole scene deforms while the camera moves: "movement of the structure not established: to monitor". [cross Displacement, NEBRASKA]
- **A Perturbation says what moved, that it came back and when, and concludes "nothing to do: points to ignore"**: its cause is stated in the three ways of section 6; "nothing to do" rests on the return seen, not on a cause. [the user, 2026-11-18]
```
- [ ] **Step 5: `scripts/validate.py:23`**: `FAMILIES` gains `"Perturbation"`.
- [ ] **Step 6: Run the checks**
Run: `python3 scripts/client.py && python3 scripts/validate.py --check && python3 scripts/validate.py | tail -1`
Expected: `ok`, `ok`, and `0 errors` on the labels' line.
- [ ] **Step 7: Commit**
```bash
git add families.md labeling-rules.md scripts/validate.py scripts/client.py docs/superpowers/specs/2026-11-18-displacement-split-design.md docs/superpowers/plans/2026-11-18-displacement-split.md
git commit -m "Families: Displacement is something that moved and never came back; Perturbation, new, what comes back without another family's cause, nothing to do (spec 2026-11-18-displacement-split-design)"
```
### Task 2: The 36 labels and the 6 rewordings
**Files:**
- Modify: `auto/<STATION>.jsonl` (ALASKA, FLORIDA, JUNGFRAU, LENZSPITZE, MICHIGAN, NEBRASKA, NIEDERHORN, NORDEND, WILDSTRUBEL, WYOMING)
- Modify: `auto/analyses/<analysis>/labels.jsonl`, the folder named by each label's `analysis`
- Create (not committed): `<scratchpad>/relabel.py`
**Interfaces:**
- Consumes: `Perturbation` known to `validate.py` (Task 1).
- [ ] **Step 1: Write `relabel.py`**
Data, from the spec:
```python
PERTURBATION = """ALASKA-1 ALASKA-2 ALASKA-3 ALASKA-5 ALASKA-6 ALASKA-9 FLORIDA-18 FLORIDA-72 JUNGFRAU-110 JUNGFRAU-112
LENZSPITZE-84 LENZSPITZE-98 LENZSPITZE-105 LENZSPITZE-106 LENZSPITZE-108 MICHIGAN-8 NEBRASKA-3 NIEDERHORN-126
NIEDERHORN-144 NIEDERHORN-157 NORDEND-80 NORDEND-112 NORDEND-243 NORDEND-251 NORDEND-257 NORDEND-271 NORDEND-278
NORDEND-279 NORDEND-310 WILDSTRUBEL-136 WILDSTRUBEL-153 WYOMING-9 WYOMING-11 WYOMING-13 WYOMING-16 WYOMING-17""".split()
REWORD = { # id: (closing sentence now, closing sentence after)
"NEBRASKA-3": ("For the customer: the camera straightens and the scene deforms with it: a movement of the structure is not established; the camera tilt is to be monitored, and restoring a third reference will make the curves reliable again.",
"For the customer: the camera straightens and the scene deforms with it, then both come back on 17 October: nothing to do over these days; restoring a third reference will make the curves reliable again."),
"NORDEND-257": ("Cause not established: a movement of PR07 itself is not excluded, but it comes back by itself; to monitor.",
"Cause not established; the curve comes back by itself the same morning: nothing to do, points to ignore."),
"NORDEND-310": ("Cause not established. It is not a movement of the structure; the camera is to be monitored.",
"Cause not established. The camera comes back after about an hour: it is not a movement of the structure, nothing to do; points to ignore."),
"FLORIDA-18": ("Small, without established movement of the structure: to be monitored.",
"The shift fades on the morning of the 15th (return labelled separately): nothing to do, points to ignore."),
"WILDSTRUBEL-136": ("This is not an established movement of the road: do not read these days as a movement, to be checked with the following days.",
"The shift fades on 04/10 (return labelled separately): nothing to do, do not read these days as a movement."),
"NORDEND-250": ("Cause not established: this is not a movement of the structure, the camera moved; point to ignore, the camera is to be watched if it happens again.",
"Cause not established: this is not a movement of the structure; the camera moved and keeps its new position: its mount is to be checked."),
}
```
Behaviour: for each station file, read it, and for each line whose `id` is in `PERTURBATION` or `REWORD`: its `family` must be `Displacement` (else stop with the id); set `family` to `Perturbation` if in `PERTURBATION`; if in `REWORD`, the description must end with the "now" sentence (else stop with the id), replace it, delete `review`. In `auto/analyses/<label["analysis"]>/labels.jsonl`, the copy is the one line with the label's `start`, `end` and old `description` (zero or two: stop with the id); apply the same change (and delete `review` if the copy has one). Write each file right after reading it, every other line byte for byte; a changed line is `json.dumps(d, ensure_ascii=False)` (the files' format: key order kept, no `\u` escape). Print the count of labels and copies changed.
- [ ] **Step 2: Run it**
Run (when the user is not validating in aimsight-app): `python3 <scratchpad>/relabel.py`
Expected: `37 labels, 37 copies` (the 36 family changes, 5 of them also reworded, plus NORDEND-250, reworded only).
- [ ] **Step 3: Check**
Run:
```bash
python3 - <<'EOF'
import json, glob
lab = {d["id"]: d for f in glob.glob("auto/*.jsonl") for d in map(json.loads, filter(str.strip, open(f)))}
import sys; sys.path.insert(0, "<scratchpad>"); from relabel import PERTURBATION, REWORD
assert all(lab[i]["family"] == "Perturbation" for i in PERTURBATION)
assert all("review" not in lab[i] and lab[i]["description"].endswith(REWORD[i][1]) for i in REWORD)
assert all("review" in lab[i] for i in set(PERTURBATION) - set(REWORD))
assert lab["NORDEND-250"]["family"] == "Displacement"
print("ok")
EOF
python3 scripts/validate.py | tail -2
git diff --numstat -- auto/ | awk '{a+=$1} END {print a}'
```
Expected: `ok`; `0 errors`; the diff counts the 74 changed lines plus whatever aimsight-app had already left uncommitted in those files (compare with `git diff --numstat` taken before Step 2).
- [ ] **Step 4: Commit**
```bash
git add auto/ALASKA.jsonl auto/FLORIDA.jsonl auto/JUNGFRAU.jsonl auto/LENZSPITZE.jsonl auto/MICHIGAN.jsonl auto/NEBRASKA.jsonl auto/NIEDERHORN.jsonl auto/NORDEND.jsonl auto/WILDSTRUBEL.jsonl auto/WYOMING.jsonl auto/analyses/*/labels.jsonl
git commit -m "auto: 36 Displacement labels that came back requalified Perturbation (review kept, a change of family the user asked for); 6 closing sentences reworded to their family, review removed (NEBRASKA-3, NORDEND-257, NORDEND-310, FLORIDA-18, WILDSTRUBEL-136, NORDEND-250)"
```
### Task 3: The cases of family Displacement
**Files:**
- Modify: `auto/cases/<id>.json`, only if `auto/cases/` exists
- [ ] **Step 1: List them**
Run: `python3 scripts/cases.py --list | grep "| Displacement |"`
If `auto/cases/` does not exist yet: skip this task and say so; the cases build reads the labels, which carry the new families.
- [ ] **Step 2: Requalify**
A case whose signature is something that comes back (the seed's "the base changes without a reference lost" and "targets move slowly, alone, then come back") gets `"family": "Perturbation"` and loses `review` if it has one (the cases rule: a change of family removes it). The others stay.
- [ ] **Step 3: Check and commit**
Run: `python3 scripts/validate.py | tail -1 && python3 scripts/cases.py --index`
Expected: `N cases: 0 errors`, the index updated.
```bash
git add auto/cases/<the changed files>
git commit -m "cases: the Displacement cases that come back are Perturbation"
```
### Task 4: The detectors
**Files (repository `~/aimsight-detectors`):**
- Modify: `scripts/displacement.py` (`jumps()`, the docstring point 2)
- Modify: `scripts/categories.py` (`CATEGORIES`, docstring line 8)
- Modify: `README.md:7`, `:15`; `CLAUDE.md` ("the six families" → "the seven families")
**Interfaces:**
- Produces: an episode of state "back" has `kind == "Perturbation"`, `label == "Sudden jump, back to normal"`; `categories.CATEGORIES["Perturbation"] == {"ignore": True, "process": "none"}`, placed right after `"Displacement"`.
- [ ] **Step 1: The benchmark before**
Run: `python3 bench/regression.py --labels ~/aimsight-labeling/auto --data ~/aimsight-data/data > <scratchpad>/before.txt`
Expected: no `skipped` line; no `Perturbation` row (no detector gives it yet).
- [ ] **Step 2: Change**
`jumps()`: the episode's kind is `"Perturbation"` when `state == "back"`, else `"Displacement"`. Docstring point 2: "greyed, its cause not established here" → "a Perturbation, greyed". `categories.py` docstring line 8: "otherwise it stays Displacement, its cause to be checked on the images" → "otherwise it is a Perturbation: nothing to do". `README.md`: "six families" → "seven families"; the `displacement.py` row: "Displacement (Lasting shift; Jump, not back yet; Slow drift), Perturbation (Sudden jump, back to normal)". `CLAUDE.md`: "six families" → "seven families".
- [ ] **Step 3: The benchmark after**
Run: `python3 bench/regression.py --labels ~/aimsight-labeling/auto --data ~/aimsight-data/data > <scratchpad>/after.txt && python3 bench/compare.py <scratchpad>/before.txt <scratchpad>/after.txt`
Expected: no `skipped` line; a `Perturbation` row with "found" above zero; the `Displacement` "correct" rises. A regression that compare.py reports is told to the user with the labels concerned (ids), per this repository's rule, not hidden.
- [ ] **Step 4: Commit**
```bash
git -C ~/aimsight-detectors add scripts/displacement.py scripts/categories.py README.md CLAUDE.md
git -C ~/aimsight-detectors commit -m "Perturbation, the seventh family: a jump back to normal is a Perturbation (nothing to do), Displacement keeps the lasting shift, the jump not back yet and the slow drift (labeling spec 2026-11-18-displacement-split-design)"
```
### Task 5: aimsight-app and the agents' instructions
**Files:**
- Modify: `~/aimsight-app/app/web/src/pages/Data.jsx:16` (`FAMILIES`), `:19` (comment)
- Modify: `~/aimsight-agents/agents/analyst.md:56`
- Modify: `~/Support-Agent/.claude/skills/support/rapport.md:20`
- [ ] **Step 1: Data.jsx**
`FAMILIES` gains `"Perturbation": "#9fd8d0"` right after `"Displacement"`; the comment "one of the six families" → "one of the seven families".
- [ ] **Step 2: Build and restart**
Run: `cd ~/aimsight-app/app/web && npm run build && grep -l Perturbation dist/assets/*.js && systemctl --user restart aimsight-app`
Expected: one bundle file named; the service active (`systemctl --user is-active aimsight-app` → `active`).
- [ ] **Step 3: The instructions**
`analyst.md:56`: add `` `Perturbation` `` after `` `Displacement` `` in the list of keys. `rapport.md:20`: `` `Displacement` (à vérifier) `` → `` `Displacement` (quelque chose a bougé : à vérifier) ``, and `` `Perturbation` (rien à faire) `` added last in the order.
- [ ] **Step 4: Commit, one per repository**
```bash
git -C ~/aimsight-app add app/web/src/pages/Data.jsx
git -C ~/aimsight-app commit -m "Data: the family Perturbation and its colour"
git -C ~/aimsight-agents add agents/analyst.md
git -C ~/aimsight-agents commit -m "analyst: the family Perturbation"
git -C ~/Support-Agent add .claude/skills/support/rapport.md
git -C ~/Support-Agent commit -m "support: la famille Perturbation"
```
### Task 6: The whole
- [ ] **Step 1: Run every check**
Run: `cd ~/aimsight-labeling && python3 scripts/validate.py | tail -2 && python3 scripts/validate.py --check && python3 scripts/client.py && python3 scripts/cases.py --check`
Expected: `0 errors` (labels and cases), `ok`, `ok`, `ok`.
- [ ] **Step 2: Look at it in the app**
Open a range with a Perturbation label (ALASKA, 2026-11-05) on the Data page: the label has the new colour, the Category menu offers `Perturbation`, a detectors' jump back shows as Perturbation, greyed.