Files
aimsight-labeling/docs/superpowers/plans/2026-11-16-analyst-knowledge.md

15 KiB
Raw Permalink Blame History

The analyst's domain knowledge: implementation plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: one hand-maintained source per kind of knowledge for the analyst (labels through the RAG, families.md with a phenomenon catalogue, labeling-rules.md, reading-rules.md, stations.md, tools/DATA.md), the old map and knowledge/ archived, then the labeling scripts moved to labeling/scripts/.

Architecture: mostly writing: markdown compiled from existing files, then path updates in four repositories. One throwaway script (in the scratchpad, never committed) converts the map's label keys to label ids. The only code change is the scripts' root path after the move.

Tech Stack: markdown; Python 3 standard library; git.

Spec: labeling/docs/superpowers/specs/2026-11-16-analyst-knowledge-design.md.

Global Constraints

  • Starts only when the English migration (Diagnostic-App/docs/superpowers/plans/2026-11-16-english-renames.md) is committed in labeling, tools, agents and Diagnostic-App: git status --short shows nothing of it in any of them (the app's own label submissions in labeling/auto/*.jsonl may remain; never stage them).
  • All new text in English. Times in UTC. No numeric value in label descriptions (not touched here anyway).
  • labeling/manual/ is never read by any step or script.
  • No station-specific rule in families.md, labeling-rules.md, reading-rules.md; stations.md is context only and says so.
  • tools holds what the data are; labeling what they mean. Nothing in tools/DATA.md says "normal", a category or a threshold to judge by.
  • Commits by path, never git add -A; English commit messages ending with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>.
  • Never stage labeling/auto/*.jsonl or labeling/auto/analyses/ (Diagnostic-App writes there concurrently).

Review Focus

  1. Knowledge silently dropped: a bullet of the map's sections 2–3, or a fact of a knowledge/ note, that lands in no new file. Pinned by the coverage table of Task 3, step 4.
  2. A map key converted to the wrong id (two labels share a start). Pinned by the ambiguity report of Task 3, step 2: every ambiguous key is resolved by hand, none guessed.
  3. A stale path left in another repository (label-map, knowledge/, labeling/<script>.py). Pinned by the repo-wide greps of Tasks 6 and 8.
  4. The RAG index not found after the move, re-embedding ~1,000 labels (~8 min). Pinned by Task 8, step 5.
  5. The analyst still told to read a removed file. Pinned by the load check of Task 6.

Task 0: Gate and baseline

  • Step 1: check the gate (Global Constraints, first line). If not met, stop and tell the user.
  • Step 2: baseline, saved in the scratchpad as baseline.txt:
    • cd ~/labeling && python3 validate.py (count of errors; the app's labels without review are expected);
    • python3 ids.py (must change nothing: git diff --stat auto/ before = after);
    • bytes the analyst reads in full today: wc -c label-map.md families.md labeling-rules.md knowledge/measurement-chain.md knowledge/displacement-columns.md knowledge/gaps.md ~/tools/GUIDE.md (plus the map is read only partly: count lines 1–60 and sections 2–4).

Task 1: The spec and the plan in labeling

Files: Create labeling/docs/superpowers/specs/2026-11-16-analyst-knowledge-design.md, labeling/docs/superpowers/plans/2026-11-16-analyst-knowledge.md (copies of the scratchpad files).

  • Step 1: copy both files.
  • Step 2: commit: git add docs/superpowers/specs/2026-11-16-analyst-knowledge-design.md docs/superpowers/plans/2026-11-16-analyst-knowledge.md && git commit -m "Analyst knowledge: design and plan (one source per kind of knowledge)".

Task 2: tools/DATA.md

Files:

  • Create: tools/DATA.md

  • Modify: tools/GUIDE.md (section "Read first", line 7 onwards), tools/scripts/data.py (docstring l. 3, l. 129), tools/scripts/inventory.py (messages l. 79, 128, 193), tools/scripts/quality.py (docstring l. 11, message ~l. 377), tools/README.md (the files of the repository), tools/CLAUDE.md (one line: DATA.md is maintained by hand, GUIDE.md by the tool-builder).

  • Sources: labeling/knowledge/displacement-columns.md, quality-metrics.md, camera-images.md.

  • Step 1: write DATA.md, sections: "The exports" (advanced, hourly, camera pose: columns and their meaning, rejected measurement keeps its row with empty corr, corr_x/corr_y in pixels, the per-pattern factor to the hourly mm, x/y are the pattern position not the spot centroid, tilt positive = camera nosing down, ref vs global_ref); "LENZSPITZE" (one line: x/y rewritten in image pixels by the extraction; 1023 and 8019 without pattern); "Image quality metrics" (the table of quality-metrics.md, as is, and its closing paragraph: quality of a target ≠ health of the reference network); "Camera images" (IRON, DIFF, acquisition modes, what is known per station). Drop: pumping dates and history, the detectors' to-do list, Support-Agent paths, the closing "causes to separate" list and the non-negotiable rules (they go to reading-rules.md, Task 3).

  • Step 2: "Read first" of GUIDE.md becomes two items: tools/DATA.md (first), and labeling/reading-rules.md. Point every docstring and message listed above at tools/DATA.md.

  • Step 3: verify: grep -rn 'knowledge/' ~/tools --include=*.py --include=*.md → nothing; cd ~/tools && python3 scripts/quality.py --self-test → passes; from ~/Diagnostic-App: python3 tools/scripts/inventory.py data/NORDEND --start 2026-11-05T06:00Z --end 2026-11-05T12:00Z runs and its messages name tools/DATA.md.

  • Step 4: commit in tools: git add DATA.md GUIDE.md README.md CLAUDE.md scripts/data.py scripts/inventory.py scripts/quality.py && git commit -m "DATA.md: what the data are, from labeling/knowledge (formats, columns, quality metrics, images)".

Task 3: labeling/reading-rules.md

Files: Create labeling/reading-rules.md; scratchpad only: convert_keys.py, coverage.md. Sources: knowledge/measurement-chain.md, knowledge/gaps.md, label-map.md sections 2 and 3, the non-negotiable rules (end of displacement-columns.md, section 4 of measurement-chain.md), knowledge/barcelona-incident.md, knowledge/expert-annotated-cases.md.

Interfaces:

  • Produces: convert_keys.py — convert(text: str) -> tuple[str, list[str]] returns the text with keys replaced and the list of unresolved or ambiguous keys; reused by Tasks 4 and 5.

  • Step 1: write convert_keys.py (scratchpad). A key is STATION DD.MM HH:MM, optionally followed by (qualifier), inside [...] citations. Year 2026. Look up labeling/auto/<STATION>.jsonl labels with start == "2026-MM-DDTHH:MMZ": one → its id; several → the one whose description contains the qualifier (or whose family equals it); none → [expert]; still several → leave the key, report it. Ends with an assert-based self-check on three keys taken from the real map: one unique, one with a qualifier (NEBRASKA 16.10 20:00 (COLJ19)), one with no match in auto/.

  • Step 2: run it on the map's sections 2–3; resolve every reported key by reading the candidate labels. No guess: if still unclear, write [expert].

  • Step 3: write reading-rules.md, sections: "How a measurement is made" (chain, from measurement-chain.md §1); "Where each cause enters" (its §2 table, without [H] cells unless a label confirms them, then cite the label id); "Thermal effects" (§3); "Gaps" (gaps.md); "Reading rules" (the map's §2, both lists, converted citations); "False friends" (the map's §3); "Non-negotiable rules" (once); "Lessons from cases" (Barcelona, the two client reports: one bullet per lesson, no narrative). Drop what families.md or labeling-rules.md already say.

  • Step 4: coverage.md (scratchpad): every bullet of the map's §2–3 and every section of the five source notes, each with where it went (reading-rules.md § / families.md / labeling-rules.md / DATA.md / dropped, and why). No row without a destination.

  • Step 5: verify: grep -nE '[A-Z0-9]{4,} [0-9]{2}\.[0-9]{2} [0-9]{2}:[0-9]{2}' reading-rules.md → nothing; grep -n 'Support-Agent\|\[H\]\|E-0' reading-rules.md → nothing.

Task 4: labeling/stations.md

Files: Create labeling/stations.md. Source: label-map.md section 4.

  • Step 1: header (two lines: context for reading a station's data, not a rule; the expert rule on the kind of structure depends on it). Then the table: Station | Structure | Camera (interval, shelter, power) | References (number, twins, local references) | Neighbouring cameras. Drop the columns "what recurs in its labels" and "Events". Convert citations with convert_keys.py.
  • Step 2: verify: one row per station of auto/ that the map describes; same key grep as Task 3, step 5 → nothing.

Task 5: The phenomenon catalogue in families.md

Files: Modify labeling/families.md (new section "Phenomena", after "Boundaries", before "What is not labelled"; the table's "Phenomena of the label map" column keeps the names, now pointing to this section). Source: the map's table of contents and cards (sections A–G).

  • Step 1: one subsection per family, then "Family of the cause" (a reference set aside, two references lost, the base changing without a lost reference). One line per phenomenon of the table of contents: name: what is seen; how to tell it from its neighbours. No citations, no station names. Each phenomenon goes under exactly one heading.
  • Step 2: verify: every phenomenon title of the map's table of contents appears once in the section (grep -c on each title, a loop in the scratchpad); the section is ≤ 60 lines; grep -nE 'NORDEND|JUNGFRAU|NIEDERHORN|WILDSTRUBEL|LENZSPITZE|CLARIDEN|BREITHORN|FLORIDA|SNCF01|MICHIGAN|NEBRASKA' families.md → nothing new.

Task 6: Archive, labeling docs, the analyst

Files:

  • Move: labeling/label-map.md, labeling/knowledge/ → labeling/docs/archive/ (git mv); one line each in labeling/docs/archive/README.md (why archived, 2026-11-16, replaced by …).

  • Modify: labeling/README.md, labeling/CLAUDE.md (the layout table of the spec: kind → file → maintained by the user, by hand).

  • Modify: agents/agents/analyst.md: l. 11 (the map sentence → "the phenomena are catalogued in labeling/families.md; the cases are the labels, through the precedents"); l. 28–31 (the reading list: families.md, labeling-rules.md, reading-rules.md, stations.md, all in full; tools/GUIDE.md and tools/DATA.md); the procedure step 2 (the catalogue gives the hypotheses, the precedents give the cases); the analysis template line "Phenomenon of the map" → "Phenomenon of the catalogue"; "Sources read" unchanged.

  • Step 1: git mv and the archive README.

  • Step 2: README, CLAUDE.md of labeling; analyst.md.

  • Step 3: verify: grep -rn 'label-map\|knowledge/' ~/labeling ~/agents ~/tools ~/Diagnostic-App ~/Optimizer ~/detectors --include=*.md --include=*.py 2>/dev/null | grep -v '/\.git/\|/archive/\|docs/superpowers\|auto/analyses' → nothing; cd ~/labeling && python3 validate.py → as in the baseline; python3 precedents.py --check → ok.

  • Step 4: load check, from ~/Diagnostic-App: claude -p "List the files you must read in full before analysing a range, and nothing else." --plugin-dir ~/agents --agent agents:analyst --model haiku → exactly the six files of step 2 (and instructions/ if it names it), no label-map, no knowledge/.

  • Step 5: commit labeling: git add reading-rules.md stations.md families.md README.md CLAUDE.md docs/archive/ label-map.md knowledge/ && git commit -m "Analyst knowledge: reading-rules.md, stations.md, a phenomenon catalogue in families.md; the label map and knowledge/ archived". Commit agents: git add agents/analyst.md && git commit -m "Analyst: reads families (with the catalogue), labeling rules, reading rules, stations, tools/DATA.md; the label map is archived".

Task 7: A real run

  • Step 1: from ~/Diagnostic-App, run the analyst on NORDEND 2026-11-05T06:00Z → 2026-11-05T12:00Z, output folder in the scratchpad (run/), not in labeling/auto/analyses/ (nothing is submitted to auto/): the request format of analyst.md, DETECTORS : no, INSTRUCTIONS: instructions/analyst.md, with --add-dir on the real targets of data/, labeling/, tools/ and the scratchpad.
  • Step 2: compare with labeling/auto/analyses/NORDEND_2026-11-05T0600_2026-11-05T1200/: the labels (count, bounds, families), and section 3 "documentation" of the new analysis: anything it says is missing that the map used to give. Note the bytes read in full against the baseline.
  • Step 3: report to the user; a missing piece of knowledge goes into the right file (Tasks 3–5) and Task 6, step 5 is amended, before Task 8.

Task 8: The scripts in labeling/scripts/

Files:

  • Move: labeling/validate.py, ids.py, precedents.py → labeling/scripts/.

  • Modify: scripts/validate.py:19, scripts/ids.py:14 (AUTO = ../auto from the script); scripts/precedents.py:29 (HERE → the labeling root, so that auto/ and .index/ stay where they are); the usage lines and messages of the three scripts (python3 labeling/scripts/<script>.py); labeling/README.md, labeling/CLAUDE.md, labeling/.claude/settings.json (the three permissions); agents/agents/analyst.md (precedents and validate lines); Diagnostic-App/README.md; the comment of Diagnostic-App/app/server/station.py:266.

  • Step 1: git mv and the path changes.

  • Step 2: python3 scripts/precedents.py --check → ok.

  • Step 3: python3 scripts/validate.py → same as the baseline; python3 labeling/scripts/validate.py from ~/Diagnostic-App (through the link) → same.

  • Step 4: python3 scripts/ids.py → git diff --stat auto/ unchanged.

  • Step 5: python3 scripts/precedents.py "a reference rejected intermittently makes the targets jump together" --k 3 → three precedents, and no "index: N/M new texts" line on stderr (nothing re-embedded).

  • Step 6: grep -rn 'labeling/\(validate\|ids\|precedents\)\.py\|python3 \(validate\|ids\|precedents\)\.py' ~/labeling ~/agents ~/tools ~/Diagnostic-App ~/Optimizer ~/detectors 2>/dev/null | grep -v '/\.git/\|/archive/\|docs/superpowers\|auto/analyses' → nothing.

  • Step 7: commits by path: labeling (scripts/ validate.py ids.py precedents.py README.md CLAUDE.md .claude/settings.json, "The labeling scripts in scripts/"), agents (agents/analyst.md), Diagnostic-App (README.md app/server/station.py).