Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it

- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/,
  weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv,
  patterns.json.
- processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the
  patterns of sensors.json, the camera, optical's current commit); never over an existing one.
- processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked
  against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns)
  and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/.
- pump_diff no longer writes targets.csv: it depends on the run.

JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within
0.034 px; a run created by new_run.sh and run as is gives run01 exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-19 10:52:46 +01:00
1 parent d3bf4b6860
commit ca366c150e
8 files changed
+185 -123

No files matched your search

@@ -3,6 +3,24 @@
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
section "data_diff").
**Replaced the same day by the user's own layout** (the layout below, `configs/` and `runs/<name>/`, is not used):
```
data_diff/<SERIAL>/
images_diff/ (index/, sensors.json, unit.json) images/ metrics/ weather/ pumped, the same for every run
runs/<run>/ one run: whatever depends on it
config.yaml the run's configuration: patterns (by sensor_name), camera and emulator.version (optical's
commit, checked), over the cloud's processing
displacement/ camera/ targets.csv its result, the files of data/ (targets.csv from its patterns)
patterns.json the patterns exactly as given to the emulator, templates included
```
`processing/run_config.sh <SERIAL> <run>` reads the run's `config.yaml` and writes its `displacement/`, `camera/`,
`targets.csv` and `patterns.json`, nothing else. Rule: what is not the same for every possible run is in the run's
folder. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml`, the cloud's processing written
in full (patterns of `sensors.json`, camera, optical's current commit); never over an existing one. Note: the reading tools (aimsight-tools `data.find_files`) read a station folder whole, subfolders
included: pointed at `data_diff/<SERIAL>/`, they see every run's curves together.
## Intent
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
@@ -1,7 +1,15 @@
# run_config: a configuration recomputed into runs/<name>/
# run_config: a run of a station recomputed from its differential images
2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`.
**Amended the same day (the user's layout, see the configurations design):** `processing/run_config.sh <SERIAL> <run>`
reads `data_diff/<SERIAL>/runs/<run>/config.yaml` (`patterns`, `camera`, `emulator.version`: optical's commit, the run
stops if `$OPTICAL` is at another one or has local changes; no `stations` level) and writes only that run's
`displacement/`, `camera/`, `targets.csv` (from its patterns; none at the station's root any more) and `patterns.json`
(the patterns as given to the emulator); the emulator's other files go through a temporary folder. No `runs/<name>/`,
no links, no `run.yaml`. The rest below holds (patterns, correction as
the cloud, columns, acceptance).
## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`)
Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw