Configurations: processing/run_config.sh recomputes configs/<name>.yaml from data_diff/ into runs/<name>/, read like data/
Patterns by sensor_name (a property, or a window redrawn from a DIFF image as the device's pattern editor does), re-measured with optical's emulator, camera correction as the cloud's process_shifts (one measurement at a time, or the series), written in the files of data/ (displacement/, camera/, targets.csv; images/, metrics/, weather/ linked to data_diff/), with run.yaml and diagnostics/ beside the stations. configs/cloud.yaml, empty, reproduces data/JUNGFRAU 2026-10-12..16: corr_x/y within 0.03 px, the same rejections, the camera pose within 5e-4 deg; 16 s. configs/jungfrau_example.yaml: PM_G back to 0.7, BL_C's window redrawn. .gitignore: data_diff/, runs/, remeasure/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
81908ca370
commit
d3bf4b6860
8 files changed
+512
No files matched your search
@@ -0,0 +1,53 @@
|
||||
# run_config Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** `processing/run_config.sh <config> [SERIAL...]` writes `runs/<name>/<SERIAL>/` with the files of `data/`.
|
||||
|
||||
**Architecture:** one script; optical imported from `$OPTICAL`; re-measurement through `reprocess_images.main`;
|
||||
correction through `remove_camera_movement`, per epoch by default.
|
||||
|
||||
**Tech Stack:** uv (numpy, pandas, pillow, pyyaml, scipy, scikit-image, opencv-python-headless).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-11-19-run-config-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Writes only under `runs/<name>/` (guarded); reads `data_diff/`, never `data/` except for the acceptance comparison.
|
||||
- Columns of `data/`'s advanced CSV, in that order: `time_utc, sensor_name, x, y, error_value, error_threshold,
|
||||
relative_x, relative_y, corr_x, corr_y, reproj_err, inlier, sigma0, pattern_type, peak, flux, sharpness, fwhm,
|
||||
dominance, snr, corr_x_mm, corr_y_mm`.
|
||||
- Camera CSV columns: `time_utc`, then `median_/min_/max_` of `pan, tilt, scale, rotation, sigma0`.
|
||||
|
||||
## Review Focus
|
||||
|
||||
- A configuration with a typo (`error_treshold`, unknown sensor): error before any work.
|
||||
- An epoch where only one reference survives: translation only, no camera row (as the cloud).
|
||||
- A day without images in the range: no file for it.
|
||||
- Re-running the same configuration: the station folder replaced, other stations untouched.
|
||||
- optical's working tree dirty: recorded in `run.yaml`.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: configuration and correction, with `--check`
|
||||
|
||||
**Files:** Create `processing/run_config.py`.
|
||||
|
||||
**Interfaces:** `load_config(path) -> dict`; `camera_settings(config, station, unit_props) -> dict`;
|
||||
`cloud_properties(sensors) -> {sid: props}`; `apply_patterns(props, overrides, images_dir) -> {sid: props}`;
|
||||
`correct(meas: DataFrame, props, cam) -> (advanced: DataFrame, camera: DataFrame)`.
|
||||
|
||||
- [ ] `check()`: unknown top / station / camera / pattern key raises; unknown sensor_name raises; camera order station >
|
||||
common > unit > defaults; synthetic correction: three static references, a static target (corr ~0), a target above
|
||||
its threshold (corr empty, x kept), relative = x - pattern x, sigma0 on every row, one camera row per minute.
|
||||
- [ ] Implement; `--check` prints ok.
|
||||
|
||||
### Task 2: the run, the cloud configuration, the acceptance
|
||||
|
||||
**Files:** Modify `processing/run_config.py` (`main`); create `processing/run_config.sh`, `configs/cloud.yaml`;
|
||||
modify `.gitignore` (`runs/`), `README.md`.
|
||||
|
||||
- [ ] `main`: stations, days, patterns, `reprocess_images.main`, `correct`, per-day files, links, `run.yaml`.
|
||||
- [ ] `run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17`; `data.py` reads `runs/cloud/JUNGFRAU`.
|
||||
- [ ] Compare with `data/JUNGFRAU`: corr_x / corr_y (all, and the rejected-reference night of 10-15), camera pose.
|
||||
- [ ] Docs.
|
||||
@@ -0,0 +1,53 @@
|
||||
# run_config: a configuration recomputed into runs/<name>/
|
||||
|
||||
2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`.
|
||||
|
||||
## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`)
|
||||
|
||||
Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw
|
||||
`x`, `y`, `error` (and the metrics) are stored; targets above their `error_threshold` get `x`/`y` blanked; a rejected
|
||||
`global_ref` gets back the `x`/`y` of the latest stored frame whose error is not above its own (in practice its own
|
||||
raw position, inserted just before), else its pattern position; `remove_camera_movement(r, focal_distance (6.0),
|
||||
pixel_size (0.0024), algorithm (ransac, ransac_threshold 0.99))`, which rejects those references from the fit again
|
||||
(error columns given) but still corrects their rows; `corr -= (x_ref, y_ref)`; `cam_info` (pan, tilt, scale,
|
||||
rotation, sigma0) and `ref_stats` (reproj_err, inlier) stored. The Advanced export adds `relative = x - pattern x`
|
||||
and copies `sigma0` onto every row; the camera export is the per-minute median / min / max of `cam_info`.
|
||||
|
||||
## Design
|
||||
|
||||
`processing/run_config.py`, through `processing/run_config.sh` (uv: numpy, pandas, pillow, pyyaml, scipy,
|
||||
scikit-image, opencv-python-headless; optical's code from `$OPTICAL`, default `~/optical`):
|
||||
|
||||
processing/run_config.sh <config> [SERIAL...] [--from D] [--to E] [--jobs N]
|
||||
|
||||
`<config>`: `configs/<name>.yaml` or a path; stations: those named, else those of the configuration; days: those
|
||||
indexed in `data_diff/<SERIAL>/images_diff/index/` (within `--from`/`--to`). For each station, `runs/<name>/<SERIAL>/`
|
||||
is replaced:
|
||||
|
||||
1. **Patterns**: `images_diff/sensors.json` (DIFF type, a pattern, not archived; latest value of each property), then
|
||||
the configuration by `sensor_name` (an unknown name or key is an error): any property replaced
|
||||
(`error_threshold`, `pattern_type`, `distance_convertion_factor`, `x`, `y`...); `redraw: {image, corners}` cuts the
|
||||
template from that DIFF image as the device's pattern editor does (`estimate_camera_pose` from the references,
|
||||
`saved_anchor_and_reference`: `x_ref` kept, `ImagePatternMono.from_image`). Written to
|
||||
`diagnostics/<SERIAL>_sensor_properties.json`.
|
||||
2. **Re-measurement**: optical's `reprocess_images.main` (emulator) on the days' images, `--times` from the index;
|
||||
rows kept in `diagnostics/<SERIAL>_remeasured.csv`.
|
||||
3. **Correction**: as the cloud, one measurement at a time (`per_epoch: true`, default): targets above their
|
||||
threshold blanked, references kept raw (rejected in the fit by `remove_camera_movement`), `corr -= x_ref`;
|
||||
`per_epoch: false`: the whole series in one call (a rejected reference interpolated in time, algorithms with
|
||||
memory). Camera block: `algorithm` (ransac), `ransac_threshold` (0.99, ransac only), `mode` (`off` | `rotation` |
|
||||
`rotation_scale` | `relative`, optical's `refine_allpoints`), `focal_distance`, `pixel_size` (unit.json, else 6.0 /
|
||||
0.0024), `per_epoch`; station block over common block over unit.json over the cloud's defaults.
|
||||
4. **Output**, the files of `data/`: `displacement/<S>_advanced_<YYYYMMDD>.csv` (columns of `data/`: `x, y,
|
||||
error_value, peak..snr` from the emulator, `relative = x - pattern x`, `corr` from the correction, `reproj_err,
|
||||
inlier` of the references, `sigma0` of the epoch on every row, `corr_*_mm = corr * distance_convertion_factor`),
|
||||
`camera/<S>_camera_<YYYYMMDD>.csv` (median / min / max per minute), `targets.csv`; `images/`, `metrics/`,
|
||||
`weather/` as relative symbolic links to `data_diff/<SERIAL>/`; `run.yaml` at `runs/<name>/` (configuration as
|
||||
resolved per station, optical's commit and state, days, date).
|
||||
|
||||
Acceptance: `configs/cloud.yaml` (empty) on JUNGFRAU 2026-10-12..16 against `data/JUNGFRAU`: positions and errors
|
||||
within 0.05 px (done by the emulator), `corr_x/y` and the camera pose close to the cloud, the differences counted.
|
||||
|
||||
Check: `run_config.py --check` in the uv environment: the configuration rules (unknown keys, order of the camera
|
||||
blocks, overrides by name) and a synthetic correction (three references, two targets, one rejected) through
|
||||
optical's `remove_camera_movement`.
|
||||
Reference in new issue
Block a user