maximeandClaude Opus 5.5 ca366c150e Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it
- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/,
  weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv,
  patterns.json.
- processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the
  patterns of sensors.json, the camera, optical's current commit); never over an existing one.
- processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked
  against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns)
  and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/.
- pump_diff no longer writes targets.csv: it depends on the run.

JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within
0.034 px; a run created by new_run.sh and run as is gives run01 exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:52:46 +01:00

aimsight-data

The data of the AIMsight stations, in a single copy on the workstation, and their pumping.

  • data/<SERIAL>/: filled by the pumping, out of git (metrics/, displacement/, camera/, images/, weather/ for a station with a Swiss position…). A lost folder is pumped again.
  • data_diff/<SERIAL>/, out of git: pumped by extraction/pump_diff.sh (never touching data/): the differential images full size under images_diff/ (with their index, the patterns sensors.json and the unit unit.json), images/ (IRON), metrics/, weather/, as in data/: only what is the same for every run. Its runs in runs/<run>/, with everything that depends on the run: config.yaml (the run's configuration), displacement/, camera/ and targets.csv (its result, the files of data/), patterns.json. Design: docs/superpowers/specs/2026-11-19-configurations-design.md.
  • processing/run_config.sh <SERIAL> <run> [--from D] [--to E]: runs data_diff/<SERIAL>/runs/<run>/config.yaml (optical's emulator and camera correction, from $OPTICAL, default ~/optical; 18 s for 6 days of JUNGFRAU) and writes the run's displacement/, camera/, targets.csv (from its patterns) and patterns.json (the patterns exactly as given to the emulator, templates included); --check without data. config.yaml states what differs from the cloud's processing (an empty file: the cloud's): patterns by sensor_name (a property, or a window redrawn from a DIFF image), camera (algorithm, ransac_threshold, mode, focal_distance, pixel_size, per_epoch) and emulator.version (optical's commit: the run stops if $OPTICAL is at another one or has local changes); its keys are checked. processing/new_run.sh <SERIAL> <run> creates a run, empty: its config.yaml only, the cloud's processing written in full (every pattern of sensors.json, the camera, optical's current commit), to edit before running it; it never overwrites an existing config.yaml. The standard configuration reproduces data/ (JUNGFRAU, 6 days: corr_x/y within 0.034 px, the same rejections). A window redrawn on a target restarts its displacement at zero at the image it is cut from (x_ref is kept, the template is new). Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
  • extraction/: the pumping, read-only on the platform; how to use it: extraction/README.md. VS Code tasks "pump" and "pump a range". extraction/pumping.py: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations).
  • weather/: the weather measured by MeteoSwiss at a place over a past range (weather/meteoswiss.py), written into data/<SERIAL>/weather/ at each pumping (extraction/weather.py); its doc in weather/README.md.
  • .env: the credentials of the database and of S3, at the root, never versioned.

The other repositories (aimsight-app, Optimizer) read the data through a link data -> ~/aimsight-data/data, ignored by git. The methods that bring data in live here; the app only calls them. Pumping is done from here only: the lock out/pump.lock holds for this folder alone.

The labels are in the neighbouring repository ~/aimsight-labeling.

Extracted from aimsight-app on 2026-11-13 (commit 462ba9e); the earlier history of the pumping is in Support-Agent.

S
Description
Données des stations AIMsight (hors de git) et leur pompage
Readme
229 KiB
0 Stars 1 Watchers 0 Forks
Languages
Python 96%
Shell 4%