Files
aimsight-data/README.md
T

16 lines
3.3 KiB
Markdown
Raw Normal View History

# aimsight-data
The data of the AIMsight stations, in a single copy on the workstation, and their pumping.
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again.
- `data_diff/<SERIAL>/`, **out of git**: pumped by `extraction/pump_diff.sh` (never touching `data/`): the differential images full size under `images_diff/` (with their index, the patterns `sensors.json` and the unit `unit.json`), `images/` (IRON), `metrics/`, `weather/`, as in `data/`: only what is the same for every run. Its runs in `runs/<run>/`, with everything that depends on the run: `config.yaml` (the run's configuration), `displacement/`, `camera/` and `targets.csv` (its result, the files of `data/`), `patterns.json`. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`.
- `processing/run_config.sh <SERIAL> <run> [--from D] [--to E]`: runs `data_diff/<SERIAL>/runs/<run>/config.yaml` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 18 s for 6 days of JUNGFRAU) and writes the run's `displacement/`, `camera/`, `targets.csv` (from its patterns) and `patterns.json` (the patterns exactly as given to the emulator, templates included); `--check` without data. `config.yaml` states what differs from the cloud's processing (an empty file: the cloud's): `patterns` by `sensor_name` (a property, or a window redrawn from a DIFF image), `camera` (algorithm, ransac_threshold, mode, focal_distance, pixel_size, per_epoch) and `emulator.version` (optical's commit: the run stops if `$OPTICAL` is at another one or has local changes); its keys are checked. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml` only, the cloud's processing written in full (every pattern of `sensors.json`, the camera, optical's current commit), to edit before running it; it never overwrites an existing `config.yaml`. The standard configuration reproduces `data/` (JUNGFRAU, 6 days: `corr_x/y` within 0.034 px, the same rejections). A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new). Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`.
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations).
- `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data/<SERIAL>/weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`.
- `.env`: the credentials of the database and of S3, at the root, never versioned.
The other repositories (aimsight-app, Optimizer) read the data through a link `data -> ~/aimsight-data/data`, ignored by git. The methods that bring data in live here; the app only calls them. Pumping is done from here only: the lock `out/pump.lock` holds for this folder alone.
The labels are in the neighbouring repository `~/aimsight-labeling`.
Extracted from aimsight-app on 2026-11-13 (commit 462ba9e); the earlier history of the pumping is in Support-Agent.