Compare commits
11
Commits
6454ffef0f
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
73567fe585 | ||
|
|
799a53c102 | ||
|
|
ca366c150e | ||
|
|
d3bf4b6860 | ||
|
|
81908ca370 | ||
|
|
1b918facd9 | ||
|
|
ea0228a73a | ||
|
|
9adc38f9b9 | ||
|
|
63df363ff3 | ||
|
|
b1790fc422 | ||
|
|
088484c376 |
No files matched your search
@@ -2,3 +2,6 @@ data/
|
||||
out/
|
||||
.env
|
||||
__pycache__/
|
||||
data_diff/
|
||||
remeasure/
|
||||
runs/
|
||||
@@ -2,11 +2,14 @@
|
||||
|
||||
The data of the AIMsight stations, in a single copy on the workstation, and their pumping.
|
||||
|
||||
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`…). A lost folder is pumped again.
|
||||
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range".
|
||||
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again.
|
||||
- `data_diff/<SERIAL>/`, **out of git**: pumped by `extraction/pump_diff.sh` (never touching `data/`): the differential images full size under `images_diff/` (with their index, the patterns `sensors.json` and the unit `unit.json`), `images/` (IRON), `metrics/`, `weather/`, as in `data/`: only what is the same for every run. Its runs in `runs/<run>/`, with everything that depends on the run: `config.yaml` (the run's configuration), `displacement/`, `camera/` and `targets.csv` (its result, the files of `data/`), `patterns.json`. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`.
|
||||
- `processing/run_config.sh <SERIAL> <run> [--from D] [--to E]`: runs `data_diff/<SERIAL>/runs/<run>/config.yaml` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 18 s for 6 days of JUNGFRAU) and writes the run's `displacement/`, `camera/`, `targets.csv` (from its patterns) and `patterns.json` (the patterns exactly as given to the emulator, templates included); `--check` without data. `config.yaml` states what differs from the cloud's processing (an empty file: the cloud's): `patterns` by `sensor_name` (a property, or a window redrawn from a DIFF image), `camera` (algorithm, ransac_threshold, mode, focal_distance, pixel_size, per_epoch) and `emulator.version` (optical's commit: the run stops if `$OPTICAL` is at another one or has local changes); its keys are checked. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml` only, the cloud's processing written in full (every pattern of `sensors.json`, the camera, optical's current commit), to edit before running it; it never overwrites an existing `config.yaml`. The standard configuration reproduces `data/` (JUNGFRAU, 6 days: `corr_x/y` within 0.034 px, the same rejections). A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new). Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`.
|
||||
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations).
|
||||
- `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data/<SERIAL>/weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`.
|
||||
- `.env`: the credentials of the database and of S3, at the root, never versioned.
|
||||
|
||||
The other repositories (aimsight-app, Optimizer) read the data through a link `data -> ~/aimsight-data/data`, ignored by git. Pumping is done from here only: the lock `out/pump.lock` holds for this folder alone.
|
||||
The other repositories (aimsight-app, Optimizer) read the data through a link `data -> ~/aimsight-data/data`, ignored by git. The methods that bring data in live here; the app only calls them. Pumping is done from here only: the lock `out/pump.lock` holds for this folder alone.
|
||||
|
||||
The labels are in the neighbouring repository `~/aimsight-labeling`.
|
||||
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
# The cloud's processing, nothing changed: the patterns of data_diff/<SERIAL>/images_diff/sensors.json, the camera
|
||||
# correction of process_shifts (ransac, ransac_threshold 0.99, the unit's focal_distance / pixel_size, else 6.0 / 0.0024),
|
||||
# one measurement at a time. The reference the other configurations are compared with; it must reproduce data/.
|
||||
#
|
||||
# A configuration states only what differs, for example:
|
||||
#
|
||||
# camera: # common to the stations of the file
|
||||
# algorithm: ransac # ransac | umeyama | danish_hybrid | trusted_danish (memory: per_epoch false)
|
||||
# mode: "off" # off | rotation | rotation_scale | relative (optical's refine_allpoints)
|
||||
# per_epoch: true # false: the whole series at once (a rejected reference interpolated in time)
|
||||
# stations:
|
||||
# JUNGFRAU:
|
||||
# camera: {focal_distance: 6.3}
|
||||
# patterns: # by sensor_name: any existing property, or a template redrawn from a DIFF image
|
||||
# PM_G: {error_threshold: 0.7}
|
||||
# REF_CT: {pattern_type: target}
|
||||
# BL_C: {redraw: {image: 20261012T1200Z_9999999.png, corners: [[1700, 929], [1724, 953]]}}
|
||||
@@ -0,0 +1,9 @@
|
||||
# Example, 2026-11-19: the two findings of the first re-measurement of JUNGFRAU (2026-10-12..16).
|
||||
# - PM_G is rejected on every image by its threshold of 0.3 (its error is about 0.43): back to the usual 0.7.
|
||||
# - BL_C is cut by its 20x20 window (edge margin -0.2 px in median): the camera moved by about 2.5 px since the
|
||||
# patterns were drawn; the window redrawn around where the target is seen now, x_ref kept (same zero).
|
||||
stations:
|
||||
JUNGFRAU:
|
||||
patterns:
|
||||
PM_G: {error_threshold: 0.7}
|
||||
BL_C: {redraw: {image: 20261012T1200Z_10163946.png, corners: [[1700, 933], [1720, 953]]}}
|
||||
@@ -0,0 +1,48 @@
|
||||
# pump_diff: the layout of data/ Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** `data_diff/<SERIAL>/` read by the other repositories as `data/<SERIAL>/`: images, metrics, weather,
|
||||
targets.csv as in `data/`; the DIFF images' extras under `images_diff/`.
|
||||
|
||||
**Architecture:** `pump_diff.py` calls `pump.py`'s `pump_metrics`, `pump_images`, `sensors`, `write_targets` per
|
||||
day / station, then `weather.main` on its `--out`; `weather.py` gets a root folder parameter.
|
||||
|
||||
**Tech Stack:** as `pump_diff.sh` (uv, sqlalchemy, psycopg2-binary, pandas, boto3).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-11-19-configurations-design.md` (section "Work, in order", 1)
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- `data/` never written; `pump.py`'s behaviour unchanged for `pump.sh` (only a return value added).
|
||||
- Extras paths: `images_diff/index/<S>_diff_<YYYYMMDD>.csv`, `images_diff/sensors.json`, `images_diff/unit.json`.
|
||||
- IRON `--every` default 60; metrics replaced for every day pumped.
|
||||
|
||||
## Review Focus
|
||||
|
||||
- Metrics failed for a day: no index, the day is read again next run.
|
||||
- `weather.py` without `--data`: writes into `data/` as before (`pump.sh` unchanged).
|
||||
- A station without position: no weather, the pumping goes on.
|
||||
- Weather raising something unexpected (network): printed, the pumping's result kept.
|
||||
- Old layout on disk (`index/` at the station root): migrated by hand once, then regenerated.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: pump_diff and weather
|
||||
|
||||
**Files:** Modify `extraction/pump_diff.py`, `extraction/pump.py` (`pump_metrics` returns `bool`),
|
||||
`extraction/weather.py` (`main(serials, data=DATA)`, `station(..., data)`, `--data DIR`), `extraction/README.md`,
|
||||
`README.md`.
|
||||
|
||||
- [ ] **Step 1:** `check()`: the index and `indexed_days` under `images_diff/index/`; fails.
|
||||
- [ ] **Step 2:** move the extras; `--every`; per day `pump_metrics` (skip the day when it returns False),
|
||||
`pump_images(..., "images", every, out)`, then the DIFF day; per station `write_targets(folder, sensors(conn, unit),
|
||||
seen)` with `seen` = {sensor_name: id} of the non-archived DIFF sensors with a pattern; at the end `weather.main(
|
||||
stations, a.out)` in a try.
|
||||
- [ ] **Step 3:** `--check` ok; `python3 extraction/pump.py --check` ok (uv); `python3 extraction/weather.py --data
|
||||
/nonexistent` prints nothing and exits 0.
|
||||
- [ ] **Step 4:** migrate `data_diff/` (old `index/` removed, `sensors.json` / `unit.json` moved; LENZSPITZE, empty,
|
||||
removed); `pump_diff.sh --from 2026-10-12 --to 2026-10-17 JUNGFRAU` and `--from 2026-09-01 --to 2026-09-02`;
|
||||
expected: no DIFF image downloaded again, metrics / images / weather / targets.csv present, `data/` unchanged.
|
||||
- [ ] **Step 5:** `data.py` on `data_diff/JUNGFRAU`: `find_files` without `others` from the extras; metrics,
|
||||
temperature and weather loaded; same counts as `data/JUNGFRAU` on those days.
|
||||
@@ -0,0 +1,86 @@
|
||||
# Pumping the differential images Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** `extraction/pump_diff.py` + `pump_diff.sh` pump a station's `OPTICAL_SHIFT-DIFF` images, their index and
|
||||
the pattern / unit properties into `data_diff/<SERIAL>/`, read-only, without touching `data/` or `pump.py`.
|
||||
|
||||
**Architecture:** One script beside `pump.py`, importing `acquisition_unit`, `root_cause`, `DB_NAMES`, `TYPES`, `UTC`
|
||||
from it; pure range logic and a per-day function testable through stubs; a wrapper copied from `pump.sh` minus
|
||||
`absolute.py` / `weather.py`.
|
||||
|
||||
**Tech Stack:** Python 3.12 in `uv` (sqlalchemy, psycopg2-binary, pandas, boto3, as `pump.sh`), standard `csv` /
|
||||
`json`.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-11-19-pump-diff-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Read-only: `default_transaction_read_only=on`, `statement_timeout=--delay` (default 60 s), as `pump.py`.
|
||||
- Never read or print the `.env`; variable names only.
|
||||
- Image names `<YYYYMMDD>T<HHMM>Z_<measurement_id>.<ext>`, as `pump.py`.
|
||||
- `--min-free-gb` default 50, checked before each day.
|
||||
- Lock `out/pump_diff.lock`, log `out/pump_diff.log`.
|
||||
|
||||
## Review Focus
|
||||
|
||||
- A crash in the middle of a day: no index file for it (temporary file + `os.replace`), so the next run reads it again.
|
||||
- Two measurements in the same minute (bursts): distinct files thanks to the id; both in the index.
|
||||
- A station without acquisition unit or unknown serial: a message, the next station goes on.
|
||||
- No `--from` and nothing indexed: the station is skipped with a message, nothing downloaded.
|
||||
- Disk below the limit: the run stops before the day, with the free space printed.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: range logic and one day, with `--check`
|
||||
|
||||
**Files:**
|
||||
- Create: `extraction/pump_diff.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `indexed_days(folder: Path) -> set[date]`; `days_to_pump(indexed: set[date], start: date | None, end: date)
|
||||
-> list[date] | None` (None: no `--from` and nothing indexed); `pump_day(rows, client, bucket: str, ext: str,
|
||||
folder: Path, station: str, day: date) -> Counter` (rows: the day's measurements, queried by `main`; counts per
|
||||
status, plus `new` and `bytes`); `check()`.
|
||||
|
||||
- [ ] **Step 1:** write `check()` with these assertions (days 1..19 of 2026-11):
|
||||
- `days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)]`;
|
||||
- `days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)]` (before the last: missing only; the
|
||||
last and after: again);
|
||||
- `days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)]`; `days_to_pump(set(), None, d(5)) is None`;
|
||||
- `pump_day` on stub `conn` / `client` with 4 measurements (one image to fetch, one already on disk, one
|
||||
`measurement_uuid` None, one where `get_object` raises): returns `{"ok": 2, "no_uuid": 1, "s3_missing": 1}`, writes
|
||||
one new file, leaves the existing one untouched, and the index has 4 rows with the right `status`, `file` empty for
|
||||
the two missing, `properties` as JSON.
|
||||
- [ ] **Step 2:** `python3 extraction/pump_diff.py --check` fails (functions missing).
|
||||
- [ ] **Step 3:** implement the three functions. `pump_day` queries `id, time_utc, measurement_group_id,
|
||||
measurement_uuid, properties` of `measurement` for the unit, type and UTC day, ordered by time; index columns
|
||||
`measurement_id, time_utc, measurement_group_id, file, status, properties`; written to `index/<S>_diff_<YYYYMMDD>.csv`
|
||||
through `<name>.tmp` and `os.replace` after the images.
|
||||
- [ ] **Step 4:** `--check` prints `ok`.
|
||||
|
||||
### Task 2: database wiring, wrapper, docs, real trial
|
||||
|
||||
**Files:**
|
||||
- Modify: `extraction/pump_diff.py` (add `sensors_json`, `unit_json`, `main`)
|
||||
- Create: `extraction/pump_diff.sh`
|
||||
- Modify: `extraction/README.md`, `README.md`, `.gitignore` (`data_diff/`)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Task 1's functions.
|
||||
- Produces: `sensors_json(conn, unit: int) -> list[dict]` (`id, name, type, time_utc, properties: [{name, value,
|
||||
time_utc, status}]`, every sensor of the unit, archived included); `unit_json(conn, unit: int) -> dict` (`id,
|
||||
serial_number, product_number, properties: [...]`).
|
||||
|
||||
- [ ] **Step 1:** `main()`: arguments `stations*`, `--from`, `--to`, `--delay`, `--min-free-gb`, `--out` (default
|
||||
`<repo>/data_diff`); without stations, the folders of `--out`; engine and S3 client as `pump.py`'s `main`; per
|
||||
station: system, unit, measurement type `OPTICAL_SHIFT-DIFF`, `sensors.json` / `unit.json` written, then each day of
|
||||
`days_to_pump` after the disk check, one line printed per day (counts, MB).
|
||||
- [ ] **Step 2:** `pump_diff.sh` from `pump.sh`: same `.env` load and `uv` packages, its own lock and log, no
|
||||
`absolute.py` / `weather.py`, no default arguments.
|
||||
- [ ] **Step 3:** `--check` still `ok`; `pump_diff.sh -h` shows the help.
|
||||
- [ ] **Step 4:** real trial: `extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-02 JUNGFRAU`; expected: 94
|
||||
images, byte-identical to `data/JUNGFRAU/images_diff/20260901*` (`cmp`), index of the day, `sensors.json`,
|
||||
`unit.json`; `data/` unchanged (`find data -newer` the start mark finds nothing).
|
||||
- [ ] **Step 5:** docs: a `pump_diff` section in `extraction/README.md`, one line in `README.md`, `data_diff/` in
|
||||
`.gitignore`.
|
||||
@@ -0,0 +1,53 @@
|
||||
# run_config Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** `processing/run_config.sh <config> [SERIAL...]` writes `runs/<name>/<SERIAL>/` with the files of `data/`.
|
||||
|
||||
**Architecture:** one script; optical imported from `$OPTICAL`; re-measurement through `reprocess_images.main`;
|
||||
correction through `remove_camera_movement`, per epoch by default.
|
||||
|
||||
**Tech Stack:** uv (numpy, pandas, pillow, pyyaml, scipy, scikit-image, opencv-python-headless).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-11-19-run-config-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Writes only under `runs/<name>/` (guarded); reads `data_diff/`, never `data/` except for the acceptance comparison.
|
||||
- Columns of `data/`'s advanced CSV, in that order: `time_utc, sensor_name, x, y, error_value, error_threshold,
|
||||
relative_x, relative_y, corr_x, corr_y, reproj_err, inlier, sigma0, pattern_type, peak, flux, sharpness, fwhm,
|
||||
dominance, snr, corr_x_mm, corr_y_mm`.
|
||||
- Camera CSV columns: `time_utc`, then `median_/min_/max_` of `pan, tilt, scale, rotation, sigma0`.
|
||||
|
||||
## Review Focus
|
||||
|
||||
- A configuration with a typo (`error_treshold`, unknown sensor): error before any work.
|
||||
- An epoch where only one reference survives: translation only, no camera row (as the cloud).
|
||||
- A day without images in the range: no file for it.
|
||||
- Re-running the same configuration: the station folder replaced, other stations untouched.
|
||||
- optical's working tree dirty: recorded in `run.yaml`.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: configuration and correction, with `--check`
|
||||
|
||||
**Files:** Create `processing/run_config.py`.
|
||||
|
||||
**Interfaces:** `load_config(path) -> dict`; `camera_settings(config, station, unit_props) -> dict`;
|
||||
`cloud_properties(sensors) -> {sid: props}`; `apply_patterns(props, overrides, images_dir) -> {sid: props}`;
|
||||
`correct(meas: DataFrame, props, cam) -> (advanced: DataFrame, camera: DataFrame)`.
|
||||
|
||||
- [ ] `check()`: unknown top / station / camera / pattern key raises; unknown sensor_name raises; camera order station >
|
||||
common > unit > defaults; synthetic correction: three static references, a static target (corr ~0), a target above
|
||||
its threshold (corr empty, x kept), relative = x - pattern x, sigma0 on every row, one camera row per minute.
|
||||
- [ ] Implement; `--check` prints ok.
|
||||
|
||||
### Task 2: the run, the cloud configuration, the acceptance
|
||||
|
||||
**Files:** Modify `processing/run_config.py` (`main`); create `processing/run_config.sh`, `configs/cloud.yaml`;
|
||||
modify `.gitignore` (`runs/`), `README.md`.
|
||||
|
||||
- [ ] `main`: stations, days, patterns, `reprocess_images.main`, `correct`, per-day files, links, `run.yaml`.
|
||||
- [ ] `run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17`; `data.py` reads `runs/cloud/JUNGFRAU`.
|
||||
- [ ] Compare with `data/JUNGFRAU`: corr_x / corr_y (all, and the rejected-reference night of 10-15), camera pose.
|
||||
- [ ] Docs.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Configurations: recomputing the stations' data from the differential images
|
||||
|
||||
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
|
||||
section "data_diff").
|
||||
|
||||
**Replaced the same day by the user's own layout** (the layout below, `configs/` and `runs/<name>/`, is not used):
|
||||
|
||||
```
|
||||
data_diff/<SERIAL>/
|
||||
images_diff/ (index/, sensors.json, unit.json) images/ metrics/ weather/ pumped, the same for every run
|
||||
runs/<run>/ one run: whatever depends on it
|
||||
config.yaml the run's configuration: patterns (by sensor_name), camera and emulator.version (optical's
|
||||
commit, checked), over the cloud's processing
|
||||
displacement/ camera/ targets.csv its result, the files of data/ (targets.csv from its patterns)
|
||||
patterns.json the patterns exactly as given to the emulator, templates included
|
||||
```
|
||||
|
||||
`processing/run_config.sh <SERIAL> <run>` reads the run's `config.yaml` and writes its `displacement/`, `camera/`,
|
||||
`targets.csv` and `patterns.json`, nothing else. Rule: what is not the same for every possible run is in the run's
|
||||
folder. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml`, the cloud's processing written
|
||||
in full (patterns of `sensors.json`, camera, optical's current commit); never over an existing one. Note: the reading tools (aimsight-tools `data.find_files`) read a station folder whole, subfolders
|
||||
included: pointed at `data_diff/<SERIAL>/`, they see every run's curves together.
|
||||
|
||||
## Intent
|
||||
|
||||
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
|
||||
thresholds, references, camera correction), compare them, and let every other repository (aimsight-app, tools,
|
||||
detectors, models, optimizer) read the result exactly as it reads `data/`. `data/` stays as it is: the cloud's
|
||||
version, the reference to compare with.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
aimsight-data/
|
||||
data/ the cloud, unchanged (pump.sh)
|
||||
data_diff/<SERIAL>/ raw material shared by every configuration (pump_diff.sh), out of git
|
||||
images_diff/ the DIFF images; and, invisible to the tools (data.find_files skips images_diff/):
|
||||
index/<SERIAL>_diff_<YYYYMMDD>.csv, sensors.json, unit.json
|
||||
images/ metrics/ weather/ targets.csv as in data/ (targets.csv: the cloud's patterns)
|
||||
configs/<name>.yaml one configuration per file, in git
|
||||
runs/<name>/ the result of a configuration, out of git, the same tree as data/
|
||||
<SERIAL>/
|
||||
displacement/ camera/ targets.csv recomputed, the columns of data/
|
||||
images/ metrics/ weather/ symbolic links to data_diff/<SERIAL>/ (nothing copied)
|
||||
run.yaml the configuration used, optical's commit, the date (outside the station folders:
|
||||
invisible to the tools)
|
||||
diagnostics/ the emulator's raw rows (edge margin, second spot)
|
||||
```
|
||||
|
||||
A component reads a configuration by pointing its `data` link at `~/aimsight-data/runs/<name>` instead of
|
||||
`~/aimsight-data/data`. Checked on 2026-11-19: `data.py` (aimsight-tools) reads through symbolic links (images,
|
||||
metrics, temperature, weather).
|
||||
|
||||
## Configuration
|
||||
|
||||
A configuration states only what differs from the cloud's patterns (`data_diff/<SERIAL>/images_diff/sensors.json`)
|
||||
and from the unit's properties (`unit.json`); the configuration `cloud`, with nothing in it, must reproduce `data/`
|
||||
(the reference: on JUNGFRAU 2026-10-12..16 the emulator matched the cloud within 0.05 px, errors identical).
|
||||
|
||||
```yaml
|
||||
stations:
|
||||
JUNGFRAU:
|
||||
patterns: # by sensor_name, over sensors.json
|
||||
PM_G: {error_threshold: 0.7}
|
||||
BL_C: {corners: [[1700, 929], [1724, 953]]}
|
||||
REF_CT: {pattern_type: target}
|
||||
camera: {focal_distance: 6.3} # over unit.json and the common camera block
|
||||
camera: # common to the stations of the file
|
||||
algorithm: ransac # ransac | danish_hybrid | trusted_danish
|
||||
mode: "off" # off | rotation | rotation_scale | relative
|
||||
```
|
||||
|
||||
Running a configuration again replaces `runs/<name>/`: to compare two settings, two files. No cache between the
|
||||
re-measurement and the correction (431 images in 11 s; about 15 min for a year of one station): added if it gets slow.
|
||||
|
||||
Rejected: a full copy per configuration (the DIFF images duplicated, 0.6 GB per station-day); a configuration
|
||||
suffix in file names (the tools recognise files by their columns and would mix configurations).
|
||||
|
||||
## Work, in order
|
||||
|
||||
1. **`pump_diff`** (this repository's `extraction/`): the extras of the DIFF images move under `images_diff/`; per day,
|
||||
before the DIFF images and their index, the metrics (`pump.py`'s `pump_metrics`) and the IRON images
|
||||
(`pump_images`, `--every`, default 60 min); per station, `targets.csv` (`write_targets`, on the DIFF patterns);
|
||||
at the end, the weather (`weather.py`, given the root folder). A day whose metrics failed gets no index: the next
|
||||
run reads it again. `pump.py`'s only change: `pump_metrics` returns whether every day succeeded.
|
||||
2. **`run_config`**: re-measurement (optical's emulator), camera correction (optical's `remove_camera_movement` and
|
||||
the all-point modes), written in the format of `data/`. Its own design once optical's correction outputs are mapped
|
||||
to the columns of `data/` (`relative_x/y, corr_x/y, reproj_err, inlier, sigma0`, the camera pose).
|
||||
@@ -0,0 +1,67 @@
|
||||
# Pumping the differential images into `data_diff/`
|
||||
|
||||
2026-11-19. Short design, approved in conversation ("A", "go with your recommendations").
|
||||
|
||||
## Intent
|
||||
|
||||
Recompute every station's data from the differential images with configurations of our own (patterns redrawn,
|
||||
thresholds, camera-correction modes), recover the frames the cloud rejected or left uncorrected, and in the end
|
||||
replace `data/`. This design covers only the first of four steps: **pump the raw material**. The next ones
|
||||
(re-measure the images with the device chain of optical, correct the camera motion, write the result in the format of
|
||||
`data/`) get their own designs.
|
||||
|
||||
## Goals, non-goals
|
||||
|
||||
- Every `OPTICAL_SHIFT-DIFF` image of a station over a range, full size, as stored on S3 (patterns may be redrawn later,
|
||||
so no crop).
|
||||
- Everything needed to redo the measurement from them: all the patterns, archived ones included, with every property
|
||||
row as stored (value, time, status); the acquisition unit's properties (focal distance, pixel size, algorithm...);
|
||||
each measurement's exact time, group and properties.
|
||||
- Gaps explained, not only seen: every measurement of the database is listed, with its file or why it has none.
|
||||
- Non-goal: touching `data/` or `pump.py` (the nightly pump stays as it is). Non-goal: scheduling (later, once the
|
||||
history is in).
|
||||
- Anti-goal: filling the disk (about 0.6 GB per station-day at 15 min; 802 GB free on 2026-11-19).
|
||||
|
||||
## Approaches
|
||||
|
||||
- **A, chosen: a separate script `extraction/pump_diff.py` + `pump_diff.sh`.** Imports what it needs from `pump.py`
|
||||
(acquisition unit, error formatting); own lock and log, so a backfill of days does not block the nightly pump; does
|
||||
not run `absolute.py` / `weather.py`.
|
||||
- B: a "diff only" mode in `pump.py`: least code, but touches the nightly pump and shares its lock.
|
||||
- C: optical's `scripts/pcc_emulator/fetch_cloud_images.py`: needs monitoring-dashboard's code and environment,
|
||||
against this repository's rule of no dependency on the app.
|
||||
|
||||
## Design
|
||||
|
||||
Output, `data_diff/<SERIAL>/` (out of git):
|
||||
|
||||
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is, named as in `data/`.
|
||||
- `index/<SERIAL>_diff_<YYYYMMDD>.csv`, one per UTC day: `measurement_id, time_utc` (exact, ISO),
|
||||
`measurement_group_id, file` (empty without image), `status` (`ok`, `no_uuid`: never reached S3, `s3_missing`: S3
|
||||
returned nothing), `properties` (the measurement's JSON). Written after the day's images, through a temporary file:
|
||||
an index on disk means its day was read through.
|
||||
- `sensors.json`: every sensor of the acquisition unit (any type, archived included): `id, name, type, time_utc` and
|
||||
all its property rows `{name, value, time_utc, status}`. Rewritten at each run (the database keeps archived
|
||||
sensors, so nothing is lost by rewriting).
|
||||
- `unit.json`: the acquisition unit (`id, serial_number, product_number`) and its property rows, same shape.
|
||||
|
||||
Runs:
|
||||
|
||||
- `pump_diff.sh --from D [--to E] SERIAL...`: the days of `[D, E)`. A day whose index exists and is before the last
|
||||
indexed day is skipped; the last indexed day and the later ones are read again (a day pumped while it was running
|
||||
is incomplete). An image already on disk is never downloaded again.
|
||||
- `pump_diff.sh [SERIAL...]` without `--from`: from the last indexed day of each station up to now. Without station:
|
||||
the stations already in `data_diff/`. A station with nothing indexed is skipped with a message (its history starts
|
||||
only on an explicit `--from`, so no 2.6 TB by accident).
|
||||
- `--min-free-gb` (default 50): checked before each day; below it, the run stops cleanly.
|
||||
- `--delay` as in `pump.py`; read-only connection; one query per day for the measurements; images one at a time.
|
||||
- Lock `out/pump_diff.lock`, log appended to `out/pump_diff.log`. Same `.env` and `uv` environment as `pump.sh`.
|
||||
|
||||
Check: `python3 extraction/pump_diff.py --check`, without database or S3: the range logic and one day pumped through
|
||||
stubs (an image downloaded, one already there, one without uuid, one refused by S3, the index written).
|
||||
|
||||
## Left to the builder
|
||||
|
||||
- The SQL of the sensors and unit properties (`property` joined to `datatype`, as `pump.py`'s `sensors`).
|
||||
- Whether a pattern's property has several rows over time is to be seen on the first real run; the format keeps them
|
||||
all either way.
|
||||
@@ -0,0 +1,61 @@
|
||||
# run_config: a run of a station recomputed from its differential images
|
||||
|
||||
2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`.
|
||||
|
||||
**Amended the same day (the user's layout, see the configurations design):** `processing/run_config.sh <SERIAL> <run>`
|
||||
reads `data_diff/<SERIAL>/runs/<run>/config.yaml` (`patterns`, `camera`, `emulator.version`: optical's commit, the run
|
||||
stops if `$OPTICAL` is at another one or has local changes; no `stations` level) and writes only that run's
|
||||
`displacement/`, `camera/`, `targets.csv` (from its patterns; none at the station's root any more) and `patterns.json`
|
||||
(the patterns as given to the emulator); the emulator's other files go through a temporary folder. No `runs/<name>/`,
|
||||
no links, no `run.yaml`. The rest below holds (patterns, correction as
|
||||
the cloud, columns, acceptance).
|
||||
|
||||
## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`)
|
||||
|
||||
Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw
|
||||
`x`, `y`, `error` (and the metrics) are stored; targets above their `error_threshold` get `x`/`y` blanked; a rejected
|
||||
`global_ref` gets back the `x`/`y` of the latest stored frame whose error is not above its own (in practice its own
|
||||
raw position, inserted just before), else its pattern position; `remove_camera_movement(r, focal_distance (6.0),
|
||||
pixel_size (0.0024), algorithm (ransac, ransac_threshold 0.99))`, which rejects those references from the fit again
|
||||
(error columns given) but still corrects their rows; `corr -= (x_ref, y_ref)`; `cam_info` (pan, tilt, scale,
|
||||
rotation, sigma0) and `ref_stats` (reproj_err, inlier) stored. The Advanced export adds `relative = x - pattern x`
|
||||
and copies `sigma0` onto every row; the camera export is the per-minute median / min / max of `cam_info`.
|
||||
|
||||
## Design
|
||||
|
||||
`processing/run_config.py`, through `processing/run_config.sh` (uv: numpy, pandas, pillow, pyyaml, scipy,
|
||||
scikit-image, opencv-python-headless; optical's code from `$OPTICAL`, default `~/optical`):
|
||||
|
||||
processing/run_config.sh <config> [SERIAL...] [--from D] [--to E] [--jobs N]
|
||||
|
||||
`<config>`: `configs/<name>.yaml` or a path; stations: those named, else those of the configuration; days: those
|
||||
indexed in `data_diff/<SERIAL>/images_diff/index/` (within `--from`/`--to`). For each station, `runs/<name>/<SERIAL>/`
|
||||
is replaced:
|
||||
|
||||
1. **Patterns**: `images_diff/sensors.json` (DIFF type, a pattern, not archived; latest value of each property), then
|
||||
the configuration by `sensor_name` (an unknown name or key is an error): any property replaced
|
||||
(`error_threshold`, `pattern_type`, `distance_convertion_factor`, `x`, `y`...); `redraw: {image, corners}` cuts the
|
||||
template from that DIFF image as the device's pattern editor does (`estimate_camera_pose` from the references,
|
||||
`saved_anchor_and_reference`: `x_ref` kept, `ImagePatternMono.from_image`). Written to
|
||||
`diagnostics/<SERIAL>_sensor_properties.json`.
|
||||
2. **Re-measurement**: optical's `reprocess_images.main` (emulator) on the days' images, `--times` from the index;
|
||||
rows kept in `diagnostics/<SERIAL>_remeasured.csv`.
|
||||
3. **Correction**: as the cloud, one measurement at a time (`per_epoch: true`, default): targets above their
|
||||
threshold blanked, references kept raw (rejected in the fit by `remove_camera_movement`), `corr -= x_ref`;
|
||||
`per_epoch: false`: the whole series in one call (a rejected reference interpolated in time, algorithms with
|
||||
memory). Camera block: `algorithm` (ransac), `ransac_threshold` (0.99, ransac only), `mode` (`off` | `rotation` |
|
||||
`rotation_scale` | `relative`, optical's `refine_allpoints`), `focal_distance`, `pixel_size` (unit.json, else 6.0 /
|
||||
0.0024), `per_epoch`; station block over common block over unit.json over the cloud's defaults.
|
||||
4. **Output**, the files of `data/`: `displacement/<S>_advanced_<YYYYMMDD>.csv` (columns of `data/`: `x, y,
|
||||
error_value, peak..snr` from the emulator, `relative = x - pattern x`, `corr` from the correction, `reproj_err,
|
||||
inlier` of the references, `sigma0` of the epoch on every row, `corr_*_mm = corr * distance_convertion_factor`),
|
||||
`camera/<S>_camera_<YYYYMMDD>.csv` (median / min / max per minute), `targets.csv`; `images/`, `metrics/`,
|
||||
`weather/` as relative symbolic links to `data_diff/<SERIAL>/`; `run.yaml` at `runs/<name>/` (configuration as
|
||||
resolved per station, optical's commit and state, days, date).
|
||||
|
||||
Acceptance: `configs/cloud.yaml` (empty) on JUNGFRAU 2026-10-12..16 against `data/JUNGFRAU`: positions and errors
|
||||
within 0.05 px (done by the emulator), `corr_x/y` and the camera pose close to the cloud, the differences counted.
|
||||
|
||||
Check: `run_config.py --check` in the uv environment: the configuration rules (unknown keys, order of the camera
|
||||
blocks, overrides by name) and a synthetic correction (three references, two targets, one rejected) through
|
||||
optical's `remove_camera_movement`.
|
||||
+33
-5
@@ -2,23 +2,25 @@
|
||||
|
||||
`pump.py` fetches, for each station (serial number of the system) and each UTC day asked, into `data/<SERIAL>/`:
|
||||
- the **system metrics** (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in `metrics/`;
|
||||
- the **displacements** ("advanced" export), one CSV per day in `displacement/`;
|
||||
- the **displacements** ("advanced" export), one CSV per day in `displacement/`, with `corr_x_mm` / `corr_y_mm` = `corr_x` / `corr_y` × `mm_per_px` of the sensor of each row (by `sensor_id`, never by name; empty without a factor); a day pumped before keeps its file: `--from D --to E --replace` pumps it again with the mm;
|
||||
- the **sensors** measured, targets and references (every sensor with a `pattern_type`), in `targets.csv`: `pattern_type`, `distance_mm` (from the camera; 1000 = not filled in on the platform) and `mm_per_px` (`distance_convertion_factor`, the factor of the dashboard's exports in mm), their current values, read again at each pumping; a name not measured in the run keeps its line;
|
||||
- the **camera movements** (`pan`, `tilt`, `scale`, `rotation`, `sigma0`), as median and bounds (`median_`, `min_`, `max_`) per minute, only at the times the camera was realigned, one CSV per day in `camera/`;
|
||||
- the **weather** (`weather.py`, after the pumping), for a station with a position in `stations.txt`: the hourly MeteoSwiss measurements nearest to it, one CSV per day with data, in `weather/` (`weather/README.md`);
|
||||
- the IRON **images** (`--images`, one per hour by default, `--every 10` or `--every 0` for more) in `images/`, and the differential ones (`--diff`) in `images_diff/`.
|
||||
|
||||
Formats: `~/aimsight-labeling/knowledge/displacement-columns.md`.
|
||||
|
||||
**Tracked stations**: `stations.txt`, one serial number per line. Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in `data/` gets the last 30 days.
|
||||
**Tracked stations**: `stations.txt`, one serial number per line, optionally followed by the position of the station (`lat,lon` or a Swiss place, for the weather pumped into `weather/`; MeteoSwiss covers Switzerland only; the app edits this list: page Stations, through `pumping.py`). Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in `data/` gets the last 30 days.
|
||||
|
||||
Resuming: with `--from`, a file already in `data/<SERIAL>/` is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without `--from` replaces it.
|
||||
|
||||
## Running (under WSL)
|
||||
|
||||
`pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `lenzspitze_absolute.py`. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump.
|
||||
`pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `absolute.py`, then `weather.py` (every station with a position: its missing days, those written for another position, and the last 2 days, fetched again; a place not found or MeteoSwiss unreachable: a line in the log, the pumping goes on). `python3 extraction/weather.py [SERIAL…]` alone fills the weather of the days already pumped. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump.
|
||||
|
||||
Variables of the `.env`: `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT`; for the images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`.
|
||||
|
||||
1. **A trial** (1 station, 2 days, in a throwaway folder): `extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND` (6 s on 2026-11-12). `lenzspitze_absolute.py` then converts `data/LENZSPITZE`, not the `--out` folder: no effect if `data/LENZSPITZE` is already converted.
|
||||
1. **A trial** (1 station, 2 days, in a throwaway folder): `extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND` (6 s on 2026-11-12). `absolute.py` then converts `data/`, not the `--out` folder: no effect on what is already converted.
|
||||
2. **Updating the tracked stations**: `extraction/pump.sh` (without argument: `--images --delay 300`), or the VS Code task **pump**. The script says, for each station, the range it pumps.
|
||||
3. **A chosen range** (history, failed day), with 3 more days before (the history of the detectors): `extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images <SERIAL>` (`--to` excluded; several stations: `<SERIAL1> <SERIAL2>`). Help: `extraction/pump.sh -h`.
|
||||
|
||||
@@ -37,6 +39,10 @@ Read-only: dedicated engine, `default_transaction_read_only`, 60 s timeout per q
|
||||
|
||||
No dependency on the `monitoring-dashboard` app: its reads are copied into `pump.py`, in SQL (its docstring says which ones, and from which commit). If the app changes its reads, the script does not follow: compare again files pumped by both.
|
||||
|
||||
## For a caller
|
||||
|
||||
`pumping.py` (standard library): `tracked()` and `save(rows)` (the list), `start(stations, start, end, images)` (`pump.sh` in the background, `--delay 300`), `running()` (the lock held) and `log()` (the end of `out/pump.log`); a refused request raises `ValueError`. Its check: `python3 extraction/pumping.py`. The page Stations of aimsight-app calls it.
|
||||
|
||||
## With Claude
|
||||
|
||||
Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small ranges; a long range or a history, outside busy hours. It never reads the `.env`.
|
||||
@@ -48,4 +54,26 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small
|
||||
- A day failed on a timeout (`QueryCanceled: … statement timeout`): run it again alone with `--delay 300`. The slowness comes from the database, not from the volume.
|
||||
- An hour without an image can have its measurement: the image never reached S3 (empty `measurement_uuid`, unfinished upload). The script skips it and takes the next image of the slot if there is one.
|
||||
- A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in `data/<SERIAL>/` and run again.
|
||||
- LENZSPITZE: its exports have relative `x` / `y` and an empty `pattern_type`; `pump.sh` then runs `lenzspitze_absolute.py` (provisional; details in `~/aimsight-labeling/knowledge/displacement-columns.md`).
|
||||
- Relative `x` / `y` (LENZSPITZE, SUSTENHORN, NADELHORN `c*_mon`): after each pumping, `pump.sh` runs `absolute.py`, which converts every sensor whose `x` / `y` medians are under 100 px, from `data/<SERIAL>/*_patterns.json` and the offset of the station (only LENZSPITZE has both today); a relative sensor it cannot place is dropped (details in `~/aimsight-labeling/knowledge/displacement-columns.md`).
|
||||
|
||||
## The differential images, into `data_diff/` (`pump_diff.sh`)
|
||||
|
||||
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the runs, `data_diff/<SERIAL>/runs/<run>/`).
|
||||
|
||||
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is;
|
||||
- `images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON);
|
||||
- `images_diff/sensors.json`: every sensor of the unit, archived included, with all its property rows (pattern, corners, x, y, error_threshold, distance_convertion_factor, start_date…);
|
||||
- `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm);
|
||||
- (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder)
|
||||
- `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`;
|
||||
- `displacement/`, `camera/` and `targets.csv` are not pumped: they depend on the run (its patterns, its settings), each run writes its own in `runs/<run>/` (`processing/run_config.sh`), never from the cloud; a pumping never touches `runs/`.
|
||||
|
||||
```bash
|
||||
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here
|
||||
extraction/pump_diff.sh JUNGFRAU # from its last indexed day up to now
|
||||
extraction/pump_diff.sh # the same for every station of data_diff/
|
||||
```
|
||||
|
||||
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`.
|
||||
|
||||
Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.
|
||||
@@ -0,0 +1,92 @@
|
||||
"""x / y of the advanced exports as absolute positions in the image, on every station. Run by pump.sh after each pumping.
|
||||
|
||||
A sensor of a file is relative when the medians of its x and y are both under RELATIVE_PX (relative: up to ~40 px seen,
|
||||
NADELHORN c*_mon, SUSTENHORN, LENZSPITZE; absolute: 429 px and more). Rewrites in place data/<STATION>/displacement/*.csv:
|
||||
x = x_pattern + DX + x_csv, y = y_pattern + DY + y_csv, where (x_pattern, y_pattern) is the centre of the pattern in
|
||||
data/<STATION>/*_patterns.json and (DX, DY) the offset between the frame of the JSON and the IRON images (OFFSETS);
|
||||
an empty pattern_type is filled from the JSON.
|
||||
A relative sensor without a position (no JSON, no offset, missing from the JSON) is dropped: its rows are removed.
|
||||
A file with no relative sensor is left as it is (already converted). Pumping again restores the originals.
|
||||
|
||||
Usage: python3 extraction/absolute.py [DATA] (default: the repository's data/)
|
||||
python3 extraction/absolute.py --check (a check of the conversion, without touching data/)
|
||||
"""
|
||||
import csv
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import statistics
|
||||
import sys
|
||||
|
||||
RELATIVE_PX = 100
|
||||
# Measured on 2026-11-04 on LENZSPITZE: median (spot − pattern of the JSON) over 9 night images from 17.10 to 03.11, 25
|
||||
# to 34 matched targets each, stable within 0.5 px. Includes the constant offset spot − (x, y) already seen on NORDEND
|
||||
# (displacement-columns.md). A station with a JSON but no offset here: its relative sensors are dropped.
|
||||
OFFSETS = {"LENZSPITZE": (330.5, 15.8)}
|
||||
|
||||
|
||||
def relative_sensors(rows):
|
||||
xy = {}
|
||||
for r in rows:
|
||||
if r["x"] and r["y"]:
|
||||
xy.setdefault(r["sensor_name"], []).append((float(r["x"]), float(r["y"])))
|
||||
return {s for s, v in xy.items()
|
||||
if max(abs(statistics.median(p[0] for p in v)), abs(statistics.median(p[1] for p in v))) < RELATIVE_PX}
|
||||
|
||||
|
||||
def convert(rows, patterns, offset):
|
||||
"""-> (rows with absolute x / y, relative sensors dropped); None if nothing is relative."""
|
||||
relative = relative_sensors(rows)
|
||||
if not relative:
|
||||
return None
|
||||
out = []
|
||||
for r in rows:
|
||||
if r["sensor_name"] in relative:
|
||||
m = patterns.get(r["sensor_name"].strip())
|
||||
if m is None or offset is None:
|
||||
continue
|
||||
r["pattern_type"] = r["pattern_type"] or m.get("pattern_type", "")
|
||||
if r["x"]:
|
||||
r["x"] = repr(m["x"] + offset[0] + float(r["x"]))
|
||||
if r["y"]:
|
||||
r["y"] = repr(m["y"] + offset[1] + float(r["y"]))
|
||||
out.append(r)
|
||||
return out
|
||||
|
||||
|
||||
def station(folder):
|
||||
name = os.path.basename(folder)
|
||||
patterns = {}
|
||||
for path in glob.glob(os.path.join(folder, "*_patterns.json")):
|
||||
patterns.update({str(v["sensor_name"]): v for v in json.load(open(path)).values()})
|
||||
offset = OFFSETS.get(name) if patterns else None
|
||||
for path in sorted(glob.glob(os.path.join(folder, "displacement", "*.csv"))):
|
||||
with open(path, newline="") as f:
|
||||
reader = csv.DictReader(f)
|
||||
fields, rows = reader.fieldnames, list(reader)
|
||||
if not fields or "x" not in fields:
|
||||
continue # an empty day
|
||||
out = convert(rows, patterns, offset)
|
||||
if out is None:
|
||||
continue
|
||||
with open(path, "w", newline="") as f:
|
||||
w = csv.DictWriter(f, fieldnames=fields)
|
||||
w.writeheader()
|
||||
w.writerows(out)
|
||||
print(f"{name} {os.path.basename(path)}: {len(out)} rows kept, {len(rows) - len(out)} relative dropped")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if sys.argv[1:] == ["--check"]:
|
||||
def row(s, x, y, t=""):
|
||||
return {"sensor_name": s, "x": x, "y": y, "pattern_type": t}
|
||||
rows = [row("a", "1.5", "-2", ""), row("a", "", "", ""), row("b", "0.1", "0.2"), row("c", "800", "600", "ref")]
|
||||
out = convert(rows, {"a": {"x": 100, "y": 200, "pattern_type": "target"}}, (10, 20))
|
||||
assert out == [row("a", "111.5", "218.0", "target"), row("a", "", "", "target"), row("c", "800", "600", "ref")], out
|
||||
assert convert(out, {}, None) is None # already absolute: left as it is
|
||||
assert convert([row("b", "0.1", "0.2")], {"b": {"x": 1, "y": 1}}, None) == [] # no offset: dropped
|
||||
print("ok")
|
||||
sys.exit()
|
||||
data = sys.argv[1] if len(sys.argv) > 1 else os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "data")
|
||||
for folder in sorted(glob.glob(os.path.join(data, "*", ""))):
|
||||
station(folder.rstrip(os.sep))
|
||||
@@ -1,49 +0,0 @@
|
||||
"""Provisional, LENZSPITZE only: x / y of the advanced exports as absolute positions in the image, as on the other stations.
|
||||
|
||||
On LENZSPITZE, x / y are relative to the reference position of the pattern (medians ~0 px). This script rewrites in place
|
||||
data/LENZSPITZE/displacement/*.csv:
|
||||
x = x_pattern + DX + x_csv, y = y_pattern + DY + y_csv, where (x_pattern, y_pattern) is the centre of the pattern in
|
||||
data/LENZSPITZE/lentzpitze_patterns.json and (DX, DY) the offset between the frame of the JSON and the IRON images;
|
||||
pattern_type, empty in these exports, is filled from the JSON.
|
||||
The sensors missing from the JSON (1023, 8019): x / y emptied (position unknown); 8019 gets global_ref (it has reproj_err).
|
||||
A file whose pattern_type is already filled is left as it is (already converted). Pumping again restores the originals.
|
||||
|
||||
Usage: python3 extraction/lenzspitze_absolute.py
|
||||
"""
|
||||
import csv
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
|
||||
STATION = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "data", "LENZSPITZE")
|
||||
# Measured on 2026-11-04: median (spot − pattern of the JSON) over 9 night images from 17.10 to 03.11, 25 to 34 matched
|
||||
# targets each, stable within 0.5 px. Includes the constant offset spot − (x, y) already seen on NORDEND (displacement-columns.md).
|
||||
DX, DY = 330.5, 15.8
|
||||
TYPE_WITHOUT_PATTERN = {"8019": "global_ref"} # reproj_err and inlier filled: reference of the compensation
|
||||
|
||||
patterns = {str(v["sensor_name"]): v for v in json.load(open(os.path.join(STATION, "lentzpitze_patterns.json"))).values()}
|
||||
|
||||
for path in sorted(glob.glob(os.path.join(STATION, "displacement", "*.csv"))):
|
||||
with open(path, newline="") as f:
|
||||
reader = csv.DictReader(f)
|
||||
fields, rows = reader.fieldnames, list(reader)
|
||||
if any(r["pattern_type"] for r in rows):
|
||||
print(f"{os.path.basename(path)}: already converted")
|
||||
continue
|
||||
for r in rows:
|
||||
s = r["sensor_name"].strip()
|
||||
m = patterns.get(s)
|
||||
if m is None:
|
||||
r["x"] = r["y"] = ""
|
||||
r["pattern_type"] = TYPE_WITHOUT_PATTERN.get(s, "")
|
||||
continue
|
||||
r["pattern_type"] = m["pattern_type"]
|
||||
if r["x"]:
|
||||
r["x"] = repr(m["x"] + DX + float(r["x"]))
|
||||
if r["y"]:
|
||||
r["y"] = repr(m["y"] + DY + float(r["y"]))
|
||||
with open(path, "w", newline="") as f:
|
||||
w = csv.DictWriter(f, fieldnames=fields)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
print(f"{os.path.basename(path)}: {len(rows)} rows converted")
|
||||
+137
-19
@@ -14,11 +14,15 @@ from the last 30 days if it has none; without `--to`: up to now.
|
||||
Output (default: the repository's `data/`, next to this folder):
|
||||
<STATION>/metrics/<STATION>_<metric>_<YYYY_MM_DD>_00_00 to <YYYY_MM_DD>_23_00.csv system metrics, per day
|
||||
<STATION>/displacement/<STATION>_advanced_<YYYYMMDD>.csv displacements, one file per day
|
||||
+ corr_x_mm, corr_y_mm: corr * mm_per_px of the sensor that measured (references included)
|
||||
<STATION>/targets.csv the targets and references measured: sensor_name, pattern_type,
|
||||
distance_mm, mm_per_px (those of the last day pumped)
|
||||
<STATION>/camera/<STATION>_camera_<YYYYMMDD>.csv camera movements, per day
|
||||
<STATION>/images/<YYYYMMDD>T<HHMM>Z_<id>.<ext> OPTICAL_SHIFT-IRON images (--images)
|
||||
<STATION>/images_diff/… differential images (--diff)
|
||||
Resumable: with `--from`, a file already written is not read again, whatever the dates asked; without `--from`, the
|
||||
days of the range are replaced (images excepted). Days in UTC, `--to` excluded; the current day is pumped as it is,
|
||||
Resumable: with `--from`, a file already written is not read again, whatever the dates asked, except from the last
|
||||
day pumped of the station (pumped during the day, it was incomplete); without `--from`, the days of the range are
|
||||
replaced (images excepted). Days in UTC, `--to` excluded; the current day is pumped as it is,
|
||||
its metrics up to the last completed hour. To redo a day, delete its files.
|
||||
|
||||
No dependency on the app (~/monitoring-dashboard): its reads are copied here, in SQL, as they are (branch `explo`,
|
||||
@@ -42,8 +46,9 @@ import pandas as pd
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.engine import URL
|
||||
|
||||
import pumping # the tracked stations (stations.txt), next to this script
|
||||
|
||||
UTC = timezone.utc
|
||||
STATIONS = Path(__file__).resolve().parent / "stations.txt" # the tracked stations, one per line
|
||||
NEW_STATION_DAYS = 30 # without --from, a station with no pumped displacements: the last 30 days
|
||||
DB_NAMES = ["AIMSIGHT_DATABASE_USER", "AIMSIGHT_DATABASE_PSWD", "AIMSIGHT_DATABASE_TURL", "AIMSIGHT_DATABASE_NAME"]
|
||||
TYPES = {"images": "OPTICAL_SHIFT-IRON", "images_diff": "OPTICAL_SHIFT-DIFF"} # folder -> measurement type
|
||||
@@ -82,13 +87,27 @@ def acquisition_unit(conn, system_id):
|
||||
return pick[0].id if pick else None
|
||||
|
||||
|
||||
def pumped_days(folder):
|
||||
"""The days of displacements already pumped."""
|
||||
return {datetime.strptime(p.stem.rsplit("_", 1)[1], "%Y%m%d").date()
|
||||
for p in (folder / "displacement").glob(f"{folder.name}_advanced_*.csv")}
|
||||
|
||||
|
||||
def given_ranges(folder, start, end):
|
||||
"""With --from, without --replace: (start, end, replace) to pump. The missing days of the range, and from the last
|
||||
day pumped (pumped during the day, it was incomplete) to the end of the range pumped again, as update_ranges."""
|
||||
last = max(pumped_days(folder), default=None)
|
||||
if last is None or not start <= last < end:
|
||||
return [(start, end, False)]
|
||||
return ([(start, last, False)] if start < last else []) + [(last, end, True)]
|
||||
|
||||
|
||||
def update_ranges(folder, end, tomorrow):
|
||||
"""Without --from: (start, end, replace) to pump, from the days of displacements already pumped. The missing days
|
||||
between the first and the last (ranges pumped separately, failed day) are filled without touching the others; the
|
||||
last day is pumped again (pumped during the day, it was incomplete) up to now; a station with no pumped
|
||||
displacements gets the last 30 days."""
|
||||
days = {datetime.strptime(p.stem.rsplit("_", 1)[1], "%Y%m%d").date()
|
||||
for p in (folder / "displacement").glob(f"{folder.name}_advanced_*.csv")}
|
||||
days = pumped_days(folder)
|
||||
if not days:
|
||||
return [(tomorrow - timedelta(days=NEW_STATION_DAYS + 1), end, True)]
|
||||
# a failed metrics day has no file at all (one query for all of them): it is a gap too; a station without any
|
||||
@@ -105,10 +124,12 @@ def pump_metrics(conn, system_id, station, start, end, out, replace):
|
||||
"""Like the download of the System page (Dashboard/System.py, get_metrics(..., bucketing="1 hour")): per hour,
|
||||
mean * transform_factor + transform_offset, or the list of values of a text metric; the labelled series of one
|
||||
metric merged; an empty row for an hour without measurement. get_metrics counts in hours before now: its query is
|
||||
redone here on a fixed day, one query per day for all the metrics."""
|
||||
redone here on a fixed day, one query per day for all the metrics. -> True when no day failed (pump_diff.py: a
|
||||
day whose metrics failed is not marked done)."""
|
||||
names = sorted(conn.execute(text("SELECT DISTINCT dt.name FROM datatype dt JOIN metric_series se "
|
||||
"ON se.datatype_id = dt.id WHERE se.system_id = :s"), {"s": system_id}).scalars())
|
||||
conn.rollback()
|
||||
ok = True
|
||||
for k in range((end - start).days):
|
||||
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
|
||||
paths = {n: out / station / "metrics"
|
||||
@@ -129,6 +150,7 @@ def pump_metrics(conn, system_id, station, start, end, out, replace):
|
||||
GROUP BY 1, dt.name""", s=system_id, a=a, b=a + timedelta(days=1))
|
||||
except Exception as e: # ponytail: like the displacements, a failed day is skipped, a new run picks it up
|
||||
print(f"{station} {a:%Y-%m-%d} metrics: failed ({root_cause(e)}), run again")
|
||||
ok = False
|
||||
continue
|
||||
finally:
|
||||
conn.rollback()
|
||||
@@ -144,6 +166,7 @@ def pump_metrics(conn, system_id, station, start, end, out, replace):
|
||||
pd.DataFrame({"time_utc": hours, "metric": col.reindex(hours).to_numpy()}).to_csv(path)
|
||||
print(f"{station} {a:%Y-%m-%d} metrics: {len(names)} files, {df['name'].nunique()} with data, "
|
||||
f"{time.time() - t0:.0f} s")
|
||||
return ok
|
||||
|
||||
|
||||
# --- the "advanced" export: build_advanced_frame (app/datapoints.py) ---
|
||||
@@ -174,7 +197,7 @@ def prepare_raw(shifts):
|
||||
"""prepare_datapoints_frame: sigma0 (camera rows, without sensor_name) copied onto each sensor at its time;
|
||||
relative = x - prop_x; error -> error_value; the full schema."""
|
||||
if shifts.empty:
|
||||
return pd.DataFrame(columns=COLUMNS)
|
||||
return pd.DataFrame(columns=COLUMNS + ["sensor_id"])
|
||||
sig = shifts.loc[shifts["sigma0"].notna(), ["time_utc", "sigma0"]]
|
||||
sigma0 = sig.groupby("time_utc")["sigma0"].median() if not sig.empty else None
|
||||
shifts = shifts.dropna(subset=["sensor_name"]).reset_index(drop=True)
|
||||
@@ -187,7 +210,7 @@ def prepare_raw(shifts):
|
||||
for c in NUMERIC:
|
||||
if c in shifts:
|
||||
shifts[c] = pd.to_numeric(shifts[c], errors="coerce")
|
||||
return shifts.reindex(columns=COLUMNS)
|
||||
return shifts.reindex(columns=COLUMNS + ["sensor_id"])
|
||||
|
||||
|
||||
def aggregate_states(conn, unit, a, b):
|
||||
@@ -196,7 +219,7 @@ def aggregate_states(conn, unit, a, b):
|
||||
In the states, x_pct holds the CORRECTED x and rx_pct the raw x."""
|
||||
thr = "COALESCE((th.value #>> '{}')::double precision, CAST(:thr AS double precision))"
|
||||
sensor = query(conn, f"""
|
||||
SELECT ss.time_utc, COALESCE(sn.value #>> '{{}}', s.name) AS name,
|
||||
SELECT ss.time_utc, ss.sensor_id, COALESCE(sn.value #>> '{{}}', s.name) AS name,
|
||||
approx_percentile(0.5, ss.x_pct) AS corr_x, approx_percentile(0.5, ss.y_pct) AS corr_y,
|
||||
approx_percentile(0.5, ss.rx_pct) AS x, approx_percentile(0.5, ss.ry_pct) AS y,
|
||||
approx_percentile(0.5, ss.error_pct) AS error, {thr} AS error_threshold,
|
||||
@@ -214,11 +237,12 @@ def aggregate_states(conn, unit, a, b):
|
||||
WHERE s.acquisition_unit_id = :au AND ss.time_utc >= :a AND ss.time_utc <= :b
|
||||
ORDER BY ss.time_utc""", au=unit, a=a, b=b, thr=DEFAULT_ERROR_THRESHOLD)
|
||||
if sensor.empty:
|
||||
return pd.DataFrame(columns=COLUMNS)
|
||||
return pd.DataFrame(columns=COLUMNS + ["sensor_id"])
|
||||
common = query(conn, """
|
||||
SELECT sc.time_utc, approx_percentile(0.5, sc.sigma0_pct) AS sigma0 FROM shift_common_1min sc
|
||||
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b""", au=unit, a=a, b=b)
|
||||
out = pd.DataFrame({"time_utc": pd.to_datetime(sensor["time_utc"]), "sensor_name": sensor["name"].astype(str)})
|
||||
out = pd.DataFrame({"time_utc": pd.to_datetime(sensor["time_utc"]), "sensor_name": sensor["name"].astype(str),
|
||||
"sensor_id": sensor["sensor_id"]})
|
||||
for col, src in [("error_value", "error"), ("error_threshold", "error_threshold"), ("corr_x", "corr_x"),
|
||||
("corr_y", "corr_y"), ("relative_x", "corr_x"), ("relative_y", "corr_y"), ("x", "x"), ("y", "y"),
|
||||
("reproj_err", "reproj_err")] + [(m, m) for m in QUALITY]:
|
||||
@@ -231,7 +255,7 @@ def aggregate_states(conn, unit, a, b):
|
||||
sig = common.loc[common["sigma0"].notna()]
|
||||
out["sigma0"] = (out["time_utc"].map(sig.assign(time_utc=pd.to_datetime(sig["time_utc"]))
|
||||
.groupby("time_utc")["sigma0"].median()) if not sig.empty else float("nan"))
|
||||
return out.reindex(columns=COLUMNS)
|
||||
return out.reindex(columns=COLUMNS + ["sensor_id"])
|
||||
|
||||
|
||||
def merge(raw, agg):
|
||||
@@ -255,11 +279,42 @@ def merge(raw, agg):
|
||||
t = utc(agg["time_utc"])
|
||||
agg["time_utc"] = t.dt.tz_convert(tz) if tz is not None else t.dt.tz_localize(None)
|
||||
return (pd.concat([raw, agg], ignore_index=True).sort_values(["sensor_name", "time_utc"], kind="stable")
|
||||
.reset_index(drop=True).reindex(columns=COLUMNS))
|
||||
.reset_index(drop=True).reindex(columns=COLUMNS + ["sensor_id"]))
|
||||
|
||||
|
||||
SENSOR_PROPS = {"pattern_type": "pattern_type", "distance_mm": "distance_mm", "distance_convertion_factor": "mm_per_px"}
|
||||
MM = {"corr_x": "corr_x_mm", "corr_y": "corr_y_mm"}
|
||||
|
||||
|
||||
def sensors(conn, unit):
|
||||
"""The sensors of the unit that have a pattern_type (targets and references), one row per sensor_id: sensor_name,
|
||||
pattern_type, distance_mm (from the camera; 1000: not filled in on the platform) and mm_per_px
|
||||
(distance_convertion_factor, the factor of the dashboard's exports in mm: DeviceOutput.get_scale,
|
||||
compute_output_signals, (corr - offset) * factor). A name may have several sensors (an older configuration of the
|
||||
unit, another lens): the factor of a measurement is that of its sensor_id, never looked up by name."""
|
||||
prop = "max(p.value #>> '{{}}') FILTER (WHERE dt.name = '{}')".format
|
||||
df = query(conn, f"""
|
||||
SELECT s.id AS sensor_id, COALESCE({prop('sensor_name')}, s.name) AS sensor_name,
|
||||
{", ".join(f"{prop(n)} AS {c}" for n, c in SENSOR_PROPS.items())}
|
||||
FROM sensor s JOIN property p ON p.sensor_id = s.id JOIN datatype dt ON dt.id = p.datatype_id
|
||||
WHERE s.acquisition_unit_id = :au
|
||||
GROUP BY s.id, s.name HAVING {prop('pattern_type')} IS NOT NULL""", au=unit)
|
||||
for c in ("distance_mm", "mm_per_px"):
|
||||
df[c] = pd.to_numeric(df[c], errors="coerce")
|
||||
return df.set_index("sensor_id")
|
||||
|
||||
|
||||
def with_mm(df, props):
|
||||
"""corr_x_mm / corr_y_mm = corr_x / corr_y * mm_per_px of the sensor_id of each row; empty without a factor."""
|
||||
factor = df["sensor_id"].map(props["mm_per_px"]).astype(float)
|
||||
for c, mm in MM.items():
|
||||
df[mm] = pd.to_numeric(df[c], errors="coerce") * factor
|
||||
return df
|
||||
|
||||
|
||||
def advanced(conn, unit, a, b):
|
||||
return merge(prepare_raw(raw_datapoints(conn, unit, a, b)), aggregate_states(conn, unit, a, b))
|
||||
df = merge(prepare_raw(raw_datapoints(conn, unit, a, b)), aggregate_states(conn, unit, a, b))
|
||||
return with_mm(df, sensors(conn, unit))
|
||||
|
||||
|
||||
def camera(conn, unit, a, b):
|
||||
@@ -278,7 +333,9 @@ DAILY_FILES = (("displacement", "advanced", advanced), ("camera", "camera", came
|
||||
|
||||
|
||||
def pump_displacements(conn, unit, station, start, end, out, replace):
|
||||
"""Displacements and camera movements: one file and one query per day, bounded, under the timeout (--delay)."""
|
||||
"""Displacements and camera movements: one file and one query per day, bounded, under the timeout (--delay)
|
||||
-> {sensor_name: sensor_id} measured, the last day pumped winning (for targets.csv)."""
|
||||
seen = {}
|
||||
for k in range((end - start).days):
|
||||
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
|
||||
for folder, name, reader in DAILY_FILES:
|
||||
@@ -295,9 +352,25 @@ def pump_displacements(conn, unit, station, start, end, out, replace):
|
||||
conn.rollback() # closes the read transaction
|
||||
if "sensor_name" in df:
|
||||
df = df.drop_duplicates(["sensor_name", "time_utc"])
|
||||
if "sensor_id" in df:
|
||||
seen.update(df.dropna(subset=["sensor_id"]).groupby("sensor_name")["sensor_id"].last().to_dict())
|
||||
df = df.drop(columns="sensor_id")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
df.to_csv(path, index=False) # an empty day is written too: the next run does not ask for it again
|
||||
print(f"{station} {a:%Y-%m-%d} {name}: {len(df)} rows, {time.time() - t0:.0f} s -> {path.name}")
|
||||
return seen
|
||||
|
||||
|
||||
def write_targets(folder, props, seen):
|
||||
"""targets.csv: the properties of the sensors measured (seen: {name: sensor_id}); the names not measured in this
|
||||
run keep their line."""
|
||||
path = folder / "targets.csv"
|
||||
new = props.loc[[i for i in seen.values() if i in props.index], ["sensor_name", *SENSOR_PROPS.values()]]
|
||||
old = pd.read_csv(path) if path.exists() else new.iloc[:0]
|
||||
t = pd.concat([old[~old["sensor_name"].astype(str).isin(new["sensor_name"])], new]).sort_values("sensor_name")
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
t.to_csv(path, index=False)
|
||||
return t
|
||||
|
||||
|
||||
def pump_images(conn, s3, unit, station, start, end, folder, every, out):
|
||||
@@ -351,6 +424,8 @@ def main():
|
||||
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD; without: the last pumped day of each "
|
||||
"station, pumped again, or the last 30 days")
|
||||
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD; without: up to now")
|
||||
p.add_argument("--replace", action="store_true", help="with --from: the days already pumped are pumped again "
|
||||
"(a file is replaced only once its day is read)")
|
||||
p.add_argument("--images", action="store_true", help="also the OPTICAL_SHIFT-IRON images")
|
||||
p.add_argument("--diff", action="store_true", help="also the OPTICAL_SHIFT-DIFF images")
|
||||
p.add_argument("--every", type=int, default=60, help="one image per slot of N minutes, 0 = all (default 60)")
|
||||
@@ -361,7 +436,7 @@ def main():
|
||||
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
|
||||
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
|
||||
end = min(datetime.strptime(a.end, "%Y-%m-%d").date(), tomorrow) if a.end else tomorrow # no day to come
|
||||
stations = a.stations or STATIONS.read_text().split()
|
||||
stations = a.stations or list(pumping.tracked())
|
||||
|
||||
missing = [n for n in DB_NAMES if not os.environ.get(n)]
|
||||
if missing:
|
||||
@@ -387,9 +462,11 @@ def main():
|
||||
if system_id is None:
|
||||
print(f"{station}: unknown system")
|
||||
continue
|
||||
ranges = [(start, end, False)] if start else update_ranges(a.out / station, end, tomorrow)
|
||||
ranges = ([(start, end, True)] if start and a.replace else given_ranges(a.out / station, start, end)
|
||||
if start else update_ranges(a.out / station, end, tomorrow))
|
||||
unit = acquisition_unit(conn, system_id)
|
||||
conn.rollback()
|
||||
seen = {}
|
||||
for range_start, range_end, replace in ranges:
|
||||
print(f"{station}: from {range_start} to {range_end - timedelta(days=1)} inclusive"
|
||||
+ ("" if replace else ", missing days only"))
|
||||
@@ -397,13 +474,54 @@ def main():
|
||||
if unit is None:
|
||||
print(f"{station}: no recognised acquisition unit, no displacements, camera or images")
|
||||
continue
|
||||
pump_displacements(conn, unit, station, range_start, range_end, a.out, replace)
|
||||
seen.update(pump_displacements(conn, unit, station, range_start, range_end, a.out, replace))
|
||||
for flag, folder in ((a.images, "images"), (a.diff, "images_diff")):
|
||||
if flag:
|
||||
pump_images(conn, s3, unit, station, range_start, range_end, folder, a.every, a.out)
|
||||
if not seen:
|
||||
continue
|
||||
try:
|
||||
props = sensors(conn, unit)
|
||||
except Exception as e: # the next run writes them
|
||||
print(f"{station} targets: failed ({root_cause(e)}), run again")
|
||||
continue
|
||||
finally:
|
||||
conn.rollback()
|
||||
t = write_targets(a.out / station, props, seen)
|
||||
print(f"{station} targets.csv: {len(t)} sensors, {t['distance_mm'].notna().sum()} with distance_mm, "
|
||||
f"{t['mm_per_px'].notna().sum()} with mm_per_px")
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def check():
|
||||
"""with_mm and write_targets without the database: `uv run --with pandas --with sqlalchemy python
|
||||
extraction/pump.py --check`."""
|
||||
import tempfile
|
||||
props = pd.DataFrame({"sensor_name": ["T1", "T1", "REF", "T2"], "pattern_type": ["target", "target", "global_ref",
|
||||
"target"], "distance_mm": [20000, 20000, 50000, 1000], "mm_per_px": [7.0, 2.0, 18.0, None]},
|
||||
index=pd.Index([11, 12, 13, 14], name="sensor_id"))
|
||||
df = pd.DataFrame({"sensor_id": [12, 13, 14, 12], "corr_x": [0.5, 0.1, 0.1, None], "corr_y": [-1, 0.2, 0.2, None]})
|
||||
df = with_mm(df, props) # T1 measured by its sensor 12 (2 mm/px), not by its older namesake 11 (7 mm/px)
|
||||
assert df["corr_x_mm"].tolist()[:2] == [1.0, 1.8] and df["corr_y_mm"].tolist()[:2] == [-2.0, 3.6], df
|
||||
assert df[["corr_x_mm", "corr_y_mm"]].iloc[2:].isna().all().all(), df # no factor, no corr: empty
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
pd.DataFrame({"sensor_name": ["OLD", "T1"], "pattern_type": "target", "distance_mm": [1, 1],
|
||||
"mm_per_px": [1.0, 1.0]}).to_csv(Path(d) / "targets.csv", index=False)
|
||||
t = write_targets(Path(d), props, {"T1": 12, "REF": 13})
|
||||
assert t.values.tolist() == [["OLD", "target", 1, 1.0], ["REF", "global_ref", 50000, 18.0],
|
||||
["T1", "target", 20000, 2.0]], t
|
||||
with tempfile.TemporaryDirectory() as d: # given_ranges: the last day pumped and after, pumped again
|
||||
f, day = Path(d) / "S", lambda n: datetime(2026, 11, n).date()
|
||||
assert given_ranges(f, day(1), day(19)) == [(day(1), day(19), False)] # nothing pumped
|
||||
(f / "displacement").mkdir(parents=True)
|
||||
for n in (2, 18):
|
||||
(f / "displacement" / f"S_advanced_202611{n:02d}.csv").touch()
|
||||
assert given_ranges(f, day(1), day(19)) == [(day(1), day(18), False), (day(18), day(19), True)]
|
||||
assert given_ranges(f, day(18), day(19)) == [(day(18), day(19), True)]
|
||||
assert given_ranges(f, day(1), day(10)) == [(day(1), day(10), False)] # the last day after the range
|
||||
print("ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
check() if sys.argv[1:] == ["--check"] else main()
|
||||
+4
-2
@@ -1,6 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Pumping under WSL. Reads the credentials from the .env at the root of the repository (never versioned),
|
||||
# runs pump.py in a throwaway uv environment (4 packages), then the post-processing of LENZSPITZE.
|
||||
# runs pump.py in a throwaway uv environment (4 packages), then absolute.py (x / y in absolute) and weather.py (the
|
||||
# MeteoSwiss weather of the stations with a position, data/<STATION>/weather/).
|
||||
# extraction/pump.sh update of the tracked stations, images included (the scheduler)
|
||||
# extraction/pump.sh --from D --to E [SERIAL] any other call: the arguments go to pump.py as they are
|
||||
# Output appended to out/pump.log; one pumping at a time (lock out/pump.lock).
|
||||
@@ -20,6 +21,7 @@ flock -n 9 || { echo "A pumping is already running." >&2; exit 1; }
|
||||
uv run --no-project --quiet --python 3.12 \
|
||||
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
|
||||
python "$here/pump.py" "$@")
|
||||
[ -f "$root/data/LENZSPITZE/lentzpitze_patterns.json" ] && python3 "$here/lenzspitze_absolute.py"
|
||||
python3 "$here/absolute.py" # x / y in absolute on every station
|
||||
python3 "$here/weather.py" # the weather of every station with a position: its missing and last days
|
||||
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end"
|
||||
} 2>&1 | tee -a "$root/out/pump.log"
|
||||
@@ -0,0 +1,307 @@
|
||||
"""Pumps the differential images (OPTICAL_SHIFT-DIFF) of stations from the AIMsight platform, full size, with what is
|
||||
needed to measure them again: every pattern of the unit (archived ones included) and the unit's properties. READ-ONLY.
|
||||
Never touches data/ nor pump.py's pumping; design: docs/superpowers/specs/2026-11-19-pump-diff-design.md.
|
||||
|
||||
Run through extraction/pump_diff.sh, under WSL, from anywhere (see README.md):
|
||||
|
||||
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU a range (its history starts here)
|
||||
extraction/pump_diff.sh JUNGFRAU from its last indexed day up to now
|
||||
extraction/pump_diff.sh the same for every station of data_diff/
|
||||
|
||||
Output (default: the repository's data_diff/, next to this folder), the tree of data/ so that the other repositories
|
||||
read it the same way (docs/superpowers/specs/2026-11-19-configurations-design.md):
|
||||
<STATION>/images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png the S3 object as is, named as in data/
|
||||
<STATION>/images_diff/index/<STATION>_diff_<YYYYMMDD>.csv every measurement of the day in the database:
|
||||
measurement_id, time_utc (exact), measurement_group_id, file (empty without image), status (ok; no_uuid: never
|
||||
reached S3; s3_missing: S3 returned nothing), properties (the measurement's JSON)
|
||||
<STATION>/images_diff/sensors.json every sensor of the unit: id, name, type, time_utc, properties [{name, value,
|
||||
time_utc, status}] (every property row as stored), rewritten at each run
|
||||
<STATION>/images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties
|
||||
(under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder)
|
||||
<STATION>/metrics/, images/ (IRON, one per --every minutes), weather/: as pump.py and weather.py write them into
|
||||
data/, with their own functions. Not targets.csv: it depends on the run (its patterns), each run writes its own
|
||||
(processing/run_config.py)
|
||||
|
||||
Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and
|
||||
the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index
|
||||
is written after its day's metrics, IRON and DIFF images, every DIFF file through a temporary one: an indexed day is
|
||||
complete; a day whose metrics failed gets no index. Without --from, a station with nothing indexed is skipped (its
|
||||
history starts only on an explicit --from). Stops before a day when the disk has less than --min-free-gb left.
|
||||
|
||||
Same reads as pump.py (measurement, acquisition_unit; the copied queries of the app, branch `explo`), plus the
|
||||
`property` rows of the sensors and of the unit (AcquisitionUnit.list_sensors, Sensor.properties of
|
||||
app/aimsight_database.py). Variables read (names only, none is printed): AIMSIGHT_DATABASE_USER / _PSWD / _TURL /
|
||||
_NAME / _PORT, AIMSIGHT_S3_URL / _ACCESS_KEY / _SECRET_KEY / _BUCKET.
|
||||
"""
|
||||
import argparse
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
UTC = timezone.utc
|
||||
MTYPE = "OPTICAL_SHIFT-DIFF"
|
||||
INDEX = ["measurement_id", "time_utc", "measurement_group_id", "file", "status", "properties"]
|
||||
|
||||
|
||||
def write_atomic(path, data):
|
||||
"""A file is either complete or absent: a run stopped in the middle leaves nothing that looks done."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_name(path.name + ".tmp")
|
||||
tmp.write_bytes(data)
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
def indexed_days(folder):
|
||||
return {datetime.strptime(p.stem.rsplit("_", 1)[1], "%Y%m%d").date()
|
||||
for p in (folder / "images_diff" / "index").glob(f"{folder.name}_diff_*.csv")}
|
||||
|
||||
|
||||
def days_to_pump(indexed, start, end):
|
||||
"""The days of [start, end) to read: the missing ones, and from the last indexed day on (perhaps incomplete).
|
||||
Without start: from the last indexed day; None when nothing is indexed."""
|
||||
last = max(indexed, default=None)
|
||||
if start is None:
|
||||
if last is None:
|
||||
return None
|
||||
start = last
|
||||
days = (start + timedelta(days=k) for k in range((end - start).days))
|
||||
return [d for d in days if last is None or d not in indexed or d >= last]
|
||||
|
||||
|
||||
def pump_day(rows, client, bucket, ext, folder, station, day):
|
||||
"""The images of one day's measurements (rows: id, time_utc, measurement_group_id, measurement_uuid, properties),
|
||||
then its index. Returns the counts per status, plus the new images and their bytes."""
|
||||
c, index = Counter(), []
|
||||
for m in rows:
|
||||
t = m.time_utc.astimezone(UTC)
|
||||
name, status = f"{t:%Y%m%dT%H%M}Z_{m.id}.{ext}", "ok"
|
||||
path = folder / "images_diff" / name
|
||||
if m.measurement_uuid is None: # the image never reached S3 (unfinished multipart upload)
|
||||
name, status = "", "no_uuid"
|
||||
elif not path.exists():
|
||||
try:
|
||||
data = client.get_object(Bucket=bucket, Key=m.measurement_uuid)["Body"].read()
|
||||
except Exception: # like the app's get_measurement_payload: S3 not answering, the image is missing
|
||||
data = None
|
||||
if data:
|
||||
write_atomic(path, data)
|
||||
c["new"], c["bytes"] = c["new"] + 1, c["bytes"] + len(data)
|
||||
else:
|
||||
name, status = "", "s3_missing"
|
||||
c[status] += 1
|
||||
index.append([m.id, t.isoformat(), "" if m.measurement_group_id is None else m.measurement_group_id, name,
|
||||
status, "" if m.properties is None else json.dumps(m.properties, sort_keys=True)])
|
||||
out = io.StringIO()
|
||||
w = csv.writer(out, lineterminator="\n")
|
||||
w.writerow(INDEX)
|
||||
w.writerows(index)
|
||||
write_atomic(folder / "images_diff" / "index" / f"{station}_diff_{day:%Y%m%d}.csv", out.getvalue().encode())
|
||||
return c
|
||||
|
||||
|
||||
def prop(name, value, t, status):
|
||||
return {"name": name, "value": value, "time_utc": t.isoformat(), "status": status}
|
||||
|
||||
|
||||
def sensors_json(conn, unit):
|
||||
"""Every sensor of the unit, any type, archived included (`archive` is one of their properties), with every
|
||||
property row: a pattern redrawn on the platform is a new sensor, the old one stays."""
|
||||
from sqlalchemy import text
|
||||
rows = conn.execute(text("""
|
||||
SELECT s.id, s.name, s.type, s.time_utc, dt.name AS prop, p.value, p.time_utc AS ptime, p.status
|
||||
FROM sensor s LEFT JOIN property p ON p.sensor_id = s.id LEFT JOIN datatype dt ON dt.id = p.datatype_id
|
||||
WHERE s.acquisition_unit_id = :au ORDER BY s.id, dt.name, p.time_utc"""), {"au": unit}).all()
|
||||
sensors = {}
|
||||
for r in rows:
|
||||
s = sensors.setdefault(r.id, {"id": r.id, "name": r.name, "type": r.type, "time_utc": r.time_utc.isoformat(),
|
||||
"properties": []})
|
||||
if r.prop is not None:
|
||||
s["properties"].append(prop(r.prop, r.value, r.ptime, r.status))
|
||||
return list(sensors.values())
|
||||
|
||||
|
||||
def unit_json(conn, unit):
|
||||
"""The acquisition unit and its own property rows (focal_distance, pixel_size, algorithm...), not its sensors'."""
|
||||
from sqlalchemy import text
|
||||
u = conn.execute(text("SELECT id, serial_number, product_number FROM acquisition_unit WHERE id = :au"),
|
||||
{"au": unit}).one()
|
||||
rows = conn.execute(text("""
|
||||
SELECT dt.name, p.value, p.time_utc, p.status FROM property p JOIN datatype dt ON dt.id = p.datatype_id
|
||||
WHERE p.acquisition_unit_id = :au AND p.sensor_id IS NULL ORDER BY dt.name, p.time_utc"""), {"au": unit}).all()
|
||||
return {"id": u.id, "serial_number": u.serial_number, "product_number": u.product_number,
|
||||
"properties": [prop(*r) for r in rows]}
|
||||
|
||||
|
||||
def to_json(obj):
|
||||
return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode()
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("stations", nargs="*", help="serial number of the system, as in the URL of the System page; "
|
||||
"none: the stations already in --out")
|
||||
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD; without: the last indexed day of each "
|
||||
"station (a station with nothing indexed is skipped)")
|
||||
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD; without: up to now")
|
||||
p.add_argument("--delay", type=int, default=60, help="maximum duration of a query, in s (default 60: database "
|
||||
"shared with production)")
|
||||
p.add_argument("--every", type=int, default=60, help="IRON images: one per slot of N minutes, 0 = all (default 60, "
|
||||
"as pump.sh)")
|
||||
p.add_argument("--min-free-gb", type=float, default=50, help="stop before a day when less is free (default 50)")
|
||||
p.add_argument("--out", type=Path, default=Path(__file__).resolve().parent.parent / "data_diff")
|
||||
a = p.parse_args()
|
||||
|
||||
import boto3
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.engine import URL
|
||||
|
||||
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause
|
||||
|
||||
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
|
||||
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
|
||||
end = min(datetime.strptime(a.end, "%Y-%m-%d").date(), tomorrow) if a.end else tomorrow
|
||||
stations = a.stations or sorted(f.name for f in a.out.glob("*") if f.is_dir())
|
||||
if not stations:
|
||||
sys.exit(f"No station: name one, with --from (nothing in {a.out} yet)")
|
||||
missing = [n for n in DB_NAMES if not os.environ.get(n)]
|
||||
if missing:
|
||||
sys.exit(f"Missing variables (names only): {', '.join(missing)}")
|
||||
|
||||
# the engine and the S3 client of pump.py's main
|
||||
url = URL.create("postgresql+psycopg2", username=os.environ["AIMSIGHT_DATABASE_USER"],
|
||||
password=os.environ["AIMSIGHT_DATABASE_PSWD"], host=os.environ["AIMSIGHT_DATABASE_TURL"],
|
||||
port=int(os.environ.get("AIMSIGHT_DATABASE_PORT", "5432")),
|
||||
database=os.environ["AIMSIGHT_DATABASE_NAME"])
|
||||
engine = create_engine(url, connect_args={
|
||||
"options": f"-c timezone=utc -c default_transaction_read_only=on -c statement_timeout={a.delay}s"})
|
||||
client = boto3.client("s3", endpoint_url=os.environ.get("AIMSIGHT_S3_URL", "http://s3:3900"),
|
||||
aws_access_key_id=os.environ.get("AIMSIGHT_S3_ACCESS_KEY", ""),
|
||||
aws_secret_access_key=os.environ.get("AIMSIGHT_S3_SECRET_KEY", ""),
|
||||
region_name="garage", use_ssl=False)
|
||||
bucket = os.environ.get("AIMSIGHT_S3_BUCKET", "measurements-assets")
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
mt = conn.execute(text("SELECT id, extension FROM measurement_type WHERE name = :n"), {"n": MTYPE}).first()
|
||||
conn.rollback()
|
||||
if mt is None:
|
||||
sys.exit(f"Unknown measurement type {MTYPE}")
|
||||
for station in stations:
|
||||
system_id = conn.execute(text("SELECT id FROM system WHERE serial_number = :s"),
|
||||
{"s": station}).scalar()
|
||||
unit = system_id and acquisition_unit(conn, system_id)
|
||||
conn.rollback()
|
||||
if not unit:
|
||||
print(f"{station}: " + ("unknown system" if system_id is None else "no recognised acquisition unit"))
|
||||
continue
|
||||
folder = a.out / station
|
||||
days = days_to_pump(indexed_days(folder), start, end)
|
||||
if days is None:
|
||||
print(f"{station}: nothing indexed yet, its history starts with --from")
|
||||
continue
|
||||
try: # what the database holds, so that an empty day is explained (LENZSPITZE: none after 2026-10-13)
|
||||
first, last = (conn.execute(text(
|
||||
"SELECT time_utc FROM measurement WHERE acquisition_unit_id = :au AND measurement_type_id = :mt "
|
||||
f"ORDER BY time_utc {o} LIMIT 1"), {"au": unit, "mt": mt.id}).scalar() for o in ("ASC", "DESC"))
|
||||
print(f"{station}: differential images in the database from {first:%Y-%m-%d} to {last:%Y-%m-%d}"
|
||||
if first else f"{station}: no differential image in the database")
|
||||
sens = sensors_json(conn, unit)
|
||||
write_atomic(folder / "images_diff" / "sensors.json", to_json(sens))
|
||||
write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit)))
|
||||
except Exception as e: # the next run writes them
|
||||
print(f"{station} images range, sensors.json, unit.json: failed ({root_cause(e)}), run again")
|
||||
finally:
|
||||
conn.rollback()
|
||||
print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}")
|
||||
for day in days:
|
||||
free = shutil.disk_usage(a.out).free / 1e9
|
||||
if free < a.min_free_gb:
|
||||
sys.exit(f"Stopped before {station} {day}: {free:.0f} GB free, under --min-free-gb "
|
||||
f"{a.min_free_gb:g}")
|
||||
nxt = day + timedelta(days=1)
|
||||
try: # as pump.py into data/; a day whose metrics or IRON images failed stays open (no index)
|
||||
if not pump_metrics(conn, system_id, station, day, nxt, a.out, True):
|
||||
print(f"{station} {day}: left open (metrics failed), run again")
|
||||
continue
|
||||
pump_images(conn, (client, bucket), unit, station, day, nxt, "images", a.every, a.out)
|
||||
except Exception as e:
|
||||
print(f"{station} {day}: left open ({root_cause(e)}), run again")
|
||||
continue
|
||||
finally:
|
||||
conn.rollback()
|
||||
a0 = datetime.combine(day, datetime.min.time(), UTC)
|
||||
try:
|
||||
rows = conn.execute(text(
|
||||
"SELECT id, time_utc, measurement_group_id, measurement_uuid, properties FROM measurement "
|
||||
"WHERE acquisition_unit_id = :au AND measurement_type_id = :mt AND time_utc >= :a "
|
||||
"AND time_utc < :b ORDER BY time_utc"),
|
||||
{"au": unit, "mt": mt.id, "a": a0, "b": a0 + timedelta(days=1)}).all()
|
||||
except Exception as e: # no index for the day: the next run reads it again
|
||||
print(f"{station} {day}: failed ({root_cause(e)}), run again")
|
||||
continue
|
||||
finally:
|
||||
conn.rollback()
|
||||
c = pump_day(rows, client, bucket, mt.extension, folder, station, day)
|
||||
print(f"{station} {day}: {c['new']} new images, {c['bytes'] / 1e6:.0f} MB ({len(rows)} in "
|
||||
f"database: {c['ok']} on disk, {c['no_uuid']} never reached S3, {c['s3_missing']} not "
|
||||
"returned by S3)")
|
||||
time.sleep(0.5)
|
||||
try: # the MeteoSwiss weather of the days now in --out, as pump.sh does for data/
|
||||
import weather
|
||||
weather.main(stations, a.out)
|
||||
except Exception as e: # the next run fetches it
|
||||
print(f"weather: failed ({type(e).__name__}: {e}), run again")
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def check():
|
||||
"""Without database or S3: `python3 extraction/pump_diff.py --check`."""
|
||||
import tempfile
|
||||
from types import SimpleNamespace
|
||||
|
||||
d = lambda n: date(2026, 11, n) # noqa: E731
|
||||
assert days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)]
|
||||
assert days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)] # the last indexed day: again
|
||||
assert days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)]
|
||||
assert days_to_pump(set(), None, d(5)) is None # no --from, nothing indexed: skipped
|
||||
|
||||
class S3:
|
||||
def get_object(self, Bucket, Key):
|
||||
if Key == "refused":
|
||||
raise OSError("NoSuchKey")
|
||||
return {"Body": io.BytesIO(b"png")}
|
||||
|
||||
t = lambda h, m: datetime(2026, 11, 3, h, m, 7, 250000, tzinfo=UTC) # noqa: E731
|
||||
rows = [SimpleNamespace(id=1, time_utc=t(10, 0), measurement_group_id=5, measurement_uuid="a", properties={"e": 1}),
|
||||
SimpleNamespace(id=2, time_utc=t(10, 0), measurement_group_id=5, measurement_uuid="b", properties=None),
|
||||
SimpleNamespace(id=3, time_utc=t(10, 15), measurement_group_id=None, measurement_uuid=None, properties={}),
|
||||
SimpleNamespace(id=4, time_utc=t(10, 30), measurement_group_id=None, measurement_uuid="refused",
|
||||
properties={})]
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
folder = Path(tmp) / "S"
|
||||
(folder / "images_diff").mkdir(parents=True)
|
||||
(folder / "images_diff" / "20261103T1000Z_2.png").write_bytes(b"old") # a burst: same minute, other id
|
||||
c = pump_day(rows, S3(), "bucket", "png", folder, "S", d(3))
|
||||
assert dict(c) == {"ok": 2, "new": 1, "bytes": 3, "no_uuid": 1, "s3_missing": 1}, c
|
||||
assert (folder / "images_diff" / "20261103T1000Z_1.png").read_bytes() == b"png"
|
||||
assert (folder / "images_diff" / "20261103T1000Z_2.png").read_bytes() == b"old" # never downloaded again
|
||||
assert not list(folder.rglob("*.tmp"))
|
||||
with open(folder / "images_diff" / "index" / "S_diff_20261103.csv", newline="") as fp:
|
||||
index = list(csv.DictReader(fp))
|
||||
assert [r["status"] for r in index] == ["ok", "ok", "no_uuid", "s3_missing"], index
|
||||
assert [r["file"] for r in index] == ["20261103T1000Z_1.png", "20261103T1000Z_2.png", "", ""], index
|
||||
assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}'
|
||||
assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index
|
||||
assert indexed_days(folder) == {d(3)}
|
||||
print("ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
check() if sys.argv[1:] == ["--check"] else main()
|
||||
Executable
+24
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
# Pumping of the differential images under WSL, into data_diff/ (never data/). Reads the credentials from the .env at
|
||||
# the root of the repository (never versioned) and runs pump_diff.py in the throwaway uv environment of pump.sh; no
|
||||
# absolute.py nor weather.py (they work on data/).
|
||||
# extraction/pump_diff.sh --from D [--to E] SERIAL... a range; the history of a station starts here
|
||||
# extraction/pump_diff.sh [SERIAL...] from the last indexed day up to now
|
||||
# Output appended to out/pump_diff.log; one at a time (lock out/pump_diff.lock), independent of pump.sh's.
|
||||
set -u
|
||||
export PATH="$HOME/.local/bin:$PATH" # uv, missing from the PATH when the Task Scheduler starts wsl.exe
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
root=$(dirname "$here")
|
||||
mkdir -p "$root/out"
|
||||
|
||||
exec 9>"$root/out/pump_diff.lock"
|
||||
flock -n 9 || { echo "A pumping of the differential images is already running." >&2; exit 1; }
|
||||
|
||||
{
|
||||
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) pump_diff.py $*"
|
||||
(set -a +u && . "$root/.env" && set +a -u && # +u: an unquoted $ in .env does not stop the load
|
||||
uv run --no-project --quiet --python 3.12 \
|
||||
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
|
||||
python "$here/pump_diff.py" "$@")
|
||||
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end"
|
||||
} 2>&1 | tee -a "$root/out/pump_diff.log"
|
||||
@@ -0,0 +1,102 @@
|
||||
"""The pumping, for its callers (aimsight-app): the tracked stations, launching pump.sh, its state.
|
||||
|
||||
- `tracked()`, `save(rows)`: extraction/stations.txt, one station per line, "SERIAL [position]" (position: "lat,lon" or a
|
||||
Swiss place, for the weather, weather/meteoswiss.py; empty: an indoor site, no weather); pump.py reads it too;
|
||||
- `start(stations, start, end, images, diff)`: pump.sh in the background (its lock: one pumping at a time); with diff,
|
||||
pump_diff.sh (the differential images into data_diff/, its own lock: it runs beside pump.sh);
|
||||
- `running(diff)`, `log(diff)`: a pumping running (any: the scheduler's, by hand, a caller's), the end of out/pump.log
|
||||
(diff: out/pump_diff.log).
|
||||
|
||||
A refused request raises ValueError, with its reason. Standard library only.
|
||||
"""
|
||||
import fcntl
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SERIAL = re.compile(r"[A-Za-z0-9_-]+$") # it becomes an argument of pump.sh and a folder of data/
|
||||
DAY = re.compile(r"\d{4}-\d{2}-\d{2}$")
|
||||
|
||||
|
||||
def tracked():
|
||||
"""{serial: position ("" without)}, in the order of the file, without duplicates."""
|
||||
f = ROOT / "extraction" / "stations.txt"
|
||||
out = {}
|
||||
for line in f.read_text().splitlines() if f.is_file() else []:
|
||||
serial, _, position = line.strip().partition(" ")
|
||||
if serial:
|
||||
out.setdefault(serial, position.strip())
|
||||
return out
|
||||
|
||||
|
||||
def save(rows):
|
||||
"""rows: [{serial, position}] -> the new stations.txt, and its content as tracked()."""
|
||||
out = {}
|
||||
for r in rows:
|
||||
serial, position = str(r.get("serial", "")).strip(), " ".join(str(r.get("position", "")).split())
|
||||
if not SERIAL.match(serial):
|
||||
raise ValueError(f"not a serial number: {serial!r}")
|
||||
out.setdefault(serial, position)
|
||||
(ROOT / "extraction" / "stations.txt").write_text("".join(f"{s} {p}".rstrip() + "\n" for s, p in out.items()))
|
||||
return out
|
||||
|
||||
|
||||
def name(diff):
|
||||
return "pump_diff" if diff else "pump"
|
||||
|
||||
|
||||
def running(diff=False):
|
||||
"""A pumping running: the lock of pump.sh (diff: pump_diff.sh) is held."""
|
||||
try:
|
||||
with open(ROOT / "out" / f"{name(diff)}.lock") as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
except BlockingIOError:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def log(lines=40, diff=False):
|
||||
"""The last lines of out/pump.log (diff: out/pump_diff.log)."""
|
||||
try:
|
||||
with open(ROOT / "out" / f"{name(diff)}.log", "rb") as f:
|
||||
f.seek(max(0, f.seek(0, os.SEEK_END) - 200 * lines))
|
||||
return f.read().decode(errors="replace").splitlines()[-lines:]
|
||||
except FileNotFoundError:
|
||||
return []
|
||||
|
||||
|
||||
def start(stations, start="", end="", images=False, diff=False):
|
||||
"""pump.sh in the background, with --from start --to end (YYYY-MM-DD, end excluded), or without (the missing days of
|
||||
each station up to now); --delay 300. With diff, pump_diff.sh (without range: from the last indexed day, a station
|
||||
never pumped is skipped; images ignored, its IRON images one per hour). -> the process."""
|
||||
if not stations or not all(isinstance(s, str) and SERIAL.match(s) for s in stations):
|
||||
raise ValueError("no station, or not a serial number")
|
||||
if bool(start) != bool(end) or (start and not (DAY.match(start) and DAY.match(end) and start < end)):
|
||||
raise ValueError("a range: From and To, YYYY-MM-DD, From before To")
|
||||
if running(diff):
|
||||
raise ValueError("a pumping is already running")
|
||||
args = ["--from", start, "--to", end] * bool(start) + ["--images"] * bool(images and not diff) + ["--delay", "300", *stations]
|
||||
return subprocess.Popen([str(ROOT / "extraction" / f"{name(diff)}.sh"), *args], cwd=ROOT,
|
||||
stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
|
||||
|
||||
if __name__ == "__main__": # a check, in a temporary folder: python3 extraction/pumping.py
|
||||
import tempfile
|
||||
ROOT = Path(tempfile.mkdtemp())
|
||||
(ROOT / "extraction").mkdir()
|
||||
(ROOT / "extraction" / "stations.txt").write_text("NORDEND\nNIEDERHORN 46.69,7.78\n\nNORDEND Zermatt\n")
|
||||
assert tracked() == {"NORDEND": "", "NIEDERHORN": "46.69,7.78"}, tracked()
|
||||
assert save([{"serial": "A", "position": " Interlaken BE "}, {"serial": "B"}, {"serial": "A"}]) == {"A": "Interlaken BE", "B": ""}
|
||||
assert (ROOT / "extraction" / "stations.txt").read_text() == "A Interlaken BE\nB\n"
|
||||
assert not running() and log() == [] and not running(diff=True) and log(diff=True) == []
|
||||
for bad in ((save, [{"serial": "../x"}]), (start, ["A;rm"]), (start, ["A"], "2026-11-10"), (start, ["A"], "2026-11-10", "2026-11-09")):
|
||||
try:
|
||||
bad[0](*bad[1:])
|
||||
raise AssertionError(f"not refused: {bad}")
|
||||
except ValueError:
|
||||
pass
|
||||
print("ok")
|
||||
@@ -1,13 +1,15 @@
|
||||
BREITHORN
|
||||
CLARIDEN
|
||||
FLORIDA
|
||||
JUNGFRAU
|
||||
LENZSPITZE
|
||||
FLORIDA Ernen
|
||||
JUNGFRAU Nyon
|
||||
LENZSPITZE Crissier
|
||||
MICHIGAN
|
||||
NADELHORN
|
||||
NADELHORN Genève
|
||||
NEBRASKA
|
||||
NIEDERHORN
|
||||
NORDEND
|
||||
WILDSTRUBEL
|
||||
SUSTENHORN
|
||||
SNCF01
|
||||
SNCF01
|
||||
ALASKA Genève
|
||||
WYOMING Genève
|
||||
@@ -0,0 +1,86 @@
|
||||
"""The weather of each tracked station with a position, into data/<STATION>/weather/. Run by pump.sh after each pumping.
|
||||
|
||||
One CSV per day that has data in the station's folder: data/<STATION>/weather/<STATION>_weather_<YYYYMMDD>.csv, the
|
||||
hourly values of the MeteoSwiss station nearest to the position (weather/meteoswiss.py), UTC, columns: time_utc, position
|
||||
(the line of stations.txt), place (found), station, station_name, distance_km, height_m, then the measurements
|
||||
(meteoswiss.PARAMETERS). A day without MeteoSwiss values gets the header only.
|
||||
Fetched again: a missing day, a day written for another position, the last 2 days (incomplete, not yet quality-checked).
|
||||
A station without a position (an indoor site) or whose place is not found: no weather/ folder, a line in the log.
|
||||
|
||||
Usage: python3 extraction/weather.py [--data DIR] [SERIAL...] (default: every tracked station, into data/; --data:
|
||||
another root with the tree of data/, e.g. data_diff/, which pump_diff.py passes)
|
||||
"""
|
||||
import csv
|
||||
import functools
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path[:0] = [str(ROOT / "extraction"), str(ROOT / "weather")]
|
||||
import meteoswiss # noqa: E402
|
||||
import pumping # noqa: E402
|
||||
|
||||
DATA = ROOT / "data"
|
||||
DAY = re.compile(r"(?<!\d)(20\d\d)_?(\d\d)_?(\d\d)(?!\d)") # ALASKA_advanced_20261017.csv, _2026_10_17_00_00 to …, 20261029T1617Z_…
|
||||
RECENT = timedelta(days=2)
|
||||
# The files of MeteoSwiss (one per station, decade, recent, now) are read once per run, for every station that shares them.
|
||||
meteoswiss.fetch = functools.lru_cache(maxsize=None)(meteoswiss.fetch)
|
||||
|
||||
|
||||
def days_with_data(folder):
|
||||
"""The days (UTC dates) of the files of the station, its weather aside."""
|
||||
return {datetime(*map(int, m.groups()), tzinfo=timezone.utc) for p in folder.rglob("*")
|
||||
if p.is_file() and "weather" not in p.relative_to(folder).parts and (m := DAY.search(p.name))}
|
||||
|
||||
|
||||
def position_of(path):
|
||||
"""The position a weather file was written for; None without a row."""
|
||||
with open(path, newline="") as f:
|
||||
return next(csv.DictReader(f), {}).get("position")
|
||||
|
||||
|
||||
def station(serial, position, now, data=DATA):
|
||||
folder = data / serial
|
||||
wanted = sorted(d for d in days_with_data(folder) if d <= now)
|
||||
out = folder / "weather"
|
||||
path = lambda d: out / f"{serial}_weather_{d:%Y%m%d}.csv" # noqa: E731
|
||||
todo = [d for d in wanted if not path(d).is_file() or now - d < RECENT or position_of(path(d)) not in (position, None)]
|
||||
if not todo:
|
||||
return
|
||||
where, hours = meteoswiss.hourly(position, todo[0], todo[-1] + timedelta(days=1))
|
||||
meta = [position, where["place"], where["station"], where["name"], f"{where['distance_km']:.1f}", f"{where['height_m']:.0f}"]
|
||||
out.mkdir(exist_ok=True)
|
||||
for d in todo:
|
||||
with open(path(d), "w", newline="") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow(["time_utc", "position", "place", "station", "station_name", "distance_km", "height_m",
|
||||
*meteoswiss.PARAMETERS.values()])
|
||||
for t, values in hours.items():
|
||||
if d <= t < d + timedelta(days=1):
|
||||
w.writerow([t.strftime("%Y-%m-%dT%H:%MZ"), *meta, *("" if v is None else f"{v:g}" for v in values.values())])
|
||||
print(f"{serial}: weather of {len(todo)} days, {where['place']} -> MeteoSwiss {where['station']} {where['name']}, "
|
||||
f"{where['distance_km']:.1f} km")
|
||||
|
||||
|
||||
def main(serials, data=DATA):
|
||||
tracked, now = pumping.tracked(), datetime.now(timezone.utc)
|
||||
for serial in serials or tracked:
|
||||
if not (Path(data) / serial).is_dir():
|
||||
continue
|
||||
if not (position := tracked.get(serial)):
|
||||
print(f"{serial}: no position in stations.txt, no weather")
|
||||
continue
|
||||
try:
|
||||
station(serial, position, now, Path(data))
|
||||
except (LookupError, OSError) as e: # the pumping goes on without the weather of this station
|
||||
print(f"{serial}: no weather: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
args = sys.argv[1:]
|
||||
if args[:1] == ["--data"]:
|
||||
main(args[2:], Path(args[1]))
|
||||
else:
|
||||
main(args)
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Creates a new, empty run: data_diff/<SERIAL>/runs/<run>/config.yaml with the cloud's processing written in full (the
|
||||
# patterns of images_diff/sensors.json, the camera, optical's current commit); no result until it is run.
|
||||
# processing/new_run.sh JUNGFRAU run02
|
||||
# processing/run_config.sh JUNGFRAU run02 then, once config.yaml is edited
|
||||
exec "$(dirname "$0")/run_config.sh" "$@" --new
|
||||
@@ -0,0 +1,411 @@
|
||||
"""Runs one configuration of a station: data_diff/<SERIAL>/runs/<run>/config.yaml in, the same folder's displacement/,
|
||||
camera/ and targets.csv out, in the files of data/. The station's pumped data, the same for every run (images_diff/,
|
||||
images/, metrics/, weather/), stay at data_diff/<SERIAL>/ and are only read; whatever depends on the run is in its folder.
|
||||
|
||||
processing/run_config.sh JUNGFRAU run02 --from 2026-10-12 --to 2026-10-17
|
||||
processing/run_config.sh JUNGFRAU run02 every indexed day
|
||||
|
||||
Days: those indexed in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). Running a run again
|
||||
replaces its displacement/, camera/, targets.csv and patterns.json; nothing else is written:
|
||||
displacement/<S>_advanced_<YYYYMMDD>.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator;
|
||||
relative = x - pattern x; corr from the camera correction; reproj_err,
|
||||
inlier of the references; sigma0 of the epoch; corr_*_mm
|
||||
camera/<S>_camera_<YYYYMMDD>.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute
|
||||
targets.csv as in data/, from the run's patterns: pattern_type, distance_mm, mm_per_px
|
||||
patterns.json the patterns exactly as given to the emulator (templates included)
|
||||
|
||||
config.yaml states what differs from the cloud's processing (an empty file: the cloud's), checked key by key:
|
||||
patterns: by sensor_name, over images_diff/sensors.json: any existing property, or
|
||||
PM_G: {error_threshold: 0.7} a template redrawn from a DIFF image of images_diff/, as the
|
||||
BL_C: {redraw: {image: <file>, corners: [[x0, y0], [x1, y1]]}} device's pattern editor does (x_ref kept)
|
||||
camera: over images_diff/unit.json, over the cloud's defaults
|
||||
algorithm: ransac ransac | umeyama | danish_hybrid | trusted_danish
|
||||
ransac_threshold: 0.99
|
||||
mode: "off" off | rotation | rotation_scale | relative (optical's refine_allpoints)
|
||||
focal_distance: 6.0 pixel_size: 0.0024
|
||||
per_epoch: true one measurement at a time, as the cloud; false: the whole series at once
|
||||
emulator:
|
||||
version: "65a4c70" optical's commit: the run stops if $OPTICAL is at another one or has local changes
|
||||
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
|
||||
"""
|
||||
import argparse
|
||||
import csv
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
# Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and
|
||||
# 8 workers x one BLAS thread per core made a 11 s re-measurement take 2 min (45 min of CPU).
|
||||
for _var in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
|
||||
os.environ.setdefault(_var, "1")
|
||||
|
||||
import numpy as np # noqa: E402
|
||||
import pandas as pd # noqa: E402
|
||||
import yaml # noqa: E402
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
OPTICAL = Path(os.environ.get("OPTICAL", Path.home() / "optical")).resolve()
|
||||
sys.path[:0] = [str(OPTICAL), str(OPTICAL / "scripts" / "pcc_emulator")]
|
||||
|
||||
MTYPE = "OPTICAL_SHIFT-DIFF"
|
||||
QUALITY = ["peak", "flux", "sharpness", "fwhm", "dominance", "snr"]
|
||||
COLUMNS = ["time_utc", "sensor_name", "x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x",
|
||||
"corr_y", "reproj_err", "inlier", "sigma0", "pattern_type"] + QUALITY + ["corr_x_mm", "corr_y_mm"]
|
||||
CAM = ["pan", "tilt", "scale", "rotation", "sigma0"]
|
||||
# the cloud's defaults (process_shifts), then the unit's properties, then the configuration
|
||||
DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0,
|
||||
"pixel_size": 0.0024, "per_epoch": True}
|
||||
UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size")
|
||||
OUTPUTS = ("displacement", "camera") # what a run writes into its folder, and replaces
|
||||
|
||||
|
||||
def _keys(d, allowed, where):
|
||||
if not isinstance(d, dict):
|
||||
raise ValueError(f"{where}: a mapping is expected")
|
||||
bad = set(d) - set(allowed)
|
||||
if bad:
|
||||
raise ValueError(f"{where}: unknown key(s) {sorted(bad)}; allowed: {sorted(allowed)}")
|
||||
|
||||
|
||||
def load_config(path):
|
||||
"""The configuration, its keys checked (a typo is an error, not a setting silently ignored)."""
|
||||
c = yaml.safe_load(Path(path).read_text()) or {}
|
||||
_keys(c, {"patterns", "camera", "emulator"}, str(path))
|
||||
_keys(c.get("camera") or {}, DEFAULTS, f"{path}: camera")
|
||||
_keys(c.get("emulator") or {}, {"version"}, f"{path}: emulator")
|
||||
_keys(c.get("patterns") or {}, set(c.get("patterns") or {}), f"{path}: patterns") # a mapping (names: later)
|
||||
return c
|
||||
|
||||
|
||||
def latest(rows):
|
||||
"""{name: value} of property rows ordered by name then time (sensors.json, unit.json): the latest wins."""
|
||||
return {r["name"]: r["value"] for r in rows}
|
||||
|
||||
|
||||
def camera_settings(config, unit_props):
|
||||
cam = dict(DEFAULTS)
|
||||
cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props})
|
||||
cam.update(config.get("camera") or {})
|
||||
return cam
|
||||
|
||||
|
||||
def cloud_properties(sensors):
|
||||
"""{sensor_id: properties} of the patterns the cloud measures the DIFF images with (process_shifts): DIFF type, a
|
||||
pattern, not archived."""
|
||||
out = {}
|
||||
for s in sensors:
|
||||
p = latest(s["properties"])
|
||||
if s["type"] == MTYPE and p.get("pattern") and not p.get("archive"):
|
||||
out[str(s["id"])] = p
|
||||
return out
|
||||
|
||||
|
||||
def apply_patterns(props, overrides, images_dir):
|
||||
"""The configuration's patterns over the cloud's, by sensor_name: a property replaced (only an existing one: a
|
||||
typo is an error), or the template redrawn from an image."""
|
||||
props = {sid: dict(p) for sid, p in props.items()}
|
||||
by_name = {p.get("sensor_name"): sid for sid, p in props.items()}
|
||||
redraws = {}
|
||||
for name, ov in (overrides or {}).items():
|
||||
if name not in by_name:
|
||||
raise ValueError(f"patterns.{name}: no such pattern; known: {sorted(map(str, by_name))}")
|
||||
sid = by_name[name]
|
||||
_keys(ov, set(props[sid]) - {"pattern"} | {"redraw"}, f"patterns.{name}")
|
||||
for k, v in ov.items():
|
||||
if k == "redraw":
|
||||
_keys(v, {"image", "corners"}, f"patterns.{name}.redraw")
|
||||
redraws[sid] = v
|
||||
else:
|
||||
props[sid][k] = v
|
||||
if redraws:
|
||||
redraw(props, redraws, images_dir)
|
||||
return props
|
||||
|
||||
|
||||
def redraw(props, redraws, images_dir):
|
||||
"""As the device's pattern editor saves a drawn box (optical_server pages/patternCreation.py): the camera pose from
|
||||
the references found in the image, the anchor and x_ref (kept) of saved_anchor_and_reference, the template cut
|
||||
at the box."""
|
||||
from aimsight_optical.imagePatternMono import ImagePatternMono, get_center, saved_anchor_and_reference
|
||||
from aimsight_optical.pattern_converter import load_pattern_from_params
|
||||
from aimsight_optical.pattern_pose import estimate_camera_pose
|
||||
from reprocess_images import read_image
|
||||
|
||||
for image in sorted({r["image"] for r in redraws.values()}):
|
||||
im = read_image(images_dir / image)
|
||||
stored = {sid: load_pattern_from_params(**dict(p)) for sid, p in props.items()}
|
||||
mine = {sid: r for sid, r in redraws.items() if r["image"] == image}
|
||||
drawn = {sid: get_center(r["corners"]) for sid, r in mine.items()
|
||||
if props[sid].get("pattern_type") == "global_ref"}
|
||||
pose, seen = estimate_camera_pose(stored, im, drawn=drawn)
|
||||
for sid, r in mine.items():
|
||||
old, corners = stored[sid], [[int(v) for v in c] for c in r["corners"]]
|
||||
anchor, (x_ref, y_ref) = saved_anchor_and_reference(old, corners, old.extra_attr.get("pattern_type", ""),
|
||||
pose=pose, seen=seen.get(sid))
|
||||
kept = {k: v for k, v in old.extra_attr.items() if k not in ("median", "p99", "x_ref", "y_ref")}
|
||||
new = ImagePatternMono.from_image(im, corners, center=anchor, x_ref=x_ref, y_ref=y_ref, **kept)
|
||||
props[sid] = json.loads(json.dumps(new.to_dict(), default=_plain))
|
||||
|
||||
|
||||
def _plain(o):
|
||||
return o.item() if hasattr(o, "item") else o.tolist() if hasattr(o, "tolist") else str(o)
|
||||
|
||||
|
||||
def correct(meas, props, cam):
|
||||
"""The emulator's rows (time_utc, pattern_id, x, y, error, metrics) -> (advanced rows in the columns of data/,
|
||||
camera rows per minute), as the cloud's process_shifts with per_epoch, else over the whole series."""
|
||||
from aimsight_optical.imagePatternMono import get_center, reference_or_anchor
|
||||
from aimsight_optical.pattern_converter import remove_camera_movement
|
||||
|
||||
m = meas.copy()
|
||||
m["sensor_id"] = m["pattern_id"].astype(str)
|
||||
m["time_utc"] = pd.to_datetime(m["time_utc"], utc=True, format="mixed")
|
||||
p = {sid: props[sid] for sid in m["sensor_id"].unique()}
|
||||
anchor = {sid: (float(q["x"]), float(q["y"])) if np.isfinite([q.get("x", np.nan), q.get("y", np.nan)]).all()
|
||||
else get_center(q["corners"]) for sid, q in p.items()}
|
||||
ref = {sid: reference_or_anchor(q.get("x_ref"), q.get("y_ref"), anchor[sid]) for sid, q in p.items()}
|
||||
sid = m["sensor_id"]
|
||||
m["sensor_name"] = sid.map({s: str(q.get("sensor_name", s)) for s, q in p.items()})
|
||||
m["error_threshold"] = pd.to_numeric(sid.map({s: q.get("error_threshold", 0.7) for s, q in p.items()}))
|
||||
m["sensor_type"] = sid.map({s: q.get("pattern_type", "") or "" for s, q in p.items()})
|
||||
m["x_ref"], m["y_ref"] = sid.map({s: r[0] for s, r in ref.items()}), sid.map({s: r[1] for s, r in ref.items()})
|
||||
|
||||
inp = m[["time_utc", "sensor_id", "x", "y", "error", "error_threshold", "sensor_type", "x_ref", "y_ref"]].copy()
|
||||
# as the cloud: a target above its threshold has no position; a reference keeps its raw one (its row is corrected,
|
||||
# remove_camera_movement leaves it out of the fit on its error)
|
||||
inp.loc[(inp["error"] > inp["error_threshold"]) & (inp["sensor_type"] != "global_ref"), ["x", "y"]] = np.nan
|
||||
kw = {"algorithm": cam["algorithm"]}
|
||||
if str(cam["algorithm"]).upper() == "RANSAC":
|
||||
kw["ransac_threshold"] = cam["ransac_threshold"]
|
||||
if cam["mode"] != "off":
|
||||
kw["refine_allpoints"] = cam["mode"]
|
||||
frames = [g for _, g in inp.groupby("time_utc")] if cam["per_epoch"] else [inp]
|
||||
corrs, cams, stats = [], [], []
|
||||
for g in frames:
|
||||
c, cp, rs = remove_camera_movement(g.copy(), cam["focal_distance"], cam["pixel_size"],
|
||||
timeserie_column="time_utc", **kw)
|
||||
corrs.append(c[["time_utc", "sensor_id", "corr_x", "corr_y"]])
|
||||
if not cp.empty:
|
||||
cams.append(cp[CAM].rename_axis("time_utc").reset_index())
|
||||
if not rs.empty:
|
||||
stats.append(rs[["time_utc", "sensor_id", "reproj_err", "inlier"]])
|
||||
m = m.merge(pd.concat(corrs), on=["time_utc", "sensor_id"], how="left")
|
||||
m = m.merge(pd.concat(stats) if stats else pd.DataFrame(columns=["time_utc", "sensor_id", "reproj_err", "inlier"]),
|
||||
on=["time_utc", "sensor_id"], how="left")
|
||||
cp = pd.concat(cams).dropna(subset=CAM, how="all") if cams else pd.DataFrame(columns=["time_utc"] + CAM)
|
||||
cp["time_utc"] = pd.to_datetime(cp["time_utc"], utc=True)
|
||||
m["sigma0"] = m["time_utc"].map(cp.groupby("time_utc")["sigma0"].median())
|
||||
|
||||
factor = pd.to_numeric(sid.map({s: q.get("distance_convertion_factor") for s, q in p.items()}), errors="coerce")
|
||||
adv = pd.DataFrame({
|
||||
"time_utc": m["time_utc"], "sensor_name": m["sensor_name"], "x": m["x"], "y": m["y"],
|
||||
"error_value": m["error"], "error_threshold": m["error_threshold"],
|
||||
"relative_x": m["x"] - sid.map({s: a[0] for s, a in anchor.items()}),
|
||||
"relative_y": m["y"] - sid.map({s: a[1] for s, a in anchor.items()}),
|
||||
"corr_x": m["corr_x"] - m["x_ref"], "corr_y": m["corr_y"] - m["y_ref"],
|
||||
"reproj_err": m["reproj_err"], "inlier": m["inlier"], "sigma0": m["sigma0"], "pattern_type": m["sensor_type"],
|
||||
**{q: m[q] if q in m else np.nan for q in QUALITY}})
|
||||
adv["corr_x_mm"], adv["corr_y_mm"] = adv["corr_x"] * factor.to_numpy(), adv["corr_y"] * factor.to_numpy()
|
||||
adv = adv.sort_values(["time_utc", "sensor_name"], kind="stable").reset_index(drop=True)[COLUMNS]
|
||||
|
||||
cp["time_utc"] = cp["time_utc"].dt.floor("1min")
|
||||
agg = cp.groupby("time_utc")[CAM].agg(["median", "min", "max"])
|
||||
agg.columns = [f"{stat}_{c}" for c, stat in agg.columns]
|
||||
return adv, agg.reset_index()
|
||||
|
||||
|
||||
def write_run(adv, camera, dest, station):
|
||||
"""displacement/ and camera/ of a run, one file per UTC day, replacing the run's previous ones."""
|
||||
for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")):
|
||||
if (dest / folder).exists():
|
||||
shutil.rmtree(dest / folder)
|
||||
(dest / folder).mkdir()
|
||||
for day, d in df.groupby(df["time_utc"].dt.date):
|
||||
d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False)
|
||||
|
||||
|
||||
def optical_state():
|
||||
git = lambda *a: subprocess.run(["git", "-C", str(OPTICAL), *a], capture_output=True, text=True).stdout.strip() # noqa: E731
|
||||
return {"path": str(OPTICAL), "commit": git("rev-parse", "HEAD"), "branch": git("rev-parse", "--abbrev-ref", "HEAD"),
|
||||
"dirty": bool(git("status", "--porcelain", "--untracked-files=no"))}
|
||||
|
||||
|
||||
PATTERN_KEYS = ["pattern_type", "corners", "x", "y", "error_threshold", "resolution", "distance_convertion_factor"]
|
||||
CAMERA_NOTES = {"algorithm": "ransac | umeyama | danish_hybrid | trusted_danish", "ransac_threshold": "px, ransac only",
|
||||
"mode": "off | rotation | rotation_scale | relative", "focal_distance": "mm; pan, tilt, sigma0 only",
|
||||
"pixel_size": "mm; pan, tilt, sigma0 only",
|
||||
"per_epoch": "one measurement at a time, as the cloud; false: the whole series at once"}
|
||||
|
||||
|
||||
def new_run(src, name):
|
||||
"""runs/<name>/config.yaml of a new, empty run (no result yet): the cloud's processing written in full, the patterns
|
||||
of images_diff/sensors.json, the camera of unit.json or the cloud's defaults, optical's current commit."""
|
||||
run = src / "runs" / name
|
||||
if (run / "config.yaml").exists():
|
||||
raise FileExistsError(f"{run / 'config.yaml'} exists: choose another name, or edit that file")
|
||||
props = sorted(cloud_properties(json.loads((src / "images_diff" / "sensors.json").read_text())).values(),
|
||||
key=lambda p: (p.get("pattern_type") != "global_ref", str(p.get("sensor_name"))))
|
||||
unit = latest(json.loads((src / "images_diff" / "unit.json").read_text())["properties"])
|
||||
cam, state = camera_settings({}, unit), optical_state()
|
||||
width = max(len(str(p.get("sensor_name"))) for p in props) + 1
|
||||
line = lambda p: f" {str(p.get('sensor_name')) + ':':{width}s} {{" + ", ".join( # noqa: E731
|
||||
f"{k}: {json.dumps(p[k])}" for k in PATTERN_KEYS if k in p) + "}"
|
||||
refs = [line(p) for p in props if p.get("pattern_type") == "global_ref"]
|
||||
targets = [line(p) for p in props if p.get("pattern_type") != "global_ref"]
|
||||
text = f"""# {name}: created on {datetime.date.today()} with the cloud's processing. Change a value, then run it:
|
||||
# processing/run_config.sh {src.name} {name}
|
||||
# The run also writes here targets.csv (as in data/, from these patterns) and patterns.json (the patterns exactly as
|
||||
# given to the emulator, templates included).
|
||||
emulator:
|
||||
version: "{state['commit'][:7]}" # optical's commit (branch {state['branch']}); the run stops if $OPTICAL is
|
||||
# at another one or has local changes
|
||||
patterns:
|
||||
# The {len(props)} patterns of the DIFF images, as in images_diff/sensors.json on {datetime.date.today()} (frozen here: a
|
||||
# later pumping does not change them). The template stays in sensors.json; to change it:
|
||||
# <name>: {{redraw: {{image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}}}}
|
||||
# pattern_type: global_ref = a reference of the camera fit, anything else a target. corners and x, y move together
|
||||
# (same window size as the template). resolution: 1/100 px. distance_convertion_factor: mm per px (corr_*_mm only).
|
||||
# references
|
||||
""" + "\n".join(refs) + "\n # targets\n" + "\n".join(targets) + "\ncamera:" + "".join(
|
||||
f"\n {f'{k}: {json.dumps(v)}':26s} # {CAMERA_NOTES[k]}" for k, v in cam.items()) + (
|
||||
"\n # focal_distance, pixel_size: the unit has none on the platform, the cloud's values"
|
||||
if not {"focal_distance", "pixel_size"} & set(unit) else "") + "\n"
|
||||
run.mkdir(parents=True, exist_ok=True)
|
||||
(run / "config.yaml").write_text(text)
|
||||
return run / "config.yaml", state
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("station", help="serial number: the folder data_diff/<SERIAL>/")
|
||||
p.add_argument("run", help="the folder data_diff/<SERIAL>/runs/<run>/, holding config.yaml")
|
||||
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD")
|
||||
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD")
|
||||
p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)")
|
||||
p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff")
|
||||
p.add_argument("--new", action="store_true", help="create the run, empty: its config.yaml only (processing/new_run.sh)")
|
||||
a = p.parse_args()
|
||||
src = a.data_diff / a.station
|
||||
run = src / "runs" / a.run
|
||||
if a.new:
|
||||
try:
|
||||
path, state = new_run(src, a.run)
|
||||
except FileExistsError as e:
|
||||
sys.exit(str(e))
|
||||
print(f"{path}: written{', but optical has local changes: the run will stop until they are committed' if state['dirty'] else ''}"
|
||||
f"\nRun it: processing/run_config.sh {a.station} {a.run}")
|
||||
return
|
||||
if not (run / "config.yaml").is_file():
|
||||
sys.exit(f"No {run / 'config.yaml'}: create the run first, processing/new_run.sh {a.station} {a.run}")
|
||||
config = load_config(run / "config.yaml")
|
||||
version = (config.get("emulator") or {}).get("version")
|
||||
if version is not None: # the code that makes the run is the one config.yaml names
|
||||
state = optical_state()
|
||||
want = subprocess.run(["git", "-C", str(OPTICAL), "rev-parse", "--verify", "--quiet", f"{version}^{{commit}}"],
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
if not want:
|
||||
sys.exit(f"emulator.version {version}: not a commit of {OPTICAL}")
|
||||
if want != state["commit"] or state["dirty"]:
|
||||
sys.exit(f"emulator.version {version}, but {OPTICAL} is at {state['commit'][:7]}"
|
||||
f"{' with local changes' if state['dirty'] else ''}: check {version} out, or change the version")
|
||||
|
||||
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{a.station}_diff_*.csv")
|
||||
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
|
||||
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
|
||||
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
|
||||
if not rows:
|
||||
sys.exit(f"{a.station}: no differential image indexed in the range, nothing done")
|
||||
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
|
||||
unit = json.loads((src / "images_diff" / "unit.json").read_text())
|
||||
cam = camera_settings(config, latest(unit["properties"]))
|
||||
props = apply_patterns(cloud_properties(sensors), config.get("patterns"), src / "images_diff")
|
||||
print(f"{a.station} {a.run}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} "
|
||||
f"patterns, camera {cam}, optical {optical_state()}")
|
||||
|
||||
import reprocess_images
|
||||
|
||||
sp = run / "patterns.json" # kept: the target configuration the run was made with, templates included
|
||||
sp.write_text(json.dumps(props, indent=1))
|
||||
pd.DataFrame([{"sensor_name": p.get("sensor_name"), "pattern_type": p.get("pattern_type"), # pump.py's columns
|
||||
"distance_mm": p.get("distance_mm"), "mm_per_px": p.get("distance_convertion_factor")}
|
||||
for p in props.values()]).sort_values("sensor_name").to_csv(run / "targets.csv", index=False)
|
||||
with tempfile.TemporaryDirectory() as tmp: # the emulator's other files: not kept
|
||||
times, remeasured = Path(tmp) / "times.csv", Path(tmp) / "rows.csv"
|
||||
with open(times, "w", newline="") as fp:
|
||||
w = csv.writer(fp)
|
||||
w.writerow(["file", "time_utc"])
|
||||
w.writerows((r["file"], r["time_utc"]) for r in rows)
|
||||
reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured),
|
||||
"--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows])
|
||||
adv, camera = correct(pd.read_csv(remeasured), props, cam)
|
||||
write_run(adv, camera, run, a.station)
|
||||
print(f"{a.station} {a.run}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows "
|
||||
f"-> {run}")
|
||||
|
||||
|
||||
def check():
|
||||
"""`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
f = Path(tmp) / "config.yaml"
|
||||
for text in ("camera: {algoritm: ransac}", "pattern: {}", "stations: {S: {}}", "patterns: [PM_G]",
|
||||
"emulator: {versio: 65a4c70}"):
|
||||
f.write_text(text)
|
||||
try:
|
||||
load_config(f)
|
||||
raise AssertionError(f"accepted: {text}")
|
||||
except ValueError:
|
||||
pass
|
||||
f.write_text("")
|
||||
assert load_config(f) == {} # an empty file: the cloud's processing
|
||||
f.write_text("camera: {mode: rotation, focal_distance: 6.3}\npatterns: {T1: {error_threshold: 0.9}}\n"
|
||||
"emulator: {version: '65a4c70'}")
|
||||
c = load_config(f)
|
||||
assert c["emulator"]["version"] == "65a4c70"
|
||||
cam = camera_settings(c, {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
|
||||
assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation",
|
||||
"ransac"), cam
|
||||
assert camera_settings({}, {})["focal_distance"] == 6.0
|
||||
|
||||
pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)}
|
||||
props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y,
|
||||
"error_threshold": 0.7, "distance_convertion_factor": 2.0, "corners": [[0, 0], [10, 10]]}
|
||||
for i, (n, (x, y)) in enumerate(pos.items())}
|
||||
sensors = [{"id": int(i), "type": MTYPE, "properties": [{"name": k, "value": v} for k, v in {**q, "pattern": "b"}
|
||||
.items()]} for i, q in props.items()] + [{"id": 9, "type": "camera_movement", "properties": []}]
|
||||
assert set(cloud_properties(sensors)) == set(props)
|
||||
for bad in ({"NOPE": {"error_threshold": 1}}, {"T1": {"error_treshold": 1}}):
|
||||
try:
|
||||
apply_patterns(props, bad, Path("."))
|
||||
raise AssertionError(f"accepted: {bad}")
|
||||
except ValueError:
|
||||
pass
|
||||
assert apply_patterns(props, {"T1": {"error_threshold": 0.9}}, Path("."))["3"]["error_threshold"] == 0.9
|
||||
|
||||
t0, t1 = "2026-11-03T10:00:07.25+00:00", "2026-11-03T10:15:07.25+00:00"
|
||||
rows = []
|
||||
for t, (dx, dy) in ((t0, (0.0, 0.0)), (t1, (1.0, 0.5))): # the whole camera shifts by (1, 0.5) px at t1
|
||||
for i, q in props.items():
|
||||
rows.append({"time_utc": t, "pattern_id": int(i), "x": q["x"] + dx, "y": q["y"] + dy, "peak": 1.0,
|
||||
"error": 0.9 if (t == t1 and q["sensor_name"] == "T2") else 0.2})
|
||||
for per_epoch in (True, False):
|
||||
adv, camera = correct(pd.DataFrame(rows), props, {**DEFAULTS, "per_epoch": per_epoch})
|
||||
assert list(adv.columns) == COLUMNS and len(adv) == 10, adv.columns
|
||||
a = adv.set_index(["time_utc", "sensor_name"])
|
||||
t1x = pd.Timestamp(t1)
|
||||
assert a.loc[(t1x, "T1"), ["corr_x", "corr_y"]].abs().max() < 1e-3, a # static target: no displacement
|
||||
assert a.loc[(t1x, "T1"), "relative_x"] == 1.0 and a.loc[(t1x, "T1"), "corr_x_mm"] == \
|
||||
2.0 * a.loc[(t1x, "T1"), "corr_x"]
|
||||
assert np.isnan(a.loc[(t1x, "T2"), "corr_x"]) and a.loc[(t1x, "T2"), "x"] == 701.0 # rejected: raw x kept
|
||||
assert a["sigma0"].notna().all() and a.loc[(t1x, "R1"), "inlier"] == a.loc[(t1x, "R1"), "inlier"], a
|
||||
assert len(camera) == 2 and str(camera["time_utc"][1]) == "2026-11-03 10:15:00+00:00", camera
|
||||
assert list(camera.columns)[:4] == ["time_utc", "median_pan", "min_pan", "max_pan"], camera.columns
|
||||
print("ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
check() if sys.argv[1:] == ["--check"] else main()
|
||||
Executable
+13
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs data_diff/<SERIAL>/runs/<run>/config.yaml (processing/run_config.py), in a throwaway uv environment with what
|
||||
# optical's emulator and camera correction need; optical's code from $OPTICAL (default ~/optical). No credentials.
|
||||
# processing/run_config.sh <SERIAL> <run> [--from D] [--to E] [--jobs N]
|
||||
# processing/run_config.sh <SERIAL> <run> --new (processing/new_run.sh): the run created, empty
|
||||
# processing/run_config.sh --check
|
||||
set -u
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
exec uv run --no-project --quiet --python 3.12 \
|
||||
--with numpy --with pandas --with pillow --with pyyaml --with scipy --with scikit-image \
|
||||
--with opencv-python-headless \
|
||||
python "$here/run_config.py" "$@"
|
||||
@@ -0,0 +1,17 @@
|
||||
# weather
|
||||
|
||||
The weather really measured at a place over a past range: the hourly values of the MeteoSwiss automatic weather station (SwissMetNet) nearest to the place. Measurements, not a model output.
|
||||
|
||||
```bash
|
||||
python3 weather/meteoswiss.py <place> <start> <end> [--out FILE]
|
||||
python3 weather/meteoswiss.py "Genève" 2026-11-15 2026-11-16 --out gve.csv
|
||||
python3 weather/meteoswiss.py 46.52,6.63 2025-06-01T12:00 2025-06-01T18:00
|
||||
```
|
||||
|
||||
The pumping writes it into `data/<SERIAL>/weather/<SERIAL>_weather_<YYYYMMDD>.csv` (`extraction/weather.py`): the columns below, `station` followed by `position` (the line of `stations.txt`), `place`, `station_name`, `distance_km`, `height_m`; the app and the agents read these files. A caller calls `hourly(place, start, end)` -> `(where, {time: {column: value or None}})`, `where` the place found and the station chosen; `LookupError` for a place not found, `OSError` when MeteoSwiss or swisstopo do not answer.
|
||||
|
||||
- `place`: a Swiss place name, resolved by the swisstopo search (`api3.geo.admin.ch`), or coordinates `lat,lon` (WGS84). The resolved place and the chosen station (name, distance, altitude) are written on stderr: check them. A best match whose label does not start with the first word asked is refused (`Toronto` gave `Trient (VS)`): a place outside Switzerland has no weather here, MeteoSwiss measures in Switzerland only.
|
||||
- `start`, `end`: `YYYY-MM-DD` or `YYYY-MM-DDTHH:MM`, in UTC, `end` excluded. The range is in the past.
|
||||
- The output: a CSV, one row per hour, `time_utc` (start of the hour) and `station`, then `precipitation_mm`, `temperature_c`, `humidity_pct`, `wind_speed_kmh`, `wind_gust_kmh`, `wind_direction_deg`, `pressure_hpa` (at the station, QFE), `sunshine_min`, `global_radiation_wm2`, `snow_depth_cm`. An empty cell: not measured at this station or at this hour.
|
||||
|
||||
The source is the Open Data of MeteoSwiss, free, without key ([terms of use](https://opendatadocs.meteoswiss.ch/fr/general/terms-of-use), [documentation of the dataset](https://opendatadocs.meteoswiss.ch/a-data-groundbased/a1-automatic-weather-stations)); the files of a station are listed by the STAC API (`data.geo.admin.ch/api/stac/v1/collections/ch.meteoschweiz.ogd-smn/items/<station>`). The values are quality-checked by MeteoSwiss, except those of the current day (`now` file). Hourly series from 1980 at most, depending on the station. Standard library only.
|
||||
@@ -0,0 +1,189 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The weather measured by MeteoSwiss at the station nearest to a place, over a past range.
|
||||
|
||||
python3 weather/meteoswiss.py <place> <start> <end> [--out FILE]
|
||||
hourly(place, start, end) for a caller (aimsight-app): where, and the values per hour
|
||||
|
||||
<place> is a Swiss place name ("Genève", "Lausanne") or coordinates "lat,lon" (WGS84);
|
||||
<start> and <end> are dates YYYY-MM-DD or YYYY-MM-DDTHH:MM, in UTC, <end> excluded.
|
||||
The output is an hourly CSV (stdout by default), one row per hour.
|
||||
|
||||
The source is the Open Data of MeteoSwiss: the automatic weather stations SwissMetNet,
|
||||
the real measurements (not a model), quality-checked except for the current day
|
||||
(https://opendatadocs.meteoswiss.ch/a-data-groundbased/a1-automatic-weather-stations).
|
||||
The place name is resolved by the search service of swisstopo (api3.geo.admin.ch).
|
||||
No key, no dependency outside the standard library.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import sys
|
||||
import unicodedata
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
BASE = "https://data.geo.admin.ch/ch.meteoschweiz.ogd-smn"
|
||||
STAC_ITEM = "https://data.geo.admin.ch/api/stac/v1/collections/ch.meteoschweiz.ogd-smn/items/{}"
|
||||
GEOCODER = "https://api3.geo.admin.ch/rest/services/api/SearchServer"
|
||||
|
||||
# MeteoSwiss parameter -> column of the output (hourly values).
|
||||
PARAMETERS = {
|
||||
"rre150h0": "precipitation_mm", # hourly total
|
||||
"tre200h0": "temperature_c", # air at 2 m, hourly mean
|
||||
"ure200h0": "humidity_pct", # relative, at 2 m, hourly mean
|
||||
"fu3010h0": "wind_speed_kmh", # hourly mean
|
||||
"fu3010h1": "wind_gust_kmh", # hourly max of the gusts
|
||||
"dkl010h0": "wind_direction_deg", # hourly mean
|
||||
"prestah0": "pressure_hpa", # at the station (QFE), hourly mean
|
||||
"sre000h0": "sunshine_min", # hourly total
|
||||
"gre000h0": "global_radiation_wm2", # hourly mean
|
||||
"htoauths": "snow_depth_cm", # instantaneous, not at every station
|
||||
}
|
||||
|
||||
|
||||
def fetch(url):
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "aimsight-weather"})
|
||||
with urllib.request.urlopen(request, timeout=60) as response:
|
||||
return response.read()
|
||||
|
||||
|
||||
def parse_time(text):
|
||||
for form in ("%Y-%m-%dT%H:%M", "%Y-%m-%d"):
|
||||
try:
|
||||
return datetime.strptime(text, form).replace(tzinfo=timezone.utc)
|
||||
except ValueError:
|
||||
pass
|
||||
raise argparse.ArgumentTypeError(f"not a date YYYY-MM-DD[THH:MM]: {text}")
|
||||
|
||||
|
||||
def locate(place):
|
||||
"""(lat, lon, label) of a place name or of "lat,lon"."""
|
||||
parts = place.split(",")
|
||||
if len(parts) == 2:
|
||||
try:
|
||||
lat, lon = float(parts[0]), float(parts[1])
|
||||
return lat, lon, f"{lat},{lon}"
|
||||
except ValueError:
|
||||
pass
|
||||
query = urllib.parse.urlencode({"searchText": place, "type": "locations", "limit": 1, "sr": 4326})
|
||||
results = json.loads(fetch(f"{GEOCODER}?{query}"))["results"]
|
||||
if not results:
|
||||
raise LookupError(f"place not found: {place}")
|
||||
attrs = results[0]["attrs"]
|
||||
label = re.sub(r"<[^>]+>", "", attrs["label"]).split("\n")[0].strip()
|
||||
# the search always returns its best match ("Toronto" -> "Trient (VS)"): the label must start with the first word asked
|
||||
# ponytail: first word only; "lat,lon" for a place whose name differs from its label
|
||||
if not plain(label).startswith(plain(place.split()[0])):
|
||||
raise LookupError(f"place not found: {place} (best match: {label})")
|
||||
return attrs["lat"], attrs["lon"], label
|
||||
|
||||
|
||||
def plain(text):
|
||||
"""Lower case, without accents: "Genève" -> "geneve"."""
|
||||
return "".join(c for c in unicodedata.normalize("NFKD", text.lower()) if not unicodedata.combining(c))
|
||||
|
||||
|
||||
def distance_km(lat1, lon1, lat2, lon2):
|
||||
p1, p2 = math.radians(lat1), math.radians(lat2)
|
||||
a = (math.sin((p2 - p1) / 2) ** 2
|
||||
+ math.cos(p1) * math.cos(p2) * math.sin(math.radians(lon2 - lon1) / 2) ** 2)
|
||||
return 2 * 6371 * math.asin(math.sqrt(a))
|
||||
|
||||
|
||||
def nearest_station(lat, lon):
|
||||
text = fetch(f"{BASE}/ogd-smn_meta_stations.csv").decode("latin-1")
|
||||
stations = list(csv.DictReader(io.StringIO(text), delimiter=";"))
|
||||
station = min(stations, key=lambda s: distance_km(
|
||||
lat, lon, float(s["station_coordinates_wgs84_lat"]), float(s["station_coordinates_wgs84_lon"])))
|
||||
station["distance_km"] = distance_km(
|
||||
lat, lon, float(station["station_coordinates_wgs84_lat"]), float(station["station_coordinates_wgs84_lon"]))
|
||||
return station
|
||||
|
||||
|
||||
def files_for(station, start, end):
|
||||
"""The hourly files of the station that cover [start, end): historical by decade, recent, now."""
|
||||
item = json.loads(fetch(STAC_ITEM.format(station.lower())))
|
||||
names = [n for n in item["assets"] if f"_{station.lower()}_h_" in n]
|
||||
chosen = []
|
||||
for name in names:
|
||||
stem = name.rsplit(".", 1)[0]
|
||||
if "_historical_" in stem:
|
||||
first, last = (int(y) for y in stem.rsplit("_", 1)[1].split("-"))
|
||||
if start.year <= last and end.year >= first:
|
||||
chosen.append(name)
|
||||
elif stem.endswith("_recent"): # from January 1 of the current year to yesterday
|
||||
if end.year >= datetime.now(timezone.utc).year - 1:
|
||||
chosen.append(name)
|
||||
elif stem.endswith("_now"): # since midnight, not yet quality-checked
|
||||
if end > datetime.now(timezone.utc) - timedelta(days=1):
|
||||
chosen.append(name)
|
||||
return [item["assets"][n]["href"] for n in sorted(chosen)]
|
||||
|
||||
|
||||
def read_rows(urls, start, end):
|
||||
"""{time: row} over [start, end); a later file (recent, now) completes an earlier one."""
|
||||
rows = {}
|
||||
for url in urls:
|
||||
text = fetch(url).decode("latin-1")
|
||||
for row in csv.DictReader(io.StringIO(text), delimiter=";"):
|
||||
time = datetime.strptime(row["reference_timestamp"], "%d.%m.%Y %H:%M").replace(tzinfo=timezone.utc)
|
||||
if start <= time < end:
|
||||
rows.setdefault(time, row)
|
||||
return rows
|
||||
|
||||
|
||||
def number(text):
|
||||
try:
|
||||
return float(text)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def hourly(place, start, end):
|
||||
"""-> (where, {time: {column: float or None}}) over [start, end) (UTC datetimes), the times sorted; where: the place
|
||||
found (place, lat, lon) and the station chosen (station, name, distance_km, height_m). Raises LookupError (place not
|
||||
found) or OSError (MeteoSwiss or swisstopo unreachable)."""
|
||||
lat, lon, label = locate(place)
|
||||
station = nearest_station(lat, lon)
|
||||
rows = read_rows(files_for(station["station_abbr"], start, end), start, end)
|
||||
where = {"place": label, "lat": lat, "lon": lon, "station": station["station_abbr"], "name": station["station_name"],
|
||||
"distance_km": station["distance_km"], "height_m": number(station["station_height_masl"])}
|
||||
return where, {t: {c: number(rows[t].get(p, "")) for p, c in PARAMETERS.items()} for t in sorted(rows)}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("place", help='a Swiss place name, or "lat,lon" (WGS84)')
|
||||
parser.add_argument("start", type=parse_time, help="YYYY-MM-DD[THH:MM], UTC, included")
|
||||
parser.add_argument("end", type=parse_time, help="YYYY-MM-DD[THH:MM], UTC, excluded")
|
||||
parser.add_argument("--out", help="the CSV file (stdout by default)")
|
||||
args = parser.parse_args()
|
||||
if args.end <= args.start:
|
||||
sys.exit("the end must be after the start")
|
||||
if args.start >= datetime.now(timezone.utc):
|
||||
sys.exit("the range must be in the past")
|
||||
|
||||
try:
|
||||
where, hours = hourly(args.place, args.start, args.end)
|
||||
except LookupError as e:
|
||||
sys.exit(str(e))
|
||||
print(f"{where['place']} ({where['lat']:.4f}, {where['lon']:.4f}) -> station {where['station']} {where['name']}, "
|
||||
f"{where['distance_km']:.1f} km, {where['height_m']:.0f} m", file=sys.stderr)
|
||||
|
||||
out = open(args.out, "w", newline="", encoding="utf-8") if args.out else sys.stdout
|
||||
writer = csv.writer(out)
|
||||
writer.writerow(["time_utc", "station", *PARAMETERS.values()])
|
||||
for time, values in hours.items():
|
||||
writer.writerow([time.strftime("%Y-%m-%dT%H:%MZ"), where["station"], *("" if v is None else f"{v:g}" for v in values.values())])
|
||||
if args.out:
|
||||
out.close()
|
||||
print(f"{len(hours)} hours", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in new issue
Block a user