Compare commits

...
11 Commits
Author SHA1 Message Date
maximeandClaude Opus 5.5 73567fe585 stations.txt: FLORIDA at Ernen, for its weather (MeteoSwiss BIN Binn, 3.6 km)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:27:00 +01:00
maximeandClaude Opus 5.5 799a53c102 pumping.py: start/running/log with diff, pump_diff.sh into data_diff/ (its own lock and log), for the Stations page of aimsight-app
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:13:27 +01:00
maximeandClaude Opus 5.5 ca366c150e Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it
- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/,
  weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv,
  patterns.json.
- processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the
  patterns of sensors.json, the camera, optical's current commit); never over an existing one.
- processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked
  against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns)
  and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/.
- pump_diff no longer writes targets.csv: it depends on the run.

JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within
0.034 px; a run created by new_run.sh and run as is gives run01 exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:52:46 +01:00
maximeandClaude Opus 5.5 d3bf4b6860 Configurations: processing/run_config.sh recomputes configs/<name>.yaml from data_diff/ into runs/<name>/, read like data/
Patterns by sensor_name (a property, or a window redrawn from a DIFF image as the device's pattern editor does),
re-measured with optical's emulator, camera correction as the cloud's process_shifts (one measurement at a time, or
the series), written in the files of data/ (displacement/, camera/, targets.csv; images/, metrics/, weather/ linked to
data_diff/), with run.yaml and diagnostics/ beside the stations. configs/cloud.yaml, empty, reproduces data/JUNGFRAU
2026-10-12..16: corr_x/y within 0.03 px, the same rejections, the camera pose within 5e-4 deg; 16 s.
configs/jungfrau_example.yaml: PM_G back to 0.7, BL_C's window redrawn. .gitignore: data_diff/, runs/, remeasure/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 10:00:02 +01:00
maximeandClaude Opus 5.5 81908ca370 Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)
The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every
measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived
included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into
data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now
returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF
images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 09:59:50 +01:00
maximeandClaude Opus 5.5 1b918facd9 Pumping: with --from, the last day pumped of each station is pumped again (pumped during the day, it was incomplete), as without --from; a range from the app no longer stops the current day at the hour of an earlier pumping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:18:57 +01:00
maximeandClaude Opus 5.5 ea0228a73a Pumping: the weather of the stations with a position written into data/<SERIAL>/weather/ (extraction/weather.py, after pump.py); the mm (corr_x_mm, corr_y_mm by the sensor_id of each row) and targets.csv with the references (pattern_type, distance_mm, mm_per_px); --replace pumps a range again
pump.py --check passes; the mm and targets.csv parts are not yet run against the database (no VPN).
stations.txt: NIEDERHORN and NORDEND without a position (outside Switzerland).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 07:17:27 +01:00
maximeandClaude Opus 5.5 9adc38f9b9 absolute.py (ex lenzspitze_absolute.py): x / y in absolute on every station after each pumping, relative sensors without a position dropped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 09:20:39 +01:00
maxime 63df363ff3 loc 2026-11-17 09:13:07 +01:00
maximeandClaude Opus 5.5 b1790fc422 The methods that bring data in, for the callers: extraction/pumping.py (tracked stations, pump.sh, its state), weather/meteoswiss.py (from aimsight-app, with hourly())
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 09:12:12 +01:00
maximeandClaude Opus 5.5 088484c376 stations.txt: SERIAL [position], the position read by the app for the weather
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 08:51:48 +01:00
25 changed files with 1866 additions and 83 deletions

No files matched your search

+3
View File
@@ -2,3 +2,6 @@ data/
out/ out/
.env .env
__pycache__/ __pycache__/
data_diff/
remeasure/
runs/
+6 -3
View File
@@ -2,11 +2,14 @@
The data of the AIMsight stations, in a single copy on the workstation, and their pumping. The data of the AIMsight stations, in a single copy on the workstation, and their pumping.
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`…). A lost folder is pumped again. - `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again.
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". - `data_diff/<SERIAL>/`, **out of git**: pumped by `extraction/pump_diff.sh` (never touching `data/`): the differential images full size under `images_diff/` (with their index, the patterns `sensors.json` and the unit `unit.json`), `images/` (IRON), `metrics/`, `weather/`, as in `data/`: only what is the same for every run. Its runs in `runs/<run>/`, with everything that depends on the run: `config.yaml` (the run's configuration), `displacement/`, `camera/` and `targets.csv` (its result, the files of `data/`), `patterns.json`. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`.
- `processing/run_config.sh <SERIAL> <run> [--from D] [--to E]`: runs `data_diff/<SERIAL>/runs/<run>/config.yaml` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 18 s for 6 days of JUNGFRAU) and writes the run's `displacement/`, `camera/`, `targets.csv` (from its patterns) and `patterns.json` (the patterns exactly as given to the emulator, templates included); `--check` without data. `config.yaml` states what differs from the cloud's processing (an empty file: the cloud's): `patterns` by `sensor_name` (a property, or a window redrawn from a DIFF image), `camera` (algorithm, ransac_threshold, mode, focal_distance, pixel_size, per_epoch) and `emulator.version` (optical's commit: the run stops if `$OPTICAL` is at another one or has local changes); its keys are checked. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml` only, the cloud's processing written in full (every pattern of `sensors.json`, the camera, optical's current commit), to edit before running it; it never overwrites an existing `config.yaml`. The standard configuration reproduces `data/` (JUNGFRAU, 6 days: `corr_x/y` within 0.034 px, the same rejections). A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new). Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`.
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations).
- `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data/<SERIAL>/weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`.
- `.env`: the credentials of the database and of S3, at the root, never versioned. - `.env`: the credentials of the database and of S3, at the root, never versioned.
The other repositories (aimsight-app, Optimizer) read the data through a link `data -> ~/aimsight-data/data`, ignored by git. Pumping is done from here only: the lock `out/pump.lock` holds for this folder alone. The other repositories (aimsight-app, Optimizer) read the data through a link `data -> ~/aimsight-data/data`, ignored by git. The methods that bring data in live here; the app only calls them. Pumping is done from here only: the lock `out/pump.lock` holds for this folder alone.
The labels are in the neighbouring repository `~/aimsight-labeling`. The labels are in the neighbouring repository `~/aimsight-labeling`.
+17
View File
@@ -0,0 +1,17 @@
# The cloud's processing, nothing changed: the patterns of data_diff/<SERIAL>/images_diff/sensors.json, the camera
# correction of process_shifts (ransac, ransac_threshold 0.99, the unit's focal_distance / pixel_size, else 6.0 / 0.0024),
# one measurement at a time. The reference the other configurations are compared with; it must reproduce data/.
#
# A configuration states only what differs, for example:
#
# camera: # common to the stations of the file
# algorithm: ransac # ransac | umeyama | danish_hybrid | trusted_danish (memory: per_epoch false)
# mode: "off" # off | rotation | rotation_scale | relative (optical's refine_allpoints)
# per_epoch: true # false: the whole series at once (a rejected reference interpolated in time)
# stations:
# JUNGFRAU:
# camera: {focal_distance: 6.3}
# patterns: # by sensor_name: any existing property, or a template redrawn from a DIFF image
# PM_G: {error_threshold: 0.7}
# REF_CT: {pattern_type: target}
# BL_C: {redraw: {image: 20261012T1200Z_9999999.png, corners: [[1700, 929], [1724, 953]]}}
+9
View File
@@ -0,0 +1,9 @@
# Example, 2026-11-19: the two findings of the first re-measurement of JUNGFRAU (2026-10-12..16).
# - PM_G is rejected on every image by its threshold of 0.3 (its error is about 0.43): back to the usual 0.7.
# - BL_C is cut by its 20x20 window (edge margin -0.2 px in median): the camera moved by about 2.5 px since the
# patterns were drawn; the window redrawn around where the target is seen now, x_ref kept (same zero).
stations:
JUNGFRAU:
patterns:
PM_G: {error_threshold: 0.7}
BL_C: {redraw: {image: 20261012T1200Z_10163946.png, corners: [[1700, 933], [1720, 953]]}}
@@ -0,0 +1,48 @@
# pump_diff: the layout of data/ Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** `data_diff/<SERIAL>/` read by the other repositories as `data/<SERIAL>/`: images, metrics, weather,
targets.csv as in `data/`; the DIFF images' extras under `images_diff/`.
**Architecture:** `pump_diff.py` calls `pump.py`'s `pump_metrics`, `pump_images`, `sensors`, `write_targets` per
day / station, then `weather.main` on its `--out`; `weather.py` gets a root folder parameter.
**Tech Stack:** as `pump_diff.sh` (uv, sqlalchemy, psycopg2-binary, pandas, boto3).
**Spec:** `docs/superpowers/specs/2026-11-19-configurations-design.md` (section "Work, in order", 1)
## Global Constraints
- `data/` never written; `pump.py`'s behaviour unchanged for `pump.sh` (only a return value added).
- Extras paths: `images_diff/index/<S>_diff_<YYYYMMDD>.csv`, `images_diff/sensors.json`, `images_diff/unit.json`.
- IRON `--every` default 60; metrics replaced for every day pumped.
## Review Focus
- Metrics failed for a day: no index, the day is read again next run.
- `weather.py` without `--data`: writes into `data/` as before (`pump.sh` unchanged).
- A station without position: no weather, the pumping goes on.
- Weather raising something unexpected (network): printed, the pumping's result kept.
- Old layout on disk (`index/` at the station root): migrated by hand once, then regenerated.
---
### Task 1: pump_diff and weather
**Files:** Modify `extraction/pump_diff.py`, `extraction/pump.py` (`pump_metrics` returns `bool`),
`extraction/weather.py` (`main(serials, data=DATA)`, `station(..., data)`, `--data DIR`), `extraction/README.md`,
`README.md`.
- [ ] **Step 1:** `check()`: the index and `indexed_days` under `images_diff/index/`; fails.
- [ ] **Step 2:** move the extras; `--every`; per day `pump_metrics` (skip the day when it returns False),
`pump_images(..., "images", every, out)`, then the DIFF day; per station `write_targets(folder, sensors(conn, unit),
seen)` with `seen` = {sensor_name: id} of the non-archived DIFF sensors with a pattern; at the end `weather.main(
stations, a.out)` in a try.
- [ ] **Step 3:** `--check` ok; `python3 extraction/pump.py --check` ok (uv); `python3 extraction/weather.py --data
/nonexistent` prints nothing and exits 0.
- [ ] **Step 4:** migrate `data_diff/` (old `index/` removed, `sensors.json` / `unit.json` moved; LENZSPITZE, empty,
removed); `pump_diff.sh --from 2026-10-12 --to 2026-10-17 JUNGFRAU` and `--from 2026-09-01 --to 2026-09-02`;
expected: no DIFF image downloaded again, metrics / images / weather / targets.csv present, `data/` unchanged.
- [ ] **Step 5:** `data.py` on `data_diff/JUNGFRAU`: `find_files` without `others` from the extras; metrics,
temperature and weather loaded; same counts as `data/JUNGFRAU` on those days.
@@ -0,0 +1,86 @@
# Pumping the differential images Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** `extraction/pump_diff.py` + `pump_diff.sh` pump a station's `OPTICAL_SHIFT-DIFF` images, their index and
the pattern / unit properties into `data_diff/<SERIAL>/`, read-only, without touching `data/` or `pump.py`.
**Architecture:** One script beside `pump.py`, importing `acquisition_unit`, `root_cause`, `DB_NAMES`, `TYPES`, `UTC`
from it; pure range logic and a per-day function testable through stubs; a wrapper copied from `pump.sh` minus
`absolute.py` / `weather.py`.
**Tech Stack:** Python 3.12 in `uv` (sqlalchemy, psycopg2-binary, pandas, boto3, as `pump.sh`), standard `csv` /
`json`.
**Spec:** `docs/superpowers/specs/2026-11-19-pump-diff-design.md`
## Global Constraints
- Read-only: `default_transaction_read_only=on`, `statement_timeout=--delay` (default 60 s), as `pump.py`.
- Never read or print the `.env`; variable names only.
- Image names `<YYYYMMDD>T<HHMM>Z_<measurement_id>.<ext>`, as `pump.py`.
- `--min-free-gb` default 50, checked before each day.
- Lock `out/pump_diff.lock`, log `out/pump_diff.log`.
## Review Focus
- A crash in the middle of a day: no index file for it (temporary file + `os.replace`), so the next run reads it again.
- Two measurements in the same minute (bursts): distinct files thanks to the id; both in the index.
- A station without acquisition unit or unknown serial: a message, the next station goes on.
- No `--from` and nothing indexed: the station is skipped with a message, nothing downloaded.
- Disk below the limit: the run stops before the day, with the free space printed.
---
### Task 1: range logic and one day, with `--check`
**Files:**
- Create: `extraction/pump_diff.py`
**Interfaces:**
- Produces: `indexed_days(folder: Path) -> set[date]`; `days_to_pump(indexed: set[date], start: date | None, end: date)
-> list[date] | None` (None: no `--from` and nothing indexed); `pump_day(rows, client, bucket: str, ext: str,
folder: Path, station: str, day: date) -> Counter` (rows: the day's measurements, queried by `main`; counts per
status, plus `new` and `bytes`); `check()`.
- [ ] **Step 1:** write `check()` with these assertions (days 1..19 of 2026-11):
- `days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)]`;
- `days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)]` (before the last: missing only; the
last and after: again);
- `days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)]`; `days_to_pump(set(), None, d(5)) is None`;
- `pump_day` on stub `conn` / `client` with 4 measurements (one image to fetch, one already on disk, one
`measurement_uuid` None, one where `get_object` raises): returns `{"ok": 2, "no_uuid": 1, "s3_missing": 1}`, writes
one new file, leaves the existing one untouched, and the index has 4 rows with the right `status`, `file` empty for
the two missing, `properties` as JSON.
- [ ] **Step 2:** `python3 extraction/pump_diff.py --check` fails (functions missing).
- [ ] **Step 3:** implement the three functions. `pump_day` queries `id, time_utc, measurement_group_id,
measurement_uuid, properties` of `measurement` for the unit, type and UTC day, ordered by time; index columns
`measurement_id, time_utc, measurement_group_id, file, status, properties`; written to `index/<S>_diff_<YYYYMMDD>.csv`
through `<name>.tmp` and `os.replace` after the images.
- [ ] **Step 4:** `--check` prints `ok`.
### Task 2: database wiring, wrapper, docs, real trial
**Files:**
- Modify: `extraction/pump_diff.py` (add `sensors_json`, `unit_json`, `main`)
- Create: `extraction/pump_diff.sh`
- Modify: `extraction/README.md`, `README.md`, `.gitignore` (`data_diff/`)
**Interfaces:**
- Consumes: Task 1's functions.
- Produces: `sensors_json(conn, unit: int) -> list[dict]` (`id, name, type, time_utc, properties: [{name, value,
time_utc, status}]`, every sensor of the unit, archived included); `unit_json(conn, unit: int) -> dict` (`id,
serial_number, product_number, properties: [...]`).
- [ ] **Step 1:** `main()`: arguments `stations*`, `--from`, `--to`, `--delay`, `--min-free-gb`, `--out` (default
`<repo>/data_diff`); without stations, the folders of `--out`; engine and S3 client as `pump.py`'s `main`; per
station: system, unit, measurement type `OPTICAL_SHIFT-DIFF`, `sensors.json` / `unit.json` written, then each day of
`days_to_pump` after the disk check, one line printed per day (counts, MB).
- [ ] **Step 2:** `pump_diff.sh` from `pump.sh`: same `.env` load and `uv` packages, its own lock and log, no
`absolute.py` / `weather.py`, no default arguments.
- [ ] **Step 3:** `--check` still `ok`; `pump_diff.sh -h` shows the help.
- [ ] **Step 4:** real trial: `extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-02 JUNGFRAU`; expected: 94
images, byte-identical to `data/JUNGFRAU/images_diff/20260901*` (`cmp`), index of the day, `sensors.json`,
`unit.json`; `data/` unchanged (`find data -newer` the start mark finds nothing).
- [ ] **Step 5:** docs: a `pump_diff` section in `extraction/README.md`, one line in `README.md`, `data_diff/` in
`.gitignore`.
@@ -0,0 +1,53 @@
# run_config Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** `processing/run_config.sh <config> [SERIAL...]` writes `runs/<name>/<SERIAL>/` with the files of `data/`.
**Architecture:** one script; optical imported from `$OPTICAL`; re-measurement through `reprocess_images.main`;
correction through `remove_camera_movement`, per epoch by default.
**Tech Stack:** uv (numpy, pandas, pillow, pyyaml, scipy, scikit-image, opencv-python-headless).
**Spec:** `docs/superpowers/specs/2026-11-19-run-config-design.md`
## Global Constraints
- Writes only under `runs/<name>/` (guarded); reads `data_diff/`, never `data/` except for the acceptance comparison.
- Columns of `data/`'s advanced CSV, in that order: `time_utc, sensor_name, x, y, error_value, error_threshold,
relative_x, relative_y, corr_x, corr_y, reproj_err, inlier, sigma0, pattern_type, peak, flux, sharpness, fwhm,
dominance, snr, corr_x_mm, corr_y_mm`.
- Camera CSV columns: `time_utc`, then `median_/min_/max_` of `pan, tilt, scale, rotation, sigma0`.
## Review Focus
- A configuration with a typo (`error_treshold`, unknown sensor): error before any work.
- An epoch where only one reference survives: translation only, no camera row (as the cloud).
- A day without images in the range: no file for it.
- Re-running the same configuration: the station folder replaced, other stations untouched.
- optical's working tree dirty: recorded in `run.yaml`.
---
### Task 1: configuration and correction, with `--check`
**Files:** Create `processing/run_config.py`.
**Interfaces:** `load_config(path) -> dict`; `camera_settings(config, station, unit_props) -> dict`;
`cloud_properties(sensors) -> {sid: props}`; `apply_patterns(props, overrides, images_dir) -> {sid: props}`;
`correct(meas: DataFrame, props, cam) -> (advanced: DataFrame, camera: DataFrame)`.
- [ ] `check()`: unknown top / station / camera / pattern key raises; unknown sensor_name raises; camera order station >
common > unit > defaults; synthetic correction: three static references, a static target (corr ~0), a target above
its threshold (corr empty, x kept), relative = x - pattern x, sigma0 on every row, one camera row per minute.
- [ ] Implement; `--check` prints ok.
### Task 2: the run, the cloud configuration, the acceptance
**Files:** Modify `processing/run_config.py` (`main`); create `processing/run_config.sh`, `configs/cloud.yaml`;
modify `.gitignore` (`runs/`), `README.md`.
- [ ] `main`: stations, days, patterns, `reprocess_images.main`, `correct`, per-day files, links, `run.yaml`.
- [ ] `run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17`; `data.py` reads `runs/cloud/JUNGFRAU`.
- [ ] Compare with `data/JUNGFRAU`: corr_x / corr_y (all, and the rejected-reference night of 10-15), camera pose.
- [ ] Docs.
@@ -0,0 +1,88 @@
# Configurations: recomputing the stations' data from the differential images
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
section "data_diff").
**Replaced the same day by the user's own layout** (the layout below, `configs/` and `runs/<name>/`, is not used):
```
data_diff/<SERIAL>/
images_diff/ (index/, sensors.json, unit.json) images/ metrics/ weather/ pumped, the same for every run
runs/<run>/ one run: whatever depends on it
config.yaml the run's configuration: patterns (by sensor_name), camera and emulator.version (optical's
commit, checked), over the cloud's processing
displacement/ camera/ targets.csv its result, the files of data/ (targets.csv from its patterns)
patterns.json the patterns exactly as given to the emulator, templates included
```
`processing/run_config.sh <SERIAL> <run>` reads the run's `config.yaml` and writes its `displacement/`, `camera/`,
`targets.csv` and `patterns.json`, nothing else. Rule: what is not the same for every possible run is in the run's
folder. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml`, the cloud's processing written
in full (patterns of `sensors.json`, camera, optical's current commit); never over an existing one. Note: the reading tools (aimsight-tools `data.find_files`) read a station folder whole, subfolders
included: pointed at `data_diff/<SERIAL>/`, they see every run's curves together.
## Intent
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
thresholds, references, camera correction), compare them, and let every other repository (aimsight-app, tools,
detectors, models, optimizer) read the result exactly as it reads `data/`. `data/` stays as it is: the cloud's
version, the reference to compare with.
## Layout
```
aimsight-data/
data/ the cloud, unchanged (pump.sh)
data_diff/<SERIAL>/ raw material shared by every configuration (pump_diff.sh), out of git
images_diff/ the DIFF images; and, invisible to the tools (data.find_files skips images_diff/):
index/<SERIAL>_diff_<YYYYMMDD>.csv, sensors.json, unit.json
images/ metrics/ weather/ targets.csv as in data/ (targets.csv: the cloud's patterns)
configs/<name>.yaml one configuration per file, in git
runs/<name>/ the result of a configuration, out of git, the same tree as data/
<SERIAL>/
displacement/ camera/ targets.csv recomputed, the columns of data/
images/ metrics/ weather/ symbolic links to data_diff/<SERIAL>/ (nothing copied)
run.yaml the configuration used, optical's commit, the date (outside the station folders:
invisible to the tools)
diagnostics/ the emulator's raw rows (edge margin, second spot)
```
A component reads a configuration by pointing its `data` link at `~/aimsight-data/runs/<name>` instead of
`~/aimsight-data/data`. Checked on 2026-11-19: `data.py` (aimsight-tools) reads through symbolic links (images,
metrics, temperature, weather).
## Configuration
A configuration states only what differs from the cloud's patterns (`data_diff/<SERIAL>/images_diff/sensors.json`)
and from the unit's properties (`unit.json`); the configuration `cloud`, with nothing in it, must reproduce `data/`
(the reference: on JUNGFRAU 2026-10-12..16 the emulator matched the cloud within 0.05 px, errors identical).
```yaml
stations:
JUNGFRAU:
patterns: # by sensor_name, over sensors.json
PM_G: {error_threshold: 0.7}
BL_C: {corners: [[1700, 929], [1724, 953]]}
REF_CT: {pattern_type: target}
camera: {focal_distance: 6.3} # over unit.json and the common camera block
camera: # common to the stations of the file
algorithm: ransac # ransac | danish_hybrid | trusted_danish
mode: "off" # off | rotation | rotation_scale | relative
```
Running a configuration again replaces `runs/<name>/`: to compare two settings, two files. No cache between the
re-measurement and the correction (431 images in 11 s; about 15 min for a year of one station): added if it gets slow.
Rejected: a full copy per configuration (the DIFF images duplicated, 0.6 GB per station-day); a configuration
suffix in file names (the tools recognise files by their columns and would mix configurations).
## Work, in order
1. **`pump_diff`** (this repository's `extraction/`): the extras of the DIFF images move under `images_diff/`; per day,
before the DIFF images and their index, the metrics (`pump.py`'s `pump_metrics`) and the IRON images
(`pump_images`, `--every`, default 60 min); per station, `targets.csv` (`write_targets`, on the DIFF patterns);
at the end, the weather (`weather.py`, given the root folder). A day whose metrics failed gets no index: the next
run reads it again. `pump.py`'s only change: `pump_metrics` returns whether every day succeeded.
2. **`run_config`**: re-measurement (optical's emulator), camera correction (optical's `remove_camera_movement` and
the all-point modes), written in the format of `data/`. Its own design once optical's correction outputs are mapped
to the columns of `data/` (`relative_x/y, corr_x/y, reproj_err, inlier, sigma0`, the camera pose).
@@ -0,0 +1,67 @@
# Pumping the differential images into `data_diff/`
2026-11-19. Short design, approved in conversation ("A", "go with your recommendations").
## Intent
Recompute every station's data from the differential images with configurations of our own (patterns redrawn,
thresholds, camera-correction modes), recover the frames the cloud rejected or left uncorrected, and in the end
replace `data/`. This design covers only the first of four steps: **pump the raw material**. The next ones
(re-measure the images with the device chain of optical, correct the camera motion, write the result in the format of
`data/`) get their own designs.
## Goals, non-goals
- Every `OPTICAL_SHIFT-DIFF` image of a station over a range, full size, as stored on S3 (patterns may be redrawn later,
so no crop).
- Everything needed to redo the measurement from them: all the patterns, archived ones included, with every property
row as stored (value, time, status); the acquisition unit's properties (focal distance, pixel size, algorithm...);
each measurement's exact time, group and properties.
- Gaps explained, not only seen: every measurement of the database is listed, with its file or why it has none.
- Non-goal: touching `data/` or `pump.py` (the nightly pump stays as it is). Non-goal: scheduling (later, once the
history is in).
- Anti-goal: filling the disk (about 0.6 GB per station-day at 15 min; 802 GB free on 2026-11-19).
## Approaches
- **A, chosen: a separate script `extraction/pump_diff.py` + `pump_diff.sh`.** Imports what it needs from `pump.py`
(acquisition unit, error formatting); own lock and log, so a backfill of days does not block the nightly pump; does
not run `absolute.py` / `weather.py`.
- B: a "diff only" mode in `pump.py`: least code, but touches the nightly pump and shares its lock.
- C: optical's `scripts/pcc_emulator/fetch_cloud_images.py`: needs monitoring-dashboard's code and environment,
against this repository's rule of no dependency on the app.
## Design
Output, `data_diff/<SERIAL>/` (out of git):
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is, named as in `data/`.
- `index/<SERIAL>_diff_<YYYYMMDD>.csv`, one per UTC day: `measurement_id, time_utc` (exact, ISO),
`measurement_group_id, file` (empty without image), `status` (`ok`, `no_uuid`: never reached S3, `s3_missing`: S3
returned nothing), `properties` (the measurement's JSON). Written after the day's images, through a temporary file:
an index on disk means its day was read through.
- `sensors.json`: every sensor of the acquisition unit (any type, archived included): `id, name, type, time_utc` and
all its property rows `{name, value, time_utc, status}`. Rewritten at each run (the database keeps archived
sensors, so nothing is lost by rewriting).
- `unit.json`: the acquisition unit (`id, serial_number, product_number`) and its property rows, same shape.
Runs:
- `pump_diff.sh --from D [--to E] SERIAL...`: the days of `[D, E)`. A day whose index exists and is before the last
indexed day is skipped; the last indexed day and the later ones are read again (a day pumped while it was running
is incomplete). An image already on disk is never downloaded again.
- `pump_diff.sh [SERIAL...]` without `--from`: from the last indexed day of each station up to now. Without station:
the stations already in `data_diff/`. A station with nothing indexed is skipped with a message (its history starts
only on an explicit `--from`, so no 2.6 TB by accident).
- `--min-free-gb` (default 50): checked before each day; below it, the run stops cleanly.
- `--delay` as in `pump.py`; read-only connection; one query per day for the measurements; images one at a time.
- Lock `out/pump_diff.lock`, log appended to `out/pump_diff.log`. Same `.env` and `uv` environment as `pump.sh`.
Check: `python3 extraction/pump_diff.py --check`, without database or S3: the range logic and one day pumped through
stubs (an image downloaded, one already there, one without uuid, one refused by S3, the index written).
## Left to the builder
- The SQL of the sensors and unit properties (`property` joined to `datatype`, as `pump.py`'s `sensors`).
- Whether a pattern's property has several rows over time is to be seen on the first real run; the format keeps them
all either way.
@@ -0,0 +1,61 @@
# run_config: a run of a station recomputed from its differential images
2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`.
**Amended the same day (the user's layout, see the configurations design):** `processing/run_config.sh <SERIAL> <run>`
reads `data_diff/<SERIAL>/runs/<run>/config.yaml` (`patterns`, `camera`, `emulator.version`: optical's commit, the run
stops if `$OPTICAL` is at another one or has local changes; no `stations` level) and writes only that run's
`displacement/`, `camera/`, `targets.csv` (from its patterns; none at the station's root any more) and `patterns.json`
(the patterns as given to the emulator); the emulator's other files go through a temporary folder. No `runs/<name>/`,
no links, no `run.yaml`. The rest below holds (patterns, correction as
the cloud, columns, acceptance).
## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`)
Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw
`x`, `y`, `error` (and the metrics) are stored; targets above their `error_threshold` get `x`/`y` blanked; a rejected
`global_ref` gets back the `x`/`y` of the latest stored frame whose error is not above its own (in practice its own
raw position, inserted just before), else its pattern position; `remove_camera_movement(r, focal_distance (6.0),
pixel_size (0.0024), algorithm (ransac, ransac_threshold 0.99))`, which rejects those references from the fit again
(error columns given) but still corrects their rows; `corr -= (x_ref, y_ref)`; `cam_info` (pan, tilt, scale,
rotation, sigma0) and `ref_stats` (reproj_err, inlier) stored. The Advanced export adds `relative = x - pattern x`
and copies `sigma0` onto every row; the camera export is the per-minute median / min / max of `cam_info`.
## Design
`processing/run_config.py`, through `processing/run_config.sh` (uv: numpy, pandas, pillow, pyyaml, scipy,
scikit-image, opencv-python-headless; optical's code from `$OPTICAL`, default `~/optical`):
processing/run_config.sh <config> [SERIAL...] [--from D] [--to E] [--jobs N]
`<config>`: `configs/<name>.yaml` or a path; stations: those named, else those of the configuration; days: those
indexed in `data_diff/<SERIAL>/images_diff/index/` (within `--from`/`--to`). For each station, `runs/<name>/<SERIAL>/`
is replaced:
1. **Patterns**: `images_diff/sensors.json` (DIFF type, a pattern, not archived; latest value of each property), then
the configuration by `sensor_name` (an unknown name or key is an error): any property replaced
(`error_threshold`, `pattern_type`, `distance_convertion_factor`, `x`, `y`...); `redraw: {image, corners}` cuts the
template from that DIFF image as the device's pattern editor does (`estimate_camera_pose` from the references,
`saved_anchor_and_reference`: `x_ref` kept, `ImagePatternMono.from_image`). Written to
`diagnostics/<SERIAL>_sensor_properties.json`.
2. **Re-measurement**: optical's `reprocess_images.main` (emulator) on the days' images, `--times` from the index;
rows kept in `diagnostics/<SERIAL>_remeasured.csv`.
3. **Correction**: as the cloud, one measurement at a time (`per_epoch: true`, default): targets above their
threshold blanked, references kept raw (rejected in the fit by `remove_camera_movement`), `corr -= x_ref`;
`per_epoch: false`: the whole series in one call (a rejected reference interpolated in time, algorithms with
memory). Camera block: `algorithm` (ransac), `ransac_threshold` (0.99, ransac only), `mode` (`off` | `rotation` |
`rotation_scale` | `relative`, optical's `refine_allpoints`), `focal_distance`, `pixel_size` (unit.json, else 6.0 /
0.0024), `per_epoch`; station block over common block over unit.json over the cloud's defaults.
4. **Output**, the files of `data/`: `displacement/<S>_advanced_<YYYYMMDD>.csv` (columns of `data/`: `x, y,
error_value, peak..snr` from the emulator, `relative = x - pattern x`, `corr` from the correction, `reproj_err,
inlier` of the references, `sigma0` of the epoch on every row, `corr_*_mm = corr * distance_convertion_factor`),
`camera/<S>_camera_<YYYYMMDD>.csv` (median / min / max per minute), `targets.csv`; `images/`, `metrics/`,
`weather/` as relative symbolic links to `data_diff/<SERIAL>/`; `run.yaml` at `runs/<name>/` (configuration as
resolved per station, optical's commit and state, days, date).
Acceptance: `configs/cloud.yaml` (empty) on JUNGFRAU 2026-10-12..16 against `data/JUNGFRAU`: positions and errors
within 0.05 px (done by the emulator), `corr_x/y` and the camera pose close to the cloud, the differences counted.
Check: `run_config.py --check` in the uv environment: the configuration rules (unknown keys, order of the camera
blocks, overrides by name) and a synthetic correction (three references, two targets, one rejected) through
optical's `remove_camera_movement`.
+33 -5
View File
@@ -2,23 +2,25 @@
`pump.py` fetches, for each station (serial number of the system) and each UTC day asked, into `data/<SERIAL>/`: `pump.py` fetches, for each station (serial number of the system) and each UTC day asked, into `data/<SERIAL>/`:
- the **system metrics** (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in `metrics/`; - the **system metrics** (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in `metrics/`;
- the **displacements** ("advanced" export), one CSV per day in `displacement/`; - the **displacements** ("advanced" export), one CSV per day in `displacement/`, with `corr_x_mm` / `corr_y_mm` = `corr_x` / `corr_y` × `mm_per_px` of the sensor of each row (by `sensor_id`, never by name; empty without a factor); a day pumped before keeps its file: `--from D --to E --replace` pumps it again with the mm;
- the **sensors** measured, targets and references (every sensor with a `pattern_type`), in `targets.csv`: `pattern_type`, `distance_mm` (from the camera; 1000 = not filled in on the platform) and `mm_per_px` (`distance_convertion_factor`, the factor of the dashboard's exports in mm), their current values, read again at each pumping; a name not measured in the run keeps its line;
- the **camera movements** (`pan`, `tilt`, `scale`, `rotation`, `sigma0`), as median and bounds (`median_`, `min_`, `max_`) per minute, only at the times the camera was realigned, one CSV per day in `camera/`; - the **camera movements** (`pan`, `tilt`, `scale`, `rotation`, `sigma0`), as median and bounds (`median_`, `min_`, `max_`) per minute, only at the times the camera was realigned, one CSV per day in `camera/`;
- the **weather** (`weather.py`, after the pumping), for a station with a position in `stations.txt`: the hourly MeteoSwiss measurements nearest to it, one CSV per day with data, in `weather/` (`weather/README.md`);
- the IRON **images** (`--images`, one per hour by default, `--every 10` or `--every 0` for more) in `images/`, and the differential ones (`--diff`) in `images_diff/`. - the IRON **images** (`--images`, one per hour by default, `--every 10` or `--every 0` for more) in `images/`, and the differential ones (`--diff`) in `images_diff/`.
Formats: `~/aimsight-labeling/knowledge/displacement-columns.md`. Formats: `~/aimsight-labeling/knowledge/displacement-columns.md`.
**Tracked stations**: `stations.txt`, one serial number per line. Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in `data/` gets the last 30 days. **Tracked stations**: `stations.txt`, one serial number per line, optionally followed by the position of the station (`lat,lon` or a Swiss place, for the weather pumped into `weather/`; MeteoSwiss covers Switzerland only; the app edits this list: page Stations, through `pumping.py`). Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in `data/` gets the last 30 days.
Resuming: with `--from`, a file already in `data/<SERIAL>/` is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without `--from` replaces it. Resuming: with `--from`, a file already in `data/<SERIAL>/` is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without `--from` replaces it.
## Running (under WSL) ## Running (under WSL)
`pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `lenzspitze_absolute.py`. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump. `pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `absolute.py`, then `weather.py` (every station with a position: its missing days, those written for another position, and the last 2 days, fetched again; a place not found or MeteoSwiss unreachable: a line in the log, the pumping goes on). `python3 extraction/weather.py [SERIAL…]` alone fills the weather of the days already pumped. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump.
Variables of the `.env`: `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT`; for the images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`. Variables of the `.env`: `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT`; for the images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`.
1. **A trial** (1 station, 2 days, in a throwaway folder): `extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND` (6 s on 2026-11-12). `lenzspitze_absolute.py` then converts `data/LENZSPITZE`, not the `--out` folder: no effect if `data/LENZSPITZE` is already converted. 1. **A trial** (1 station, 2 days, in a throwaway folder): `extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND` (6 s on 2026-11-12). `absolute.py` then converts `data/`, not the `--out` folder: no effect on what is already converted.
2. **Updating the tracked stations**: `extraction/pump.sh` (without argument: `--images --delay 300`), or the VS Code task **pump**. The script says, for each station, the range it pumps. 2. **Updating the tracked stations**: `extraction/pump.sh` (without argument: `--images --delay 300`), or the VS Code task **pump**. The script says, for each station, the range it pumps.
3. **A chosen range** (history, failed day), with 3 more days before (the history of the detectors): `extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images <SERIAL>` (`--to` excluded; several stations: `<SERIAL1> <SERIAL2>`). Help: `extraction/pump.sh -h`. 3. **A chosen range** (history, failed day), with 3 more days before (the history of the detectors): `extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images <SERIAL>` (`--to` excluded; several stations: `<SERIAL1> <SERIAL2>`). Help: `extraction/pump.sh -h`.
@@ -37,6 +39,10 @@ Read-only: dedicated engine, `default_transaction_read_only`, 60 s timeout per q
No dependency on the `monitoring-dashboard` app: its reads are copied into `pump.py`, in SQL (its docstring says which ones, and from which commit). If the app changes its reads, the script does not follow: compare again files pumped by both. No dependency on the `monitoring-dashboard` app: its reads are copied into `pump.py`, in SQL (its docstring says which ones, and from which commit). If the app changes its reads, the script does not follow: compare again files pumped by both.
## For a caller
`pumping.py` (standard library): `tracked()` and `save(rows)` (the list), `start(stations, start, end, images)` (`pump.sh` in the background, `--delay 300`), `running()` (the lock held) and `log()` (the end of `out/pump.log`); a refused request raises `ValueError`. Its check: `python3 extraction/pumping.py`. The page Stations of aimsight-app calls it.
## With Claude ## With Claude
Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small ranges; a long range or a history, outside busy hours. It never reads the `.env`. Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small ranges; a long range or a history, outside busy hours. It never reads the `.env`.
@@ -48,4 +54,26 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small
- A day failed on a timeout (`QueryCanceled: … statement timeout`): run it again alone with `--delay 300`. The slowness comes from the database, not from the volume. - A day failed on a timeout (`QueryCanceled: … statement timeout`): run it again alone with `--delay 300`. The slowness comes from the database, not from the volume.
- An hour without an image can have its measurement: the image never reached S3 (empty `measurement_uuid`, unfinished upload). The script skips it and takes the next image of the slot if there is one. - An hour without an image can have its measurement: the image never reached S3 (empty `measurement_uuid`, unfinished upload). The script skips it and takes the next image of the slot if there is one.
- A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in `data/<SERIAL>/` and run again. - A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in `data/<SERIAL>/` and run again.
- LENZSPITZE: its exports have relative `x` / `y` and an empty `pattern_type`; `pump.sh` then runs `lenzspitze_absolute.py` (provisional; details in `~/aimsight-labeling/knowledge/displacement-columns.md`). - Relative `x` / `y` (LENZSPITZE, SUSTENHORN, NADELHORN `c*_mon`): after each pumping, `pump.sh` runs `absolute.py`, which converts every sensor whose `x` / `y` medians are under 100 px, from `data/<SERIAL>/*_patterns.json` and the offset of the station (only LENZSPITZE has both today); a relative sensor it cannot place is dropped (details in `~/aimsight-labeling/knowledge/displacement-columns.md`).
## The differential images, into `data_diff/` (`pump_diff.sh`)
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the runs, `data_diff/<SERIAL>/runs/<run>/`).
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is;
- `images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON);
- `images_diff/sensors.json`: every sensor of the unit, archived included, with all its property rows (pattern, corners, x, y, error_threshold, distance_convertion_factor, start_date…);
- `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm);
- (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder)
- `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`;
- `displacement/`, `camera/` and `targets.csv` are not pumped: they depend on the run (its patterns, its settings), each run writes its own in `runs/<run>/` (`processing/run_config.sh`), never from the cloud; a pumping never touches `runs/`.
```bash
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here
extraction/pump_diff.sh JUNGFRAU # from its last indexed day up to now
extraction/pump_diff.sh # the same for every station of data_diff/
```
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`.
Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.
+92
View File
@@ -0,0 +1,92 @@
"""x / y of the advanced exports as absolute positions in the image, on every station. Run by pump.sh after each pumping.
A sensor of a file is relative when the medians of its x and y are both under RELATIVE_PX (relative: up to ~40 px seen,
NADELHORN c*_mon, SUSTENHORN, LENZSPITZE; absolute: 429 px and more). Rewrites in place data/<STATION>/displacement/*.csv:
x = x_pattern + DX + x_csv, y = y_pattern + DY + y_csv, where (x_pattern, y_pattern) is the centre of the pattern in
data/<STATION>/*_patterns.json and (DX, DY) the offset between the frame of the JSON and the IRON images (OFFSETS);
an empty pattern_type is filled from the JSON.
A relative sensor without a position (no JSON, no offset, missing from the JSON) is dropped: its rows are removed.
A file with no relative sensor is left as it is (already converted). Pumping again restores the originals.
Usage: python3 extraction/absolute.py [DATA] (default: the repository's data/)
python3 extraction/absolute.py --check (a check of the conversion, without touching data/)
"""
import csv
import glob
import json
import os
import statistics
import sys
RELATIVE_PX = 100
# Measured on 2026-11-04 on LENZSPITZE: median (spot − pattern of the JSON) over 9 night images from 17.10 to 03.11, 25
# to 34 matched targets each, stable within 0.5 px. Includes the constant offset spot − (x, y) already seen on NORDEND
# (displacement-columns.md). A station with a JSON but no offset here: its relative sensors are dropped.
OFFSETS = {"LENZSPITZE": (330.5, 15.8)}
def relative_sensors(rows):
xy = {}
for r in rows:
if r["x"] and r["y"]:
xy.setdefault(r["sensor_name"], []).append((float(r["x"]), float(r["y"])))
return {s for s, v in xy.items()
if max(abs(statistics.median(p[0] for p in v)), abs(statistics.median(p[1] for p in v))) < RELATIVE_PX}
def convert(rows, patterns, offset):
"""-> (rows with absolute x / y, relative sensors dropped); None if nothing is relative."""
relative = relative_sensors(rows)
if not relative:
return None
out = []
for r in rows:
if r["sensor_name"] in relative:
m = patterns.get(r["sensor_name"].strip())
if m is None or offset is None:
continue
r["pattern_type"] = r["pattern_type"] or m.get("pattern_type", "")
if r["x"]:
r["x"] = repr(m["x"] + offset[0] + float(r["x"]))
if r["y"]:
r["y"] = repr(m["y"] + offset[1] + float(r["y"]))
out.append(r)
return out
def station(folder):
name = os.path.basename(folder)
patterns = {}
for path in glob.glob(os.path.join(folder, "*_patterns.json")):
patterns.update({str(v["sensor_name"]): v for v in json.load(open(path)).values()})
offset = OFFSETS.get(name) if patterns else None
for path in sorted(glob.glob(os.path.join(folder, "displacement", "*.csv"))):
with open(path, newline="") as f:
reader = csv.DictReader(f)
fields, rows = reader.fieldnames, list(reader)
if not fields or "x" not in fields:
continue # an empty day
out = convert(rows, patterns, offset)
if out is None:
continue
with open(path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=fields)
w.writeheader()
w.writerows(out)
print(f"{name} {os.path.basename(path)}: {len(out)} rows kept, {len(rows) - len(out)} relative dropped")
if __name__ == "__main__":
if sys.argv[1:] == ["--check"]:
def row(s, x, y, t=""):
return {"sensor_name": s, "x": x, "y": y, "pattern_type": t}
rows = [row("a", "1.5", "-2", ""), row("a", "", "", ""), row("b", "0.1", "0.2"), row("c", "800", "600", "ref")]
out = convert(rows, {"a": {"x": 100, "y": 200, "pattern_type": "target"}}, (10, 20))
assert out == [row("a", "111.5", "218.0", "target"), row("a", "", "", "target"), row("c", "800", "600", "ref")], out
assert convert(out, {}, None) is None # already absolute: left as it is
assert convert([row("b", "0.1", "0.2")], {"b": {"x": 1, "y": 1}}, None) == [] # no offset: dropped
print("ok")
sys.exit()
data = sys.argv[1] if len(sys.argv) > 1 else os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "data")
for folder in sorted(glob.glob(os.path.join(data, "*", ""))):
station(folder.rstrip(os.sep))
-49
View File
@@ -1,49 +0,0 @@
"""Provisional, LENZSPITZE only: x / y of the advanced exports as absolute positions in the image, as on the other stations.
On LENZSPITZE, x / y are relative to the reference position of the pattern (medians ~0 px). This script rewrites in place
data/LENZSPITZE/displacement/*.csv:
x = x_pattern + DX + x_csv, y = y_pattern + DY + y_csv, where (x_pattern, y_pattern) is the centre of the pattern in
data/LENZSPITZE/lentzpitze_patterns.json and (DX, DY) the offset between the frame of the JSON and the IRON images;
pattern_type, empty in these exports, is filled from the JSON.
The sensors missing from the JSON (1023, 8019): x / y emptied (position unknown); 8019 gets global_ref (it has reproj_err).
A file whose pattern_type is already filled is left as it is (already converted). Pumping again restores the originals.
Usage: python3 extraction/lenzspitze_absolute.py
"""
import csv
import glob
import json
import os
STATION = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "data", "LENZSPITZE")
# Measured on 2026-11-04: median (spot − pattern of the JSON) over 9 night images from 17.10 to 03.11, 25 to 34 matched
# targets each, stable within 0.5 px. Includes the constant offset spot − (x, y) already seen on NORDEND (displacement-columns.md).
DX, DY = 330.5, 15.8
TYPE_WITHOUT_PATTERN = {"8019": "global_ref"} # reproj_err and inlier filled: reference of the compensation
patterns = {str(v["sensor_name"]): v for v in json.load(open(os.path.join(STATION, "lentzpitze_patterns.json"))).values()}
for path in sorted(glob.glob(os.path.join(STATION, "displacement", "*.csv"))):
with open(path, newline="") as f:
reader = csv.DictReader(f)
fields, rows = reader.fieldnames, list(reader)
if any(r["pattern_type"] for r in rows):
print(f"{os.path.basename(path)}: already converted")
continue
for r in rows:
s = r["sensor_name"].strip()
m = patterns.get(s)
if m is None:
r["x"] = r["y"] = ""
r["pattern_type"] = TYPE_WITHOUT_PATTERN.get(s, "")
continue
r["pattern_type"] = m["pattern_type"]
if r["x"]:
r["x"] = repr(m["x"] + DX + float(r["x"]))
if r["y"]:
r["y"] = repr(m["y"] + DY + float(r["y"]))
with open(path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=fields)
w.writeheader()
w.writerows(rows)
print(f"{os.path.basename(path)}: {len(rows)} rows converted")
+137 -19
View File
@@ -14,11 +14,15 @@ from the last 30 days if it has none; without `--to`: up to now.
Output (default: the repository's `data/`, next to this folder): Output (default: the repository's `data/`, next to this folder):
<STATION>/metrics/<STATION>_<metric>_<YYYY_MM_DD>_00_00 to <YYYY_MM_DD>_23_00.csv system metrics, per day <STATION>/metrics/<STATION>_<metric>_<YYYY_MM_DD>_00_00 to <YYYY_MM_DD>_23_00.csv system metrics, per day
<STATION>/displacement/<STATION>_advanced_<YYYYMMDD>.csv displacements, one file per day <STATION>/displacement/<STATION>_advanced_<YYYYMMDD>.csv displacements, one file per day
+ corr_x_mm, corr_y_mm: corr * mm_per_px of the sensor that measured (references included)
<STATION>/targets.csv the targets and references measured: sensor_name, pattern_type,
distance_mm, mm_per_px (those of the last day pumped)
<STATION>/camera/<STATION>_camera_<YYYYMMDD>.csv camera movements, per day <STATION>/camera/<STATION>_camera_<YYYYMMDD>.csv camera movements, per day
<STATION>/images/<YYYYMMDD>T<HHMM>Z_<id>.<ext> OPTICAL_SHIFT-IRON images (--images) <STATION>/images/<YYYYMMDD>T<HHMM>Z_<id>.<ext> OPTICAL_SHIFT-IRON images (--images)
<STATION>/images_diff/… differential images (--diff) <STATION>/images_diff/… differential images (--diff)
Resumable: with `--from`, a file already written is not read again, whatever the dates asked; without `--from`, the Resumable: with `--from`, a file already written is not read again, whatever the dates asked, except from the last
days of the range are replaced (images excepted). Days in UTC, `--to` excluded; the current day is pumped as it is, day pumped of the station (pumped during the day, it was incomplete); without `--from`, the days of the range are
replaced (images excepted). Days in UTC, `--to` excluded; the current day is pumped as it is,
its metrics up to the last completed hour. To redo a day, delete its files. its metrics up to the last completed hour. To redo a day, delete its files.
No dependency on the app (~/monitoring-dashboard): its reads are copied here, in SQL, as they are (branch `explo`, No dependency on the app (~/monitoring-dashboard): its reads are copied here, in SQL, as they are (branch `explo`,
@@ -42,8 +46,9 @@ import pandas as pd
from sqlalchemy import create_engine, text from sqlalchemy import create_engine, text
from sqlalchemy.engine import URL from sqlalchemy.engine import URL
import pumping # the tracked stations (stations.txt), next to this script
UTC = timezone.utc UTC = timezone.utc
STATIONS = Path(__file__).resolve().parent / "stations.txt" # the tracked stations, one per line
NEW_STATION_DAYS = 30 # without --from, a station with no pumped displacements: the last 30 days NEW_STATION_DAYS = 30 # without --from, a station with no pumped displacements: the last 30 days
DB_NAMES = ["AIMSIGHT_DATABASE_USER", "AIMSIGHT_DATABASE_PSWD", "AIMSIGHT_DATABASE_TURL", "AIMSIGHT_DATABASE_NAME"] DB_NAMES = ["AIMSIGHT_DATABASE_USER", "AIMSIGHT_DATABASE_PSWD", "AIMSIGHT_DATABASE_TURL", "AIMSIGHT_DATABASE_NAME"]
TYPES = {"images": "OPTICAL_SHIFT-IRON", "images_diff": "OPTICAL_SHIFT-DIFF"} # folder -> measurement type TYPES = {"images": "OPTICAL_SHIFT-IRON", "images_diff": "OPTICAL_SHIFT-DIFF"} # folder -> measurement type
@@ -82,13 +87,27 @@ def acquisition_unit(conn, system_id):
return pick[0].id if pick else None return pick[0].id if pick else None
def pumped_days(folder):
"""The days of displacements already pumped."""
return {datetime.strptime(p.stem.rsplit("_", 1)[1], "%Y%m%d").date()
for p in (folder / "displacement").glob(f"{folder.name}_advanced_*.csv")}
def given_ranges(folder, start, end):
"""With --from, without --replace: (start, end, replace) to pump. The missing days of the range, and from the last
day pumped (pumped during the day, it was incomplete) to the end of the range pumped again, as update_ranges."""
last = max(pumped_days(folder), default=None)
if last is None or not start <= last < end:
return [(start, end, False)]
return ([(start, last, False)] if start < last else []) + [(last, end, True)]
def update_ranges(folder, end, tomorrow): def update_ranges(folder, end, tomorrow):
"""Without --from: (start, end, replace) to pump, from the days of displacements already pumped. The missing days """Without --from: (start, end, replace) to pump, from the days of displacements already pumped. The missing days
between the first and the last (ranges pumped separately, failed day) are filled without touching the others; the between the first and the last (ranges pumped separately, failed day) are filled without touching the others; the
last day is pumped again (pumped during the day, it was incomplete) up to now; a station with no pumped last day is pumped again (pumped during the day, it was incomplete) up to now; a station with no pumped
displacements gets the last 30 days.""" displacements gets the last 30 days."""
days = {datetime.strptime(p.stem.rsplit("_", 1)[1], "%Y%m%d").date() days = pumped_days(folder)
for p in (folder / "displacement").glob(f"{folder.name}_advanced_*.csv")}
if not days: if not days:
return [(tomorrow - timedelta(days=NEW_STATION_DAYS + 1), end, True)] return [(tomorrow - timedelta(days=NEW_STATION_DAYS + 1), end, True)]
# a failed metrics day has no file at all (one query for all of them): it is a gap too; a station without any # a failed metrics day has no file at all (one query for all of them): it is a gap too; a station without any
@@ -105,10 +124,12 @@ def pump_metrics(conn, system_id, station, start, end, out, replace):
"""Like the download of the System page (Dashboard/System.py, get_metrics(..., bucketing="1 hour")): per hour, """Like the download of the System page (Dashboard/System.py, get_metrics(..., bucketing="1 hour")): per hour,
mean * transform_factor + transform_offset, or the list of values of a text metric; the labelled series of one mean * transform_factor + transform_offset, or the list of values of a text metric; the labelled series of one
metric merged; an empty row for an hour without measurement. get_metrics counts in hours before now: its query is metric merged; an empty row for an hour without measurement. get_metrics counts in hours before now: its query is
redone here on a fixed day, one query per day for all the metrics.""" redone here on a fixed day, one query per day for all the metrics. -> True when no day failed (pump_diff.py: a
day whose metrics failed is not marked done)."""
names = sorted(conn.execute(text("SELECT DISTINCT dt.name FROM datatype dt JOIN metric_series se " names = sorted(conn.execute(text("SELECT DISTINCT dt.name FROM datatype dt JOIN metric_series se "
"ON se.datatype_id = dt.id WHERE se.system_id = :s"), {"s": system_id}).scalars()) "ON se.datatype_id = dt.id WHERE se.system_id = :s"), {"s": system_id}).scalars())
conn.rollback() conn.rollback()
ok = True
for k in range((end - start).days): for k in range((end - start).days):
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC) a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
paths = {n: out / station / "metrics" paths = {n: out / station / "metrics"
@@ -129,6 +150,7 @@ def pump_metrics(conn, system_id, station, start, end, out, replace):
GROUP BY 1, dt.name""", s=system_id, a=a, b=a + timedelta(days=1)) GROUP BY 1, dt.name""", s=system_id, a=a, b=a + timedelta(days=1))
except Exception as e: # ponytail: like the displacements, a failed day is skipped, a new run picks it up except Exception as e: # ponytail: like the displacements, a failed day is skipped, a new run picks it up
print(f"{station} {a:%Y-%m-%d} metrics: failed ({root_cause(e)}), run again") print(f"{station} {a:%Y-%m-%d} metrics: failed ({root_cause(e)}), run again")
ok = False
continue continue
finally: finally:
conn.rollback() conn.rollback()
@@ -144,6 +166,7 @@ def pump_metrics(conn, system_id, station, start, end, out, replace):
pd.DataFrame({"time_utc": hours, "metric": col.reindex(hours).to_numpy()}).to_csv(path) pd.DataFrame({"time_utc": hours, "metric": col.reindex(hours).to_numpy()}).to_csv(path)
print(f"{station} {a:%Y-%m-%d} metrics: {len(names)} files, {df['name'].nunique()} with data, " print(f"{station} {a:%Y-%m-%d} metrics: {len(names)} files, {df['name'].nunique()} with data, "
f"{time.time() - t0:.0f} s") f"{time.time() - t0:.0f} s")
return ok
# --- the "advanced" export: build_advanced_frame (app/datapoints.py) --- # --- the "advanced" export: build_advanced_frame (app/datapoints.py) ---
@@ -174,7 +197,7 @@ def prepare_raw(shifts):
"""prepare_datapoints_frame: sigma0 (camera rows, without sensor_name) copied onto each sensor at its time; """prepare_datapoints_frame: sigma0 (camera rows, without sensor_name) copied onto each sensor at its time;
relative = x - prop_x; error -> error_value; the full schema.""" relative = x - prop_x; error -> error_value; the full schema."""
if shifts.empty: if shifts.empty:
return pd.DataFrame(columns=COLUMNS) return pd.DataFrame(columns=COLUMNS + ["sensor_id"])
sig = shifts.loc[shifts["sigma0"].notna(), ["time_utc", "sigma0"]] sig = shifts.loc[shifts["sigma0"].notna(), ["time_utc", "sigma0"]]
sigma0 = sig.groupby("time_utc")["sigma0"].median() if not sig.empty else None sigma0 = sig.groupby("time_utc")["sigma0"].median() if not sig.empty else None
shifts = shifts.dropna(subset=["sensor_name"]).reset_index(drop=True) shifts = shifts.dropna(subset=["sensor_name"]).reset_index(drop=True)
@@ -187,7 +210,7 @@ def prepare_raw(shifts):
for c in NUMERIC: for c in NUMERIC:
if c in shifts: if c in shifts:
shifts[c] = pd.to_numeric(shifts[c], errors="coerce") shifts[c] = pd.to_numeric(shifts[c], errors="coerce")
return shifts.reindex(columns=COLUMNS) return shifts.reindex(columns=COLUMNS + ["sensor_id"])
def aggregate_states(conn, unit, a, b): def aggregate_states(conn, unit, a, b):
@@ -196,7 +219,7 @@ def aggregate_states(conn, unit, a, b):
In the states, x_pct holds the CORRECTED x and rx_pct the raw x.""" In the states, x_pct holds the CORRECTED x and rx_pct the raw x."""
thr = "COALESCE((th.value #>> '{}')::double precision, CAST(:thr AS double precision))" thr = "COALESCE((th.value #>> '{}')::double precision, CAST(:thr AS double precision))"
sensor = query(conn, f""" sensor = query(conn, f"""
SELECT ss.time_utc, COALESCE(sn.value #>> '{{}}', s.name) AS name, SELECT ss.time_utc, ss.sensor_id, COALESCE(sn.value #>> '{{}}', s.name) AS name,
approx_percentile(0.5, ss.x_pct) AS corr_x, approx_percentile(0.5, ss.y_pct) AS corr_y, approx_percentile(0.5, ss.x_pct) AS corr_x, approx_percentile(0.5, ss.y_pct) AS corr_y,
approx_percentile(0.5, ss.rx_pct) AS x, approx_percentile(0.5, ss.ry_pct) AS y, approx_percentile(0.5, ss.rx_pct) AS x, approx_percentile(0.5, ss.ry_pct) AS y,
approx_percentile(0.5, ss.error_pct) AS error, {thr} AS error_threshold, approx_percentile(0.5, ss.error_pct) AS error, {thr} AS error_threshold,
@@ -214,11 +237,12 @@ def aggregate_states(conn, unit, a, b):
WHERE s.acquisition_unit_id = :au AND ss.time_utc >= :a AND ss.time_utc <= :b WHERE s.acquisition_unit_id = :au AND ss.time_utc >= :a AND ss.time_utc <= :b
ORDER BY ss.time_utc""", au=unit, a=a, b=b, thr=DEFAULT_ERROR_THRESHOLD) ORDER BY ss.time_utc""", au=unit, a=a, b=b, thr=DEFAULT_ERROR_THRESHOLD)
if sensor.empty: if sensor.empty:
return pd.DataFrame(columns=COLUMNS) return pd.DataFrame(columns=COLUMNS + ["sensor_id"])
common = query(conn, """ common = query(conn, """
SELECT sc.time_utc, approx_percentile(0.5, sc.sigma0_pct) AS sigma0 FROM shift_common_1min sc SELECT sc.time_utc, approx_percentile(0.5, sc.sigma0_pct) AS sigma0 FROM shift_common_1min sc
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b""", au=unit, a=a, b=b) WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b""", au=unit, a=a, b=b)
out = pd.DataFrame({"time_utc": pd.to_datetime(sensor["time_utc"]), "sensor_name": sensor["name"].astype(str)}) out = pd.DataFrame({"time_utc": pd.to_datetime(sensor["time_utc"]), "sensor_name": sensor["name"].astype(str),
"sensor_id": sensor["sensor_id"]})
for col, src in [("error_value", "error"), ("error_threshold", "error_threshold"), ("corr_x", "corr_x"), for col, src in [("error_value", "error"), ("error_threshold", "error_threshold"), ("corr_x", "corr_x"),
("corr_y", "corr_y"), ("relative_x", "corr_x"), ("relative_y", "corr_y"), ("x", "x"), ("y", "y"), ("corr_y", "corr_y"), ("relative_x", "corr_x"), ("relative_y", "corr_y"), ("x", "x"), ("y", "y"),
("reproj_err", "reproj_err")] + [(m, m) for m in QUALITY]: ("reproj_err", "reproj_err")] + [(m, m) for m in QUALITY]:
@@ -231,7 +255,7 @@ def aggregate_states(conn, unit, a, b):
sig = common.loc[common["sigma0"].notna()] sig = common.loc[common["sigma0"].notna()]
out["sigma0"] = (out["time_utc"].map(sig.assign(time_utc=pd.to_datetime(sig["time_utc"])) out["sigma0"] = (out["time_utc"].map(sig.assign(time_utc=pd.to_datetime(sig["time_utc"]))
.groupby("time_utc")["sigma0"].median()) if not sig.empty else float("nan")) .groupby("time_utc")["sigma0"].median()) if not sig.empty else float("nan"))
return out.reindex(columns=COLUMNS) return out.reindex(columns=COLUMNS + ["sensor_id"])
def merge(raw, agg): def merge(raw, agg):
@@ -255,11 +279,42 @@ def merge(raw, agg):
t = utc(agg["time_utc"]) t = utc(agg["time_utc"])
agg["time_utc"] = t.dt.tz_convert(tz) if tz is not None else t.dt.tz_localize(None) agg["time_utc"] = t.dt.tz_convert(tz) if tz is not None else t.dt.tz_localize(None)
return (pd.concat([raw, agg], ignore_index=True).sort_values(["sensor_name", "time_utc"], kind="stable") return (pd.concat([raw, agg], ignore_index=True).sort_values(["sensor_name", "time_utc"], kind="stable")
.reset_index(drop=True).reindex(columns=COLUMNS)) .reset_index(drop=True).reindex(columns=COLUMNS + ["sensor_id"]))
SENSOR_PROPS = {"pattern_type": "pattern_type", "distance_mm": "distance_mm", "distance_convertion_factor": "mm_per_px"}
MM = {"corr_x": "corr_x_mm", "corr_y": "corr_y_mm"}
def sensors(conn, unit):
"""The sensors of the unit that have a pattern_type (targets and references), one row per sensor_id: sensor_name,
pattern_type, distance_mm (from the camera; 1000: not filled in on the platform) and mm_per_px
(distance_convertion_factor, the factor of the dashboard's exports in mm: DeviceOutput.get_scale,
compute_output_signals, (corr - offset) * factor). A name may have several sensors (an older configuration of the
unit, another lens): the factor of a measurement is that of its sensor_id, never looked up by name."""
prop = "max(p.value #>> '{{}}') FILTER (WHERE dt.name = '{}')".format
df = query(conn, f"""
SELECT s.id AS sensor_id, COALESCE({prop('sensor_name')}, s.name) AS sensor_name,
{", ".join(f"{prop(n)} AS {c}" for n, c in SENSOR_PROPS.items())}
FROM sensor s JOIN property p ON p.sensor_id = s.id JOIN datatype dt ON dt.id = p.datatype_id
WHERE s.acquisition_unit_id = :au
GROUP BY s.id, s.name HAVING {prop('pattern_type')} IS NOT NULL""", au=unit)
for c in ("distance_mm", "mm_per_px"):
df[c] = pd.to_numeric(df[c], errors="coerce")
return df.set_index("sensor_id")
def with_mm(df, props):
"""corr_x_mm / corr_y_mm = corr_x / corr_y * mm_per_px of the sensor_id of each row; empty without a factor."""
factor = df["sensor_id"].map(props["mm_per_px"]).astype(float)
for c, mm in MM.items():
df[mm] = pd.to_numeric(df[c], errors="coerce") * factor
return df
def advanced(conn, unit, a, b): def advanced(conn, unit, a, b):
return merge(prepare_raw(raw_datapoints(conn, unit, a, b)), aggregate_states(conn, unit, a, b)) df = merge(prepare_raw(raw_datapoints(conn, unit, a, b)), aggregate_states(conn, unit, a, b))
return with_mm(df, sensors(conn, unit))
def camera(conn, unit, a, b): def camera(conn, unit, a, b):
@@ -278,7 +333,9 @@ DAILY_FILES = (("displacement", "advanced", advanced), ("camera", "camera", came
def pump_displacements(conn, unit, station, start, end, out, replace): def pump_displacements(conn, unit, station, start, end, out, replace):
"""Displacements and camera movements: one file and one query per day, bounded, under the timeout (--delay).""" """Displacements and camera movements: one file and one query per day, bounded, under the timeout (--delay)
-> {sensor_name: sensor_id} measured, the last day pumped winning (for targets.csv)."""
seen = {}
for k in range((end - start).days): for k in range((end - start).days):
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC) a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
for folder, name, reader in DAILY_FILES: for folder, name, reader in DAILY_FILES:
@@ -295,9 +352,25 @@ def pump_displacements(conn, unit, station, start, end, out, replace):
conn.rollback() # closes the read transaction conn.rollback() # closes the read transaction
if "sensor_name" in df: if "sensor_name" in df:
df = df.drop_duplicates(["sensor_name", "time_utc"]) df = df.drop_duplicates(["sensor_name", "time_utc"])
if "sensor_id" in df:
seen.update(df.dropna(subset=["sensor_id"]).groupby("sensor_name")["sensor_id"].last().to_dict())
df = df.drop(columns="sensor_id")
path.parent.mkdir(parents=True, exist_ok=True) path.parent.mkdir(parents=True, exist_ok=True)
df.to_csv(path, index=False) # an empty day is written too: the next run does not ask for it again df.to_csv(path, index=False) # an empty day is written too: the next run does not ask for it again
print(f"{station} {a:%Y-%m-%d} {name}: {len(df)} rows, {time.time() - t0:.0f} s -> {path.name}") print(f"{station} {a:%Y-%m-%d} {name}: {len(df)} rows, {time.time() - t0:.0f} s -> {path.name}")
return seen
def write_targets(folder, props, seen):
"""targets.csv: the properties of the sensors measured (seen: {name: sensor_id}); the names not measured in this
run keep their line."""
path = folder / "targets.csv"
new = props.loc[[i for i in seen.values() if i in props.index], ["sensor_name", *SENSOR_PROPS.values()]]
old = pd.read_csv(path) if path.exists() else new.iloc[:0]
t = pd.concat([old[~old["sensor_name"].astype(str).isin(new["sensor_name"])], new]).sort_values("sensor_name")
folder.mkdir(parents=True, exist_ok=True)
t.to_csv(path, index=False)
return t
def pump_images(conn, s3, unit, station, start, end, folder, every, out): def pump_images(conn, s3, unit, station, start, end, folder, every, out):
@@ -351,6 +424,8 @@ def main():
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD; without: the last pumped day of each " p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD; without: the last pumped day of each "
"station, pumped again, or the last 30 days") "station, pumped again, or the last 30 days")
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD; without: up to now") p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD; without: up to now")
p.add_argument("--replace", action="store_true", help="with --from: the days already pumped are pumped again "
"(a file is replaced only once its day is read)")
p.add_argument("--images", action="store_true", help="also the OPTICAL_SHIFT-IRON images") p.add_argument("--images", action="store_true", help="also the OPTICAL_SHIFT-IRON images")
p.add_argument("--diff", action="store_true", help="also the OPTICAL_SHIFT-DIFF images") p.add_argument("--diff", action="store_true", help="also the OPTICAL_SHIFT-DIFF images")
p.add_argument("--every", type=int, default=60, help="one image per slot of N minutes, 0 = all (default 60)") p.add_argument("--every", type=int, default=60, help="one image per slot of N minutes, 0 = all (default 60)")
@@ -361,7 +436,7 @@ def main():
tomorrow = datetime.now(UTC).date() + timedelta(days=1) tomorrow = datetime.now(UTC).date() + timedelta(days=1)
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
end = min(datetime.strptime(a.end, "%Y-%m-%d").date(), tomorrow) if a.end else tomorrow # no day to come end = min(datetime.strptime(a.end, "%Y-%m-%d").date(), tomorrow) if a.end else tomorrow # no day to come
stations = a.stations or STATIONS.read_text().split() stations = a.stations or list(pumping.tracked())
missing = [n for n in DB_NAMES if not os.environ.get(n)] missing = [n for n in DB_NAMES if not os.environ.get(n)]
if missing: if missing:
@@ -387,9 +462,11 @@ def main():
if system_id is None: if system_id is None:
print(f"{station}: unknown system") print(f"{station}: unknown system")
continue continue
ranges = [(start, end, False)] if start else update_ranges(a.out / station, end, tomorrow) ranges = ([(start, end, True)] if start and a.replace else given_ranges(a.out / station, start, end)
if start else update_ranges(a.out / station, end, tomorrow))
unit = acquisition_unit(conn, system_id) unit = acquisition_unit(conn, system_id)
conn.rollback() conn.rollback()
seen = {}
for range_start, range_end, replace in ranges: for range_start, range_end, replace in ranges:
print(f"{station}: from {range_start} to {range_end - timedelta(days=1)} inclusive" print(f"{station}: from {range_start} to {range_end - timedelta(days=1)} inclusive"
+ ("" if replace else ", missing days only")) + ("" if replace else ", missing days only"))
@@ -397,13 +474,54 @@ def main():
if unit is None: if unit is None:
print(f"{station}: no recognised acquisition unit, no displacements, camera or images") print(f"{station}: no recognised acquisition unit, no displacements, camera or images")
continue continue
pump_displacements(conn, unit, station, range_start, range_end, a.out, replace) seen.update(pump_displacements(conn, unit, station, range_start, range_end, a.out, replace))
for flag, folder in ((a.images, "images"), (a.diff, "images_diff")): for flag, folder in ((a.images, "images"), (a.diff, "images_diff")):
if flag: if flag:
pump_images(conn, s3, unit, station, range_start, range_end, folder, a.every, a.out) pump_images(conn, s3, unit, station, range_start, range_end, folder, a.every, a.out)
if not seen:
continue
try:
props = sensors(conn, unit)
except Exception as e: # the next run writes them
print(f"{station} targets: failed ({root_cause(e)}), run again")
continue
finally:
conn.rollback()
t = write_targets(a.out / station, props, seen)
print(f"{station} targets.csv: {len(t)} sensors, {t['distance_mm'].notna().sum()} with distance_mm, "
f"{t['mm_per_px'].notna().sum()} with mm_per_px")
finally: finally:
engine.dispose() engine.dispose()
def check():
"""with_mm and write_targets without the database: `uv run --with pandas --with sqlalchemy python
extraction/pump.py --check`."""
import tempfile
props = pd.DataFrame({"sensor_name": ["T1", "T1", "REF", "T2"], "pattern_type": ["target", "target", "global_ref",
"target"], "distance_mm": [20000, 20000, 50000, 1000], "mm_per_px": [7.0, 2.0, 18.0, None]},
index=pd.Index([11, 12, 13, 14], name="sensor_id"))
df = pd.DataFrame({"sensor_id": [12, 13, 14, 12], "corr_x": [0.5, 0.1, 0.1, None], "corr_y": [-1, 0.2, 0.2, None]})
df = with_mm(df, props) # T1 measured by its sensor 12 (2 mm/px), not by its older namesake 11 (7 mm/px)
assert df["corr_x_mm"].tolist()[:2] == [1.0, 1.8] and df["corr_y_mm"].tolist()[:2] == [-2.0, 3.6], df
assert df[["corr_x_mm", "corr_y_mm"]].iloc[2:].isna().all().all(), df # no factor, no corr: empty
with tempfile.TemporaryDirectory() as d:
pd.DataFrame({"sensor_name": ["OLD", "T1"], "pattern_type": "target", "distance_mm": [1, 1],
"mm_per_px": [1.0, 1.0]}).to_csv(Path(d) / "targets.csv", index=False)
t = write_targets(Path(d), props, {"T1": 12, "REF": 13})
assert t.values.tolist() == [["OLD", "target", 1, 1.0], ["REF", "global_ref", 50000, 18.0],
["T1", "target", 20000, 2.0]], t
with tempfile.TemporaryDirectory() as d: # given_ranges: the last day pumped and after, pumped again
f, day = Path(d) / "S", lambda n: datetime(2026, 11, n).date()
assert given_ranges(f, day(1), day(19)) == [(day(1), day(19), False)] # nothing pumped
(f / "displacement").mkdir(parents=True)
for n in (2, 18):
(f / "displacement" / f"S_advanced_202611{n:02d}.csv").touch()
assert given_ranges(f, day(1), day(19)) == [(day(1), day(18), False), (day(18), day(19), True)]
assert given_ranges(f, day(18), day(19)) == [(day(18), day(19), True)]
assert given_ranges(f, day(1), day(10)) == [(day(1), day(10), False)] # the last day after the range
print("ok")
if __name__ == "__main__": if __name__ == "__main__":
main() check() if sys.argv[1:] == ["--check"] else main()
+4 -2
View File
@@ -1,6 +1,7 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Pumping under WSL. Reads the credentials from the .env at the root of the repository (never versioned), # Pumping under WSL. Reads the credentials from the .env at the root of the repository (never versioned),
# runs pump.py in a throwaway uv environment (4 packages), then the post-processing of LENZSPITZE. # runs pump.py in a throwaway uv environment (4 packages), then absolute.py (x / y in absolute) and weather.py (the
# MeteoSwiss weather of the stations with a position, data/<STATION>/weather/).
# extraction/pump.sh update of the tracked stations, images included (the scheduler) # extraction/pump.sh update of the tracked stations, images included (the scheduler)
# extraction/pump.sh --from D --to E [SERIAL] any other call: the arguments go to pump.py as they are # extraction/pump.sh --from D --to E [SERIAL] any other call: the arguments go to pump.py as they are
# Output appended to out/pump.log; one pumping at a time (lock out/pump.lock). # Output appended to out/pump.log; one pumping at a time (lock out/pump.lock).
@@ -20,6 +21,7 @@ flock -n 9 || { echo "A pumping is already running." >&2; exit 1; }
uv run --no-project --quiet --python 3.12 \ uv run --no-project --quiet --python 3.12 \
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \ --with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
python "$here/pump.py" "$@") python "$here/pump.py" "$@")
[ -f "$root/data/LENZSPITZE/lentzpitze_patterns.json" ] && python3 "$here/lenzspitze_absolute.py" python3 "$here/absolute.py" # x / y in absolute on every station
python3 "$here/weather.py" # the weather of every station with a position: its missing and last days
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end" echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end"
} 2>&1 | tee -a "$root/out/pump.log" } 2>&1 | tee -a "$root/out/pump.log"
+307
View File
@@ -0,0 +1,307 @@
"""Pumps the differential images (OPTICAL_SHIFT-DIFF) of stations from the AIMsight platform, full size, with what is
needed to measure them again: every pattern of the unit (archived ones included) and the unit's properties. READ-ONLY.
Never touches data/ nor pump.py's pumping; design: docs/superpowers/specs/2026-11-19-pump-diff-design.md.
Run through extraction/pump_diff.sh, under WSL, from anywhere (see README.md):
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU a range (its history starts here)
extraction/pump_diff.sh JUNGFRAU from its last indexed day up to now
extraction/pump_diff.sh the same for every station of data_diff/
Output (default: the repository's data_diff/, next to this folder), the tree of data/ so that the other repositories
read it the same way (docs/superpowers/specs/2026-11-19-configurations-design.md):
<STATION>/images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png the S3 object as is, named as in data/
<STATION>/images_diff/index/<STATION>_diff_<YYYYMMDD>.csv every measurement of the day in the database:
measurement_id, time_utc (exact), measurement_group_id, file (empty without image), status (ok; no_uuid: never
reached S3; s3_missing: S3 returned nothing), properties (the measurement's JSON)
<STATION>/images_diff/sensors.json every sensor of the unit: id, name, type, time_utc, properties [{name, value,
time_utc, status}] (every property row as stored), rewritten at each run
<STATION>/images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties
(under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder)
<STATION>/metrics/, images/ (IRON, one per --every minutes), weather/: as pump.py and weather.py write them into
data/, with their own functions. Not targets.csv: it depends on the run (its patterns), each run writes its own
(processing/run_config.py)
Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and
the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index
is written after its day's metrics, IRON and DIFF images, every DIFF file through a temporary one: an indexed day is
complete; a day whose metrics failed gets no index. Without --from, a station with nothing indexed is skipped (its
history starts only on an explicit --from). Stops before a day when the disk has less than --min-free-gb left.
Same reads as pump.py (measurement, acquisition_unit; the copied queries of the app, branch `explo`), plus the
`property` rows of the sensors and of the unit (AcquisitionUnit.list_sensors, Sensor.properties of
app/aimsight_database.py). Variables read (names only, none is printed): AIMSIGHT_DATABASE_USER / _PSWD / _TURL /
_NAME / _PORT, AIMSIGHT_S3_URL / _ACCESS_KEY / _SECRET_KEY / _BUCKET.
"""
import argparse
import csv
import io
import json
import os
import shutil
import sys
import time
from collections import Counter
from datetime import date, datetime, timedelta, timezone
from pathlib import Path
UTC = timezone.utc
MTYPE = "OPTICAL_SHIFT-DIFF"
INDEX = ["measurement_id", "time_utc", "measurement_group_id", "file", "status", "properties"]
def write_atomic(path, data):
"""A file is either complete or absent: a run stopped in the middle leaves nothing that looks done."""
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_name(path.name + ".tmp")
tmp.write_bytes(data)
os.replace(tmp, path)
def indexed_days(folder):
return {datetime.strptime(p.stem.rsplit("_", 1)[1], "%Y%m%d").date()
for p in (folder / "images_diff" / "index").glob(f"{folder.name}_diff_*.csv")}
def days_to_pump(indexed, start, end):
"""The days of [start, end) to read: the missing ones, and from the last indexed day on (perhaps incomplete).
Without start: from the last indexed day; None when nothing is indexed."""
last = max(indexed, default=None)
if start is None:
if last is None:
return None
start = last
days = (start + timedelta(days=k) for k in range((end - start).days))
return [d for d in days if last is None or d not in indexed or d >= last]
def pump_day(rows, client, bucket, ext, folder, station, day):
"""The images of one day's measurements (rows: id, time_utc, measurement_group_id, measurement_uuid, properties),
then its index. Returns the counts per status, plus the new images and their bytes."""
c, index = Counter(), []
for m in rows:
t = m.time_utc.astimezone(UTC)
name, status = f"{t:%Y%m%dT%H%M}Z_{m.id}.{ext}", "ok"
path = folder / "images_diff" / name
if m.measurement_uuid is None: # the image never reached S3 (unfinished multipart upload)
name, status = "", "no_uuid"
elif not path.exists():
try:
data = client.get_object(Bucket=bucket, Key=m.measurement_uuid)["Body"].read()
except Exception: # like the app's get_measurement_payload: S3 not answering, the image is missing
data = None
if data:
write_atomic(path, data)
c["new"], c["bytes"] = c["new"] + 1, c["bytes"] + len(data)
else:
name, status = "", "s3_missing"
c[status] += 1
index.append([m.id, t.isoformat(), "" if m.measurement_group_id is None else m.measurement_group_id, name,
status, "" if m.properties is None else json.dumps(m.properties, sort_keys=True)])
out = io.StringIO()
w = csv.writer(out, lineterminator="\n")
w.writerow(INDEX)
w.writerows(index)
write_atomic(folder / "images_diff" / "index" / f"{station}_diff_{day:%Y%m%d}.csv", out.getvalue().encode())
return c
def prop(name, value, t, status):
return {"name": name, "value": value, "time_utc": t.isoformat(), "status": status}
def sensors_json(conn, unit):
"""Every sensor of the unit, any type, archived included (`archive` is one of their properties), with every
property row: a pattern redrawn on the platform is a new sensor, the old one stays."""
from sqlalchemy import text
rows = conn.execute(text("""
SELECT s.id, s.name, s.type, s.time_utc, dt.name AS prop, p.value, p.time_utc AS ptime, p.status
FROM sensor s LEFT JOIN property p ON p.sensor_id = s.id LEFT JOIN datatype dt ON dt.id = p.datatype_id
WHERE s.acquisition_unit_id = :au ORDER BY s.id, dt.name, p.time_utc"""), {"au": unit}).all()
sensors = {}
for r in rows:
s = sensors.setdefault(r.id, {"id": r.id, "name": r.name, "type": r.type, "time_utc": r.time_utc.isoformat(),
"properties": []})
if r.prop is not None:
s["properties"].append(prop(r.prop, r.value, r.ptime, r.status))
return list(sensors.values())
def unit_json(conn, unit):
"""The acquisition unit and its own property rows (focal_distance, pixel_size, algorithm...), not its sensors'."""
from sqlalchemy import text
u = conn.execute(text("SELECT id, serial_number, product_number FROM acquisition_unit WHERE id = :au"),
{"au": unit}).one()
rows = conn.execute(text("""
SELECT dt.name, p.value, p.time_utc, p.status FROM property p JOIN datatype dt ON dt.id = p.datatype_id
WHERE p.acquisition_unit_id = :au AND p.sensor_id IS NULL ORDER BY dt.name, p.time_utc"""), {"au": unit}).all()
return {"id": u.id, "serial_number": u.serial_number, "product_number": u.product_number,
"properties": [prop(*r) for r in rows]}
def to_json(obj):
return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode()
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("stations", nargs="*", help="serial number of the system, as in the URL of the System page; "
"none: the stations already in --out")
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD; without: the last indexed day of each "
"station (a station with nothing indexed is skipped)")
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD; without: up to now")
p.add_argument("--delay", type=int, default=60, help="maximum duration of a query, in s (default 60: database "
"shared with production)")
p.add_argument("--every", type=int, default=60, help="IRON images: one per slot of N minutes, 0 = all (default 60, "
"as pump.sh)")
p.add_argument("--min-free-gb", type=float, default=50, help="stop before a day when less is free (default 50)")
p.add_argument("--out", type=Path, default=Path(__file__).resolve().parent.parent / "data_diff")
a = p.parse_args()
import boto3
from sqlalchemy import create_engine, text
from sqlalchemy.engine import URL
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
end = min(datetime.strptime(a.end, "%Y-%m-%d").date(), tomorrow) if a.end else tomorrow
stations = a.stations or sorted(f.name for f in a.out.glob("*") if f.is_dir())
if not stations:
sys.exit(f"No station: name one, with --from (nothing in {a.out} yet)")
missing = [n for n in DB_NAMES if not os.environ.get(n)]
if missing:
sys.exit(f"Missing variables (names only): {', '.join(missing)}")
# the engine and the S3 client of pump.py's main
url = URL.create("postgresql+psycopg2", username=os.environ["AIMSIGHT_DATABASE_USER"],
password=os.environ["AIMSIGHT_DATABASE_PSWD"], host=os.environ["AIMSIGHT_DATABASE_TURL"],
port=int(os.environ.get("AIMSIGHT_DATABASE_PORT", "5432")),
database=os.environ["AIMSIGHT_DATABASE_NAME"])
engine = create_engine(url, connect_args={
"options": f"-c timezone=utc -c default_transaction_read_only=on -c statement_timeout={a.delay}s"})
client = boto3.client("s3", endpoint_url=os.environ.get("AIMSIGHT_S3_URL", "http://s3:3900"),
aws_access_key_id=os.environ.get("AIMSIGHT_S3_ACCESS_KEY", ""),
aws_secret_access_key=os.environ.get("AIMSIGHT_S3_SECRET_KEY", ""),
region_name="garage", use_ssl=False)
bucket = os.environ.get("AIMSIGHT_S3_BUCKET", "measurements-assets")
try:
with engine.connect() as conn:
mt = conn.execute(text("SELECT id, extension FROM measurement_type WHERE name = :n"), {"n": MTYPE}).first()
conn.rollback()
if mt is None:
sys.exit(f"Unknown measurement type {MTYPE}")
for station in stations:
system_id = conn.execute(text("SELECT id FROM system WHERE serial_number = :s"),
{"s": station}).scalar()
unit = system_id and acquisition_unit(conn, system_id)
conn.rollback()
if not unit:
print(f"{station}: " + ("unknown system" if system_id is None else "no recognised acquisition unit"))
continue
folder = a.out / station
days = days_to_pump(indexed_days(folder), start, end)
if days is None:
print(f"{station}: nothing indexed yet, its history starts with --from")
continue
try: # what the database holds, so that an empty day is explained (LENZSPITZE: none after 2026-10-13)
first, last = (conn.execute(text(
"SELECT time_utc FROM measurement WHERE acquisition_unit_id = :au AND measurement_type_id = :mt "
f"ORDER BY time_utc {o} LIMIT 1"), {"au": unit, "mt": mt.id}).scalar() for o in ("ASC", "DESC"))
print(f"{station}: differential images in the database from {first:%Y-%m-%d} to {last:%Y-%m-%d}"
if first else f"{station}: no differential image in the database")
sens = sensors_json(conn, unit)
write_atomic(folder / "images_diff" / "sensors.json", to_json(sens))
write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit)))
except Exception as e: # the next run writes them
print(f"{station} images range, sensors.json, unit.json: failed ({root_cause(e)}), run again")
finally:
conn.rollback()
print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}")
for day in days:
free = shutil.disk_usage(a.out).free / 1e9
if free < a.min_free_gb:
sys.exit(f"Stopped before {station} {day}: {free:.0f} GB free, under --min-free-gb "
f"{a.min_free_gb:g}")
nxt = day + timedelta(days=1)
try: # as pump.py into data/; a day whose metrics or IRON images failed stays open (no index)
if not pump_metrics(conn, system_id, station, day, nxt, a.out, True):
print(f"{station} {day}: left open (metrics failed), run again")
continue
pump_images(conn, (client, bucket), unit, station, day, nxt, "images", a.every, a.out)
except Exception as e:
print(f"{station} {day}: left open ({root_cause(e)}), run again")
continue
finally:
conn.rollback()
a0 = datetime.combine(day, datetime.min.time(), UTC)
try:
rows = conn.execute(text(
"SELECT id, time_utc, measurement_group_id, measurement_uuid, properties FROM measurement "
"WHERE acquisition_unit_id = :au AND measurement_type_id = :mt AND time_utc >= :a "
"AND time_utc < :b ORDER BY time_utc"),
{"au": unit, "mt": mt.id, "a": a0, "b": a0 + timedelta(days=1)}).all()
except Exception as e: # no index for the day: the next run reads it again
print(f"{station} {day}: failed ({root_cause(e)}), run again")
continue
finally:
conn.rollback()
c = pump_day(rows, client, bucket, mt.extension, folder, station, day)
print(f"{station} {day}: {c['new']} new images, {c['bytes'] / 1e6:.0f} MB ({len(rows)} in "
f"database: {c['ok']} on disk, {c['no_uuid']} never reached S3, {c['s3_missing']} not "
"returned by S3)")
time.sleep(0.5)
try: # the MeteoSwiss weather of the days now in --out, as pump.sh does for data/
import weather
weather.main(stations, a.out)
except Exception as e: # the next run fetches it
print(f"weather: failed ({type(e).__name__}: {e}), run again")
finally:
engine.dispose()
def check():
"""Without database or S3: `python3 extraction/pump_diff.py --check`."""
import tempfile
from types import SimpleNamespace
d = lambda n: date(2026, 11, n) # noqa: E731
assert days_to_pump(set(), d(1), d(4)) == [d(1), d(2), d(3)]
assert days_to_pump({d(1), d(2), d(5)}, d(1), d(7)) == [d(3), d(4), d(5), d(6)] # the last indexed day: again
assert days_to_pump({d(2)}, None, d(5)) == [d(2), d(3), d(4)]
assert days_to_pump(set(), None, d(5)) is None # no --from, nothing indexed: skipped
class S3:
def get_object(self, Bucket, Key):
if Key == "refused":
raise OSError("NoSuchKey")
return {"Body": io.BytesIO(b"png")}
t = lambda h, m: datetime(2026, 11, 3, h, m, 7, 250000, tzinfo=UTC) # noqa: E731
rows = [SimpleNamespace(id=1, time_utc=t(10, 0), measurement_group_id=5, measurement_uuid="a", properties={"e": 1}),
SimpleNamespace(id=2, time_utc=t(10, 0), measurement_group_id=5, measurement_uuid="b", properties=None),
SimpleNamespace(id=3, time_utc=t(10, 15), measurement_group_id=None, measurement_uuid=None, properties={}),
SimpleNamespace(id=4, time_utc=t(10, 30), measurement_group_id=None, measurement_uuid="refused",
properties={})]
with tempfile.TemporaryDirectory() as tmp:
folder = Path(tmp) / "S"
(folder / "images_diff").mkdir(parents=True)
(folder / "images_diff" / "20261103T1000Z_2.png").write_bytes(b"old") # a burst: same minute, other id
c = pump_day(rows, S3(), "bucket", "png", folder, "S", d(3))
assert dict(c) == {"ok": 2, "new": 1, "bytes": 3, "no_uuid": 1, "s3_missing": 1}, c
assert (folder / "images_diff" / "20261103T1000Z_1.png").read_bytes() == b"png"
assert (folder / "images_diff" / "20261103T1000Z_2.png").read_bytes() == b"old" # never downloaded again
assert not list(folder.rglob("*.tmp"))
with open(folder / "images_diff" / "index" / "S_diff_20261103.csv", newline="") as fp:
index = list(csv.DictReader(fp))
assert [r["status"] for r in index] == ["ok", "ok", "no_uuid", "s3_missing"], index
assert [r["file"] for r in index] == ["20261103T1000Z_1.png", "20261103T1000Z_2.png", "", ""], index
assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}'
assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index
assert indexed_days(folder) == {d(3)}
print("ok")
if __name__ == "__main__":
check() if sys.argv[1:] == ["--check"] else main()
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env bash
# Pumping of the differential images under WSL, into data_diff/ (never data/). Reads the credentials from the .env at
# the root of the repository (never versioned) and runs pump_diff.py in the throwaway uv environment of pump.sh; no
# absolute.py nor weather.py (they work on data/).
# extraction/pump_diff.sh --from D [--to E] SERIAL... a range; the history of a station starts here
# extraction/pump_diff.sh [SERIAL...] from the last indexed day up to now
# Output appended to out/pump_diff.log; one at a time (lock out/pump_diff.lock), independent of pump.sh's.
set -u
export PATH="$HOME/.local/bin:$PATH" # uv, missing from the PATH when the Task Scheduler starts wsl.exe
here=$(cd "$(dirname "$0")" && pwd)
root=$(dirname "$here")
mkdir -p "$root/out"
exec 9>"$root/out/pump_diff.lock"
flock -n 9 || { echo "A pumping of the differential images is already running." >&2; exit 1; }
{
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) pump_diff.py $*"
(set -a +u && . "$root/.env" && set +a -u && # +u: an unquoted $ in .env does not stop the load
uv run --no-project --quiet --python 3.12 \
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
python "$here/pump_diff.py" "$@")
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end"
} 2>&1 | tee -a "$root/out/pump_diff.log"
+102
View File
@@ -0,0 +1,102 @@
"""The pumping, for its callers (aimsight-app): the tracked stations, launching pump.sh, its state.
- `tracked()`, `save(rows)`: extraction/stations.txt, one station per line, "SERIAL [position]" (position: "lat,lon" or a
Swiss place, for the weather, weather/meteoswiss.py; empty: an indoor site, no weather); pump.py reads it too;
- `start(stations, start, end, images, diff)`: pump.sh in the background (its lock: one pumping at a time); with diff,
pump_diff.sh (the differential images into data_diff/, its own lock: it runs beside pump.sh);
- `running(diff)`, `log(diff)`: a pumping running (any: the scheduler's, by hand, a caller's), the end of out/pump.log
(diff: out/pump_diff.log).
A refused request raises ValueError, with its reason. Standard library only.
"""
import fcntl
import os
import re
import subprocess
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SERIAL = re.compile(r"[A-Za-z0-9_-]+$") # it becomes an argument of pump.sh and a folder of data/
DAY = re.compile(r"\d{4}-\d{2}-\d{2}$")
def tracked():
"""{serial: position ("" without)}, in the order of the file, without duplicates."""
f = ROOT / "extraction" / "stations.txt"
out = {}
for line in f.read_text().splitlines() if f.is_file() else []:
serial, _, position = line.strip().partition(" ")
if serial:
out.setdefault(serial, position.strip())
return out
def save(rows):
"""rows: [{serial, position}] -> the new stations.txt, and its content as tracked()."""
out = {}
for r in rows:
serial, position = str(r.get("serial", "")).strip(), " ".join(str(r.get("position", "")).split())
if not SERIAL.match(serial):
raise ValueError(f"not a serial number: {serial!r}")
out.setdefault(serial, position)
(ROOT / "extraction" / "stations.txt").write_text("".join(f"{s} {p}".rstrip() + "\n" for s, p in out.items()))
return out
def name(diff):
return "pump_diff" if diff else "pump"
def running(diff=False):
"""A pumping running: the lock of pump.sh (diff: pump_diff.sh) is held."""
try:
with open(ROOT / "out" / f"{name(diff)}.lock") as lock:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except FileNotFoundError:
return False
except BlockingIOError:
return True
return False
def log(lines=40, diff=False):
"""The last lines of out/pump.log (diff: out/pump_diff.log)."""
try:
with open(ROOT / "out" / f"{name(diff)}.log", "rb") as f:
f.seek(max(0, f.seek(0, os.SEEK_END) - 200 * lines))
return f.read().decode(errors="replace").splitlines()[-lines:]
except FileNotFoundError:
return []
def start(stations, start="", end="", images=False, diff=False):
"""pump.sh in the background, with --from start --to end (YYYY-MM-DD, end excluded), or without (the missing days of
each station up to now); --delay 300. With diff, pump_diff.sh (without range: from the last indexed day, a station
never pumped is skipped; images ignored, its IRON images one per hour). -> the process."""
if not stations or not all(isinstance(s, str) and SERIAL.match(s) for s in stations):
raise ValueError("no station, or not a serial number")
if bool(start) != bool(end) or (start and not (DAY.match(start) and DAY.match(end) and start < end)):
raise ValueError("a range: From and To, YYYY-MM-DD, From before To")
if running(diff):
raise ValueError("a pumping is already running")
args = ["--from", start, "--to", end] * bool(start) + ["--images"] * bool(images and not diff) + ["--delay", "300", *stations]
return subprocess.Popen([str(ROOT / "extraction" / f"{name(diff)}.sh"), *args], cwd=ROOT,
stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if __name__ == "__main__": # a check, in a temporary folder: python3 extraction/pumping.py
import tempfile
ROOT = Path(tempfile.mkdtemp())
(ROOT / "extraction").mkdir()
(ROOT / "extraction" / "stations.txt").write_text("NORDEND\nNIEDERHORN 46.69,7.78\n\nNORDEND Zermatt\n")
assert tracked() == {"NORDEND": "", "NIEDERHORN": "46.69,7.78"}, tracked()
assert save([{"serial": "A", "position": " Interlaken BE "}, {"serial": "B"}, {"serial": "A"}]) == {"A": "Interlaken BE", "B": ""}
assert (ROOT / "extraction" / "stations.txt").read_text() == "A Interlaken BE\nB\n"
assert not running() and log() == [] and not running(diff=True) and log(diff=True) == []
for bad in ((save, [{"serial": "../x"}]), (start, ["A;rm"]), (start, ["A"], "2026-11-10"), (start, ["A"], "2026-11-10", "2026-11-09")):
try:
bad[0](*bad[1:])
raise AssertionError(f"not refused: {bad}")
except ValueError:
pass
print("ok")
+7 -5
View File
@@ -1,13 +1,15 @@
BREITHORN BREITHORN
CLARIDEN CLARIDEN
FLORIDA FLORIDA Ernen
JUNGFRAU JUNGFRAU Nyon
LENZSPITZE LENZSPITZE Crissier
MICHIGAN MICHIGAN
NADELHORN NADELHORN Genève
NEBRASKA NEBRASKA
NIEDERHORN NIEDERHORN
NORDEND NORDEND
WILDSTRUBEL WILDSTRUBEL
SUSTENHORN SUSTENHORN
SNCF01 SNCF01
ALASKA Genève
WYOMING Genève
+86
View File
@@ -0,0 +1,86 @@
"""The weather of each tracked station with a position, into data/<STATION>/weather/. Run by pump.sh after each pumping.
One CSV per day that has data in the station's folder: data/<STATION>/weather/<STATION>_weather_<YYYYMMDD>.csv, the
hourly values of the MeteoSwiss station nearest to the position (weather/meteoswiss.py), UTC, columns: time_utc, position
(the line of stations.txt), place (found), station, station_name, distance_km, height_m, then the measurements
(meteoswiss.PARAMETERS). A day without MeteoSwiss values gets the header only.
Fetched again: a missing day, a day written for another position, the last 2 days (incomplete, not yet quality-checked).
A station without a position (an indoor site) or whose place is not found: no weather/ folder, a line in the log.
Usage: python3 extraction/weather.py [--data DIR] [SERIAL...] (default: every tracked station, into data/; --data:
another root with the tree of data/, e.g. data_diff/, which pump_diff.py passes)
"""
import csv
import functools
import re
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path[:0] = [str(ROOT / "extraction"), str(ROOT / "weather")]
import meteoswiss # noqa: E402
import pumping # noqa: E402
DATA = ROOT / "data"
DAY = re.compile(r"(?<!\d)(20\d\d)_?(\d\d)_?(\d\d)(?!\d)") # ALASKA_advanced_20261017.csv, _2026_10_17_00_00 to …, 20261029T1617Z_…
RECENT = timedelta(days=2)
# The files of MeteoSwiss (one per station, decade, recent, now) are read once per run, for every station that shares them.
meteoswiss.fetch = functools.lru_cache(maxsize=None)(meteoswiss.fetch)
def days_with_data(folder):
"""The days (UTC dates) of the files of the station, its weather aside."""
return {datetime(*map(int, m.groups()), tzinfo=timezone.utc) for p in folder.rglob("*")
if p.is_file() and "weather" not in p.relative_to(folder).parts and (m := DAY.search(p.name))}
def position_of(path):
"""The position a weather file was written for; None without a row."""
with open(path, newline="") as f:
return next(csv.DictReader(f), {}).get("position")
def station(serial, position, now, data=DATA):
folder = data / serial
wanted = sorted(d for d in days_with_data(folder) if d <= now)
out = folder / "weather"
path = lambda d: out / f"{serial}_weather_{d:%Y%m%d}.csv" # noqa: E731
todo = [d for d in wanted if not path(d).is_file() or now - d < RECENT or position_of(path(d)) not in (position, None)]
if not todo:
return
where, hours = meteoswiss.hourly(position, todo[0], todo[-1] + timedelta(days=1))
meta = [position, where["place"], where["station"], where["name"], f"{where['distance_km']:.1f}", f"{where['height_m']:.0f}"]
out.mkdir(exist_ok=True)
for d in todo:
with open(path(d), "w", newline="") as f:
w = csv.writer(f)
w.writerow(["time_utc", "position", "place", "station", "station_name", "distance_km", "height_m",
*meteoswiss.PARAMETERS.values()])
for t, values in hours.items():
if d <= t < d + timedelta(days=1):
w.writerow([t.strftime("%Y-%m-%dT%H:%MZ"), *meta, *("" if v is None else f"{v:g}" for v in values.values())])
print(f"{serial}: weather of {len(todo)} days, {where['place']} -> MeteoSwiss {where['station']} {where['name']}, "
f"{where['distance_km']:.1f} km")
def main(serials, data=DATA):
tracked, now = pumping.tracked(), datetime.now(timezone.utc)
for serial in serials or tracked:
if not (Path(data) / serial).is_dir():
continue
if not (position := tracked.get(serial)):
print(f"{serial}: no position in stations.txt, no weather")
continue
try:
station(serial, position, now, Path(data))
except (LookupError, OSError) as e: # the pumping goes on without the weather of this station
print(f"{serial}: no weather: {e}")
if __name__ == "__main__":
args = sys.argv[1:]
if args[:1] == ["--data"]:
main(args[2:], Path(args[1]))
else:
main(args)
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Creates a new, empty run: data_diff/<SERIAL>/runs/<run>/config.yaml with the cloud's processing written in full (the
# patterns of images_diff/sensors.json, the camera, optical's current commit); no result until it is run.
# processing/new_run.sh JUNGFRAU run02
# processing/run_config.sh JUNGFRAU run02 then, once config.yaml is edited
exec "$(dirname "$0")/run_config.sh" "$@" --new
+411
View File
@@ -0,0 +1,411 @@
"""Runs one configuration of a station: data_diff/<SERIAL>/runs/<run>/config.yaml in, the same folder's displacement/,
camera/ and targets.csv out, in the files of data/. The station's pumped data, the same for every run (images_diff/,
images/, metrics/, weather/), stay at data_diff/<SERIAL>/ and are only read; whatever depends on the run is in its folder.
processing/run_config.sh JUNGFRAU run02 --from 2026-10-12 --to 2026-10-17
processing/run_config.sh JUNGFRAU run02 every indexed day
Days: those indexed in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). Running a run again
replaces its displacement/, camera/, targets.csv and patterns.json; nothing else is written:
displacement/<S>_advanced_<YYYYMMDD>.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator;
relative = x - pattern x; corr from the camera correction; reproj_err,
inlier of the references; sigma0 of the epoch; corr_*_mm
camera/<S>_camera_<YYYYMMDD>.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute
targets.csv as in data/, from the run's patterns: pattern_type, distance_mm, mm_per_px
patterns.json the patterns exactly as given to the emulator (templates included)
config.yaml states what differs from the cloud's processing (an empty file: the cloud's), checked key by key:
patterns: by sensor_name, over images_diff/sensors.json: any existing property, or
PM_G: {error_threshold: 0.7} a template redrawn from a DIFF image of images_diff/, as the
BL_C: {redraw: {image: <file>, corners: [[x0, y0], [x1, y1]]}} device's pattern editor does (x_ref kept)
camera: over images_diff/unit.json, over the cloud's defaults
algorithm: ransac ransac | umeyama | danish_hybrid | trusted_danish
ransac_threshold: 0.99
mode: "off" off | rotation | rotation_scale | relative (optical's refine_allpoints)
focal_distance: 6.0 pixel_size: 0.0024
per_epoch: true one measurement at a time, as the cloud; false: the whole series at once
emulator:
version: "65a4c70" optical's commit: the run stops if $OPTICAL is at another one or has local changes
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
"""
import argparse
import csv
import datetime
import json
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
# Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and
# 8 workers x one BLAS thread per core made a 11 s re-measurement take 2 min (45 min of CPU).
for _var in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS"):
os.environ.setdefault(_var, "1")
import numpy as np # noqa: E402
import pandas as pd # noqa: E402
import yaml # noqa: E402
ROOT = Path(__file__).resolve().parents[1]
OPTICAL = Path(os.environ.get("OPTICAL", Path.home() / "optical")).resolve()
sys.path[:0] = [str(OPTICAL), str(OPTICAL / "scripts" / "pcc_emulator")]
MTYPE = "OPTICAL_SHIFT-DIFF"
QUALITY = ["peak", "flux", "sharpness", "fwhm", "dominance", "snr"]
COLUMNS = ["time_utc", "sensor_name", "x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x",
"corr_y", "reproj_err", "inlier", "sigma0", "pattern_type"] + QUALITY + ["corr_x_mm", "corr_y_mm"]
CAM = ["pan", "tilt", "scale", "rotation", "sigma0"]
# the cloud's defaults (process_shifts), then the unit's properties, then the configuration
DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0,
"pixel_size": 0.0024, "per_epoch": True}
UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size")
OUTPUTS = ("displacement", "camera") # what a run writes into its folder, and replaces
def _keys(d, allowed, where):
if not isinstance(d, dict):
raise ValueError(f"{where}: a mapping is expected")
bad = set(d) - set(allowed)
if bad:
raise ValueError(f"{where}: unknown key(s) {sorted(bad)}; allowed: {sorted(allowed)}")
def load_config(path):
"""The configuration, its keys checked (a typo is an error, not a setting silently ignored)."""
c = yaml.safe_load(Path(path).read_text()) or {}
_keys(c, {"patterns", "camera", "emulator"}, str(path))
_keys(c.get("camera") or {}, DEFAULTS, f"{path}: camera")
_keys(c.get("emulator") or {}, {"version"}, f"{path}: emulator")
_keys(c.get("patterns") or {}, set(c.get("patterns") or {}), f"{path}: patterns") # a mapping (names: later)
return c
def latest(rows):
"""{name: value} of property rows ordered by name then time (sensors.json, unit.json): the latest wins."""
return {r["name"]: r["value"] for r in rows}
def camera_settings(config, unit_props):
cam = dict(DEFAULTS)
cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props})
cam.update(config.get("camera") or {})
return cam
def cloud_properties(sensors):
"""{sensor_id: properties} of the patterns the cloud measures the DIFF images with (process_shifts): DIFF type, a
pattern, not archived."""
out = {}
for s in sensors:
p = latest(s["properties"])
if s["type"] == MTYPE and p.get("pattern") and not p.get("archive"):
out[str(s["id"])] = p
return out
def apply_patterns(props, overrides, images_dir):
"""The configuration's patterns over the cloud's, by sensor_name: a property replaced (only an existing one: a
typo is an error), or the template redrawn from an image."""
props = {sid: dict(p) for sid, p in props.items()}
by_name = {p.get("sensor_name"): sid for sid, p in props.items()}
redraws = {}
for name, ov in (overrides or {}).items():
if name not in by_name:
raise ValueError(f"patterns.{name}: no such pattern; known: {sorted(map(str, by_name))}")
sid = by_name[name]
_keys(ov, set(props[sid]) - {"pattern"} | {"redraw"}, f"patterns.{name}")
for k, v in ov.items():
if k == "redraw":
_keys(v, {"image", "corners"}, f"patterns.{name}.redraw")
redraws[sid] = v
else:
props[sid][k] = v
if redraws:
redraw(props, redraws, images_dir)
return props
def redraw(props, redraws, images_dir):
"""As the device's pattern editor saves a drawn box (optical_server pages/patternCreation.py): the camera pose from
the references found in the image, the anchor and x_ref (kept) of saved_anchor_and_reference, the template cut
at the box."""
from aimsight_optical.imagePatternMono import ImagePatternMono, get_center, saved_anchor_and_reference
from aimsight_optical.pattern_converter import load_pattern_from_params
from aimsight_optical.pattern_pose import estimate_camera_pose
from reprocess_images import read_image
for image in sorted({r["image"] for r in redraws.values()}):
im = read_image(images_dir / image)
stored = {sid: load_pattern_from_params(**dict(p)) for sid, p in props.items()}
mine = {sid: r for sid, r in redraws.items() if r["image"] == image}
drawn = {sid: get_center(r["corners"]) for sid, r in mine.items()
if props[sid].get("pattern_type") == "global_ref"}
pose, seen = estimate_camera_pose(stored, im, drawn=drawn)
for sid, r in mine.items():
old, corners = stored[sid], [[int(v) for v in c] for c in r["corners"]]
anchor, (x_ref, y_ref) = saved_anchor_and_reference(old, corners, old.extra_attr.get("pattern_type", ""),
pose=pose, seen=seen.get(sid))
kept = {k: v for k, v in old.extra_attr.items() if k not in ("median", "p99", "x_ref", "y_ref")}
new = ImagePatternMono.from_image(im, corners, center=anchor, x_ref=x_ref, y_ref=y_ref, **kept)
props[sid] = json.loads(json.dumps(new.to_dict(), default=_plain))
def _plain(o):
return o.item() if hasattr(o, "item") else o.tolist() if hasattr(o, "tolist") else str(o)
def correct(meas, props, cam):
"""The emulator's rows (time_utc, pattern_id, x, y, error, metrics) -> (advanced rows in the columns of data/,
camera rows per minute), as the cloud's process_shifts with per_epoch, else over the whole series."""
from aimsight_optical.imagePatternMono import get_center, reference_or_anchor
from aimsight_optical.pattern_converter import remove_camera_movement
m = meas.copy()
m["sensor_id"] = m["pattern_id"].astype(str)
m["time_utc"] = pd.to_datetime(m["time_utc"], utc=True, format="mixed")
p = {sid: props[sid] for sid in m["sensor_id"].unique()}
anchor = {sid: (float(q["x"]), float(q["y"])) if np.isfinite([q.get("x", np.nan), q.get("y", np.nan)]).all()
else get_center(q["corners"]) for sid, q in p.items()}
ref = {sid: reference_or_anchor(q.get("x_ref"), q.get("y_ref"), anchor[sid]) for sid, q in p.items()}
sid = m["sensor_id"]
m["sensor_name"] = sid.map({s: str(q.get("sensor_name", s)) for s, q in p.items()})
m["error_threshold"] = pd.to_numeric(sid.map({s: q.get("error_threshold", 0.7) for s, q in p.items()}))
m["sensor_type"] = sid.map({s: q.get("pattern_type", "") or "" for s, q in p.items()})
m["x_ref"], m["y_ref"] = sid.map({s: r[0] for s, r in ref.items()}), sid.map({s: r[1] for s, r in ref.items()})
inp = m[["time_utc", "sensor_id", "x", "y", "error", "error_threshold", "sensor_type", "x_ref", "y_ref"]].copy()
# as the cloud: a target above its threshold has no position; a reference keeps its raw one (its row is corrected,
# remove_camera_movement leaves it out of the fit on its error)
inp.loc[(inp["error"] > inp["error_threshold"]) & (inp["sensor_type"] != "global_ref"), ["x", "y"]] = np.nan
kw = {"algorithm": cam["algorithm"]}
if str(cam["algorithm"]).upper() == "RANSAC":
kw["ransac_threshold"] = cam["ransac_threshold"]
if cam["mode"] != "off":
kw["refine_allpoints"] = cam["mode"]
frames = [g for _, g in inp.groupby("time_utc")] if cam["per_epoch"] else [inp]
corrs, cams, stats = [], [], []
for g in frames:
c, cp, rs = remove_camera_movement(g.copy(), cam["focal_distance"], cam["pixel_size"],
timeserie_column="time_utc", **kw)
corrs.append(c[["time_utc", "sensor_id", "corr_x", "corr_y"]])
if not cp.empty:
cams.append(cp[CAM].rename_axis("time_utc").reset_index())
if not rs.empty:
stats.append(rs[["time_utc", "sensor_id", "reproj_err", "inlier"]])
m = m.merge(pd.concat(corrs), on=["time_utc", "sensor_id"], how="left")
m = m.merge(pd.concat(stats) if stats else pd.DataFrame(columns=["time_utc", "sensor_id", "reproj_err", "inlier"]),
on=["time_utc", "sensor_id"], how="left")
cp = pd.concat(cams).dropna(subset=CAM, how="all") if cams else pd.DataFrame(columns=["time_utc"] + CAM)
cp["time_utc"] = pd.to_datetime(cp["time_utc"], utc=True)
m["sigma0"] = m["time_utc"].map(cp.groupby("time_utc")["sigma0"].median())
factor = pd.to_numeric(sid.map({s: q.get("distance_convertion_factor") for s, q in p.items()}), errors="coerce")
adv = pd.DataFrame({
"time_utc": m["time_utc"], "sensor_name": m["sensor_name"], "x": m["x"], "y": m["y"],
"error_value": m["error"], "error_threshold": m["error_threshold"],
"relative_x": m["x"] - sid.map({s: a[0] for s, a in anchor.items()}),
"relative_y": m["y"] - sid.map({s: a[1] for s, a in anchor.items()}),
"corr_x": m["corr_x"] - m["x_ref"], "corr_y": m["corr_y"] - m["y_ref"],
"reproj_err": m["reproj_err"], "inlier": m["inlier"], "sigma0": m["sigma0"], "pattern_type": m["sensor_type"],
**{q: m[q] if q in m else np.nan for q in QUALITY}})
adv["corr_x_mm"], adv["corr_y_mm"] = adv["corr_x"] * factor.to_numpy(), adv["corr_y"] * factor.to_numpy()
adv = adv.sort_values(["time_utc", "sensor_name"], kind="stable").reset_index(drop=True)[COLUMNS]
cp["time_utc"] = cp["time_utc"].dt.floor("1min")
agg = cp.groupby("time_utc")[CAM].agg(["median", "min", "max"])
agg.columns = [f"{stat}_{c}" for c, stat in agg.columns]
return adv, agg.reset_index()
def write_run(adv, camera, dest, station):
"""displacement/ and camera/ of a run, one file per UTC day, replacing the run's previous ones."""
for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")):
if (dest / folder).exists():
shutil.rmtree(dest / folder)
(dest / folder).mkdir()
for day, d in df.groupby(df["time_utc"].dt.date):
d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False)
def optical_state():
git = lambda *a: subprocess.run(["git", "-C", str(OPTICAL), *a], capture_output=True, text=True).stdout.strip() # noqa: E731
return {"path": str(OPTICAL), "commit": git("rev-parse", "HEAD"), "branch": git("rev-parse", "--abbrev-ref", "HEAD"),
"dirty": bool(git("status", "--porcelain", "--untracked-files=no"))}
PATTERN_KEYS = ["pattern_type", "corners", "x", "y", "error_threshold", "resolution", "distance_convertion_factor"]
CAMERA_NOTES = {"algorithm": "ransac | umeyama | danish_hybrid | trusted_danish", "ransac_threshold": "px, ransac only",
"mode": "off | rotation | rotation_scale | relative", "focal_distance": "mm; pan, tilt, sigma0 only",
"pixel_size": "mm; pan, tilt, sigma0 only",
"per_epoch": "one measurement at a time, as the cloud; false: the whole series at once"}
def new_run(src, name):
"""runs/<name>/config.yaml of a new, empty run (no result yet): the cloud's processing written in full, the patterns
of images_diff/sensors.json, the camera of unit.json or the cloud's defaults, optical's current commit."""
run = src / "runs" / name
if (run / "config.yaml").exists():
raise FileExistsError(f"{run / 'config.yaml'} exists: choose another name, or edit that file")
props = sorted(cloud_properties(json.loads((src / "images_diff" / "sensors.json").read_text())).values(),
key=lambda p: (p.get("pattern_type") != "global_ref", str(p.get("sensor_name"))))
unit = latest(json.loads((src / "images_diff" / "unit.json").read_text())["properties"])
cam, state = camera_settings({}, unit), optical_state()
width = max(len(str(p.get("sensor_name"))) for p in props) + 1
line = lambda p: f" {str(p.get('sensor_name')) + ':':{width}s} {{" + ", ".join( # noqa: E731
f"{k}: {json.dumps(p[k])}" for k in PATTERN_KEYS if k in p) + "}"
refs = [line(p) for p in props if p.get("pattern_type") == "global_ref"]
targets = [line(p) for p in props if p.get("pattern_type") != "global_ref"]
text = f"""# {name}: created on {datetime.date.today()} with the cloud's processing. Change a value, then run it:
# processing/run_config.sh {src.name} {name}
# The run also writes here targets.csv (as in data/, from these patterns) and patterns.json (the patterns exactly as
# given to the emulator, templates included).
emulator:
version: "{state['commit'][:7]}" # optical's commit (branch {state['branch']}); the run stops if $OPTICAL is
# at another one or has local changes
patterns:
# The {len(props)} patterns of the DIFF images, as in images_diff/sensors.json on {datetime.date.today()} (frozen here: a
# later pumping does not change them). The template stays in sensors.json; to change it:
# <name>: {{redraw: {{image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}}}}
# pattern_type: global_ref = a reference of the camera fit, anything else a target. corners and x, y move together
# (same window size as the template). resolution: 1/100 px. distance_convertion_factor: mm per px (corr_*_mm only).
# references
""" + "\n".join(refs) + "\n # targets\n" + "\n".join(targets) + "\ncamera:" + "".join(
f"\n {f'{k}: {json.dumps(v)}':26s} # {CAMERA_NOTES[k]}" for k, v in cam.items()) + (
"\n # focal_distance, pixel_size: the unit has none on the platform, the cloud's values"
if not {"focal_distance", "pixel_size"} & set(unit) else "") + "\n"
run.mkdir(parents=True, exist_ok=True)
(run / "config.yaml").write_text(text)
return run / "config.yaml", state
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("station", help="serial number: the folder data_diff/<SERIAL>/")
p.add_argument("run", help="the folder data_diff/<SERIAL>/runs/<run>/, holding config.yaml")
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD")
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD")
p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)")
p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff")
p.add_argument("--new", action="store_true", help="create the run, empty: its config.yaml only (processing/new_run.sh)")
a = p.parse_args()
src = a.data_diff / a.station
run = src / "runs" / a.run
if a.new:
try:
path, state = new_run(src, a.run)
except FileExistsError as e:
sys.exit(str(e))
print(f"{path}: written{', but optical has local changes: the run will stop until they are committed' if state['dirty'] else ''}"
f"\nRun it: processing/run_config.sh {a.station} {a.run}")
return
if not (run / "config.yaml").is_file():
sys.exit(f"No {run / 'config.yaml'}: create the run first, processing/new_run.sh {a.station} {a.run}")
config = load_config(run / "config.yaml")
version = (config.get("emulator") or {}).get("version")
if version is not None: # the code that makes the run is the one config.yaml names
state = optical_state()
want = subprocess.run(["git", "-C", str(OPTICAL), "rev-parse", "--verify", "--quiet", f"{version}^{{commit}}"],
capture_output=True, text=True).stdout.strip()
if not want:
sys.exit(f"emulator.version {version}: not a commit of {OPTICAL}")
if want != state["commit"] or state["dirty"]:
sys.exit(f"emulator.version {version}, but {OPTICAL} is at {state['commit'][:7]}"
f"{' with local changes' if state['dirty'] else ''}: check {version} out, or change the version")
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{a.station}_diff_*.csv")
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
if not rows:
sys.exit(f"{a.station}: no differential image indexed in the range, nothing done")
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
unit = json.loads((src / "images_diff" / "unit.json").read_text())
cam = camera_settings(config, latest(unit["properties"]))
props = apply_patterns(cloud_properties(sensors), config.get("patterns"), src / "images_diff")
print(f"{a.station} {a.run}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} "
f"patterns, camera {cam}, optical {optical_state()}")
import reprocess_images
sp = run / "patterns.json" # kept: the target configuration the run was made with, templates included
sp.write_text(json.dumps(props, indent=1))
pd.DataFrame([{"sensor_name": p.get("sensor_name"), "pattern_type": p.get("pattern_type"), # pump.py's columns
"distance_mm": p.get("distance_mm"), "mm_per_px": p.get("distance_convertion_factor")}
for p in props.values()]).sort_values("sensor_name").to_csv(run / "targets.csv", index=False)
with tempfile.TemporaryDirectory() as tmp: # the emulator's other files: not kept
times, remeasured = Path(tmp) / "times.csv", Path(tmp) / "rows.csv"
with open(times, "w", newline="") as fp:
w = csv.writer(fp)
w.writerow(["file", "time_utc"])
w.writerows((r["file"], r["time_utc"]) for r in rows)
reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured),
"--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows])
adv, camera = correct(pd.read_csv(remeasured), props, cam)
write_run(adv, camera, run, a.station)
print(f"{a.station} {a.run}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows "
f"-> {run}")
def check():
"""`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical."""
with tempfile.TemporaryDirectory() as tmp:
f = Path(tmp) / "config.yaml"
for text in ("camera: {algoritm: ransac}", "pattern: {}", "stations: {S: {}}", "patterns: [PM_G]",
"emulator: {versio: 65a4c70}"):
f.write_text(text)
try:
load_config(f)
raise AssertionError(f"accepted: {text}")
except ValueError:
pass
f.write_text("")
assert load_config(f) == {} # an empty file: the cloud's processing
f.write_text("camera: {mode: rotation, focal_distance: 6.3}\npatterns: {T1: {error_threshold: 0.9}}\n"
"emulator: {version: '65a4c70'}")
c = load_config(f)
assert c["emulator"]["version"] == "65a4c70"
cam = camera_settings(c, {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation",
"ransac"), cam
assert camera_settings({}, {})["focal_distance"] == 6.0
pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)}
props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y,
"error_threshold": 0.7, "distance_convertion_factor": 2.0, "corners": [[0, 0], [10, 10]]}
for i, (n, (x, y)) in enumerate(pos.items())}
sensors = [{"id": int(i), "type": MTYPE, "properties": [{"name": k, "value": v} for k, v in {**q, "pattern": "b"}
.items()]} for i, q in props.items()] + [{"id": 9, "type": "camera_movement", "properties": []}]
assert set(cloud_properties(sensors)) == set(props)
for bad in ({"NOPE": {"error_threshold": 1}}, {"T1": {"error_treshold": 1}}):
try:
apply_patterns(props, bad, Path("."))
raise AssertionError(f"accepted: {bad}")
except ValueError:
pass
assert apply_patterns(props, {"T1": {"error_threshold": 0.9}}, Path("."))["3"]["error_threshold"] == 0.9
t0, t1 = "2026-11-03T10:00:07.25+00:00", "2026-11-03T10:15:07.25+00:00"
rows = []
for t, (dx, dy) in ((t0, (0.0, 0.0)), (t1, (1.0, 0.5))): # the whole camera shifts by (1, 0.5) px at t1
for i, q in props.items():
rows.append({"time_utc": t, "pattern_id": int(i), "x": q["x"] + dx, "y": q["y"] + dy, "peak": 1.0,
"error": 0.9 if (t == t1 and q["sensor_name"] == "T2") else 0.2})
for per_epoch in (True, False):
adv, camera = correct(pd.DataFrame(rows), props, {**DEFAULTS, "per_epoch": per_epoch})
assert list(adv.columns) == COLUMNS and len(adv) == 10, adv.columns
a = adv.set_index(["time_utc", "sensor_name"])
t1x = pd.Timestamp(t1)
assert a.loc[(t1x, "T1"), ["corr_x", "corr_y"]].abs().max() < 1e-3, a # static target: no displacement
assert a.loc[(t1x, "T1"), "relative_x"] == 1.0 and a.loc[(t1x, "T1"), "corr_x_mm"] == \
2.0 * a.loc[(t1x, "T1"), "corr_x"]
assert np.isnan(a.loc[(t1x, "T2"), "corr_x"]) and a.loc[(t1x, "T2"), "x"] == 701.0 # rejected: raw x kept
assert a["sigma0"].notna().all() and a.loc[(t1x, "R1"), "inlier"] == a.loc[(t1x, "R1"), "inlier"], a
assert len(camera) == 2 and str(camera["time_utc"][1]) == "2026-11-03 10:15:00+00:00", camera
assert list(camera.columns)[:4] == ["time_utc", "median_pan", "min_pan", "max_pan"], camera.columns
print("ok")
if __name__ == "__main__":
check() if sys.argv[1:] == ["--check"] else main()
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
# Runs data_diff/<SERIAL>/runs/<run>/config.yaml (processing/run_config.py), in a throwaway uv environment with what
# optical's emulator and camera correction need; optical's code from $OPTICAL (default ~/optical). No credentials.
# processing/run_config.sh <SERIAL> <run> [--from D] [--to E] [--jobs N]
# processing/run_config.sh <SERIAL> <run> --new (processing/new_run.sh): the run created, empty
# processing/run_config.sh --check
set -u
export PATH="$HOME/.local/bin:$PATH"
here=$(cd "$(dirname "$0")" && pwd)
exec uv run --no-project --quiet --python 3.12 \
--with numpy --with pandas --with pillow --with pyyaml --with scipy --with scikit-image \
--with opencv-python-headless \
python "$here/run_config.py" "$@"
+17
View File
@@ -0,0 +1,17 @@
# weather
The weather really measured at a place over a past range: the hourly values of the MeteoSwiss automatic weather station (SwissMetNet) nearest to the place. Measurements, not a model output.
```bash
python3 weather/meteoswiss.py <place> <start> <end> [--out FILE]
python3 weather/meteoswiss.py "Genève" 2026-11-15 2026-11-16 --out gve.csv
python3 weather/meteoswiss.py 46.52,6.63 2025-06-01T12:00 2025-06-01T18:00
```
The pumping writes it into `data/<SERIAL>/weather/<SERIAL>_weather_<YYYYMMDD>.csv` (`extraction/weather.py`): the columns below, `station` followed by `position` (the line of `stations.txt`), `place`, `station_name`, `distance_km`, `height_m`; the app and the agents read these files. A caller calls `hourly(place, start, end)` -> `(where, {time: {column: value or None}})`, `where` the place found and the station chosen; `LookupError` for a place not found, `OSError` when MeteoSwiss or swisstopo do not answer.
- `place`: a Swiss place name, resolved by the swisstopo search (`api3.geo.admin.ch`), or coordinates `lat,lon` (WGS84). The resolved place and the chosen station (name, distance, altitude) are written on stderr: check them. A best match whose label does not start with the first word asked is refused (`Toronto` gave `Trient (VS)`): a place outside Switzerland has no weather here, MeteoSwiss measures in Switzerland only.
- `start`, `end`: `YYYY-MM-DD` or `YYYY-MM-DDTHH:MM`, in UTC, `end` excluded. The range is in the past.
- The output: a CSV, one row per hour, `time_utc` (start of the hour) and `station`, then `precipitation_mm`, `temperature_c`, `humidity_pct`, `wind_speed_kmh`, `wind_gust_kmh`, `wind_direction_deg`, `pressure_hpa` (at the station, QFE), `sunshine_min`, `global_radiation_wm2`, `snow_depth_cm`. An empty cell: not measured at this station or at this hour.
The source is the Open Data of MeteoSwiss, free, without key ([terms of use](https://opendatadocs.meteoswiss.ch/fr/general/terms-of-use), [documentation of the dataset](https://opendatadocs.meteoswiss.ch/a-data-groundbased/a1-automatic-weather-stations)); the files of a station are listed by the STAC API (`data.geo.admin.ch/api/stac/v1/collections/ch.meteoschweiz.ogd-smn/items/<station>`). The values are quality-checked by MeteoSwiss, except those of the current day (`now` file). Hourly series from 1980 at most, depending on the station. Standard library only.
+189
View File
@@ -0,0 +1,189 @@
#!/usr/bin/env python3
"""The weather measured by MeteoSwiss at the station nearest to a place, over a past range.
python3 weather/meteoswiss.py <place> <start> <end> [--out FILE]
hourly(place, start, end) for a caller (aimsight-app): where, and the values per hour
<place> is a Swiss place name ("Genève", "Lausanne") or coordinates "lat,lon" (WGS84);
<start> and <end> are dates YYYY-MM-DD or YYYY-MM-DDTHH:MM, in UTC, <end> excluded.
The output is an hourly CSV (stdout by default), one row per hour.
The source is the Open Data of MeteoSwiss: the automatic weather stations SwissMetNet,
the real measurements (not a model), quality-checked except for the current day
(https://opendatadocs.meteoswiss.ch/a-data-groundbased/a1-automatic-weather-stations).
The place name is resolved by the search service of swisstopo (api3.geo.admin.ch).
No key, no dependency outside the standard library.
"""
import argparse
import csv
import io
import json
import math
import re
import sys
import unicodedata
import urllib.parse
import urllib.request
from datetime import datetime, timedelta, timezone
BASE = "https://data.geo.admin.ch/ch.meteoschweiz.ogd-smn"
STAC_ITEM = "https://data.geo.admin.ch/api/stac/v1/collections/ch.meteoschweiz.ogd-smn/items/{}"
GEOCODER = "https://api3.geo.admin.ch/rest/services/api/SearchServer"
# MeteoSwiss parameter -> column of the output (hourly values).
PARAMETERS = {
"rre150h0": "precipitation_mm", # hourly total
"tre200h0": "temperature_c", # air at 2 m, hourly mean
"ure200h0": "humidity_pct", # relative, at 2 m, hourly mean
"fu3010h0": "wind_speed_kmh", # hourly mean
"fu3010h1": "wind_gust_kmh", # hourly max of the gusts
"dkl010h0": "wind_direction_deg", # hourly mean
"prestah0": "pressure_hpa", # at the station (QFE), hourly mean
"sre000h0": "sunshine_min", # hourly total
"gre000h0": "global_radiation_wm2", # hourly mean
"htoauths": "snow_depth_cm", # instantaneous, not at every station
}
def fetch(url):
request = urllib.request.Request(url, headers={"User-Agent": "aimsight-weather"})
with urllib.request.urlopen(request, timeout=60) as response:
return response.read()
def parse_time(text):
for form in ("%Y-%m-%dT%H:%M", "%Y-%m-%d"):
try:
return datetime.strptime(text, form).replace(tzinfo=timezone.utc)
except ValueError:
pass
raise argparse.ArgumentTypeError(f"not a date YYYY-MM-DD[THH:MM]: {text}")
def locate(place):
"""(lat, lon, label) of a place name or of "lat,lon"."""
parts = place.split(",")
if len(parts) == 2:
try:
lat, lon = float(parts[0]), float(parts[1])
return lat, lon, f"{lat},{lon}"
except ValueError:
pass
query = urllib.parse.urlencode({"searchText": place, "type": "locations", "limit": 1, "sr": 4326})
results = json.loads(fetch(f"{GEOCODER}?{query}"))["results"]
if not results:
raise LookupError(f"place not found: {place}")
attrs = results[0]["attrs"]
label = re.sub(r"<[^>]+>", "", attrs["label"]).split("\n")[0].strip()
# the search always returns its best match ("Toronto" -> "Trient (VS)"): the label must start with the first word asked
# ponytail: first word only; "lat,lon" for a place whose name differs from its label
if not plain(label).startswith(plain(place.split()[0])):
raise LookupError(f"place not found: {place} (best match: {label})")
return attrs["lat"], attrs["lon"], label
def plain(text):
"""Lower case, without accents: "Genève" -> "geneve"."""
return "".join(c for c in unicodedata.normalize("NFKD", text.lower()) if not unicodedata.combining(c))
def distance_km(lat1, lon1, lat2, lon2):
p1, p2 = math.radians(lat1), math.radians(lat2)
a = (math.sin((p2 - p1) / 2) ** 2
+ math.cos(p1) * math.cos(p2) * math.sin(math.radians(lon2 - lon1) / 2) ** 2)
return 2 * 6371 * math.asin(math.sqrt(a))
def nearest_station(lat, lon):
text = fetch(f"{BASE}/ogd-smn_meta_stations.csv").decode("latin-1")
stations = list(csv.DictReader(io.StringIO(text), delimiter=";"))
station = min(stations, key=lambda s: distance_km(
lat, lon, float(s["station_coordinates_wgs84_lat"]), float(s["station_coordinates_wgs84_lon"])))
station["distance_km"] = distance_km(
lat, lon, float(station["station_coordinates_wgs84_lat"]), float(station["station_coordinates_wgs84_lon"]))
return station
def files_for(station, start, end):
"""The hourly files of the station that cover [start, end): historical by decade, recent, now."""
item = json.loads(fetch(STAC_ITEM.format(station.lower())))
names = [n for n in item["assets"] if f"_{station.lower()}_h_" in n]
chosen = []
for name in names:
stem = name.rsplit(".", 1)[0]
if "_historical_" in stem:
first, last = (int(y) for y in stem.rsplit("_", 1)[1].split("-"))
if start.year <= last and end.year >= first:
chosen.append(name)
elif stem.endswith("_recent"): # from January 1 of the current year to yesterday
if end.year >= datetime.now(timezone.utc).year - 1:
chosen.append(name)
elif stem.endswith("_now"): # since midnight, not yet quality-checked
if end > datetime.now(timezone.utc) - timedelta(days=1):
chosen.append(name)
return [item["assets"][n]["href"] for n in sorted(chosen)]
def read_rows(urls, start, end):
"""{time: row} over [start, end); a later file (recent, now) completes an earlier one."""
rows = {}
for url in urls:
text = fetch(url).decode("latin-1")
for row in csv.DictReader(io.StringIO(text), delimiter=";"):
time = datetime.strptime(row["reference_timestamp"], "%d.%m.%Y %H:%M").replace(tzinfo=timezone.utc)
if start <= time < end:
rows.setdefault(time, row)
return rows
def number(text):
try:
return float(text)
except ValueError:
return None
def hourly(place, start, end):
"""-> (where, {time: {column: float or None}}) over [start, end) (UTC datetimes), the times sorted; where: the place
found (place, lat, lon) and the station chosen (station, name, distance_km, height_m). Raises LookupError (place not
found) or OSError (MeteoSwiss or swisstopo unreachable)."""
lat, lon, label = locate(place)
station = nearest_station(lat, lon)
rows = read_rows(files_for(station["station_abbr"], start, end), start, end)
where = {"place": label, "lat": lat, "lon": lon, "station": station["station_abbr"], "name": station["station_name"],
"distance_km": station["distance_km"], "height_m": number(station["station_height_masl"])}
return where, {t: {c: number(rows[t].get(p, "")) for p, c in PARAMETERS.items()} for t in sorted(rows)}
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("place", help='a Swiss place name, or "lat,lon" (WGS84)')
parser.add_argument("start", type=parse_time, help="YYYY-MM-DD[THH:MM], UTC, included")
parser.add_argument("end", type=parse_time, help="YYYY-MM-DD[THH:MM], UTC, excluded")
parser.add_argument("--out", help="the CSV file (stdout by default)")
args = parser.parse_args()
if args.end <= args.start:
sys.exit("the end must be after the start")
if args.start >= datetime.now(timezone.utc):
sys.exit("the range must be in the past")
try:
where, hours = hourly(args.place, args.start, args.end)
except LookupError as e:
sys.exit(str(e))
print(f"{where['place']} ({where['lat']:.4f}, {where['lon']:.4f}) -> station {where['station']} {where['name']}, "
f"{where['distance_km']:.1f} km, {where['height_m']:.0f} m", file=sys.stderr)
out = open(args.out, "w", newline="", encoding="utf-8") if args.out else sys.stdout
writer = csv.writer(out)
writer.writerow(["time_utc", "station", *PARAMETERS.values()])
for time, values in hours.items():
writer.writerow([time.strftime("%Y-%m-%dT%H:%MZ"), where["station"], *("" if v is None else f"{v:g}" for v in values.values())])
if args.out:
out.close()
print(f"{len(hours)} hours", file=sys.stderr)
if __name__ == "__main__":
main()