The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
80 lines
11 KiB
Markdown
80 lines
11 KiB
Markdown
# extraction/: pumping the data of the platform (read-only)
|
||
|
||
`pump.py` fetches, for each station (serial number of the system) and each UTC day asked, into `data/<SERIAL>/`:
|
||
- the **system metrics** (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in `metrics/`;
|
||
- the **displacements** ("advanced" export), one CSV per day in `displacement/`, with `corr_x_mm` / `corr_y_mm` = `corr_x` / `corr_y` × `mm_per_px` of the sensor of each row (by `sensor_id`, never by name; empty without a factor); a day pumped before keeps its file: `--from D --to E --replace` pumps it again with the mm;
|
||
- the **sensors** measured, targets and references (every sensor with a `pattern_type`), in `targets.csv`: `pattern_type`, `distance_mm` (from the camera; 1000 = not filled in on the platform) and `mm_per_px` (`distance_convertion_factor`, the factor of the dashboard's exports in mm), their current values, read again at each pumping; a name not measured in the run keeps its line;
|
||
- the **camera movements** (`pan`, `tilt`, `scale`, `rotation`, `sigma0`), as median and bounds (`median_`, `min_`, `max_`) per minute, only at the times the camera was realigned, one CSV per day in `camera/`;
|
||
- the **weather** (`weather.py`, after the pumping), for a station with a position in `stations.txt`: the hourly MeteoSwiss measurements nearest to it, one CSV per day with data, in `weather/` (`weather/README.md`);
|
||
- the IRON **images** (`--images`, one per hour by default, `--every 10` or `--every 0` for more) in `images/`, and the differential ones (`--diff`) in `images_diff/`.
|
||
|
||
Formats: `~/aimsight-labeling/knowledge/displacement-columns.md`.
|
||
|
||
**Tracked stations**: `stations.txt`, one serial number per line, optionally followed by the position of the station (`lat,lon` or a Swiss place, for the weather pumped into `weather/`; MeteoSwiss covers Switzerland only; the app edits this list: page Stations, through `pumping.py`). Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in `data/` gets the last 30 days.
|
||
|
||
Resuming: with `--from`, a file already in `data/<SERIAL>/` is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without `--from` replaces it.
|
||
|
||
## Running (under WSL)
|
||
|
||
`pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `absolute.py`, then `weather.py` (every station with a position: its missing days, those written for another position, and the last 2 days, fetched again; a place not found or MeteoSwiss unreachable: a line in the log, the pumping goes on). `python3 extraction/weather.py [SERIAL…]` alone fills the weather of the days already pumped. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump.
|
||
|
||
Variables of the `.env`: `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT`; for the images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`.
|
||
|
||
1. **A trial** (1 station, 2 days, in a throwaway folder): `extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND` (6 s on 2026-11-12). `absolute.py` then converts `data/`, not the `--out` folder: no effect on what is already converted.
|
||
2. **Updating the tracked stations**: `extraction/pump.sh` (without argument: `--images --delay 300`), or the VS Code task **pump**. The script says, for each station, the range it pumps.
|
||
3. **A chosen range** (history, failed day), with 3 more days before (the history of the detectors): `extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images <SERIAL>` (`--to` excluded; several stations: `<SERIAL1> <SERIAL2>`). Help: `extraction/pump.sh -h`.
|
||
|
||
**Every night, automatically**: a task of the Windows Task Scheduler, to create once in PowerShell (not as administrator). It starts WSL if needed and, PC off or asleep at the set time, runs as soon as it wakes up; a missed day is caught up at the next run.
|
||
|
||
```powershell
|
||
$a = New-ScheduledTaskAction -Execute "wsl.exe" -Argument "-d Ubuntu -e bash -lc /home/maxime/aimsight-data/extraction/pump.sh"
|
||
$t = New-ScheduledTaskTrigger -Daily -At 04:00
|
||
$s = New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Hours 6)
|
||
Register-ScheduledTask -TaskName "AIMsight pump" -Action $a -Trigger $t -Settings $s
|
||
```
|
||
|
||
Trying the task right away: `Start-ScheduledTask "AIMsight pump"`, then read `out/pump.log`. Removing it: `Unregister-ScheduledTask "AIMsight pump"`.
|
||
|
||
Read-only: dedicated engine, `default_transaction_read_only`, 60 s timeout per query, one query per day; outside busy hours (database shared with production).
|
||
|
||
No dependency on the `monitoring-dashboard` app: its reads are copied into `pump.py`, in SQL (its docstring says which ones, and from which commit). If the app changes its reads, the script does not follow: compare again files pumped by both.
|
||
|
||
## For a caller
|
||
|
||
`pumping.py` (standard library): `tracked()` and `save(rows)` (the list), `start(stations, start, end, images)` (`pump.sh` in the background, `--delay 300`), `running()` (the lock held) and `log()` (the end of `out/pump.log`); a refused request raises `ValueError`. Its check: `python3 extraction/pumping.py`. The page Stations of aimsight-app calls it.
|
||
|
||
## With Claude
|
||
|
||
Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small ranges; a long range or a history, outside busy hours. It never reads the `.env`.
|
||
|
||
## Good to know
|
||
|
||
- `<SERIAL>` is the serial number of the **system**, the one in the URL of the System page (`/explore/system/<SERIAL>`) and at the start of the CSVs downloaded there (`NORDEND_BusVoltage_…`: `NORDEND`); not the name of the device on the Explore page.
|
||
- `data/` is out of git: a lost folder is pumped again.
|
||
- A day failed on a timeout (`QueryCanceled: … statement timeout`): run it again alone with `--delay 300`. The slowness comes from the database, not from the volume.
|
||
- An hour without an image can have its measurement: the image never reached S3 (empty `measurement_uuid`, unfinished upload). The script skips it and takes the next image of the slot if there is one.
|
||
- A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in `data/<SERIAL>/` and run again.
|
||
- Relative `x` / `y` (LENZSPITZE, SUSTENHORN, NADELHORN `c*_mon`): after each pumping, `pump.sh` runs `absolute.py`, which converts every sensor whose `x` / `y` medians are under 100 px, from `data/<SERIAL>/*_patterns.json` and the offset of the station (only LENZSPITZE has both today); a relative sensor it cannot place is dropped (details in `~/aimsight-labeling/knowledge/displacement-columns.md`).
|
||
|
||
## The differential images, into `data_diff/` (`pump_diff.sh`)
|
||
|
||
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the configurations, `runs/<name>/`, read like `data/`).
|
||
|
||
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is;
|
||
- `images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON);
|
||
- `images_diff/sensors.json`: every sensor of the unit, archived included, with all its property rows (pattern, corners, x, y, error_threshold, distance_convertion_factor, start_date…);
|
||
- `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm);
|
||
- (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder)
|
||
- `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`;
|
||
- `targets.csv`: as in `data/`, for the patterns the DIFF images are measured with (not archived); `displacement/` and `camera/` come only from the recomputation of a configuration (`runs/<name>/`), never from the cloud.
|
||
|
||
```bash
|
||
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here
|
||
extraction/pump_diff.sh JUNGFRAU # from its last indexed day up to now
|
||
extraction/pump_diff.sh # the same for every station of data_diff/
|
||
```
|
||
|
||
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`; `targets.csv` the same without the two non-DIFF sensors `target_01`, `target_02`.
|
||
|
||
Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.
|