Files
aimsight-data/extraction/README.md
T

79 lines
11 KiB
Markdown
Raw Normal View History

# extraction/: pumping the data of the platform (read-only)
`pump.py` fetches, for each station (serial number of the system) and each UTC day asked, into `data/<SERIAL>/`:
- the **system metrics** (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in `metrics/`;
- the **displacements** ("advanced" export), one CSV per day in `displacement/`, with `corr_x_mm` / `corr_y_mm` = `corr_x` / `corr_y` × `mm_per_px` of the sensor of each row (by `sensor_id`, never by name; empty without a factor); a day pumped before keeps its file: `--from D --to E --replace` pumps it again with the mm;
- the **sensors** measured, targets and references (every sensor with a `pattern_type`), in `targets.csv`: `pattern_type`, `distance_mm` (from the camera; 1000 = not filled in on the platform) and `mm_per_px` (`distance_convertion_factor`, the factor of the dashboard's exports in mm), their current values, read again at each pumping; a name not measured in the run keeps its line;
- the **camera movements** (`pan`, `tilt`, `scale`, `rotation`, `sigma0`), as median and bounds (`median_`, `min_`, `max_`) per minute, only at the times the camera was realigned, one CSV per day in `camera/`;
- the **weather** (`weather.py`, after the pumping), for a station with a position in `stations.txt`: the hourly MeteoSwiss measurements nearest to it, one CSV per day with data, in `weather/` (`weather/README.md`);
- the IRON **images** (`--images`, one per hour by default, `--every 10` or `--every 0` for more) in `images/`, and the differential ones (`--diff`) in `images_diff/`.
Formats: `~/aimsight-labeling/knowledge/displacement-columns.md`.
**Tracked stations**: `stations.txt`, one serial number per line, optionally followed by the position of the station (`lat,lon` or a Swiss place, for the weather pumped into `weather/`; MeteoSwiss covers Switzerland only; the app edits this list: page Stations, through `pumping.py`). Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in `data/` gets the last 30 days.
Resuming: with `--from`, a file already in `data/<SERIAL>/` is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without `--from` replaces it.
## Running (under WSL)
`pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `absolute.py`, then `weather.py` (every station with a position: its missing days, those written for another position, and the last 2 days, fetched again; a place not found or MeteoSwiss unreachable: a line in the log, the pumping goes on). `python3 extraction/weather.py [SERIAL…]` alone fills the weather of the days already pumped. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump.
Variables of the `.env`: `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT`; for the images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`.
1. **A trial** (1 station, 2 days, in a throwaway folder): `extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND` (6 s on 2026-11-12). `absolute.py` then converts `data/`, not the `--out` folder: no effect on what is already converted.
2. **Updating the tracked stations**: `extraction/pump.sh` (without argument: `--images --delay 300`), or the VS Code task **pump**. The script says, for each station, the range it pumps.
3. **A chosen range** (history, failed day), with 3 more days before (the history of the detectors): `extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images <SERIAL>` (`--to` excluded; several stations: `<SERIAL1> <SERIAL2>`). Help: `extraction/pump.sh -h`.
**Every night, automatically**: a task of the Windows Task Scheduler, to create once in PowerShell (not as administrator). It starts WSL if needed and, PC off or asleep at the set time, runs as soon as it wakes up; a missed day is caught up at the next run.
```powershell
$a = New-ScheduledTaskAction -Execute "wsl.exe" -Argument "-d Ubuntu -e bash -lc /home/maxime/aimsight-data/extraction/pump.sh"
$t = New-ScheduledTaskTrigger -Daily -At 04:00
$s = New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Hours 6)
Register-ScheduledTask -TaskName "AIMsight pump" -Action $a -Trigger $t -Settings $s
```
Trying the task right away: `Start-ScheduledTask "AIMsight pump"`, then read `out/pump.log`. Removing it: `Unregister-ScheduledTask "AIMsight pump"`.
Read-only: dedicated engine, `default_transaction_read_only`, 60 s timeout per query, one query per day; outside busy hours (database shared with production).
No dependency on the `monitoring-dashboard` app: its reads are copied into `pump.py`, in SQL (its docstring says which ones, and from which commit). If the app changes its reads, the script does not follow: compare again files pumped by both.
## For a caller
`pumping.py` (standard library): `tracked()` and `save(rows)` (the list), `start(stations, start, end, images)` (`pump.sh` in the background, `--delay 300`), `running()` (the lock held) and `log()` (the end of `out/pump.log`); a refused request raises `ValueError`. Its check: `python3 extraction/pumping.py`. The page Stations of aimsight-app calls it.
## With Claude
Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small ranges; a long range or a history, outside busy hours. It never reads the `.env`.
## Good to know
- `<SERIAL>` is the serial number of the **system**, the one in the URL of the System page (`/explore/system/<SERIAL>`) and at the start of the CSVs downloaded there (`NORDEND_BusVoltage_…`: `NORDEND`); not the name of the device on the Explore page.
- `data/` is out of git: a lost folder is pumped again.
- A day failed on a timeout (`QueryCanceled: … statement timeout`): run it again alone with `--delay 300`. The slowness comes from the database, not from the volume.
- An hour without an image can have its measurement: the image never reached S3 (empty `measurement_uuid`, unfinished upload). The script skips it and takes the next image of the slot if there is one.
- A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in `data/<SERIAL>/` and run again.
- Relative `x` / `y` (LENZSPITZE, SUSTENHORN, NADELHORN `c*_mon`): after each pumping, `pump.sh` runs `absolute.py`, which converts every sensor whose `x` / `y` medians are under 100 px, from `data/<SERIAL>/*_patterns.json` and the offset of the station (only LENZSPITZE has both today); a relative sensor it cannot place is dropped (details in `~/aimsight-labeling/knowledge/displacement-columns.md`).
## The differential images, into `data_diff/` (`pump_diff.sh`)
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the runs, `data_diff/<SERIAL>/runs/<run>/`).
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is;
- `images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON);
- `images_diff/sensors.json`: every sensor of the unit, archived included, with all its property rows (pattern, corners, x, y, error_threshold, distance_convertion_factor, start_date…);
- `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm);
- (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder)
- `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`;
- `displacement/`, `camera/` and `targets.csv` are not pumped: they depend on the run (its patterns, its settings), each run writes its own in `runs/<run>/` (`processing/run_config.sh`), never from the cloud; a pumping never touches `runs/`.
```bash
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here
extraction/pump_diff.sh JUNGFRAU # from its last indexed day up to now
extraction/pump_diff.sh # the same for every station of data_diff/
```
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`.
Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.