Files
aimsight-data/extraction
maximeandClaude Opus 5.5 81908ca370 Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)
The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every
measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived
included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into
data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now
returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF
images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 09:59:50 +01:00
..

extraction/: pumping the data of the platform (read-only)

pump.py fetches, for each station (serial number of the system) and each UTC day asked, into data/<SERIAL>/:

  • the system metrics (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in metrics/;
  • the displacements ("advanced" export), one CSV per day in displacement/, with corr_x_mm / corr_y_mm = corr_x / corr_y × mm_per_px of the sensor of each row (by sensor_id, never by name; empty without a factor); a day pumped before keeps its file: --from D --to E --replace pumps it again with the mm;
  • the sensors measured, targets and references (every sensor with a pattern_type), in targets.csv: pattern_type, distance_mm (from the camera; 1000 = not filled in on the platform) and mm_per_px (distance_convertion_factor, the factor of the dashboard's exports in mm), their current values, read again at each pumping; a name not measured in the run keeps its line;
  • the camera movements (pan, tilt, scale, rotation, sigma0), as median and bounds (median_, min_, max_) per minute, only at the times the camera was realigned, one CSV per day in camera/;
  • the weather (weather.py, after the pumping), for a station with a position in stations.txt: the hourly MeteoSwiss measurements nearest to it, one CSV per day with data, in weather/ (weather/README.md);
  • the IRON images (--images, one per hour by default, --every 10 or --every 0 for more) in images/, and the differential ones (--diff) in images_diff/.

Formats: ~/aimsight-labeling/knowledge/displacement-columns.md.

Tracked stations: stations.txt, one serial number per line, optionally followed by the position of the station (lat,lon or a Swiss place, for the weather pumped into weather/; MeteoSwiss covers Switzerland only; the app edits this list: page Stations, through pumping.py). Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in data/ gets the last 30 days.

Resuming: with --from, a file already in data/<SERIAL>/ is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without --from replaces it.

Running (under WSL)

pump.sh reads the credentials from the .env at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (sqlalchemy, psycopg2-binary, pandas, boto3) in a throwaway uv environment (Python 3.12), runs pump.py, then absolute.py, then weather.py (every station with a position: its missing days, those written for another position, and the last 2 days, fetched again; a place not found or MeteoSwiss unreachable: a line in the log, the pumping goes on). python3 extraction/weather.py [SERIAL…] alone fills the weather of the days already pumped. Its output is appended to out/pump.log; a second run during a pumping stops at once (lock out/pump.lock). The lock holds for this folder only: the other repositories read data/ through a link, they do not pump.

Variables of the .env: AIMSIGHT_DATABASE_USER, _PSWD, _TURL, _NAME, _PORT; for the images, AIMSIGHT_S3_URL, _ACCESS_KEY, _SECRET_KEY, _BUCKET.

  1. A trial (1 station, 2 days, in a throwaway folder): extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND (6 s on 2026-11-12). absolute.py then converts data/, not the --out folder: no effect on what is already converted.
  2. Updating the tracked stations: extraction/pump.sh (without argument: --images --delay 300), or the VS Code task pump. The script says, for each station, the range it pumps.
  3. A chosen range (history, failed day), with 3 more days before (the history of the detectors): extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images <SERIAL> (--to excluded; several stations: <SERIAL1> <SERIAL2>). Help: extraction/pump.sh -h.

Every night, automatically: a task of the Windows Task Scheduler, to create once in PowerShell (not as administrator). It starts WSL if needed and, PC off or asleep at the set time, runs as soon as it wakes up; a missed day is caught up at the next run.

$a = New-ScheduledTaskAction -Execute "wsl.exe" -Argument "-d Ubuntu -e bash -lc /home/maxime/aimsight-data/extraction/pump.sh"
$t = New-ScheduledTaskTrigger -Daily -At 04:00
$s = New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Hours 6)
Register-ScheduledTask -TaskName "AIMsight pump" -Action $a -Trigger $t -Settings $s

Trying the task right away: Start-ScheduledTask "AIMsight pump", then read out/pump.log. Removing it: Unregister-ScheduledTask "AIMsight pump".

Read-only: dedicated engine, default_transaction_read_only, 60 s timeout per query, one query per day; outside busy hours (database shared with production).

No dependency on the monitoring-dashboard app: its reads are copied into pump.py, in SQL (its docstring says which ones, and from which commit). If the app changes its reads, the script does not follow: compare again files pumped by both.

For a caller

pumping.py (standard library): tracked() and save(rows) (the list), start(stations, start, end, images) (pump.sh in the background, --delay 300), running() (the lock held) and log() (the end of out/pump.log); a refused request raises ValueError. Its check: python3 extraction/pumping.py. The page Stations of aimsight-app calls it.

With Claude

Claude may run pump.sh itself (user's agreement, 2026-11-12): read-only, small ranges; a long range or a history, outside busy hours. It never reads the .env.

Good to know

  • <SERIAL> is the serial number of the system, the one in the URL of the System page (/explore/system/<SERIAL>) and at the start of the CSVs downloaded there (NORDEND_BusVoltage_…: NORDEND); not the name of the device on the Explore page.
  • data/ is out of git: a lost folder is pumped again.
  • A day failed on a timeout (QueryCanceled: … statement timeout): run it again alone with --delay 300. The slowness comes from the database, not from the volume.
  • An hour without an image can have its measurement: the image never reached S3 (empty measurement_uuid, unfinished upload). The script skips it and takes the next image of the slot if there is one.
  • A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in data/<SERIAL>/ and run again.
  • Relative x / y (LENZSPITZE, SUSTENHORN, NADELHORN c*_mon): after each pumping, pump.sh runs absolute.py, which converts every sensor whose x / y medians are under 100 px, from data/<SERIAL>/*_patterns.json and the offset of the station (only LENZSPITZE has both today); a relative sensor it cannot place is dropped (details in ~/aimsight-labeling/knowledge/displacement-columns.md).

The differential images, into data_diff/ (pump_diff.sh)

pump_diff.py, through pump_diff.sh, fetches the OPTICAL_SHIFT-DIFF images of a station, full size, with what is needed to measure them again, into data_diff/<SERIAL>/ (out of git), in the tree of data/, so that the other repositories read it the same way; it never touches data/ and does not run absolute.py (the recomputed positions are absolute). Its own lock (out/pump_diff.lock) and log (out/pump_diff.log): a long history does not block the nightly pump.sh. Designs: docs/superpowers/specs/2026-11-19-pump-diff-design.md, 2026-11-19-configurations-design.md (the configurations, runs/<name>/, read like data/).

  • images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png: the S3 object as is;
  • images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv: every DIFF measurement of the day in the database, measurement_id, time_utc (exact), measurement_group_id, file, status (ok; no_uuid: never reached S3; s3_missing: S3 returned nothing), properties (JSON);
  • images_diff/sensors.json: every sensor of the unit, archived included, with all its property rows (pattern, corners, x, y, error_threshold, distance_convertion_factor, start_date…);
  • images_diff/unit.json: the acquisition unit and its properties (focal_distance, pixel_size, image_shape when filled in; none: the platform takes 6.0 mm and 0.0024 mm);
  • (under images_diff/, the reading tools do not see these files: data.find_files skips the folder)
  • metrics/, images/ (IRON, one per --every minutes, default 60), weather/: with the functions of pump.py and weather.py (weather.py --data data_diff), the same files as in data/;
  • targets.csv: as in data/, for the patterns the DIFF images are measured with (not archived); displacement/ and camera/ come only from the recomputation of a configuration (runs/<name>/), never from the cloud.
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU   # a range; the history of a station starts here
extraction/pump_diff.sh JUNGFRAU                                     # from its last indexed day up to now
extraction/pump_diff.sh                                              # the same for every station of data_diff/

Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). --min-free-gb (default 50) stops the run before a day. Check without database: python3 extraction/pump_diff.py --check. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of pump.py --diff; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to data/; targets.csv the same without the two non-DIFF sensors target_01, target_02.

Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit pump.py picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.