5.3 KiB
extraction/: pumping the data of the platform (read-only)
pump.py fetches, for each station (serial number of the system) and each UTC day asked, into data/<SERIAL>/:
- the system metrics (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in
metrics/; - the displacements ("advanced" export), one CSV per day in
displacement/; - the camera movements (
pan,tilt,scale,rotation,sigma0), as median and bounds (median_,min_,max_) per minute, only at the times the camera was realigned, one CSV per day incamera/; - the IRON images (
--images, one per hour by default,--every 10or--every 0for more) inimages/, and the differential ones (--diff) inimages_diff/.
Formats: ~/labeling/knowledge/displacement-columns.md.
Tracked stations: stations.txt, one serial number per line. Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in data/ gets the last 30 days.
Resuming: with --from, a file already in data/<SERIAL>/ is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without --from replaces it.
Running (under WSL)
pump.sh reads the credentials from the .env at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (sqlalchemy, psycopg2-binary, pandas, boto3) in a throwaway uv environment (Python 3.12), runs pump.py, then lenzspitze_absolute.py. Its output is appended to out/pump.log; a second run during a pumping stops at once (lock out/pump.lock). The lock holds for this folder only: the other repositories read data/ through a link, they do not pump.
Variables of the .env: AIMSIGHT_DATABASE_USER, _PSWD, _TURL, _NAME, _PORT; for the images, AIMSIGHT_S3_URL, _ACCESS_KEY, _SECRET_KEY, _BUCKET.
- A trial (1 station, 2 days, in a throwaway folder):
extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND(6 s on 2026-11-12).lenzspitze_absolute.pythen convertsdata/LENZSPITZE, not the--outfolder: no effect ifdata/LENZSPITZEis already converted. - Updating the tracked stations:
extraction/pump.sh(without argument:--images --delay 300), or the VS Code task pump. The script says, for each station, the range it pumps. - A chosen range (history, failed day), with 3 more days before (the history of the detectors):
extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images <SERIAL>(--toexcluded; several stations:<SERIAL1> <SERIAL2>). Help:extraction/pump.sh -h.
Every night, automatically: a task of the Windows Task Scheduler, to create once in PowerShell (not as administrator). It starts WSL if needed and, PC off or asleep at the set time, runs as soon as it wakes up; a missed day is caught up at the next run.
$a = New-ScheduledTaskAction -Execute "wsl.exe" -Argument "-d Ubuntu -e bash -lc /home/maxime/aimsight-data/extraction/pump.sh"
$t = New-ScheduledTaskTrigger -Daily -At 04:00
$s = New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Hours 6)
Register-ScheduledTask -TaskName "AIMsight pump" -Action $a -Trigger $t -Settings $s
Trying the task right away: Start-ScheduledTask "AIMsight pump", then read out/pump.log. Removing it: Unregister-ScheduledTask "AIMsight pump".
Read-only: dedicated engine, default_transaction_read_only, 60 s timeout per query, one query per day; outside busy hours (database shared with production).
No dependency on the monitoring-dashboard app: its reads are copied into pump.py, in SQL (its docstring says which ones, and from which commit). If the app changes its reads, the script does not follow: compare again files pumped by both.
With Claude
Claude may run pump.sh itself (user's agreement, 2026-11-12): read-only, small ranges; a long range or a history, outside busy hours. It never reads the .env.
Good to know
<SERIAL>is the serial number of the system, the one in the URL of the System page (/explore/system/<SERIAL>) and at the start of the CSVs downloaded there (NORDEND_BusVoltage_…:NORDEND); not the name of the device on the Explore page.data/is out of git: a lost folder is pumped again.- A day failed on a timeout (
QueryCanceled: … statement timeout): run it again alone with--delay 300. The slowness comes from the database, not from the volume. - An hour without an image can have its measurement: the image never reached S3 (empty
measurement_uuid, unfinished upload). The script skips it and takes the next image of the slot if there is one. - A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in
data/<SERIAL>/and run again. - LENZSPITZE: its exports have relative
x/yand an emptypattern_type;pump.shthen runslenzspitze_absolute.py(provisional; details in~/labeling/knowledge/displacement-columns.md).