- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/, weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv, patterns.json. - processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the patterns of sensors.json, the camera, optical's current commit); never over an existing one. - processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns) and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/. - pump_diff no longer writes targets.csv: it depends on the run. JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within 0.034 px; a run created by new_run.sh and run as is gives run01 exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extraction/: pumping the data of the platform (read-only)
pump.py fetches, for each station (serial number of the system) and each UTC day asked, into data/<SERIAL>/:
- the system metrics (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in
metrics/; - the displacements ("advanced" export), one CSV per day in
displacement/, withcorr_x_mm/corr_y_mm=corr_x/corr_y×mm_per_pxof the sensor of each row (bysensor_id, never by name; empty without a factor); a day pumped before keeps its file:--from D --to E --replacepumps it again with the mm; - the sensors measured, targets and references (every sensor with a
pattern_type), intargets.csv:pattern_type,distance_mm(from the camera; 1000 = not filled in on the platform) andmm_per_px(distance_convertion_factor, the factor of the dashboard's exports in mm), their current values, read again at each pumping; a name not measured in the run keeps its line; - the camera movements (
pan,tilt,scale,rotation,sigma0), as median and bounds (median_,min_,max_) per minute, only at the times the camera was realigned, one CSV per day incamera/; - the weather (
weather.py, after the pumping), for a station with a position instations.txt: the hourly MeteoSwiss measurements nearest to it, one CSV per day with data, inweather/(weather/README.md); - the IRON images (
--images, one per hour by default,--every 10or--every 0for more) inimages/, and the differential ones (--diff) inimages_diff/.
Formats: ~/aimsight-labeling/knowledge/displacement-columns.md.
Tracked stations: stations.txt, one serial number per line, optionally followed by the position of the station (lat,lon or a Swiss place, for the weather pumped into weather/; MeteoSwiss covers Switzerland only; the app edits this list: page Stations, through pumping.py). Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in data/ gets the last 30 days.
Resuming: with --from, a file already in data/<SERIAL>/ is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without --from replaces it.
Running (under WSL)
pump.sh reads the credentials from the .env at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (sqlalchemy, psycopg2-binary, pandas, boto3) in a throwaway uv environment (Python 3.12), runs pump.py, then absolute.py, then weather.py (every station with a position: its missing days, those written for another position, and the last 2 days, fetched again; a place not found or MeteoSwiss unreachable: a line in the log, the pumping goes on). python3 extraction/weather.py [SERIAL…] alone fills the weather of the days already pumped. Its output is appended to out/pump.log; a second run during a pumping stops at once (lock out/pump.lock). The lock holds for this folder only: the other repositories read data/ through a link, they do not pump.
Variables of the .env: AIMSIGHT_DATABASE_USER, _PSWD, _TURL, _NAME, _PORT; for the images, AIMSIGHT_S3_URL, _ACCESS_KEY, _SECRET_KEY, _BUCKET.
- A trial (1 station, 2 days, in a throwaway folder):
extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND(6 s on 2026-11-12).absolute.pythen convertsdata/, not the--outfolder: no effect on what is already converted. - Updating the tracked stations:
extraction/pump.sh(without argument:--images --delay 300), or the VS Code task pump. The script says, for each station, the range it pumps. - A chosen range (history, failed day), with 3 more days before (the history of the detectors):
extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images <SERIAL>(--toexcluded; several stations:<SERIAL1> <SERIAL2>). Help:extraction/pump.sh -h.
Every night, automatically: a task of the Windows Task Scheduler, to create once in PowerShell (not as administrator). It starts WSL if needed and, PC off or asleep at the set time, runs as soon as it wakes up; a missed day is caught up at the next run.
$a = New-ScheduledTaskAction -Execute "wsl.exe" -Argument "-d Ubuntu -e bash -lc /home/maxime/aimsight-data/extraction/pump.sh"
$t = New-ScheduledTaskTrigger -Daily -At 04:00
$s = New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Hours 6)
Register-ScheduledTask -TaskName "AIMsight pump" -Action $a -Trigger $t -Settings $s
Trying the task right away: Start-ScheduledTask "AIMsight pump", then read out/pump.log. Removing it: Unregister-ScheduledTask "AIMsight pump".
Read-only: dedicated engine, default_transaction_read_only, 60 s timeout per query, one query per day; outside busy hours (database shared with production).
No dependency on the monitoring-dashboard app: its reads are copied into pump.py, in SQL (its docstring says which ones, and from which commit). If the app changes its reads, the script does not follow: compare again files pumped by both.
For a caller
pumping.py (standard library): tracked() and save(rows) (the list), start(stations, start, end, images) (pump.sh in the background, --delay 300), running() (the lock held) and log() (the end of out/pump.log); a refused request raises ValueError. Its check: python3 extraction/pumping.py. The page Stations of aimsight-app calls it.
With Claude
Claude may run pump.sh itself (user's agreement, 2026-11-12): read-only, small ranges; a long range or a history, outside busy hours. It never reads the .env.
Good to know
<SERIAL>is the serial number of the system, the one in the URL of the System page (/explore/system/<SERIAL>) and at the start of the CSVs downloaded there (NORDEND_BusVoltage_…:NORDEND); not the name of the device on the Explore page.data/is out of git: a lost folder is pumped again.- A day failed on a timeout (
QueryCanceled: … statement timeout): run it again alone with--delay 300. The slowness comes from the database, not from the volume. - An hour without an image can have its measurement: the image never reached S3 (empty
measurement_uuid, unfinished upload). The script skips it and takes the next image of the slot if there is one. - A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in
data/<SERIAL>/and run again. - Relative
x/y(LENZSPITZE, SUSTENHORN, NADELHORNc*_mon): after each pumping,pump.shrunsabsolute.py, which converts every sensor whosex/ymedians are under 100 px, fromdata/<SERIAL>/*_patterns.jsonand the offset of the station (only LENZSPITZE has both today); a relative sensor it cannot place is dropped (details in~/aimsight-labeling/knowledge/displacement-columns.md).
The differential images, into data_diff/ (pump_diff.sh)
pump_diff.py, through pump_diff.sh, fetches the OPTICAL_SHIFT-DIFF images of a station, full size, with what is needed to measure them again, into data_diff/<SERIAL>/ (out of git), in the tree of data/, so that the other repositories read it the same way; it never touches data/ and does not run absolute.py (the recomputed positions are absolute). Its own lock (out/pump_diff.lock) and log (out/pump_diff.log): a long history does not block the nightly pump.sh. Designs: docs/superpowers/specs/2026-11-19-pump-diff-design.md, 2026-11-19-configurations-design.md (the runs, data_diff/<SERIAL>/runs/<run>/).
images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png: the S3 object as is;images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv: every DIFF measurement of the day in the database,measurement_id, time_utc(exact),measurement_group_id, file, status(ok;no_uuid: never reached S3;s3_missing: S3 returned nothing),properties(JSON);images_diff/sensors.json: every sensor of the unit, archived included, with all its property rows (pattern, corners, x, y, error_threshold, distance_convertion_factor, start_date…);images_diff/unit.json: the acquisition unit and its properties (focal_distance,pixel_size,image_shapewhen filled in; none: the platform takes 6.0 mm and 0.0024 mm);- (under
images_diff/, the reading tools do not see these files:data.find_filesskips the folder) metrics/,images/(IRON, one per--everyminutes, default 60),weather/: with the functions ofpump.pyandweather.py(weather.py --data data_diff), the same files as indata/;displacement/,camera/andtargets.csvare not pumped: they depend on the run (its patterns, its settings), each run writes its own inruns/<run>/(processing/run_config.sh), never from the cloud; a pumping never touchesruns/.
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here
extraction/pump_diff.sh JUNGFRAU # from its last indexed day up to now
extraction/pump_diff.sh # the same for every station of data_diff/
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). --min-free-gb (default 50) stops the run before a day. Check without database: python3 extraction/pump_diff.py --check. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of pump.py --diff; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to data/.
Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit pump.py picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.