English: pump.py and pump.sh (ex pomper), lenzspitze_absolute.py, options --from/--to/--delay, out/pump.log; .env loaded with +u (pump.sh -h works)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-16 12:40:03 +01:00
1 parent 260cd5e522
commit 5ed9f5a97a
9 files changed
+531 -530

No files matched your search

+11 -11
View File
@@ -2,35 +2,35 @@
"version": "2.0.0", "version": "2.0.0",
"tasks": [ "tasks": [
{ {
"label": "pomper", "label": "pump",
"detail": "Met à jour les stations suivies (extraction/stations.txt), images comprises ; extraction/README.md", "detail": "Updates the tracked stations (extraction/stations.txt), images included; extraction/README.md",
"type": "shell", "type": "shell",
"command": "extraction/pomper.sh", "command": "extraction/pump.sh",
"problemMatcher": [] "problemMatcher": []
}, },
{ {
"label": "pomper une plage", "label": "pump a range",
"detail": "Pompe une plage de jours UTC (--au exclu) pour une ou plusieurs stations, images comprises", "detail": "Pumps a range of UTC days (--to excluded) for one or more stations, images included",
"type": "shell", "type": "shell",
"command": "extraction/pomper.sh --du ${input:du} --au ${input:au} --images --delai 300 ${input:stations}", "command": "extraction/pump.sh --from ${input:from} --to ${input:to} --images --delay 300 ${input:stations}",
"problemMatcher": [] "problemMatcher": []
} }
], ],
"inputs": [ "inputs": [
{ {
"id": "du", "id": "from",
"type": "promptString", "type": "promptString",
"description": "Premier jour UTC (AAAA-MM-JJ) ; 3 jours avant la période voulue, pour l'historique des détecteurs" "description": "First UTC day (YYYY-MM-DD); 3 days before the wanted period, for the history of the detectors"
}, },
{ {
"id": "au", "id": "to",
"type": "promptString", "type": "promptString",
"description": "Dernier jour UTC, exclu (AAAA-MM-JJ)" "description": "Last UTC day, excluded (YYYY-MM-DD)"
}, },
{ {
"id": "stations", "id": "stations",
"type": "promptString", "type": "promptString",
"description": "Numéros de série des stations, séparés par des espaces (ex. NORDEND)" "description": "Serial numbers of the stations, separated by spaces (e.g. NORDEND)"
} }
] ]
} }
+7 -7
View File
@@ -1,13 +1,13 @@
# aimsight-data # aimsight-data
Les données des stations AIMsight, en un seul exemplaire sur le poste, et leur pompage. The data of the AIMsight stations, in a single copy on the workstation, and their pumping.
- `data/<SERIE>/` : rempli par le pompage, **hors de git** (`metrics/`, `displacement/`, `camera/`, `images/`…). Un dossier perdu se repompe. - `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`…). A lost folder is pumped again.
- `extraction/` : le pompage, en lecture seule sur la plateforme ; mode d'emploi `extraction/README.md`. Tâches VS Code « pomper » et « pomper une plage ». - `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range".
- `.env` : les identifiants de la base et de S3, à la racine, jamais versionné. - `.env`: the credentials of the database and of S3, at the root, never versioned.
Les autres dépôts (Diagnostic-App, Optimizer) lisent les données par un lien `data -> ~/aimsight-data/data`, ignoré par git. On ne pompe que d'ici : le verrou `out/pompage.lock` vaut pour ce seul dossier. The other repositories (Diagnostic-App, Optimizer) read the data through a link `data -> ~/aimsight-data/data`, ignored by git. Pumping is done from here only: the lock `out/pump.lock` holds for this folder alone.
Les étiquettes sont dans le dépôt voisin `~/labeling`. The labels are in the neighbouring repository `~/labeling`.
Extrait de Diagnostic-App le 2026-11-13 (commit 462ba9e) ; l'historique antérieur du pompage est dans Support-Agent. Extracted from Diagnostic-App on 2026-11-13 (commit 462ba9e); the earlier history of the pumping is in Support-Agent.
+30 -30
View File
@@ -1,51 +1,51 @@
# extraction/ : pomper les données de la plateforme (lecture seule) # extraction/: pumping the data of the platform (read-only)
`pomper.py` récupère, pour chaque station (numéro de série du système) et chaque jour UTC demandé, dans `data/<SERIE>/` : `pump.py` fetches, for each station (serial number of the system) and each UTC day asked, into `data/<SERIAL>/`:
- les **métriques système** (énergie, disque, 4G, températures…), moyennes horaires, un CSV par métrique et par jour dans `metrics/` ; - the **system metrics** (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in `metrics/`;
- les **déplacements** (export « advanced »), un CSV par jour dans `displacement/` ; - the **displacements** ("advanced" export), one CSV per day in `displacement/`;
- les **mouvements de la caméra** (`pan`, `tilt`, `scale`, `rotation`, `sigma0`), en médiane et bornes (`median_`, `min_`, `max_`) par minute, aux seuls instants où la caméra a été recalée, un CSV par jour dans `camera/` ; - the **camera movements** (`pan`, `tilt`, `scale`, `rotation`, `sigma0`), as median and bounds (`median_`, `min_`, `max_`) per minute, only at the times the camera was realigned, one CSV per day in `camera/`;
- les **images** IRON (`--images`, une par heure par défaut, `--every 10` ou `--every 0` pour plus) dans `images/`, et différentielles (`--diff`) dans `images_diff/`. - the IRON **images** (`--images`, one per hour by default, `--every 10` or `--every 0` for more) in `images/`, and the differential ones (`--diff`) in `images_diff/`.
Formats : `~/labeling/connaissances/colonnes-des-deplacements.md`. Formats: `~/labeling/knowledge/displacement-columns.md`.
**Stations suivies** : `stations.txt`, un numéro de série par ligne. Sans station ni dates, le script les met toutes à jour jusqu'à maintenant : chaque station comble d'abord ses jours manquants entre le premier et le dernier jour pompé (sans déplacements, ou sans aucune métrique), puis repart de son dernier jour, repompé (il était peut-être incomplet) ; une station sans dossier dans `data/` reçoit les 30 derniers jours. **Tracked stations**: `stations.txt`, one serial number per line. Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in `data/` gets the last 30 days.
Reprise : avec `--du`, un fichier déjà dans `data/<SERIE>/` n'est pas redemandé ; relancer sur une plage plus large ne lit que les jours et les images qui manquent. Le jour en cours est pompé tel qu'il est, ses métriques jusqu'à la dernière heure terminée ; le lancement suivant sans `--du` le remplace. Resuming: with `--from`, a file already in `data/<SERIAL>/` is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without `--from` replaces it.
## Lancer (sous WSL) ## Running (under WSL)
`pomper.sh` lit les identifiants dans le `.env` à la racine du dépôt (base PostgreSQL et S3 ; jamais versionné, interdit en lecture à Claude), installe les 4 paquets nécessaires (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) dans un environnement `uv` jetable (Python 3.12), lance `pomper.py`, puis `lenzspitze_absolu.py`. Sa sortie s'ajoute à `out/pompage.log` ; un second lancement pendant un pompage s'arrête aussitôt (verrou `out/pompage.lock`). Le verrou ne vaut que pour ce dossier : les autres dépôts lisent `data/` par un lien, ils ne pompent pas. `pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `lenzspitze_absolute.py`. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump.
Variables du `.env` : `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT` ; pour les images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`. Variables of the `.env`: `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT`; for the images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`.
1. **Un essai** (1 station, 2 jours, dans un dossier jetable) : `extraction/pomper.sh --du 2026-11-09 --au 2026-11-11 --images --out /tmp/essai-pompage NORDEND` (6 s le 2026-11-12). `lenzspitze_absolu.py` convertit ensuite `data/LENZSPITZE`, pas le dossier `--out` : sans effet si `data/LENZSPITZE` est déjà converti. 1. **A trial** (1 station, 2 days, in a throwaway folder): `extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND` (6 s on 2026-11-12). `lenzspitze_absolute.py` then converts `data/LENZSPITZE`, not the `--out` folder: no effect if `data/LENZSPITZE` is already converted.
2. **Mettre à jour les stations suivies** : `extraction/pomper.sh` (sans argument : `--images --delai 300`), ou la tâche VS Code **pomper**. Le script dit, pour chaque station, la plage qu'il pompe. 2. **Updating the tracked stations**: `extraction/pump.sh` (without argument: `--images --delay 300`), or the VS Code task **pump**. The script says, for each station, the range it pumps.
3. **Une plage choisie** (historique, jour en échec), avec 3 jours de plus avant (l'historique des détecteurs) : `extraction/pomper.sh --du 2026-10-02 --au 2026-10-26 --images <SERIE>` (`--au` exclu ; plusieurs stations : `<SERIE1> <SERIE2>`). Aide : `extraction/pomper.sh -h`. 3. **A chosen range** (history, failed day), with 3 more days before (the history of the detectors): `extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images <SERIAL>` (`--to` excluded; several stations: `<SERIAL1> <SERIAL2>`). Help: `extraction/pump.sh -h`.
**Chaque nuit, automatiquement** : une tâche du Planificateur de tâches de Windows, à créer une fois dans PowerShell (pas en administrateur). Elle démarre WSL au besoin et, PC éteint ou en veille à l'heure dite, part dès qu'il se rallume ; un jour manqué est rattrapé au lancement suivant. **Every night, automatically**: a task of the Windows Task Scheduler, to create once in PowerShell (not as administrator). It starts WSL if needed and, PC off or asleep at the set time, runs as soon as it wakes up; a missed day is caught up at the next run.
```powershell ```powershell
$a = New-ScheduledTaskAction -Execute "wsl.exe" -Argument "-d Ubuntu -e bash -lc /home/maxime/aimsight-data/extraction/pomper.sh" $a = New-ScheduledTaskAction -Execute "wsl.exe" -Argument "-d Ubuntu -e bash -lc /home/maxime/aimsight-data/extraction/pump.sh"
$t = New-ScheduledTaskTrigger -Daily -At 04:00 $t = New-ScheduledTaskTrigger -Daily -At 04:00
$s = New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Hours 6) $s = New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Hours 6)
Register-ScheduledTask -TaskName "AIMsight pompage" -Action $a -Trigger $t -Settings $s Register-ScheduledTask -TaskName "AIMsight pump" -Action $a -Trigger $t -Settings $s
``` ```
Essayer la tâche tout de suite : `Start-ScheduledTask "AIMsight pompage"`, puis lire `out/pompage.log`. La retirer : `Unregister-ScheduledTask "AIMsight pompage"`. Trying the task right away: `Start-ScheduledTask "AIMsight pump"`, then read `out/pump.log`. Removing it: `Unregister-ScheduledTask "AIMsight pump"`.
Lecture seule : moteur dédié, `default_transaction_read_only`, délai de 60 s par requête, une requête par jour ; hors des heures chargées (base partagée avec la production). Read-only: dedicated engine, `default_transaction_read_only`, 60 s timeout per query, one query per day; outside busy hours (database shared with production).
Sans dépendance à l'appli `monitoring-dashboard` : ses lectures sont recopiées dans `pomper.py`, en SQL (sa docstring dit lesquelles, et de quel commit). Si l'appli change ses lectures, le script ne suit pas : comparer à nouveau des fichiers pompés aux deux. No dependency on the `monitoring-dashboard` app: its reads are copied into `pump.py`, in SQL (its docstring says which ones, and from which commit). If the app changes its reads, the script does not follow: compare again files pumped by both.
## Avec Claude ## With Claude
Claude peut lancer `pomper.sh` lui-même (accord de l'utilisateur, 2026-11-12) : lecture seule, petites plages ; une longue plage ou un historique, hors des heures chargées. Il ne lit jamais le `.env`. Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small ranges; a long range or a history, outside busy hours. It never reads the `.env`.
## À savoir ## Good to know
- `<SERIE>` est le numéro de série du **système**, celui de l'URL de la page System (`/explore/system/<SERIE>`) et du début des CSV qu'on y télécharge (`NORDEND_BusVoltage_…` : `NORDEND`) ; pas le nom de l'appareil de la page Explore. - `<SERIAL>` is the serial number of the **system**, the one in the URL of the System page (`/explore/system/<SERIAL>`) and at the start of the CSVs downloaded there (`NORDEND_BusVoltage_…`: `NORDEND`); not the name of the device on the Explore page.
- `data/` est hors de git : un dossier perdu se repompe. - `data/` is out of git: a lost folder is pumped again.
- Un jour en échec par délai dépassé (`QueryCanceled: … statement timeout`) : le relancer seul avec `--delai 300`. La lenteur vient de la base, pas du volume. - A day failed on a timeout (`QueryCanceled: … statement timeout`): run it again alone with `--delay 300`. The slowness comes from the database, not from the volume.
- Une heure sans image peut avoir sa mesure : l'image n'est jamais arrivée sur S3 (`measurement_uuid` vide, envoi inachevé). Le script la saute et prend l'image suivante de la tranche s'il y en a une. - An hour without an image can have its measurement: the image never reached S3 (empty `measurement_uuid`, unfinished upload). The script skips it and takes the next image of the slot if there is one.
- Un jour passé, pompé avant que la station ait fini d'envoyer ses données (coupure, synchronisation en retard), reste incomplet : supprimer ses fichiers dans `data/<SERIE>/` et relancer. - A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in `data/<SERIAL>/` and run again.
- LENZSPITZE : ses exports ont `x` / `y` relatifs et `pattern_type` vide ; `pomper.sh` lance ensuite `lenzspitze_absolu.py` (provisoire ; détail dans `~/labeling/connaissances/colonnes-des-deplacements.md`). - LENZSPITZE: its exports have relative `x` / `y` and an empty `pattern_type`; `pump.sh` then runs `lenzspitze_absolute.py` (provisional; details in `~/labeling/knowledge/displacement-columns.md`).
-49
View File
@@ -1,49 +0,0 @@
"""Provisoire, LENZSPITZE seulement : x / y des exports advanced en position absolue dans l'image, comme sur les autres stations.
Sur LENZSPITZE, x / y sont relatifs à la position de référence du motif (médianes ~0 px). Ce script réécrit en place
data/LENZSPITZE/displacement/*.csv :
x = x_motif + DX + x_csv, y = y_motif + DY + y_csv, où (x_motif, y_motif) est le centre du motif dans
data/LENZSPITZE/lentzpitze_patterns.json et (DX, DY) le décalage entre le repère du JSON et les images IRON ;
pattern_type, vide dans ces exports, est rempli depuis le JSON.
Les capteurs absents du JSON (1023, 8019) : x / y vidés (position inconnue) ; 8019 reçoit global_ref (il a des reproj_err).
Un fichier dont pattern_type est déjà rempli est laissé tel quel (déjà converti). Un repompage rend les originaux.
Usage : python3 extraction/lenzspitze_absolu.py
"""
import csv
import glob
import json
import os
STATION = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "data", "LENZSPITZE")
# Mesuré le 2026-11-04 : médiane (tache − motif du JSON) sur 9 images de nuit du 17.10 au 03.11, 25 à 34 cibles appariées
# chacune, stable à 0,5 px près. Comprend l'écart constant tache − (x, y) déjà vu sur NORDEND (colonnes-des-deplacements.md).
DX, DY = 330.5, 15.8
TYPE_SANS_MOTIF = {"8019": "global_ref"} # reproj_err et inlier remplis : référence de la compensation
motifs = {str(v["sensor_name"]): v for v in json.load(open(os.path.join(STATION, "lentzpitze_patterns.json"))).values()}
for chemin in sorted(glob.glob(os.path.join(STATION, "displacement", "*.csv"))):
with open(chemin, newline="") as f:
lecteur = csv.DictReader(f)
champs, lignes = lecteur.fieldnames, list(lecteur)
if any(r["pattern_type"] for r in lignes):
print(f"{os.path.basename(chemin)} : déjà converti")
continue
for r in lignes:
s = r["sensor_name"].strip()
m = motifs.get(s)
if m is None:
r["x"] = r["y"] = ""
r["pattern_type"] = TYPE_SANS_MOTIF.get(s, "")
continue
r["pattern_type"] = m["pattern_type"]
if r["x"]:
r["x"] = repr(m["x"] + DX + float(r["x"]))
if r["y"]:
r["y"] = repr(m["y"] + DY + float(r["y"]))
with open(chemin, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=champs)
w.writeheader()
w.writerows(lignes)
print(f"{os.path.basename(chemin)} : {len(lignes)} lignes converties")
+49
View File
@@ -0,0 +1,49 @@
"""Provisional, LENZSPITZE only: x / y of the advanced exports as absolute positions in the image, as on the other stations.
On LENZSPITZE, x / y are relative to the reference position of the pattern (medians ~0 px). This script rewrites in place
data/LENZSPITZE/displacement/*.csv:
x = x_pattern + DX + x_csv, y = y_pattern + DY + y_csv, where (x_pattern, y_pattern) is the centre of the pattern in
data/LENZSPITZE/lentzpitze_patterns.json and (DX, DY) the offset between the frame of the JSON and the IRON images;
pattern_type, empty in these exports, is filled from the JSON.
The sensors missing from the JSON (1023, 8019): x / y emptied (position unknown); 8019 gets global_ref (it has reproj_err).
A file whose pattern_type is already filled is left as it is (already converted). Pumping again restores the originals.
Usage: python3 extraction/lenzspitze_absolute.py
"""
import csv
import glob
import json
import os
STATION = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "data", "LENZSPITZE")
# Measured on 2026-11-04: median (spot − pattern of the JSON) over 9 night images from 17.10 to 03.11, 25 to 34 matched
# targets each, stable within 0.5 px. Includes the constant offset spot − (x, y) already seen on NORDEND (displacement-columns.md).
DX, DY = 330.5, 15.8
TYPE_WITHOUT_PATTERN = {"8019": "global_ref"} # reproj_err and inlier filled: reference of the compensation
patterns = {str(v["sensor_name"]): v for v in json.load(open(os.path.join(STATION, "lentzpitze_patterns.json"))).values()}
for path in sorted(glob.glob(os.path.join(STATION, "displacement", "*.csv"))):
with open(path, newline="") as f:
reader = csv.DictReader(f)
fields, rows = reader.fieldnames, list(reader)
if any(r["pattern_type"] for r in rows):
print(f"{os.path.basename(path)}: already converted")
continue
for r in rows:
s = r["sensor_name"].strip()
m = patterns.get(s)
if m is None:
r["x"] = r["y"] = ""
r["pattern_type"] = TYPE_WITHOUT_PATTERN.get(s, "")
continue
r["pattern_type"] = m["pattern_type"]
if r["x"]:
r["x"] = repr(m["x"] + DX + float(r["x"]))
if r["y"]:
r["y"] = repr(m["y"] + DY + float(r["y"]))
with open(path, "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=fields)
w.writeheader()
w.writerows(rows)
print(f"{os.path.basename(path)}: {len(rows)} rows converted")
-408
View File
@@ -1,408 +0,0 @@
"""Pompe les métriques système, les déplacements (export « advanced ») et les images de stations de la plateforme
AIMsight, dans le format que lisent l'app, les outils de lecture et les détecteurs. LECTURE SEULE. Essayé le
2026-10-26 (NORDEND, 2 jours) ; autonome depuis le 2026-11-12.
À lancer par extraction/pomper.sh, sous WSL, depuis n'importe où (voir README.md) :
extraction/pomper.sh
extraction/pomper.sh --du 2026-08-01 --au 2026-08-08 --images WEISSHORN
Sans station : celles de `stations.txt`. Sans `--du` : chaque station comble ses jours manquants (déplacements ou
métriques) avant le dernier pompé, puis repart de ce dernier jour, repompé (il était peut-être incomplet), ou des 30
derniers jours si elle n'en a pas ; sans `--au` : jusqu'à maintenant.
Sortie (défaut : le `data/` du dépôt, à côté de ce dossier) :
<STATION>/metrics/<STATION>_<métrique>_<AAAA_MM_JJ>_00_00 to <AAAA_MM_JJ>_23_00.csv métriques système, par jour
<STATION>/displacement/<STATION>_advanced_<AAAAMMJJ>.csv déplacements, un fichier par jour
<STATION>/camera/<STATION>_camera_<AAAAMMJJ>.csv mouvements de la caméra, par jour
<STATION>/images/<AAAAMMJJ>T<HHMM>Z_<id>.<ext> images OPTICAL_SHIFT-IRON (--images)
<STATION>/images_diff/… images différentielles (--diff)
Reprenable : avec `--du`, un fichier déjà écrit n'est pas relu, quelles que soient les dates demandées ; sans `--du`,
les jours de la plage sont remplacés (images exceptées). Jours en UTC, `--au` exclu ; le jour en cours est pompé tel
qu'il est, ses métriques jusqu'à la dernière heure terminée. Pour reprendre un jour, supprimer ses fichiers.
Sans dépendance à l'appli (~/monitoring-dashboard) : ses lectures sont recopiées ici, en SQL, telles quelles (branche
`explo`, commit 3dce73d du 2026-11-12) : la requête de System.get_metrics (le téléchargement de la page System) ;
build_advanced_frame (app/datapoints.py, ce que tracent l'onglet Advanced et l'export parquet), avec
AcquisitionUnit.get_datapoints et AggregateEngine.get_sensor_aggregate / get_common_aggregate ; les mouvements de la
caméra de fetch_cam_movements_frame (AggregateEngine.get_cam_scalar_aggregate, ce que trace le visualiseur) ;
AcquisitionUnit.measurements et get_measurement_payload (S3). Si l'appli change ses lectures, ce fichier ne suit pas.
Connexion en lecture seule (`default_transaction_read_only`), délai de 60 s par requête (`--delai`). Variables lues
(noms seulement, aucune n'est affichée) : AIMSIGHT_DATABASE_USER / _PSWD / _TURL / _NAME / _PORT, et pour les images
AIMSIGHT_S3_URL / _ACCESS_KEY / _SECRET_KEY / _BUCKET.
"""
import argparse
import os
import sys
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pandas as pd
from sqlalchemy import create_engine, text
from sqlalchemy.engine import URL
UTC = timezone.utc
STATIONS = Path(__file__).resolve().parent / "stations.txt" # les stations suivies, une par ligne
JOURS_NOUVELLE = 30 # sans --du, une station sans déplacements pompés : les 30 derniers jours
DB_NAMES = ["AIMSIGHT_DATABASE_USER", "AIMSIGHT_DATABASE_PSWD", "AIMSIGHT_DATABASE_TURL", "AIMSIGHT_DATABASE_NAME"]
TYPES = {"images": "OPTICAL_SHIFT-IRON", "images_diff": "OPTICAL_SHIFT-DIFF"} # dossier -> type de mesure
# --- constantes de l'appli (app/datapoints.py, app/aggregates.py) ---
QUALITY = ["peak", "flux", "sharpness", "fwhm", "dominance", "snr"]
CAM_SIGNALS = ["pan", "tilt", "scale", "rotation", "sigma0"]
DEFAULT_ERROR_THRESHOLD = 0.7
DATATYPES = ["x", "y", "error", "corr_x", "corr_y", "reproj_err", "inlier", "sigma0"]
PROPERTIES = ["x", "y", "pattern_type", "sensor_name", "error_threshold"]
COLUMNS = ["time_utc", "sensor_name", "x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x",
"corr_y", "reproj_err", "inlier", "sigma0", "pattern_type"] + QUALITY
NUMERIC = ["x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x", "corr_y", "reproj_err",
"inlier", "sigma0"] + QUALITY
# bandes d'inlier de shift_sensor_1min -> code ; les deux derniers NOMS sont faux (-1 rejeté, -2 interpolé), les codes
# sont justes ; ordre du meilleur au pire
INLIER_BANDS = {"inlier_in_fit": 1.0, "inlier_out_fit": 0.0, "inlier_no_redundancy": -1.0, "inlier_rejected": -2.0}
def raison(e):
"""L'erreur d'origine (celle de la base) : pandas et SQLAlchemy l'enveloppent, la requête SQL d'abord."""
while e.__cause__ is not None:
e = e.__cause__
return f"{type(e).__name__}: {(str(e).splitlines() or [''])[0][:300]}"
def lire(conn, sql, **params):
return pd.read_sql(text(sql), conn, params=params)
def acquisition_unit(conn, system_id):
"""Même choix que Device.get_acquisition_unit (app/base/models.py:268) : l'unique unité, sinon MER* / ME2S*."""
units = conn.execute(text("SELECT id, product_number FROM acquisition_unit WHERE system_id = :s"),
{"s": system_id}).all()
pick = units if len(units) == 1 else [u for u in units if u.product_number.startswith(("MER", "ME2S"))]
return pick[0].id if pick else None
def plages_suivi(dossier, fin, demain):
"""Sans --du : (début, fin, remplacer) à pomper, d'après les jours de déplacements déjà pompés. Les jours manquants
entre le premier et le dernier (plages pompées à part, jour en échec) sont comblés sans toucher aux autres ; le
dernier jour est repompé (pompé en cours de journée, il était incomplet) jusqu'à maintenant ; une station sans
déplacements pompés reçoit les 30 derniers jours."""
jours = {datetime.strptime(p.stem.rsplit("_", 1)[1], "%Y%m%d").date()
for p in (dossier / "displacement").glob(f"{dossier.name}_advanced_*.csv")}
if not jours:
return [(demain - timedelta(days=JOURS_NOUVELLE + 1), fin, True)]
# un jour de métriques en échec n'a aucun fichier (une requête pour toutes) : c'est aussi un trou ; une station
# sans aucune métrique n'en a jamais, ses jours n'en sont pas
metriques = {datetime.strptime(p.name.split(" to ")[0][-16:-6], "%Y_%m_%d").date()
for p in (dossier / "metrics").glob(f"{dossier.name}_*_00_00 to *.csv")}
premier, dernier = min(jours), max(jours)
trous = [premier + timedelta(days=k) for k in range((dernier - premier).days)
if premier + timedelta(days=k) not in jours or (metriques and premier + timedelta(days=k) not in metriques)]
return ([(trous[0], dernier, False)] if trous else []) + [(dernier, fin, True)]
def pomper_metriques(conn, system_id, station, debut, fin, out, remplacer):
"""Comme le téléchargement de la page System (Dashboard/System.py, get_metrics(..., bucketing="1 hour")) : par
heure, moyenne * transform_factor + transform_offset, ou la liste des valeurs d'une métrique texte ; les séries à
labels d'une même métrique confondues ; une ligne vide pour une heure sans mesure. get_metrics compte en heures
avant maintenant : sa requête est refaite ici sur un jour fixe, une requête par jour pour toutes les métriques."""
noms = sorted(conn.execute(text("SELECT DISTINCT dt.name FROM datatype dt JOIN metric_series se "
"ON se.datatype_id = dt.id WHERE se.system_id = :s"), {"s": system_id}).scalars())
conn.rollback()
for k in range((fin - debut).days):
a = datetime.combine(debut + timedelta(days=k), datetime.min.time(), UTC)
paths = {n: out / station / "metrics"
/ f"{station}_{n.replace(os.sep, '-')}_{a:%Y_%m_%d}_00_00 to {a:%Y_%m_%d}_23_00.csv" for n in noms}
if not remplacer and all(p.exists() for p in paths.values()):
continue
t0 = time.time()
try:
# date_trunc('hour') : même découpage que time_bucket('1 hour')
df = lire(conn, """
SELECT date_trunc('hour', ms.time_utc) AS heure, dt.name AS nom,
avg(ms.value_num) * any_value(dt.transform_factor) + any_value(dt.transform_offset) AS num,
jsonb_agg(ms.value_text) FILTER (WHERE ms.value_text IS NOT NULL) AS txt
FROM metric_sample ms
JOIN metric_series se ON ms.series_id = se.id
JOIN datatype dt ON se.datatype_id = dt.id
WHERE se.system_id = :s AND ms.time_utc >= :a AND ms.time_utc < :b
GROUP BY 1, dt.name""", s=system_id, a=a, b=a + timedelta(days=1))
except Exception as e: # ponytail : comme les déplacements, un jour en échec est sauté, relancer le reprend
print(f"{station} {a:%Y-%m-%d} métriques : échec ({raison(e)}), à relancer")
continue
finally:
conn.rollback()
df["heure"] = pd.to_datetime(df["heure"], utc=True)
heures = pd.to_datetime(pd.date_range(a, periods=24, freq="h"), utc=True) # même fuseau que df, pour reindex
# jour en cours : les heures terminées seulement, sinon les heures à venir seraient lues comme un trou
heures = heures[heures + pd.Timedelta(hours=1) <= pd.Timestamp.now(tz=UTC)]
for n, path in paths.items():
d = df[df["nom"] == n].set_index("heure")
col = d["num"] if d["num"].notna().any() else d["txt"] # métrique texte : la liste de l'heure
path.parent.mkdir(parents=True, exist_ok=True)
# CSV de la page System : ',time_utc,metric', l'index en première colonne ; un jour vide est écrit aussi
pd.DataFrame({"time_utc": heures, "metric": col.reindex(heures).to_numpy()}).to_csv(path)
print(f"{station} {a:%Y-%m-%d} métriques : {len(noms)} fichiers, {df['nom'].nunique()} avec mesure, "
f"{time.time() - t0:.0f} s")
# --- l'export « advanced » : build_advanced_frame (app/datapoints.py) ---
def datapoints_bruts(conn, unit, a, b):
"""AcquisitionUnit.get_datapoints : les datapoints de l'unité, un type par colonne, les propriétés des capteurs
à côté (prop_x / prop_y : la position de référence, homonyme d'un type)."""
types = ",\n".join(f"max(d.value) FILTER (WHERE dt.name = '{t}') AS {t}" for t in DATATYPES)
noms = {p: f"prop_{p}" if p in DATATYPES else p for p in PROPERTIES}
props = ",\n".join(f"any_value(p_{p}.value) AS {noms[p]}" for p in PROPERTIES)
joints = "\n".join(f"LEFT JOIN (SELECT p.sensor_id, p.value FROM property p JOIN datatype pdt "
f"ON p.datatype_id = pdt.id WHERE pdt.name = '{p}') p_{p} ON d.sensor_id = p_{p}.sensor_id"
for p in PROPERTIES)
return lire(conn, f"""
SELECT d.time_utc, d.sensor_id, any_value(d.measurement_id) AS measurement_id,
{types},
{props}
FROM datapoint d
JOIN sensor s ON d.sensor_id = s.id
JOIN datatype dt ON d.datatype_id = dt.id
{joints}
WHERE s.acquisition_unit_id = :au AND d.time_utc >= :a AND d.time_utc <= :b
GROUP BY d.time_utc, d.sensor_id
ORDER BY d.sensor_id, d.time_utc""", au=unit, a=a, b=b)
def preparer_bruts(shifts):
"""prepare_datapoints_frame : sigma0 (lignes de la caméra, sans sensor_name) recopié sur chaque capteur à son
instant ; relative = x - prop_x ; error -> error_value ; le schéma complet."""
if shifts.empty:
return pd.DataFrame(columns=COLUMNS)
sig = shifts.loc[shifts["sigma0"].notna(), ["time_utc", "sigma0"]]
sigma0 = sig.groupby("time_utc")["sigma0"].median() if not sig.empty else None
shifts = shifts.dropna(subset=["sensor_name"]).reset_index(drop=True)
shifts["sigma0"] = shifts["time_utc"].map(sigma0) if sigma0 is not None else pd.NA
shifts["relative_x"] = shifts["x"] - pd.to_numeric(shifts["prop_x"], errors="coerce")
shifts["relative_y"] = shifts["y"] - pd.to_numeric(shifts["prop_y"], errors="coerce")
shifts = shifts.drop(columns=["prop_x", "prop_y"]).rename(columns={"error": "error_value"})
shifts["time_utc"] = pd.to_datetime(shifts["time_utc"])
shifts["sensor_name"] = shifts["sensor_name"].astype(str)
for c in NUMERIC:
if c in shifts:
shifts[c] = pd.to_numeric(shifts[c], errors="coerce")
return shifts.reindex(columns=COLUMNS)
def etats_agreges(conn, unit, a, b):
"""fetch_aggregate_datapoints_frame : les médianes par minute des tables d'états (shift_sensor_1min,
shift_common_1min), seule source des appareils « burst », au schéma de l'export (prepare_aggregate_datapoints_frame).
Dans les états, x_pct porte le x CORRIGÉ et rx_pct le x brut."""
thr = "COALESCE((th.value #>> '{}')::double precision, CAST(:thr AS double precision))"
sensor = lire(conn, f"""
SELECT ss.time_utc, COALESCE(sn.value #>> '{{}}', s.name) AS name,
approx_percentile(0.5, ss.x_pct) AS corr_x, approx_percentile(0.5, ss.y_pct) AS corr_y,
approx_percentile(0.5, ss.rx_pct) AS x, approx_percentile(0.5, ss.ry_pct) AS y,
approx_percentile(0.5, ss.error_pct) AS error, {thr} AS error_threshold,
approx_percentile(0.5, ss.reproj_err_pct) AS reproj_err,
{", ".join(f"ss.{n}" for n in INLIER_BANDS)},
{", ".join(f"approx_percentile(0.5, ss.{m}_pct) AS {m}" for m in QUALITY)}
FROM shift_sensor_1min ss
JOIN sensor s ON s.id = ss.sensor_id
LEFT JOIN (SELECT p.sensor_id, any_value(p.value) AS value FROM property p JOIN datatype dt
ON dt.id = p.datatype_id WHERE dt.name = 'sensor_name' GROUP BY p.sensor_id) sn
ON sn.sensor_id = ss.sensor_id
LEFT JOIN (SELECT p.sensor_id, any_value(p.value) AS value FROM property p JOIN datatype dt
ON dt.id = p.datatype_id WHERE dt.name = 'error_threshold' GROUP BY p.sensor_id) th
ON th.sensor_id = ss.sensor_id
WHERE s.acquisition_unit_id = :au AND ss.time_utc >= :a AND ss.time_utc <= :b
ORDER BY ss.time_utc""", au=unit, a=a, b=b, thr=DEFAULT_ERROR_THRESHOLD)
if sensor.empty:
return pd.DataFrame(columns=COLUMNS)
common = lire(conn, """
SELECT sc.time_utc, approx_percentile(0.5, sc.sigma0_pct) AS sigma0 FROM shift_common_1min sc
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b""", au=unit, a=a, b=b)
out = pd.DataFrame({"time_utc": pd.to_datetime(sensor["time_utc"]), "sensor_name": sensor["name"].astype(str)})
for col, src in [("error_value", "error"), ("error_threshold", "error_threshold"), ("corr_x", "corr_x"),
("corr_y", "corr_y"), ("relative_x", "corr_x"), ("relative_y", "corr_y"), ("x", "x"), ("y", "y"),
("reproj_err", "reproj_err")] + [(m, m) for m in QUALITY]:
out[col] = pd.to_numeric(sensor[src], errors="coerce")
# inlier_code_from_bands : la bande majoritaire, l'égalité à la pire ; toutes à 0 -> NaN (pas de redondance)
counts = sensor[list(INLIER_BANDS)].apply(pd.to_numeric, errors="coerce").fillna(0)
code = counts[list(INLIER_BANDS)[::-1]].idxmax(axis=1).map(INLIER_BANDS)
out["inlier"] = code.where(counts.sum(axis=1) > 0).to_numpy()
out["pattern_type"] = None
sig = common.loc[common["sigma0"].notna()]
out["sigma0"] = (out["time_utc"].map(sig.assign(time_utc=pd.to_datetime(sig["time_utc"]))
.groupby("time_utc")["sigma0"].median()) if not sig.empty else float("nan"))
return out.reindex(columns=COLUMNS)
def fusionner(raw, agg):
"""merge_raw_and_aggregate_frames : le brut gagne ; un état n'est gardé que là où son capteur n'a pas de ligne
brute dans la même minute."""
if agg.empty:
return raw
if raw.empty:
return agg
def utc(t):
t = pd.to_datetime(t)
return t.dt.tz_localize("UTC") if t.dt.tz is None else t.dt.tz_convert("UTC")
cles = pd.MultiIndex.from_arrays([raw["sensor_name"], utc(raw["time_utc"]).dt.floor("1min")])
agg = agg.loc[~pd.MultiIndex.from_arrays([agg["sensor_name"], utc(agg["time_utc"]).dt.floor("1min")]).isin(cles)]
if agg.empty:
return raw
agg = agg.copy()
tz = pd.to_datetime(raw["time_utc"]).dt.tz
t = utc(agg["time_utc"])
agg["time_utc"] = t.dt.tz_convert(tz) if tz is not None else t.dt.tz_localize(None)
return (pd.concat([raw, agg], ignore_index=True).sort_values(["sensor_name", "time_utc"], kind="stable")
.reset_index(drop=True).reindex(columns=COLUMNS))
def advanced(conn, unit, a, b):
return fusionner(preparer_bruts(datapoints_bruts(conn, unit, a, b)), etats_agreges(conn, unit, a, b))
def camera(conn, unit, a, b):
"""Pan, tilt, scale, rotation et sigma0 de la caméra, en médiane et bornes (median_ / min_ / max_) par minute, donc
une ligne par mesure de la caméra (toutes les 15 min à NORDEND, min = max) : ce que trace le visualiseur
(fetch_cam_movements_frame), sans son except, qui rendrait vide un jour en échec au lieu de le laisser à relancer."""
sigs = ", ".join(f"approx_percentile(0.5, sc.{s}_pct) AS median_{s}, min_val(sc.{s}_pct) AS min_{s}, "
f"max_val(sc.{s}_pct) AS max_{s}" for s in CAM_SIGNALS)
return lire(conn, f"""SELECT sc.time_utc, {sigs} FROM shift_common_1min sc
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b
ORDER BY sc.time_utc""", au=unit, a=a, b=b)
JOURS = (("displacement", "advanced", advanced), ("camera", "camera", camera)) # dossier, fichier, lecture
def pomper_deplacements(conn, unit, station, debut, fin, out, remplacer):
"""Déplacements et mouvements de la caméra : un fichier et une requête par jour, bornée, sous le délai (--delai)."""
for k in range((fin - debut).days):
a = datetime.combine(debut + timedelta(days=k), datetime.min.time(), UTC)
for dossier, nom, lecture in JOURS:
path = out / station / dossier / f"{station}_{nom}_{a:%Y%m%d}.csv"
if path.exists() and not remplacer:
continue
t0 = time.time()
try:
df = lecture(conn, unit, a, a + timedelta(days=1) - timedelta(microseconds=1))
except Exception as e: # ponytail : un jour en échec est sauté, relancer le reprend ; pas de nouvel essai
print(f"{station} {a:%Y-%m-%d} {nom} : échec ({raison(e)}), à relancer")
continue
finally:
conn.rollback() # ferme la transaction de lecture
if "sensor_name" in df:
df = df.drop_duplicates(["sensor_name", "time_utc"])
path.parent.mkdir(parents=True, exist_ok=True)
df.to_csv(path, index=False) # un jour vide est écrit aussi : la relance ne le redemande pas
print(f"{station} {a:%Y-%m-%d} {nom}: {len(df)} lignes, {time.time() - t0:.0f} s -> {path.name}")
def pomper_images(conn, s3, unit, station, debut, fin, dossier, every, out):
mt = conn.execute(text("SELECT id, extension FROM measurement_type WHERE name = :n"), {"n": TYPES[dossier]}).first()
if mt is None:
print(f"{station} : type {TYPES[dossier]} inconnu")
return
client, bucket_s3 = s3
for k in range((fin - debut).days):
a = datetime.combine(debut + timedelta(days=k), datetime.min.time(), UTC)
mesures = conn.execute(text(
"SELECT id, time_utc, measurement_uuid FROM measurement WHERE acquisition_unit_id = :au "
"AND measurement_type_id = :mt AND time_utc >= :a AND time_utc < :b ORDER BY time_utc"),
{"au": unit, "mt": mt.id, "a": a, "b": a + timedelta(days=1)}).all()
conn.rollback()
seen, n, octets, sans_uuid, absentes = set(), 0, 0, 0, 0
for m in mesures:
# measurement_uuid vide : l'image n'est jamais arrivée sur S3 (envoi en plusieurs parties inachevé) ;
# écartée, la tranche prend l'image suivante s'il y en a une
if m.measurement_uuid is None:
sans_uuid += 1
continue
t = m.time_utc.astimezone(UTC)
bucket = (t.hour * 60 + t.minute) // every if every else m.id # une image par tranche de `every` min
path = out / station / dossier / f"{t:%Y%m%dT%H%M}Z_{m.id}.{mt.extension}"
if bucket in seen:
continue
seen.add(bucket) # même déjà là : sinon la relance prendrait l'image suivante de la tranche
if path.exists():
continue
try:
data = client.get_object(Bucket=bucket_s3, Key=m.measurement_uuid)["Body"].read()
except Exception: # comme get_measurement_payload : S3 qui ne répond pas, l'image est comptée absente
data = None
if data:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
n, octets = n + 1, octets + len(data)
else:
absentes += 1
# le détail dit pourquoi un jour rend 0 : rien en base, jamais arrivées sur S3, ou S3 qui ne les rend pas
print(f"{station} {a:%Y-%m-%d} {dossier}: {n} nouvelles images, {octets / 1e6:.0f} Mo "
f"({len(mesures)} en base, {sans_uuid} jamais arrivées sur S3, {absentes} non rendues par S3)")
time.sleep(0.5)
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("stations", nargs="*", help="numéro de série du système, comme dans l'URL de la page System ; "
"aucun : ceux de stations.txt")
p.add_argument("--du", help="premier jour UTC, AAAA-MM-JJ ; sans : le dernier jour pompé de chaque station, "
"repompé, ou les 30 derniers jours")
p.add_argument("--au", help="dernier jour UTC exclu, AAAA-MM-JJ ; sans : jusqu'à maintenant")
p.add_argument("--images", action="store_true", help="aussi les images OPTICAL_SHIFT-IRON")
p.add_argument("--diff", action="store_true", help="aussi les images OPTICAL_SHIFT-DIFF")
p.add_argument("--every", type=int, default=60, help="une image par tranche de N minutes, 0 = toutes (défaut 60)")
p.add_argument("--delai", type=int, default=60, help="délai maximal d'une requête, en s (défaut 60 : base partagée "
"avec la production) ; plus long seulement pour un jour en échec par délai dépassé")
p.add_argument("--out", type=Path, default=Path(__file__).resolve().parent.parent / "data")
a = p.parse_args()
demain = datetime.now(UTC).date() + timedelta(days=1)
du = datetime.strptime(a.du, "%Y-%m-%d").date() if a.du else None
fin = min(datetime.strptime(a.au, "%Y-%m-%d").date(), demain) if a.au else demain # pas de jour à venir
stations = a.stations or STATIONS.read_text().split()
missing = [n for n in DB_NAMES if not os.environ.get(n)]
if missing:
sys.exit(f"Variables absentes (noms seulement) : {', '.join(missing)}")
url = URL.create("postgresql+psycopg2", username=os.environ["AIMSIGHT_DATABASE_USER"],
password=os.environ["AIMSIGHT_DATABASE_PSWD"], host=os.environ["AIMSIGHT_DATABASE_TURL"],
port=int(os.environ.get("AIMSIGHT_DATABASE_PORT", "5432")),
database=os.environ["AIMSIGHT_DATABASE_NAME"])
engine = create_engine(url, connect_args={
"options": f"-c timezone=utc -c default_transaction_read_only=on -c statement_timeout={a.delai}s"})
s3 = None
if a.images or a.diff:
import boto3 # comme S3client de l'appli (Garage)
s3 = (boto3.client("s3", endpoint_url=os.environ.get("AIMSIGHT_S3_URL", "http://s3:3900"),
aws_access_key_id=os.environ.get("AIMSIGHT_S3_ACCESS_KEY", ""),
aws_secret_access_key=os.environ.get("AIMSIGHT_S3_SECRET_KEY", ""),
region_name="garage", use_ssl=False),
os.environ.get("AIMSIGHT_S3_BUCKET", "measurements-assets"))
try:
with engine.connect() as conn:
for station in stations:
system_id = conn.execute(text("SELECT id FROM system WHERE serial_number = :s"), {"s": station}).scalar()
if system_id is None:
print(f"{station} : système inconnu")
continue
plages = [(du, fin, False)] if du else plages_suivi(a.out / station, fin, demain)
unit = acquisition_unit(conn, system_id)
conn.rollback()
for debut, fin_plage, remplacer in plages:
print(f"{station} : du {debut} au {fin_plage - timedelta(days=1)} inclus"
+ ("" if remplacer else ", jours manquants seulement"))
pomper_metriques(conn, system_id, station, debut, fin_plage, a.out, remplacer)
if unit is None:
print(f"{station} : pas d'unité d'acquisition reconnue, ni déplacements, ni caméra, ni images")
continue
pomper_deplacements(conn, unit, station, debut, fin_plage, a.out, remplacer)
for flag, dossier in ((a.images, "images"), (a.diff, "images_diff")):
if flag:
pomper_images(conn, s3, unit, station, debut, fin_plage, dossier, a.every, a.out)
finally:
engine.dispose()
if __name__ == "__main__":
main()
-25
View File
@@ -1,25 +0,0 @@
#!/usr/bin/env bash
# Pompage sous WSL. Lit les identifiants dans le .env à la racine du dépôt (jamais versionné),
# lance pomper.py dans un environnement uv jetable (4 paquets), puis le post-traitement de LENZSPITZE.
# extraction/pomper.sh mise à jour des stations suivies, images comprises (le planificateur)
# extraction/pomper.sh --du D --au F [SERIE] tout autre appel : les arguments vont tels quels à pomper.py
# Sortie ajoutée à out/pompage.log ; un seul pompage à la fois (verrou out/pompage.lock).
set -u
export PATH="$HOME/.local/bin:$PATH" # uv, absent du PATH quand le Planificateur de tâches lance wsl.exe
ici=$(cd "$(dirname "$0")" && pwd)
racine=$(dirname "$ici")
mkdir -p "$racine/out"
exec 9>"$racine/out/pompage.lock"
flock -n 9 || { echo "Un pompage tourne déjà." >&2; exit 1; }
[ $# -eq 0 ] && set -- --images --delai 300
{
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) pomper.py $*"
(set -a && . "$racine/.env" && set +a &&
uv run --no-project --quiet --python 3.12 \
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
python "$ici/pomper.py" "$@")
[ -f "$racine/data/LENZSPITZE/lentzpitze_patterns.json" ] && python3 "$ici/lenzspitze_absolu.py"
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) fin"
} 2>&1 | tee -a "$racine/out/pompage.log"
+409
View File
@@ -0,0 +1,409 @@
"""Pumps the system metrics, the displacements ("advanced" export) and the images of stations from the AIMsight
platform, in the format read by the app, the reading tools and the detectors. READ-ONLY. Tried on 2026-10-26
(NORDEND, 2 days); standalone since 2026-11-12.
Run through extraction/pump.sh, under WSL, from anywhere (see README.md):
extraction/pump.sh
extraction/pump.sh --from 2026-08-01 --to 2026-08-08 --images WEISSHORN
Without a station: those of `stations.txt`. Without `--from`: each station fills its missing days (displacements or
metrics) before the last one pumped, then restarts from that last day, pumped again (it was perhaps incomplete), or
from the last 30 days if it has none; without `--to`: up to now.
Output (default: the repository's `data/`, next to this folder):
<STATION>/metrics/<STATION>_<metric>_<YYYY_MM_DD>_00_00 to <YYYY_MM_DD>_23_00.csv system metrics, per day
<STATION>/displacement/<STATION>_advanced_<YYYYMMDD>.csv displacements, one file per day
<STATION>/camera/<STATION>_camera_<YYYYMMDD>.csv camera movements, per day
<STATION>/images/<YYYYMMDD>T<HHMM>Z_<id>.<ext> OPTICAL_SHIFT-IRON images (--images)
<STATION>/images_diff/… differential images (--diff)
Resumable: with `--from`, a file already written is not read again, whatever the dates asked; without `--from`, the
days of the range are replaced (images excepted). Days in UTC, `--to` excluded; the current day is pumped as it is,
its metrics up to the last completed hour. To redo a day, delete its files.
No dependency on the app (~/monitoring-dashboard): its reads are copied here, in SQL, as they are (branch `explo`,
commit 3dce73d of 2026-11-12): the query of System.get_metrics (the download of the System page);
build_advanced_frame (app/datapoints.py, what the Advanced tab and the parquet export plot), with
AcquisitionUnit.get_datapoints and AggregateEngine.get_sensor_aggregate / get_common_aggregate; the camera movements
of fetch_cam_movements_frame (AggregateEngine.get_cam_scalar_aggregate, what the viewer plots);
AcquisitionUnit.measurements and get_measurement_payload (S3). If the app changes its reads, this file does not follow.
Read-only connection (`default_transaction_read_only`), 60 s timeout per query (`--delay`). Variables read (names
only, none is printed): AIMSIGHT_DATABASE_USER / _PSWD / _TURL / _NAME / _PORT, and for the images
AIMSIGHT_S3_URL / _ACCESS_KEY / _SECRET_KEY / _BUCKET.
"""
import argparse
import os
import sys
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pandas as pd
from sqlalchemy import create_engine, text
from sqlalchemy.engine import URL
UTC = timezone.utc
STATIONS = Path(__file__).resolve().parent / "stations.txt" # the tracked stations, one per line
NEW_STATION_DAYS = 30 # without --from, a station with no pumped displacements: the last 30 days
DB_NAMES = ["AIMSIGHT_DATABASE_USER", "AIMSIGHT_DATABASE_PSWD", "AIMSIGHT_DATABASE_TURL", "AIMSIGHT_DATABASE_NAME"]
TYPES = {"images": "OPTICAL_SHIFT-IRON", "images_diff": "OPTICAL_SHIFT-DIFF"} # folder -> measurement type
# --- constants of the app (app/datapoints.py, app/aggregates.py) ---
QUALITY = ["peak", "flux", "sharpness", "fwhm", "dominance", "snr"]
CAM_SIGNALS = ["pan", "tilt", "scale", "rotation", "sigma0"]
DEFAULT_ERROR_THRESHOLD = 0.7
DATATYPES = ["x", "y", "error", "corr_x", "corr_y", "reproj_err", "inlier", "sigma0"]
PROPERTIES = ["x", "y", "pattern_type", "sensor_name", "error_threshold"]
COLUMNS = ["time_utc", "sensor_name", "x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x",
"corr_y", "reproj_err", "inlier", "sigma0", "pattern_type"] + QUALITY
NUMERIC = ["x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x", "corr_y", "reproj_err",
"inlier", "sigma0"] + QUALITY
# inlier bands of shift_sensor_1min -> code; the last two NAMES are wrong (-1 rejected, -2 interpolated), the codes
# are right; ordered from best to worst
INLIER_BANDS = {"inlier_in_fit": 1.0, "inlier_out_fit": 0.0, "inlier_no_redundancy": -1.0, "inlier_rejected": -2.0}
def root_cause(e):
"""The original error (the database's): pandas and SQLAlchemy wrap it, the SQL query first."""
while e.__cause__ is not None:
e = e.__cause__
return f"{type(e).__name__}: {(str(e).splitlines() or [''])[0][:300]}"
def query(conn, sql, **params):
return pd.read_sql(text(sql), conn, params=params)
def acquisition_unit(conn, system_id):
"""Same choice as Device.get_acquisition_unit (app/base/models.py:268): the only unit, otherwise MER* / ME2S*."""
units = conn.execute(text("SELECT id, product_number FROM acquisition_unit WHERE system_id = :s"),
{"s": system_id}).all()
pick = units if len(units) == 1 else [u for u in units if u.product_number.startswith(("MER", "ME2S"))]
return pick[0].id if pick else None
def update_ranges(folder, end, tomorrow):
"""Without --from: (start, end, replace) to pump, from the days of displacements already pumped. The missing days
between the first and the last (ranges pumped separately, failed day) are filled without touching the others; the
last day is pumped again (pumped during the day, it was incomplete) up to now; a station with no pumped
displacements gets the last 30 days."""
days = {datetime.strptime(p.stem.rsplit("_", 1)[1], "%Y%m%d").date()
for p in (folder / "displacement").glob(f"{folder.name}_advanced_*.csv")}
if not days:
return [(tomorrow - timedelta(days=NEW_STATION_DAYS + 1), end, True)]
# a failed metrics day has no file at all (one query for all of them): it is a gap too; a station without any
# metric never has one, its days are not gaps
metric_days = {datetime.strptime(p.name.split(" to ")[0][-16:-6], "%Y_%m_%d").date()
for p in (folder / "metrics").glob(f"{folder.name}_*_00_00 to *.csv")}
first, last = min(days), max(days)
gaps = [first + timedelta(days=k) for k in range((last - first).days)
if first + timedelta(days=k) not in days or (metric_days and first + timedelta(days=k) not in metric_days)]
return ([(gaps[0], last, False)] if gaps else []) + [(last, end, True)]
def pump_metrics(conn, system_id, station, start, end, out, replace):
"""Like the download of the System page (Dashboard/System.py, get_metrics(..., bucketing="1 hour")): per hour,
mean * transform_factor + transform_offset, or the list of values of a text metric; the labelled series of one
metric merged; an empty row for an hour without measurement. get_metrics counts in hours before now: its query is
redone here on a fixed day, one query per day for all the metrics."""
names = sorted(conn.execute(text("SELECT DISTINCT dt.name FROM datatype dt JOIN metric_series se "
"ON se.datatype_id = dt.id WHERE se.system_id = :s"), {"s": system_id}).scalars())
conn.rollback()
for k in range((end - start).days):
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
paths = {n: out / station / "metrics"
/ f"{station}_{n.replace(os.sep, '-')}_{a:%Y_%m_%d}_00_00 to {a:%Y_%m_%d}_23_00.csv" for n in names}
if not replace and all(p.exists() for p in paths.values()):
continue
t0 = time.time()
try:
# date_trunc('hour'): same bucketing as time_bucket('1 hour')
df = query(conn, """
SELECT date_trunc('hour', ms.time_utc) AS hour, dt.name AS name,
avg(ms.value_num) * any_value(dt.transform_factor) + any_value(dt.transform_offset) AS num,
jsonb_agg(ms.value_text) FILTER (WHERE ms.value_text IS NOT NULL) AS txt
FROM metric_sample ms
JOIN metric_series se ON ms.series_id = se.id
JOIN datatype dt ON se.datatype_id = dt.id
WHERE se.system_id = :s AND ms.time_utc >= :a AND ms.time_utc < :b
GROUP BY 1, dt.name""", s=system_id, a=a, b=a + timedelta(days=1))
except Exception as e: # ponytail: like the displacements, a failed day is skipped, a new run picks it up
print(f"{station} {a:%Y-%m-%d} metrics: failed ({root_cause(e)}), run again")
continue
finally:
conn.rollback()
df["hour"] = pd.to_datetime(df["hour"], utc=True)
hours = pd.to_datetime(pd.date_range(a, periods=24, freq="h"), utc=True) # same time zone as df, for reindex
# current day: the completed hours only, otherwise the hours to come would read as a gap
hours = hours[hours + pd.Timedelta(hours=1) <= pd.Timestamp.now(tz=UTC)]
for n, path in paths.items():
d = df[df["name"] == n].set_index("hour")
col = d["num"] if d["num"].notna().any() else d["txt"] # text metric: the list of the hour
path.parent.mkdir(parents=True, exist_ok=True)
# CSV of the System page: ',time_utc,metric', the index as first column; an empty day is written too
pd.DataFrame({"time_utc": hours, "metric": col.reindex(hours).to_numpy()}).to_csv(path)
print(f"{station} {a:%Y-%m-%d} metrics: {len(names)} files, {df['name'].nunique()} with data, "
f"{time.time() - t0:.0f} s")
# --- the "advanced" export: build_advanced_frame (app/datapoints.py) ---
def raw_datapoints(conn, unit, a, b):
"""AcquisitionUnit.get_datapoints: the datapoints of the unit, one type per column, the sensor properties next to
them (prop_x / prop_y: the reference position, namesake of a type)."""
types = ",\n".join(f"max(d.value) FILTER (WHERE dt.name = '{t}') AS {t}" for t in DATATYPES)
names = {p: f"prop_{p}" if p in DATATYPES else p for p in PROPERTIES}
props = ",\n".join(f"any_value(p_{p}.value) AS {names[p]}" for p in PROPERTIES)
joins = "\n".join(f"LEFT JOIN (SELECT p.sensor_id, p.value FROM property p JOIN datatype pdt "
f"ON p.datatype_id = pdt.id WHERE pdt.name = '{p}') p_{p} ON d.sensor_id = p_{p}.sensor_id"
for p in PROPERTIES)
return query(conn, f"""
SELECT d.time_utc, d.sensor_id, any_value(d.measurement_id) AS measurement_id,
{types},
{props}
FROM datapoint d
JOIN sensor s ON d.sensor_id = s.id
JOIN datatype dt ON d.datatype_id = dt.id
{joins}
WHERE s.acquisition_unit_id = :au AND d.time_utc >= :a AND d.time_utc <= :b
GROUP BY d.time_utc, d.sensor_id
ORDER BY d.sensor_id, d.time_utc""", au=unit, a=a, b=b)
def prepare_raw(shifts):
"""prepare_datapoints_frame: sigma0 (camera rows, without sensor_name) copied onto each sensor at its time;
relative = x - prop_x; error -> error_value; the full schema."""
if shifts.empty:
return pd.DataFrame(columns=COLUMNS)
sig = shifts.loc[shifts["sigma0"].notna(), ["time_utc", "sigma0"]]
sigma0 = sig.groupby("time_utc")["sigma0"].median() if not sig.empty else None
shifts = shifts.dropna(subset=["sensor_name"]).reset_index(drop=True)
shifts["sigma0"] = shifts["time_utc"].map(sigma0) if sigma0 is not None else pd.NA
shifts["relative_x"] = shifts["x"] - pd.to_numeric(shifts["prop_x"], errors="coerce")
shifts["relative_y"] = shifts["y"] - pd.to_numeric(shifts["prop_y"], errors="coerce")
shifts = shifts.drop(columns=["prop_x", "prop_y"]).rename(columns={"error": "error_value"})
shifts["time_utc"] = pd.to_datetime(shifts["time_utc"])
shifts["sensor_name"] = shifts["sensor_name"].astype(str)
for c in NUMERIC:
if c in shifts:
shifts[c] = pd.to_numeric(shifts[c], errors="coerce")
return shifts.reindex(columns=COLUMNS)
def aggregate_states(conn, unit, a, b):
"""fetch_aggregate_datapoints_frame: the per-minute medians of the state tables (shift_sensor_1min,
shift_common_1min), only source for "burst" devices, in the schema of the export (prepare_aggregate_datapoints_frame).
In the states, x_pct holds the CORRECTED x and rx_pct the raw x."""
thr = "COALESCE((th.value #>> '{}')::double precision, CAST(:thr AS double precision))"
sensor = query(conn, f"""
SELECT ss.time_utc, COALESCE(sn.value #>> '{{}}', s.name) AS name,
approx_percentile(0.5, ss.x_pct) AS corr_x, approx_percentile(0.5, ss.y_pct) AS corr_y,
approx_percentile(0.5, ss.rx_pct) AS x, approx_percentile(0.5, ss.ry_pct) AS y,
approx_percentile(0.5, ss.error_pct) AS error, {thr} AS error_threshold,
approx_percentile(0.5, ss.reproj_err_pct) AS reproj_err,
{", ".join(f"ss.{n}" for n in INLIER_BANDS)},
{", ".join(f"approx_percentile(0.5, ss.{m}_pct) AS {m}" for m in QUALITY)}
FROM shift_sensor_1min ss
JOIN sensor s ON s.id = ss.sensor_id
LEFT JOIN (SELECT p.sensor_id, any_value(p.value) AS value FROM property p JOIN datatype dt
ON dt.id = p.datatype_id WHERE dt.name = 'sensor_name' GROUP BY p.sensor_id) sn
ON sn.sensor_id = ss.sensor_id
LEFT JOIN (SELECT p.sensor_id, any_value(p.value) AS value FROM property p JOIN datatype dt
ON dt.id = p.datatype_id WHERE dt.name = 'error_threshold' GROUP BY p.sensor_id) th
ON th.sensor_id = ss.sensor_id
WHERE s.acquisition_unit_id = :au AND ss.time_utc >= :a AND ss.time_utc <= :b
ORDER BY ss.time_utc""", au=unit, a=a, b=b, thr=DEFAULT_ERROR_THRESHOLD)
if sensor.empty:
return pd.DataFrame(columns=COLUMNS)
common = query(conn, """
SELECT sc.time_utc, approx_percentile(0.5, sc.sigma0_pct) AS sigma0 FROM shift_common_1min sc
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b""", au=unit, a=a, b=b)
out = pd.DataFrame({"time_utc": pd.to_datetime(sensor["time_utc"]), "sensor_name": sensor["name"].astype(str)})
for col, src in [("error_value", "error"), ("error_threshold", "error_threshold"), ("corr_x", "corr_x"),
("corr_y", "corr_y"), ("relative_x", "corr_x"), ("relative_y", "corr_y"), ("x", "x"), ("y", "y"),
("reproj_err", "reproj_err")] + [(m, m) for m in QUALITY]:
out[col] = pd.to_numeric(sensor[src], errors="coerce")
# inlier_code_from_bands: the majority band, a tie goes to the worst; all at 0 -> NaN (no redundancy)
counts = sensor[list(INLIER_BANDS)].apply(pd.to_numeric, errors="coerce").fillna(0)
code = counts[list(INLIER_BANDS)[::-1]].idxmax(axis=1).map(INLIER_BANDS)
out["inlier"] = code.where(counts.sum(axis=1) > 0).to_numpy()
out["pattern_type"] = None
sig = common.loc[common["sigma0"].notna()]
out["sigma0"] = (out["time_utc"].map(sig.assign(time_utc=pd.to_datetime(sig["time_utc"]))
.groupby("time_utc")["sigma0"].median()) if not sig.empty else float("nan"))
return out.reindex(columns=COLUMNS)
def merge(raw, agg):
"""merge_raw_and_aggregate_frames: the raw wins; a state is kept only where its sensor has no raw row in the same
minute."""
if agg.empty:
return raw
if raw.empty:
return agg
def utc(t):
t = pd.to_datetime(t)
return t.dt.tz_localize("UTC") if t.dt.tz is None else t.dt.tz_convert("UTC")
keys = pd.MultiIndex.from_arrays([raw["sensor_name"], utc(raw["time_utc"]).dt.floor("1min")])
agg = agg.loc[~pd.MultiIndex.from_arrays([agg["sensor_name"], utc(agg["time_utc"]).dt.floor("1min")]).isin(keys)]
if agg.empty:
return raw
agg = agg.copy()
tz = pd.to_datetime(raw["time_utc"]).dt.tz
t = utc(agg["time_utc"])
agg["time_utc"] = t.dt.tz_convert(tz) if tz is not None else t.dt.tz_localize(None)
return (pd.concat([raw, agg], ignore_index=True).sort_values(["sensor_name", "time_utc"], kind="stable")
.reset_index(drop=True).reindex(columns=COLUMNS))
def advanced(conn, unit, a, b):
return merge(prepare_raw(raw_datapoints(conn, unit, a, b)), aggregate_states(conn, unit, a, b))
def camera(conn, unit, a, b):
"""Pan, tilt, scale, rotation and sigma0 of the camera, as median and bounds (median_ / min_ / max_) per minute,
hence one row per camera measurement (every 15 min at NORDEND, min = max): what the viewer plots
(fetch_cam_movements_frame), without its except, which would return a failed day as empty instead of leaving it
to run again."""
sigs = ", ".join(f"approx_percentile(0.5, sc.{s}_pct) AS median_{s}, min_val(sc.{s}_pct) AS min_{s}, "
f"max_val(sc.{s}_pct) AS max_{s}" for s in CAM_SIGNALS)
return query(conn, f"""SELECT sc.time_utc, {sigs} FROM shift_common_1min sc
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b
ORDER BY sc.time_utc""", au=unit, a=a, b=b)
DAILY_FILES = (("displacement", "advanced", advanced), ("camera", "camera", camera)) # folder, file, reader
def pump_displacements(conn, unit, station, start, end, out, replace):
"""Displacements and camera movements: one file and one query per day, bounded, under the timeout (--delay)."""
for k in range((end - start).days):
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
for folder, name, reader in DAILY_FILES:
path = out / station / folder / f"{station}_{name}_{a:%Y%m%d}.csv"
if path.exists() and not replace:
continue
t0 = time.time()
try:
df = reader(conn, unit, a, a + timedelta(days=1) - timedelta(microseconds=1))
except Exception as e: # ponytail: a failed day is skipped, a new run picks it up; no retry
print(f"{station} {a:%Y-%m-%d} {name}: failed ({root_cause(e)}), run again")
continue
finally:
conn.rollback() # closes the read transaction
if "sensor_name" in df:
df = df.drop_duplicates(["sensor_name", "time_utc"])
path.parent.mkdir(parents=True, exist_ok=True)
df.to_csv(path, index=False) # an empty day is written too: the next run does not ask for it again
print(f"{station} {a:%Y-%m-%d} {name}: {len(df)} rows, {time.time() - t0:.0f} s -> {path.name}")
def pump_images(conn, s3, unit, station, start, end, folder, every, out):
mt = conn.execute(text("SELECT id, extension FROM measurement_type WHERE name = :n"), {"n": TYPES[folder]}).first()
if mt is None:
print(f"{station}: unknown type {TYPES[folder]}")
return
client, bucket_name = s3
for k in range((end - start).days):
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
measurements = conn.execute(text(
"SELECT id, time_utc, measurement_uuid FROM measurement WHERE acquisition_unit_id = :au "
"AND measurement_type_id = :mt AND time_utc >= :a AND time_utc < :b ORDER BY time_utc"),
{"au": unit, "mt": mt.id, "a": a, "b": a + timedelta(days=1)}).all()
conn.rollback()
seen, n, nbytes, no_uuid, unreturned = set(), 0, 0, 0, 0
for m in measurements:
# empty measurement_uuid: the image never reached S3 (unfinished multipart upload); skipped, the slot
# takes the next image if there is one
if m.measurement_uuid is None:
no_uuid += 1
continue
t = m.time_utc.astimezone(UTC)
bucket = (t.hour * 60 + t.minute) // every if every else m.id # one image per slot of `every` min
path = out / station / folder / f"{t:%Y%m%dT%H%M}Z_{m.id}.{mt.extension}"
if bucket in seen:
continue
seen.add(bucket) # even if already there: otherwise the next run would take the next image of the slot
if path.exists():
continue
try:
data = client.get_object(Bucket=bucket_name, Key=m.measurement_uuid)["Body"].read()
except Exception: # like get_measurement_payload: S3 not answering, the image is counted as missing
data = None
if data:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
n, nbytes = n + 1, nbytes + len(data)
else:
unreturned += 1
# the detail says why a day gives 0: nothing in the database, never reached S3, or S3 not returning them
print(f"{station} {a:%Y-%m-%d} {folder}: {n} new images, {nbytes / 1e6:.0f} MB "
f"({len(measurements)} in database, {no_uuid} never reached S3, {unreturned} not returned by S3)")
time.sleep(0.5)
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("stations", nargs="*", help="serial number of the system, as in the URL of the System page; "
"none: those of stations.txt")
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD; without: the last pumped day of each "
"station, pumped again, or the last 30 days")
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD; without: up to now")
p.add_argument("--images", action="store_true", help="also the OPTICAL_SHIFT-IRON images")
p.add_argument("--diff", action="store_true", help="also the OPTICAL_SHIFT-DIFF images")
p.add_argument("--every", type=int, default=60, help="one image per slot of N minutes, 0 = all (default 60)")
p.add_argument("--delay", type=int, default=60, help="maximum duration of a query, in s (default 60: database "
"shared with production); longer only for a day that failed on a timeout")
p.add_argument("--out", type=Path, default=Path(__file__).resolve().parent.parent / "data")
a = p.parse_args()
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
end = min(datetime.strptime(a.end, "%Y-%m-%d").date(), tomorrow) if a.end else tomorrow # no day to come
stations = a.stations or STATIONS.read_text().split()
missing = [n for n in DB_NAMES if not os.environ.get(n)]
if missing:
sys.exit(f"Missing variables (names only): {', '.join(missing)}")
url = URL.create("postgresql+psycopg2", username=os.environ["AIMSIGHT_DATABASE_USER"],
password=os.environ["AIMSIGHT_DATABASE_PSWD"], host=os.environ["AIMSIGHT_DATABASE_TURL"],
port=int(os.environ.get("AIMSIGHT_DATABASE_PORT", "5432")),
database=os.environ["AIMSIGHT_DATABASE_NAME"])
engine = create_engine(url, connect_args={
"options": f"-c timezone=utc -c default_transaction_read_only=on -c statement_timeout={a.delay}s"})
s3 = None
if a.images or a.diff:
import boto3 # like the app's S3client (Garage)
s3 = (boto3.client("s3", endpoint_url=os.environ.get("AIMSIGHT_S3_URL", "http://s3:3900"),
aws_access_key_id=os.environ.get("AIMSIGHT_S3_ACCESS_KEY", ""),
aws_secret_access_key=os.environ.get("AIMSIGHT_S3_SECRET_KEY", ""),
region_name="garage", use_ssl=False),
os.environ.get("AIMSIGHT_S3_BUCKET", "measurements-assets"))
try:
with engine.connect() as conn:
for station in stations:
system_id = conn.execute(text("SELECT id FROM system WHERE serial_number = :s"), {"s": station}).scalar()
if system_id is None:
print(f"{station}: unknown system")
continue
ranges = [(start, end, False)] if start else update_ranges(a.out / station, end, tomorrow)
unit = acquisition_unit(conn, system_id)
conn.rollback()
for range_start, range_end, replace in ranges:
print(f"{station}: from {range_start} to {range_end - timedelta(days=1)} inclusive"
+ ("" if replace else ", missing days only"))
pump_metrics(conn, system_id, station, range_start, range_end, a.out, replace)
if unit is None:
print(f"{station}: no recognised acquisition unit, no displacements, camera or images")
continue
pump_displacements(conn, unit, station, range_start, range_end, a.out, replace)
for flag, folder in ((a.images, "images"), (a.diff, "images_diff")):
if flag:
pump_images(conn, s3, unit, station, range_start, range_end, folder, a.every, a.out)
finally:
engine.dispose()
if __name__ == "__main__":
main()
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
# Pumping under WSL. Reads the credentials from the .env at the root of the repository (never versioned),
# runs pump.py in a throwaway uv environment (4 packages), then the post-processing of LENZSPITZE.
# extraction/pump.sh update of the tracked stations, images included (the scheduler)
# extraction/pump.sh --from D --to E [SERIAL] any other call: the arguments go to pump.py as they are
# Output appended to out/pump.log; one pumping at a time (lock out/pump.lock).
set -u
export PATH="$HOME/.local/bin:$PATH" # uv, missing from the PATH when the Task Scheduler starts wsl.exe
here=$(cd "$(dirname "$0")" && pwd)
root=$(dirname "$here")
mkdir -p "$root/out"
exec 9>"$root/out/pump.lock"
flock -n 9 || { echo "A pumping is already running." >&2; exit 1; }
[ $# -eq 0 ] && set -- --images --delay 300
{
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) pump.py $*"
(set -a +u && . "$root/.env" && set +a -u && # +u: an unquoted $ in .env does not stop the load
uv run --no-project --quiet --python 3.12 \
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
python "$here/pump.py" "$@")
[ -f "$root/data/LENZSPITZE/lentzpitze_patterns.json" ] && python3 "$here/lenzspitze_absolute.py"
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end"
} 2>&1 | tee -a "$root/out/pump.log"