English: pump.py and pump.sh (ex pomper), lenzspitze_absolute.py, options --from/--to/--delay, out/pump.log; .env loaded with +u (pump.sh -h works)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
260cd5e522
commit
5ed9f5a97a
9 files changed
+531
-530
No files matched your search
Vendored
+11
-11
@@ -2,35 +2,35 @@
|
||||
"version": "2.0.0",
|
||||
"tasks": [
|
||||
{
|
||||
"label": "pomper",
|
||||
"detail": "Met à jour les stations suivies (extraction/stations.txt), images comprises ; extraction/README.md",
|
||||
"label": "pump",
|
||||
"detail": "Updates the tracked stations (extraction/stations.txt), images included; extraction/README.md",
|
||||
"type": "shell",
|
||||
"command": "extraction/pomper.sh",
|
||||
"command": "extraction/pump.sh",
|
||||
"problemMatcher": []
|
||||
},
|
||||
{
|
||||
"label": "pomper une plage",
|
||||
"detail": "Pompe une plage de jours UTC (--au exclu) pour une ou plusieurs stations, images comprises",
|
||||
"label": "pump a range",
|
||||
"detail": "Pumps a range of UTC days (--to excluded) for one or more stations, images included",
|
||||
"type": "shell",
|
||||
"command": "extraction/pomper.sh --du ${input:du} --au ${input:au} --images --delai 300 ${input:stations}",
|
||||
"command": "extraction/pump.sh --from ${input:from} --to ${input:to} --images --delay 300 ${input:stations}",
|
||||
"problemMatcher": []
|
||||
}
|
||||
],
|
||||
"inputs": [
|
||||
{
|
||||
"id": "du",
|
||||
"id": "from",
|
||||
"type": "promptString",
|
||||
"description": "Premier jour UTC (AAAA-MM-JJ) ; 3 jours avant la période voulue, pour l'historique des détecteurs"
|
||||
"description": "First UTC day (YYYY-MM-DD); 3 days before the wanted period, for the history of the detectors"
|
||||
},
|
||||
{
|
||||
"id": "au",
|
||||
"id": "to",
|
||||
"type": "promptString",
|
||||
"description": "Dernier jour UTC, exclu (AAAA-MM-JJ)"
|
||||
"description": "Last UTC day, excluded (YYYY-MM-DD)"
|
||||
},
|
||||
{
|
||||
"id": "stations",
|
||||
"type": "promptString",
|
||||
"description": "Numéros de série des stations, séparés par des espaces (ex. NORDEND)"
|
||||
"description": "Serial numbers of the stations, separated by spaces (e.g. NORDEND)"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,13 +1,13 @@
|
||||
# aimsight-data
|
||||
|
||||
Les données des stations AIMsight, en un seul exemplaire sur le poste, et leur pompage.
|
||||
The data of the AIMsight stations, in a single copy on the workstation, and their pumping.
|
||||
|
||||
- `data/<SERIE>/` : rempli par le pompage, **hors de git** (`metrics/`, `displacement/`, `camera/`, `images/`…). Un dossier perdu se repompe.
|
||||
- `extraction/` : le pompage, en lecture seule sur la plateforme ; mode d'emploi `extraction/README.md`. Tâches VS Code « pomper » et « pomper une plage ».
|
||||
- `.env` : les identifiants de la base et de S3, à la racine, jamais versionné.
|
||||
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`…). A lost folder is pumped again.
|
||||
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range".
|
||||
- `.env`: the credentials of the database and of S3, at the root, never versioned.
|
||||
|
||||
Les autres dépôts (Diagnostic-App, Optimizer) lisent les données par un lien `data -> ~/aimsight-data/data`, ignoré par git. On ne pompe que d'ici : le verrou `out/pompage.lock` vaut pour ce seul dossier.
|
||||
The other repositories (Diagnostic-App, Optimizer) read the data through a link `data -> ~/aimsight-data/data`, ignored by git. Pumping is done from here only: the lock `out/pump.lock` holds for this folder alone.
|
||||
|
||||
Les étiquettes sont dans le dépôt voisin `~/labeling`.
|
||||
The labels are in the neighbouring repository `~/labeling`.
|
||||
|
||||
Extrait de Diagnostic-App le 2026-11-13 (commit 462ba9e) ; l'historique antérieur du pompage est dans Support-Agent.
|
||||
Extracted from Diagnostic-App on 2026-11-13 (commit 462ba9e); the earlier history of the pumping is in Support-Agent.
|
||||
+30
-30
@@ -1,51 +1,51 @@
|
||||
# extraction/ : pomper les données de la plateforme (lecture seule)
|
||||
# extraction/: pumping the data of the platform (read-only)
|
||||
|
||||
`pomper.py` récupère, pour chaque station (numéro de série du système) et chaque jour UTC demandé, dans `data/<SERIE>/` :
|
||||
- les **métriques système** (énergie, disque, 4G, températures…), moyennes horaires, un CSV par métrique et par jour dans `metrics/` ;
|
||||
- les **déplacements** (export « advanced »), un CSV par jour dans `displacement/` ;
|
||||
- les **mouvements de la caméra** (`pan`, `tilt`, `scale`, `rotation`, `sigma0`), en médiane et bornes (`median_`, `min_`, `max_`) par minute, aux seuls instants où la caméra a été recalée, un CSV par jour dans `camera/` ;
|
||||
- les **images** IRON (`--images`, une par heure par défaut, `--every 10` ou `--every 0` pour plus) dans `images/`, et différentielles (`--diff`) dans `images_diff/`.
|
||||
`pump.py` fetches, for each station (serial number of the system) and each UTC day asked, into `data/<SERIAL>/`:
|
||||
- the **system metrics** (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in `metrics/`;
|
||||
- the **displacements** ("advanced" export), one CSV per day in `displacement/`;
|
||||
- the **camera movements** (`pan`, `tilt`, `scale`, `rotation`, `sigma0`), as median and bounds (`median_`, `min_`, `max_`) per minute, only at the times the camera was realigned, one CSV per day in `camera/`;
|
||||
- the IRON **images** (`--images`, one per hour by default, `--every 10` or `--every 0` for more) in `images/`, and the differential ones (`--diff`) in `images_diff/`.
|
||||
|
||||
Formats : `~/labeling/connaissances/colonnes-des-deplacements.md`.
|
||||
Formats: `~/labeling/knowledge/displacement-columns.md`.
|
||||
|
||||
**Stations suivies** : `stations.txt`, un numéro de série par ligne. Sans station ni dates, le script les met toutes à jour jusqu'à maintenant : chaque station comble d'abord ses jours manquants entre le premier et le dernier jour pompé (sans déplacements, ou sans aucune métrique), puis repart de son dernier jour, repompé (il était peut-être incomplet) ; une station sans dossier dans `data/` reçoit les 30 derniers jours.
|
||||
**Tracked stations**: `stations.txt`, one serial number per line. Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in `data/` gets the last 30 days.
|
||||
|
||||
Reprise : avec `--du`, un fichier déjà dans `data/<SERIE>/` n'est pas redemandé ; relancer sur une plage plus large ne lit que les jours et les images qui manquent. Le jour en cours est pompé tel qu'il est, ses métriques jusqu'à la dernière heure terminée ; le lancement suivant sans `--du` le remplace.
|
||||
Resuming: with `--from`, a file already in `data/<SERIAL>/` is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without `--from` replaces it.
|
||||
|
||||
## Lancer (sous WSL)
|
||||
## Running (under WSL)
|
||||
|
||||
`pomper.sh` lit les identifiants dans le `.env` à la racine du dépôt (base PostgreSQL et S3 ; jamais versionné, interdit en lecture à Claude), installe les 4 paquets nécessaires (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) dans un environnement `uv` jetable (Python 3.12), lance `pomper.py`, puis `lenzspitze_absolu.py`. Sa sortie s'ajoute à `out/pompage.log` ; un second lancement pendant un pompage s'arrête aussitôt (verrou `out/pompage.lock`). Le verrou ne vaut que pour ce dossier : les autres dépôts lisent `data/` par un lien, ils ne pompent pas.
|
||||
`pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `lenzspitze_absolute.py`. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump.
|
||||
|
||||
Variables du `.env` : `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT` ; pour les images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`.
|
||||
Variables of the `.env`: `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT`; for the images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`.
|
||||
|
||||
1. **Un essai** (1 station, 2 jours, dans un dossier jetable) : `extraction/pomper.sh --du 2026-11-09 --au 2026-11-11 --images --out /tmp/essai-pompage NORDEND` (6 s le 2026-11-12). `lenzspitze_absolu.py` convertit ensuite `data/LENZSPITZE`, pas le dossier `--out` : sans effet si `data/LENZSPITZE` est déjà converti.
|
||||
2. **Mettre à jour les stations suivies** : `extraction/pomper.sh` (sans argument : `--images --delai 300`), ou la tâche VS Code **pomper**. Le script dit, pour chaque station, la plage qu'il pompe.
|
||||
3. **Une plage choisie** (historique, jour en échec), avec 3 jours de plus avant (l'historique des détecteurs) : `extraction/pomper.sh --du 2026-10-02 --au 2026-10-26 --images <SERIE>` (`--au` exclu ; plusieurs stations : `<SERIE1> <SERIE2>`). Aide : `extraction/pomper.sh -h`.
|
||||
1. **A trial** (1 station, 2 days, in a throwaway folder): `extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND` (6 s on 2026-11-12). `lenzspitze_absolute.py` then converts `data/LENZSPITZE`, not the `--out` folder: no effect if `data/LENZSPITZE` is already converted.
|
||||
2. **Updating the tracked stations**: `extraction/pump.sh` (without argument: `--images --delay 300`), or the VS Code task **pump**. The script says, for each station, the range it pumps.
|
||||
3. **A chosen range** (history, failed day), with 3 more days before (the history of the detectors): `extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images <SERIAL>` (`--to` excluded; several stations: `<SERIAL1> <SERIAL2>`). Help: `extraction/pump.sh -h`.
|
||||
|
||||
**Chaque nuit, automatiquement** : une tâche du Planificateur de tâches de Windows, à créer une fois dans PowerShell (pas en administrateur). Elle démarre WSL au besoin et, PC éteint ou en veille à l'heure dite, part dès qu'il se rallume ; un jour manqué est rattrapé au lancement suivant.
|
||||
**Every night, automatically**: a task of the Windows Task Scheduler, to create once in PowerShell (not as administrator). It starts WSL if needed and, PC off or asleep at the set time, runs as soon as it wakes up; a missed day is caught up at the next run.
|
||||
|
||||
```powershell
|
||||
$a = New-ScheduledTaskAction -Execute "wsl.exe" -Argument "-d Ubuntu -e bash -lc /home/maxime/aimsight-data/extraction/pomper.sh"
|
||||
$a = New-ScheduledTaskAction -Execute "wsl.exe" -Argument "-d Ubuntu -e bash -lc /home/maxime/aimsight-data/extraction/pump.sh"
|
||||
$t = New-ScheduledTaskTrigger -Daily -At 04:00
|
||||
$s = New-ScheduledTaskSettingsSet -StartWhenAvailable -ExecutionTimeLimit (New-TimeSpan -Hours 6)
|
||||
Register-ScheduledTask -TaskName "AIMsight pompage" -Action $a -Trigger $t -Settings $s
|
||||
Register-ScheduledTask -TaskName "AIMsight pump" -Action $a -Trigger $t -Settings $s
|
||||
```
|
||||
|
||||
Essayer la tâche tout de suite : `Start-ScheduledTask "AIMsight pompage"`, puis lire `out/pompage.log`. La retirer : `Unregister-ScheduledTask "AIMsight pompage"`.
|
||||
Trying the task right away: `Start-ScheduledTask "AIMsight pump"`, then read `out/pump.log`. Removing it: `Unregister-ScheduledTask "AIMsight pump"`.
|
||||
|
||||
Lecture seule : moteur dédié, `default_transaction_read_only`, délai de 60 s par requête, une requête par jour ; hors des heures chargées (base partagée avec la production).
|
||||
Read-only: dedicated engine, `default_transaction_read_only`, 60 s timeout per query, one query per day; outside busy hours (database shared with production).
|
||||
|
||||
Sans dépendance à l'appli `monitoring-dashboard` : ses lectures sont recopiées dans `pomper.py`, en SQL (sa docstring dit lesquelles, et de quel commit). Si l'appli change ses lectures, le script ne suit pas : comparer à nouveau des fichiers pompés aux deux.
|
||||
No dependency on the `monitoring-dashboard` app: its reads are copied into `pump.py`, in SQL (its docstring says which ones, and from which commit). If the app changes its reads, the script does not follow: compare again files pumped by both.
|
||||
|
||||
## Avec Claude
|
||||
## With Claude
|
||||
|
||||
Claude peut lancer `pomper.sh` lui-même (accord de l'utilisateur, 2026-11-12) : lecture seule, petites plages ; une longue plage ou un historique, hors des heures chargées. Il ne lit jamais le `.env`.
|
||||
Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small ranges; a long range or a history, outside busy hours. It never reads the `.env`.
|
||||
|
||||
## À savoir
|
||||
## Good to know
|
||||
|
||||
- `<SERIE>` est le numéro de série du **système**, celui de l'URL de la page System (`/explore/system/<SERIE>`) et du début des CSV qu'on y télécharge (`NORDEND_BusVoltage_…` : `NORDEND`) ; pas le nom de l'appareil de la page Explore.
|
||||
- `data/` est hors de git : un dossier perdu se repompe.
|
||||
- Un jour en échec par délai dépassé (`QueryCanceled: … statement timeout`) : le relancer seul avec `--delai 300`. La lenteur vient de la base, pas du volume.
|
||||
- Une heure sans image peut avoir sa mesure : l'image n'est jamais arrivée sur S3 (`measurement_uuid` vide, envoi inachevé). Le script la saute et prend l'image suivante de la tranche s'il y en a une.
|
||||
- Un jour passé, pompé avant que la station ait fini d'envoyer ses données (coupure, synchronisation en retard), reste incomplet : supprimer ses fichiers dans `data/<SERIE>/` et relancer.
|
||||
- LENZSPITZE : ses exports ont `x` / `y` relatifs et `pattern_type` vide ; `pomper.sh` lance ensuite `lenzspitze_absolu.py` (provisoire ; détail dans `~/labeling/connaissances/colonnes-des-deplacements.md`).
|
||||
- `<SERIAL>` is the serial number of the **system**, the one in the URL of the System page (`/explore/system/<SERIAL>`) and at the start of the CSVs downloaded there (`NORDEND_BusVoltage_…`: `NORDEND`); not the name of the device on the Explore page.
|
||||
- `data/` is out of git: a lost folder is pumped again.
|
||||
- A day failed on a timeout (`QueryCanceled: … statement timeout`): run it again alone with `--delay 300`. The slowness comes from the database, not from the volume.
|
||||
- An hour without an image can have its measurement: the image never reached S3 (empty `measurement_uuid`, unfinished upload). The script skips it and takes the next image of the slot if there is one.
|
||||
- A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in `data/<SERIAL>/` and run again.
|
||||
- LENZSPITZE: its exports have relative `x` / `y` and an empty `pattern_type`; `pump.sh` then runs `lenzspitze_absolute.py` (provisional; details in `~/labeling/knowledge/displacement-columns.md`).
|
||||
@@ -1,49 +0,0 @@
|
||||
"""Provisoire, LENZSPITZE seulement : x / y des exports advanced en position absolue dans l'image, comme sur les autres stations.
|
||||
|
||||
Sur LENZSPITZE, x / y sont relatifs à la position de référence du motif (médianes ~0 px). Ce script réécrit en place
|
||||
data/LENZSPITZE/displacement/*.csv :
|
||||
x = x_motif + DX + x_csv, y = y_motif + DY + y_csv, où (x_motif, y_motif) est le centre du motif dans
|
||||
data/LENZSPITZE/lentzpitze_patterns.json et (DX, DY) le décalage entre le repère du JSON et les images IRON ;
|
||||
pattern_type, vide dans ces exports, est rempli depuis le JSON.
|
||||
Les capteurs absents du JSON (1023, 8019) : x / y vidés (position inconnue) ; 8019 reçoit global_ref (il a des reproj_err).
|
||||
Un fichier dont pattern_type est déjà rempli est laissé tel quel (déjà converti). Un repompage rend les originaux.
|
||||
|
||||
Usage : python3 extraction/lenzspitze_absolu.py
|
||||
"""
|
||||
import csv
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
|
||||
STATION = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "data", "LENZSPITZE")
|
||||
# Mesuré le 2026-11-04 : médiane (tache − motif du JSON) sur 9 images de nuit du 17.10 au 03.11, 25 à 34 cibles appariées
|
||||
# chacune, stable à 0,5 px près. Comprend l'écart constant tache − (x, y) déjà vu sur NORDEND (colonnes-des-deplacements.md).
|
||||
DX, DY = 330.5, 15.8
|
||||
TYPE_SANS_MOTIF = {"8019": "global_ref"} # reproj_err et inlier remplis : référence de la compensation
|
||||
|
||||
motifs = {str(v["sensor_name"]): v for v in json.load(open(os.path.join(STATION, "lentzpitze_patterns.json"))).values()}
|
||||
|
||||
for chemin in sorted(glob.glob(os.path.join(STATION, "displacement", "*.csv"))):
|
||||
with open(chemin, newline="") as f:
|
||||
lecteur = csv.DictReader(f)
|
||||
champs, lignes = lecteur.fieldnames, list(lecteur)
|
||||
if any(r["pattern_type"] for r in lignes):
|
||||
print(f"{os.path.basename(chemin)} : déjà converti")
|
||||
continue
|
||||
for r in lignes:
|
||||
s = r["sensor_name"].strip()
|
||||
m = motifs.get(s)
|
||||
if m is None:
|
||||
r["x"] = r["y"] = ""
|
||||
r["pattern_type"] = TYPE_SANS_MOTIF.get(s, "")
|
||||
continue
|
||||
r["pattern_type"] = m["pattern_type"]
|
||||
if r["x"]:
|
||||
r["x"] = repr(m["x"] + DX + float(r["x"]))
|
||||
if r["y"]:
|
||||
r["y"] = repr(m["y"] + DY + float(r["y"]))
|
||||
with open(chemin, "w", newline="") as f:
|
||||
w = csv.DictWriter(f, fieldnames=champs)
|
||||
w.writeheader()
|
||||
w.writerows(lignes)
|
||||
print(f"{os.path.basename(chemin)} : {len(lignes)} lignes converties")
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Provisional, LENZSPITZE only: x / y of the advanced exports as absolute positions in the image, as on the other stations.
|
||||
|
||||
On LENZSPITZE, x / y are relative to the reference position of the pattern (medians ~0 px). This script rewrites in place
|
||||
data/LENZSPITZE/displacement/*.csv:
|
||||
x = x_pattern + DX + x_csv, y = y_pattern + DY + y_csv, where (x_pattern, y_pattern) is the centre of the pattern in
|
||||
data/LENZSPITZE/lentzpitze_patterns.json and (DX, DY) the offset between the frame of the JSON and the IRON images;
|
||||
pattern_type, empty in these exports, is filled from the JSON.
|
||||
The sensors missing from the JSON (1023, 8019): x / y emptied (position unknown); 8019 gets global_ref (it has reproj_err).
|
||||
A file whose pattern_type is already filled is left as it is (already converted). Pumping again restores the originals.
|
||||
|
||||
Usage: python3 extraction/lenzspitze_absolute.py
|
||||
"""
|
||||
import csv
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
|
||||
STATION = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "data", "LENZSPITZE")
|
||||
# Measured on 2026-11-04: median (spot − pattern of the JSON) over 9 night images from 17.10 to 03.11, 25 to 34 matched
|
||||
# targets each, stable within 0.5 px. Includes the constant offset spot − (x, y) already seen on NORDEND (displacement-columns.md).
|
||||
DX, DY = 330.5, 15.8
|
||||
TYPE_WITHOUT_PATTERN = {"8019": "global_ref"} # reproj_err and inlier filled: reference of the compensation
|
||||
|
||||
patterns = {str(v["sensor_name"]): v for v in json.load(open(os.path.join(STATION, "lentzpitze_patterns.json"))).values()}
|
||||
|
||||
for path in sorted(glob.glob(os.path.join(STATION, "displacement", "*.csv"))):
|
||||
with open(path, newline="") as f:
|
||||
reader = csv.DictReader(f)
|
||||
fields, rows = reader.fieldnames, list(reader)
|
||||
if any(r["pattern_type"] for r in rows):
|
||||
print(f"{os.path.basename(path)}: already converted")
|
||||
continue
|
||||
for r in rows:
|
||||
s = r["sensor_name"].strip()
|
||||
m = patterns.get(s)
|
||||
if m is None:
|
||||
r["x"] = r["y"] = ""
|
||||
r["pattern_type"] = TYPE_WITHOUT_PATTERN.get(s, "")
|
||||
continue
|
||||
r["pattern_type"] = m["pattern_type"]
|
||||
if r["x"]:
|
||||
r["x"] = repr(m["x"] + DX + float(r["x"]))
|
||||
if r["y"]:
|
||||
r["y"] = repr(m["y"] + DY + float(r["y"]))
|
||||
with open(path, "w", newline="") as f:
|
||||
w = csv.DictWriter(f, fieldnames=fields)
|
||||
w.writeheader()
|
||||
w.writerows(rows)
|
||||
print(f"{os.path.basename(path)}: {len(rows)} rows converted")
|
||||
@@ -1,408 +0,0 @@
|
||||
"""Pompe les métriques système, les déplacements (export « advanced ») et les images de stations de la plateforme
|
||||
AIMsight, dans le format que lisent l'app, les outils de lecture et les détecteurs. LECTURE SEULE. Essayé le
|
||||
2026-10-26 (NORDEND, 2 jours) ; autonome depuis le 2026-11-12.
|
||||
|
||||
À lancer par extraction/pomper.sh, sous WSL, depuis n'importe où (voir README.md) :
|
||||
|
||||
extraction/pomper.sh
|
||||
extraction/pomper.sh --du 2026-08-01 --au 2026-08-08 --images WEISSHORN
|
||||
|
||||
Sans station : celles de `stations.txt`. Sans `--du` : chaque station comble ses jours manquants (déplacements ou
|
||||
métriques) avant le dernier pompé, puis repart de ce dernier jour, repompé (il était peut-être incomplet), ou des 30
|
||||
derniers jours si elle n'en a pas ; sans `--au` : jusqu'à maintenant.
|
||||
|
||||
Sortie (défaut : le `data/` du dépôt, à côté de ce dossier) :
|
||||
<STATION>/metrics/<STATION>_<métrique>_<AAAA_MM_JJ>_00_00 to <AAAA_MM_JJ>_23_00.csv métriques système, par jour
|
||||
<STATION>/displacement/<STATION>_advanced_<AAAAMMJJ>.csv déplacements, un fichier par jour
|
||||
<STATION>/camera/<STATION>_camera_<AAAAMMJJ>.csv mouvements de la caméra, par jour
|
||||
<STATION>/images/<AAAAMMJJ>T<HHMM>Z_<id>.<ext> images OPTICAL_SHIFT-IRON (--images)
|
||||
<STATION>/images_diff/… images différentielles (--diff)
|
||||
Reprenable : avec `--du`, un fichier déjà écrit n'est pas relu, quelles que soient les dates demandées ; sans `--du`,
|
||||
les jours de la plage sont remplacés (images exceptées). Jours en UTC, `--au` exclu ; le jour en cours est pompé tel
|
||||
qu'il est, ses métriques jusqu'à la dernière heure terminée. Pour reprendre un jour, supprimer ses fichiers.
|
||||
|
||||
Sans dépendance à l'appli (~/monitoring-dashboard) : ses lectures sont recopiées ici, en SQL, telles quelles (branche
|
||||
`explo`, commit 3dce73d du 2026-11-12) : la requête de System.get_metrics (le téléchargement de la page System) ;
|
||||
build_advanced_frame (app/datapoints.py, ce que tracent l'onglet Advanced et l'export parquet), avec
|
||||
AcquisitionUnit.get_datapoints et AggregateEngine.get_sensor_aggregate / get_common_aggregate ; les mouvements de la
|
||||
caméra de fetch_cam_movements_frame (AggregateEngine.get_cam_scalar_aggregate, ce que trace le visualiseur) ;
|
||||
AcquisitionUnit.measurements et get_measurement_payload (S3). Si l'appli change ses lectures, ce fichier ne suit pas.
|
||||
Connexion en lecture seule (`default_transaction_read_only`), délai de 60 s par requête (`--delai`). Variables lues
|
||||
(noms seulement, aucune n'est affichée) : AIMSIGHT_DATABASE_USER / _PSWD / _TURL / _NAME / _PORT, et pour les images
|
||||
AIMSIGHT_S3_URL / _ACCESS_KEY / _SECRET_KEY / _BUCKET.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.engine import URL
|
||||
|
||||
UTC = timezone.utc
|
||||
STATIONS = Path(__file__).resolve().parent / "stations.txt" # les stations suivies, une par ligne
|
||||
JOURS_NOUVELLE = 30 # sans --du, une station sans déplacements pompés : les 30 derniers jours
|
||||
DB_NAMES = ["AIMSIGHT_DATABASE_USER", "AIMSIGHT_DATABASE_PSWD", "AIMSIGHT_DATABASE_TURL", "AIMSIGHT_DATABASE_NAME"]
|
||||
TYPES = {"images": "OPTICAL_SHIFT-IRON", "images_diff": "OPTICAL_SHIFT-DIFF"} # dossier -> type de mesure
|
||||
|
||||
# --- constantes de l'appli (app/datapoints.py, app/aggregates.py) ---
|
||||
QUALITY = ["peak", "flux", "sharpness", "fwhm", "dominance", "snr"]
|
||||
CAM_SIGNALS = ["pan", "tilt", "scale", "rotation", "sigma0"]
|
||||
DEFAULT_ERROR_THRESHOLD = 0.7
|
||||
DATATYPES = ["x", "y", "error", "corr_x", "corr_y", "reproj_err", "inlier", "sigma0"]
|
||||
PROPERTIES = ["x", "y", "pattern_type", "sensor_name", "error_threshold"]
|
||||
COLUMNS = ["time_utc", "sensor_name", "x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x",
|
||||
"corr_y", "reproj_err", "inlier", "sigma0", "pattern_type"] + QUALITY
|
||||
NUMERIC = ["x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x", "corr_y", "reproj_err",
|
||||
"inlier", "sigma0"] + QUALITY
|
||||
# bandes d'inlier de shift_sensor_1min -> code ; les deux derniers NOMS sont faux (-1 rejeté, -2 interpolé), les codes
|
||||
# sont justes ; ordre du meilleur au pire
|
||||
INLIER_BANDS = {"inlier_in_fit": 1.0, "inlier_out_fit": 0.0, "inlier_no_redundancy": -1.0, "inlier_rejected": -2.0}
|
||||
|
||||
|
||||
def raison(e):
|
||||
"""L'erreur d'origine (celle de la base) : pandas et SQLAlchemy l'enveloppent, la requête SQL d'abord."""
|
||||
while e.__cause__ is not None:
|
||||
e = e.__cause__
|
||||
return f"{type(e).__name__}: {(str(e).splitlines() or [''])[0][:300]}"
|
||||
|
||||
|
||||
def lire(conn, sql, **params):
|
||||
return pd.read_sql(text(sql), conn, params=params)
|
||||
|
||||
|
||||
def acquisition_unit(conn, system_id):
|
||||
"""Même choix que Device.get_acquisition_unit (app/base/models.py:268) : l'unique unité, sinon MER* / ME2S*."""
|
||||
units = conn.execute(text("SELECT id, product_number FROM acquisition_unit WHERE system_id = :s"),
|
||||
{"s": system_id}).all()
|
||||
pick = units if len(units) == 1 else [u for u in units if u.product_number.startswith(("MER", "ME2S"))]
|
||||
return pick[0].id if pick else None
|
||||
|
||||
|
||||
def plages_suivi(dossier, fin, demain):
|
||||
"""Sans --du : (début, fin, remplacer) à pomper, d'après les jours de déplacements déjà pompés. Les jours manquants
|
||||
entre le premier et le dernier (plages pompées à part, jour en échec) sont comblés sans toucher aux autres ; le
|
||||
dernier jour est repompé (pompé en cours de journée, il était incomplet) jusqu'à maintenant ; une station sans
|
||||
déplacements pompés reçoit les 30 derniers jours."""
|
||||
jours = {datetime.strptime(p.stem.rsplit("_", 1)[1], "%Y%m%d").date()
|
||||
for p in (dossier / "displacement").glob(f"{dossier.name}_advanced_*.csv")}
|
||||
if not jours:
|
||||
return [(demain - timedelta(days=JOURS_NOUVELLE + 1), fin, True)]
|
||||
# un jour de métriques en échec n'a aucun fichier (une requête pour toutes) : c'est aussi un trou ; une station
|
||||
# sans aucune métrique n'en a jamais, ses jours n'en sont pas
|
||||
metriques = {datetime.strptime(p.name.split(" to ")[0][-16:-6], "%Y_%m_%d").date()
|
||||
for p in (dossier / "metrics").glob(f"{dossier.name}_*_00_00 to *.csv")}
|
||||
premier, dernier = min(jours), max(jours)
|
||||
trous = [premier + timedelta(days=k) for k in range((dernier - premier).days)
|
||||
if premier + timedelta(days=k) not in jours or (metriques and premier + timedelta(days=k) not in metriques)]
|
||||
return ([(trous[0], dernier, False)] if trous else []) + [(dernier, fin, True)]
|
||||
|
||||
|
||||
def pomper_metriques(conn, system_id, station, debut, fin, out, remplacer):
|
||||
"""Comme le téléchargement de la page System (Dashboard/System.py, get_metrics(..., bucketing="1 hour")) : par
|
||||
heure, moyenne * transform_factor + transform_offset, ou la liste des valeurs d'une métrique texte ; les séries à
|
||||
labels d'une même métrique confondues ; une ligne vide pour une heure sans mesure. get_metrics compte en heures
|
||||
avant maintenant : sa requête est refaite ici sur un jour fixe, une requête par jour pour toutes les métriques."""
|
||||
noms = sorted(conn.execute(text("SELECT DISTINCT dt.name FROM datatype dt JOIN metric_series se "
|
||||
"ON se.datatype_id = dt.id WHERE se.system_id = :s"), {"s": system_id}).scalars())
|
||||
conn.rollback()
|
||||
for k in range((fin - debut).days):
|
||||
a = datetime.combine(debut + timedelta(days=k), datetime.min.time(), UTC)
|
||||
paths = {n: out / station / "metrics"
|
||||
/ f"{station}_{n.replace(os.sep, '-')}_{a:%Y_%m_%d}_00_00 to {a:%Y_%m_%d}_23_00.csv" for n in noms}
|
||||
if not remplacer and all(p.exists() for p in paths.values()):
|
||||
continue
|
||||
t0 = time.time()
|
||||
try:
|
||||
# date_trunc('hour') : même découpage que time_bucket('1 hour')
|
||||
df = lire(conn, """
|
||||
SELECT date_trunc('hour', ms.time_utc) AS heure, dt.name AS nom,
|
||||
avg(ms.value_num) * any_value(dt.transform_factor) + any_value(dt.transform_offset) AS num,
|
||||
jsonb_agg(ms.value_text) FILTER (WHERE ms.value_text IS NOT NULL) AS txt
|
||||
FROM metric_sample ms
|
||||
JOIN metric_series se ON ms.series_id = se.id
|
||||
JOIN datatype dt ON se.datatype_id = dt.id
|
||||
WHERE se.system_id = :s AND ms.time_utc >= :a AND ms.time_utc < :b
|
||||
GROUP BY 1, dt.name""", s=system_id, a=a, b=a + timedelta(days=1))
|
||||
except Exception as e: # ponytail : comme les déplacements, un jour en échec est sauté, relancer le reprend
|
||||
print(f"{station} {a:%Y-%m-%d} métriques : échec ({raison(e)}), à relancer")
|
||||
continue
|
||||
finally:
|
||||
conn.rollback()
|
||||
df["heure"] = pd.to_datetime(df["heure"], utc=True)
|
||||
heures = pd.to_datetime(pd.date_range(a, periods=24, freq="h"), utc=True) # même fuseau que df, pour reindex
|
||||
# jour en cours : les heures terminées seulement, sinon les heures à venir seraient lues comme un trou
|
||||
heures = heures[heures + pd.Timedelta(hours=1) <= pd.Timestamp.now(tz=UTC)]
|
||||
for n, path in paths.items():
|
||||
d = df[df["nom"] == n].set_index("heure")
|
||||
col = d["num"] if d["num"].notna().any() else d["txt"] # métrique texte : la liste de l'heure
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
# CSV de la page System : ',time_utc,metric', l'index en première colonne ; un jour vide est écrit aussi
|
||||
pd.DataFrame({"time_utc": heures, "metric": col.reindex(heures).to_numpy()}).to_csv(path)
|
||||
print(f"{station} {a:%Y-%m-%d} métriques : {len(noms)} fichiers, {df['nom'].nunique()} avec mesure, "
|
||||
f"{time.time() - t0:.0f} s")
|
||||
|
||||
|
||||
# --- l'export « advanced » : build_advanced_frame (app/datapoints.py) ---
|
||||
|
||||
def datapoints_bruts(conn, unit, a, b):
|
||||
"""AcquisitionUnit.get_datapoints : les datapoints de l'unité, un type par colonne, les propriétés des capteurs
|
||||
à côté (prop_x / prop_y : la position de référence, homonyme d'un type)."""
|
||||
types = ",\n".join(f"max(d.value) FILTER (WHERE dt.name = '{t}') AS {t}" for t in DATATYPES)
|
||||
noms = {p: f"prop_{p}" if p in DATATYPES else p for p in PROPERTIES}
|
||||
props = ",\n".join(f"any_value(p_{p}.value) AS {noms[p]}" for p in PROPERTIES)
|
||||
joints = "\n".join(f"LEFT JOIN (SELECT p.sensor_id, p.value FROM property p JOIN datatype pdt "
|
||||
f"ON p.datatype_id = pdt.id WHERE pdt.name = '{p}') p_{p} ON d.sensor_id = p_{p}.sensor_id"
|
||||
for p in PROPERTIES)
|
||||
return lire(conn, f"""
|
||||
SELECT d.time_utc, d.sensor_id, any_value(d.measurement_id) AS measurement_id,
|
||||
{types},
|
||||
{props}
|
||||
FROM datapoint d
|
||||
JOIN sensor s ON d.sensor_id = s.id
|
||||
JOIN datatype dt ON d.datatype_id = dt.id
|
||||
{joints}
|
||||
WHERE s.acquisition_unit_id = :au AND d.time_utc >= :a AND d.time_utc <= :b
|
||||
GROUP BY d.time_utc, d.sensor_id
|
||||
ORDER BY d.sensor_id, d.time_utc""", au=unit, a=a, b=b)
|
||||
|
||||
|
||||
def preparer_bruts(shifts):
|
||||
"""prepare_datapoints_frame : sigma0 (lignes de la caméra, sans sensor_name) recopié sur chaque capteur à son
|
||||
instant ; relative = x - prop_x ; error -> error_value ; le schéma complet."""
|
||||
if shifts.empty:
|
||||
return pd.DataFrame(columns=COLUMNS)
|
||||
sig = shifts.loc[shifts["sigma0"].notna(), ["time_utc", "sigma0"]]
|
||||
sigma0 = sig.groupby("time_utc")["sigma0"].median() if not sig.empty else None
|
||||
shifts = shifts.dropna(subset=["sensor_name"]).reset_index(drop=True)
|
||||
shifts["sigma0"] = shifts["time_utc"].map(sigma0) if sigma0 is not None else pd.NA
|
||||
shifts["relative_x"] = shifts["x"] - pd.to_numeric(shifts["prop_x"], errors="coerce")
|
||||
shifts["relative_y"] = shifts["y"] - pd.to_numeric(shifts["prop_y"], errors="coerce")
|
||||
shifts = shifts.drop(columns=["prop_x", "prop_y"]).rename(columns={"error": "error_value"})
|
||||
shifts["time_utc"] = pd.to_datetime(shifts["time_utc"])
|
||||
shifts["sensor_name"] = shifts["sensor_name"].astype(str)
|
||||
for c in NUMERIC:
|
||||
if c in shifts:
|
||||
shifts[c] = pd.to_numeric(shifts[c], errors="coerce")
|
||||
return shifts.reindex(columns=COLUMNS)
|
||||
|
||||
|
||||
def etats_agreges(conn, unit, a, b):
|
||||
"""fetch_aggregate_datapoints_frame : les médianes par minute des tables d'états (shift_sensor_1min,
|
||||
shift_common_1min), seule source des appareils « burst », au schéma de l'export (prepare_aggregate_datapoints_frame).
|
||||
Dans les états, x_pct porte le x CORRIGÉ et rx_pct le x brut."""
|
||||
thr = "COALESCE((th.value #>> '{}')::double precision, CAST(:thr AS double precision))"
|
||||
sensor = lire(conn, f"""
|
||||
SELECT ss.time_utc, COALESCE(sn.value #>> '{{}}', s.name) AS name,
|
||||
approx_percentile(0.5, ss.x_pct) AS corr_x, approx_percentile(0.5, ss.y_pct) AS corr_y,
|
||||
approx_percentile(0.5, ss.rx_pct) AS x, approx_percentile(0.5, ss.ry_pct) AS y,
|
||||
approx_percentile(0.5, ss.error_pct) AS error, {thr} AS error_threshold,
|
||||
approx_percentile(0.5, ss.reproj_err_pct) AS reproj_err,
|
||||
{", ".join(f"ss.{n}" for n in INLIER_BANDS)},
|
||||
{", ".join(f"approx_percentile(0.5, ss.{m}_pct) AS {m}" for m in QUALITY)}
|
||||
FROM shift_sensor_1min ss
|
||||
JOIN sensor s ON s.id = ss.sensor_id
|
||||
LEFT JOIN (SELECT p.sensor_id, any_value(p.value) AS value FROM property p JOIN datatype dt
|
||||
ON dt.id = p.datatype_id WHERE dt.name = 'sensor_name' GROUP BY p.sensor_id) sn
|
||||
ON sn.sensor_id = ss.sensor_id
|
||||
LEFT JOIN (SELECT p.sensor_id, any_value(p.value) AS value FROM property p JOIN datatype dt
|
||||
ON dt.id = p.datatype_id WHERE dt.name = 'error_threshold' GROUP BY p.sensor_id) th
|
||||
ON th.sensor_id = ss.sensor_id
|
||||
WHERE s.acquisition_unit_id = :au AND ss.time_utc >= :a AND ss.time_utc <= :b
|
||||
ORDER BY ss.time_utc""", au=unit, a=a, b=b, thr=DEFAULT_ERROR_THRESHOLD)
|
||||
if sensor.empty:
|
||||
return pd.DataFrame(columns=COLUMNS)
|
||||
common = lire(conn, """
|
||||
SELECT sc.time_utc, approx_percentile(0.5, sc.sigma0_pct) AS sigma0 FROM shift_common_1min sc
|
||||
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b""", au=unit, a=a, b=b)
|
||||
out = pd.DataFrame({"time_utc": pd.to_datetime(sensor["time_utc"]), "sensor_name": sensor["name"].astype(str)})
|
||||
for col, src in [("error_value", "error"), ("error_threshold", "error_threshold"), ("corr_x", "corr_x"),
|
||||
("corr_y", "corr_y"), ("relative_x", "corr_x"), ("relative_y", "corr_y"), ("x", "x"), ("y", "y"),
|
||||
("reproj_err", "reproj_err")] + [(m, m) for m in QUALITY]:
|
||||
out[col] = pd.to_numeric(sensor[src], errors="coerce")
|
||||
# inlier_code_from_bands : la bande majoritaire, l'égalité à la pire ; toutes à 0 -> NaN (pas de redondance)
|
||||
counts = sensor[list(INLIER_BANDS)].apply(pd.to_numeric, errors="coerce").fillna(0)
|
||||
code = counts[list(INLIER_BANDS)[::-1]].idxmax(axis=1).map(INLIER_BANDS)
|
||||
out["inlier"] = code.where(counts.sum(axis=1) > 0).to_numpy()
|
||||
out["pattern_type"] = None
|
||||
sig = common.loc[common["sigma0"].notna()]
|
||||
out["sigma0"] = (out["time_utc"].map(sig.assign(time_utc=pd.to_datetime(sig["time_utc"]))
|
||||
.groupby("time_utc")["sigma0"].median()) if not sig.empty else float("nan"))
|
||||
return out.reindex(columns=COLUMNS)
|
||||
|
||||
|
||||
def fusionner(raw, agg):
|
||||
"""merge_raw_and_aggregate_frames : le brut gagne ; un état n'est gardé que là où son capteur n'a pas de ligne
|
||||
brute dans la même minute."""
|
||||
if agg.empty:
|
||||
return raw
|
||||
if raw.empty:
|
||||
return agg
|
||||
|
||||
def utc(t):
|
||||
t = pd.to_datetime(t)
|
||||
return t.dt.tz_localize("UTC") if t.dt.tz is None else t.dt.tz_convert("UTC")
|
||||
|
||||
cles = pd.MultiIndex.from_arrays([raw["sensor_name"], utc(raw["time_utc"]).dt.floor("1min")])
|
||||
agg = agg.loc[~pd.MultiIndex.from_arrays([agg["sensor_name"], utc(agg["time_utc"]).dt.floor("1min")]).isin(cles)]
|
||||
if agg.empty:
|
||||
return raw
|
||||
agg = agg.copy()
|
||||
tz = pd.to_datetime(raw["time_utc"]).dt.tz
|
||||
t = utc(agg["time_utc"])
|
||||
agg["time_utc"] = t.dt.tz_convert(tz) if tz is not None else t.dt.tz_localize(None)
|
||||
return (pd.concat([raw, agg], ignore_index=True).sort_values(["sensor_name", "time_utc"], kind="stable")
|
||||
.reset_index(drop=True).reindex(columns=COLUMNS))
|
||||
|
||||
|
||||
def advanced(conn, unit, a, b):
|
||||
return fusionner(preparer_bruts(datapoints_bruts(conn, unit, a, b)), etats_agreges(conn, unit, a, b))
|
||||
|
||||
|
||||
def camera(conn, unit, a, b):
|
||||
"""Pan, tilt, scale, rotation et sigma0 de la caméra, en médiane et bornes (median_ / min_ / max_) par minute, donc
|
||||
une ligne par mesure de la caméra (toutes les 15 min à NORDEND, min = max) : ce que trace le visualiseur
|
||||
(fetch_cam_movements_frame), sans son except, qui rendrait vide un jour en échec au lieu de le laisser à relancer."""
|
||||
sigs = ", ".join(f"approx_percentile(0.5, sc.{s}_pct) AS median_{s}, min_val(sc.{s}_pct) AS min_{s}, "
|
||||
f"max_val(sc.{s}_pct) AS max_{s}" for s in CAM_SIGNALS)
|
||||
return lire(conn, f"""SELECT sc.time_utc, {sigs} FROM shift_common_1min sc
|
||||
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b
|
||||
ORDER BY sc.time_utc""", au=unit, a=a, b=b)
|
||||
|
||||
|
||||
JOURS = (("displacement", "advanced", advanced), ("camera", "camera", camera)) # dossier, fichier, lecture
|
||||
|
||||
|
||||
def pomper_deplacements(conn, unit, station, debut, fin, out, remplacer):
|
||||
"""Déplacements et mouvements de la caméra : un fichier et une requête par jour, bornée, sous le délai (--delai)."""
|
||||
for k in range((fin - debut).days):
|
||||
a = datetime.combine(debut + timedelta(days=k), datetime.min.time(), UTC)
|
||||
for dossier, nom, lecture in JOURS:
|
||||
path = out / station / dossier / f"{station}_{nom}_{a:%Y%m%d}.csv"
|
||||
if path.exists() and not remplacer:
|
||||
continue
|
||||
t0 = time.time()
|
||||
try:
|
||||
df = lecture(conn, unit, a, a + timedelta(days=1) - timedelta(microseconds=1))
|
||||
except Exception as e: # ponytail : un jour en échec est sauté, relancer le reprend ; pas de nouvel essai
|
||||
print(f"{station} {a:%Y-%m-%d} {nom} : échec ({raison(e)}), à relancer")
|
||||
continue
|
||||
finally:
|
||||
conn.rollback() # ferme la transaction de lecture
|
||||
if "sensor_name" in df:
|
||||
df = df.drop_duplicates(["sensor_name", "time_utc"])
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
df.to_csv(path, index=False) # un jour vide est écrit aussi : la relance ne le redemande pas
|
||||
print(f"{station} {a:%Y-%m-%d} {nom}: {len(df)} lignes, {time.time() - t0:.0f} s -> {path.name}")
|
||||
|
||||
|
||||
def pomper_images(conn, s3, unit, station, debut, fin, dossier, every, out):
|
||||
mt = conn.execute(text("SELECT id, extension FROM measurement_type WHERE name = :n"), {"n": TYPES[dossier]}).first()
|
||||
if mt is None:
|
||||
print(f"{station} : type {TYPES[dossier]} inconnu")
|
||||
return
|
||||
client, bucket_s3 = s3
|
||||
for k in range((fin - debut).days):
|
||||
a = datetime.combine(debut + timedelta(days=k), datetime.min.time(), UTC)
|
||||
mesures = conn.execute(text(
|
||||
"SELECT id, time_utc, measurement_uuid FROM measurement WHERE acquisition_unit_id = :au "
|
||||
"AND measurement_type_id = :mt AND time_utc >= :a AND time_utc < :b ORDER BY time_utc"),
|
||||
{"au": unit, "mt": mt.id, "a": a, "b": a + timedelta(days=1)}).all()
|
||||
conn.rollback()
|
||||
seen, n, octets, sans_uuid, absentes = set(), 0, 0, 0, 0
|
||||
for m in mesures:
|
||||
# measurement_uuid vide : l'image n'est jamais arrivée sur S3 (envoi en plusieurs parties inachevé) ;
|
||||
# écartée, la tranche prend l'image suivante s'il y en a une
|
||||
if m.measurement_uuid is None:
|
||||
sans_uuid += 1
|
||||
continue
|
||||
t = m.time_utc.astimezone(UTC)
|
||||
bucket = (t.hour * 60 + t.minute) // every if every else m.id # une image par tranche de `every` min
|
||||
path = out / station / dossier / f"{t:%Y%m%dT%H%M}Z_{m.id}.{mt.extension}"
|
||||
if bucket in seen:
|
||||
continue
|
||||
seen.add(bucket) # même déjà là : sinon la relance prendrait l'image suivante de la tranche
|
||||
if path.exists():
|
||||
continue
|
||||
try:
|
||||
data = client.get_object(Bucket=bucket_s3, Key=m.measurement_uuid)["Body"].read()
|
||||
except Exception: # comme get_measurement_payload : S3 qui ne répond pas, l'image est comptée absente
|
||||
data = None
|
||||
if data:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
n, octets = n + 1, octets + len(data)
|
||||
else:
|
||||
absentes += 1
|
||||
# le détail dit pourquoi un jour rend 0 : rien en base, jamais arrivées sur S3, ou S3 qui ne les rend pas
|
||||
print(f"{station} {a:%Y-%m-%d} {dossier}: {n} nouvelles images, {octets / 1e6:.0f} Mo "
|
||||
f"({len(mesures)} en base, {sans_uuid} jamais arrivées sur S3, {absentes} non rendues par S3)")
|
||||
time.sleep(0.5)
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("stations", nargs="*", help="numéro de série du système, comme dans l'URL de la page System ; "
|
||||
"aucun : ceux de stations.txt")
|
||||
p.add_argument("--du", help="premier jour UTC, AAAA-MM-JJ ; sans : le dernier jour pompé de chaque station, "
|
||||
"repompé, ou les 30 derniers jours")
|
||||
p.add_argument("--au", help="dernier jour UTC exclu, AAAA-MM-JJ ; sans : jusqu'à maintenant")
|
||||
p.add_argument("--images", action="store_true", help="aussi les images OPTICAL_SHIFT-IRON")
|
||||
p.add_argument("--diff", action="store_true", help="aussi les images OPTICAL_SHIFT-DIFF")
|
||||
p.add_argument("--every", type=int, default=60, help="une image par tranche de N minutes, 0 = toutes (défaut 60)")
|
||||
p.add_argument("--delai", type=int, default=60, help="délai maximal d'une requête, en s (défaut 60 : base partagée "
|
||||
"avec la production) ; plus long seulement pour un jour en échec par délai dépassé")
|
||||
p.add_argument("--out", type=Path, default=Path(__file__).resolve().parent.parent / "data")
|
||||
a = p.parse_args()
|
||||
demain = datetime.now(UTC).date() + timedelta(days=1)
|
||||
du = datetime.strptime(a.du, "%Y-%m-%d").date() if a.du else None
|
||||
fin = min(datetime.strptime(a.au, "%Y-%m-%d").date(), demain) if a.au else demain # pas de jour à venir
|
||||
stations = a.stations or STATIONS.read_text().split()
|
||||
|
||||
missing = [n for n in DB_NAMES if not os.environ.get(n)]
|
||||
if missing:
|
||||
sys.exit(f"Variables absentes (noms seulement) : {', '.join(missing)}")
|
||||
url = URL.create("postgresql+psycopg2", username=os.environ["AIMSIGHT_DATABASE_USER"],
|
||||
password=os.environ["AIMSIGHT_DATABASE_PSWD"], host=os.environ["AIMSIGHT_DATABASE_TURL"],
|
||||
port=int(os.environ.get("AIMSIGHT_DATABASE_PORT", "5432")),
|
||||
database=os.environ["AIMSIGHT_DATABASE_NAME"])
|
||||
engine = create_engine(url, connect_args={
|
||||
"options": f"-c timezone=utc -c default_transaction_read_only=on -c statement_timeout={a.delai}s"})
|
||||
s3 = None
|
||||
if a.images or a.diff:
|
||||
import boto3 # comme S3client de l'appli (Garage)
|
||||
s3 = (boto3.client("s3", endpoint_url=os.environ.get("AIMSIGHT_S3_URL", "http://s3:3900"),
|
||||
aws_access_key_id=os.environ.get("AIMSIGHT_S3_ACCESS_KEY", ""),
|
||||
aws_secret_access_key=os.environ.get("AIMSIGHT_S3_SECRET_KEY", ""),
|
||||
region_name="garage", use_ssl=False),
|
||||
os.environ.get("AIMSIGHT_S3_BUCKET", "measurements-assets"))
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
for station in stations:
|
||||
system_id = conn.execute(text("SELECT id FROM system WHERE serial_number = :s"), {"s": station}).scalar()
|
||||
if system_id is None:
|
||||
print(f"{station} : système inconnu")
|
||||
continue
|
||||
plages = [(du, fin, False)] if du else plages_suivi(a.out / station, fin, demain)
|
||||
unit = acquisition_unit(conn, system_id)
|
||||
conn.rollback()
|
||||
for debut, fin_plage, remplacer in plages:
|
||||
print(f"{station} : du {debut} au {fin_plage - timedelta(days=1)} inclus"
|
||||
+ ("" if remplacer else ", jours manquants seulement"))
|
||||
pomper_metriques(conn, system_id, station, debut, fin_plage, a.out, remplacer)
|
||||
if unit is None:
|
||||
print(f"{station} : pas d'unité d'acquisition reconnue, ni déplacements, ni caméra, ni images")
|
||||
continue
|
||||
pomper_deplacements(conn, unit, station, debut, fin_plage, a.out, remplacer)
|
||||
for flag, dossier in ((a.images, "images"), (a.diff, "images_diff")):
|
||||
if flag:
|
||||
pomper_images(conn, s3, unit, station, debut, fin_plage, dossier, a.every, a.out)
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,25 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Pompage sous WSL. Lit les identifiants dans le .env à la racine du dépôt (jamais versionné),
|
||||
# lance pomper.py dans un environnement uv jetable (4 paquets), puis le post-traitement de LENZSPITZE.
|
||||
# extraction/pomper.sh mise à jour des stations suivies, images comprises (le planificateur)
|
||||
# extraction/pomper.sh --du D --au F [SERIE] tout autre appel : les arguments vont tels quels à pomper.py
|
||||
# Sortie ajoutée à out/pompage.log ; un seul pompage à la fois (verrou out/pompage.lock).
|
||||
set -u
|
||||
export PATH="$HOME/.local/bin:$PATH" # uv, absent du PATH quand le Planificateur de tâches lance wsl.exe
|
||||
ici=$(cd "$(dirname "$0")" && pwd)
|
||||
racine=$(dirname "$ici")
|
||||
mkdir -p "$racine/out"
|
||||
|
||||
exec 9>"$racine/out/pompage.lock"
|
||||
flock -n 9 || { echo "Un pompage tourne déjà." >&2; exit 1; }
|
||||
|
||||
[ $# -eq 0 ] && set -- --images --delai 300
|
||||
{
|
||||
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) pomper.py $*"
|
||||
(set -a && . "$racine/.env" && set +a &&
|
||||
uv run --no-project --quiet --python 3.12 \
|
||||
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
|
||||
python "$ici/pomper.py" "$@")
|
||||
[ -f "$racine/data/LENZSPITZE/lentzpitze_patterns.json" ] && python3 "$ici/lenzspitze_absolu.py"
|
||||
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) fin"
|
||||
} 2>&1 | tee -a "$racine/out/pompage.log"
|
||||
@@ -0,0 +1,409 @@
|
||||
"""Pumps the system metrics, the displacements ("advanced" export) and the images of stations from the AIMsight
|
||||
platform, in the format read by the app, the reading tools and the detectors. READ-ONLY. Tried on 2026-10-26
|
||||
(NORDEND, 2 days); standalone since 2026-11-12.
|
||||
|
||||
Run through extraction/pump.sh, under WSL, from anywhere (see README.md):
|
||||
|
||||
extraction/pump.sh
|
||||
extraction/pump.sh --from 2026-08-01 --to 2026-08-08 --images WEISSHORN
|
||||
|
||||
Without a station: those of `stations.txt`. Without `--from`: each station fills its missing days (displacements or
|
||||
metrics) before the last one pumped, then restarts from that last day, pumped again (it was perhaps incomplete), or
|
||||
from the last 30 days if it has none; without `--to`: up to now.
|
||||
|
||||
Output (default: the repository's `data/`, next to this folder):
|
||||
<STATION>/metrics/<STATION>_<metric>_<YYYY_MM_DD>_00_00 to <YYYY_MM_DD>_23_00.csv system metrics, per day
|
||||
<STATION>/displacement/<STATION>_advanced_<YYYYMMDD>.csv displacements, one file per day
|
||||
<STATION>/camera/<STATION>_camera_<YYYYMMDD>.csv camera movements, per day
|
||||
<STATION>/images/<YYYYMMDD>T<HHMM>Z_<id>.<ext> OPTICAL_SHIFT-IRON images (--images)
|
||||
<STATION>/images_diff/… differential images (--diff)
|
||||
Resumable: with `--from`, a file already written is not read again, whatever the dates asked; without `--from`, the
|
||||
days of the range are replaced (images excepted). Days in UTC, `--to` excluded; the current day is pumped as it is,
|
||||
its metrics up to the last completed hour. To redo a day, delete its files.
|
||||
|
||||
No dependency on the app (~/monitoring-dashboard): its reads are copied here, in SQL, as they are (branch `explo`,
|
||||
commit 3dce73d of 2026-11-12): the query of System.get_metrics (the download of the System page);
|
||||
build_advanced_frame (app/datapoints.py, what the Advanced tab and the parquet export plot), with
|
||||
AcquisitionUnit.get_datapoints and AggregateEngine.get_sensor_aggregate / get_common_aggregate; the camera movements
|
||||
of fetch_cam_movements_frame (AggregateEngine.get_cam_scalar_aggregate, what the viewer plots);
|
||||
AcquisitionUnit.measurements and get_measurement_payload (S3). If the app changes its reads, this file does not follow.
|
||||
Read-only connection (`default_transaction_read_only`), 60 s timeout per query (`--delay`). Variables read (names
|
||||
only, none is printed): AIMSIGHT_DATABASE_USER / _PSWD / _TURL / _NAME / _PORT, and for the images
|
||||
AIMSIGHT_S3_URL / _ACCESS_KEY / _SECRET_KEY / _BUCKET.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.engine import URL
|
||||
|
||||
UTC = timezone.utc
|
||||
STATIONS = Path(__file__).resolve().parent / "stations.txt" # the tracked stations, one per line
|
||||
NEW_STATION_DAYS = 30 # without --from, a station with no pumped displacements: the last 30 days
|
||||
DB_NAMES = ["AIMSIGHT_DATABASE_USER", "AIMSIGHT_DATABASE_PSWD", "AIMSIGHT_DATABASE_TURL", "AIMSIGHT_DATABASE_NAME"]
|
||||
TYPES = {"images": "OPTICAL_SHIFT-IRON", "images_diff": "OPTICAL_SHIFT-DIFF"} # folder -> measurement type
|
||||
|
||||
# --- constants of the app (app/datapoints.py, app/aggregates.py) ---
|
||||
QUALITY = ["peak", "flux", "sharpness", "fwhm", "dominance", "snr"]
|
||||
CAM_SIGNALS = ["pan", "tilt", "scale", "rotation", "sigma0"]
|
||||
DEFAULT_ERROR_THRESHOLD = 0.7
|
||||
DATATYPES = ["x", "y", "error", "corr_x", "corr_y", "reproj_err", "inlier", "sigma0"]
|
||||
PROPERTIES = ["x", "y", "pattern_type", "sensor_name", "error_threshold"]
|
||||
COLUMNS = ["time_utc", "sensor_name", "x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x",
|
||||
"corr_y", "reproj_err", "inlier", "sigma0", "pattern_type"] + QUALITY
|
||||
NUMERIC = ["x", "y", "error_value", "error_threshold", "relative_x", "relative_y", "corr_x", "corr_y", "reproj_err",
|
||||
"inlier", "sigma0"] + QUALITY
|
||||
# inlier bands of shift_sensor_1min -> code; the last two NAMES are wrong (-1 rejected, -2 interpolated), the codes
|
||||
# are right; ordered from best to worst
|
||||
INLIER_BANDS = {"inlier_in_fit": 1.0, "inlier_out_fit": 0.0, "inlier_no_redundancy": -1.0, "inlier_rejected": -2.0}
|
||||
|
||||
|
||||
def root_cause(e):
|
||||
"""The original error (the database's): pandas and SQLAlchemy wrap it, the SQL query first."""
|
||||
while e.__cause__ is not None:
|
||||
e = e.__cause__
|
||||
return f"{type(e).__name__}: {(str(e).splitlines() or [''])[0][:300]}"
|
||||
|
||||
|
||||
def query(conn, sql, **params):
|
||||
return pd.read_sql(text(sql), conn, params=params)
|
||||
|
||||
|
||||
def acquisition_unit(conn, system_id):
|
||||
"""Same choice as Device.get_acquisition_unit (app/base/models.py:268): the only unit, otherwise MER* / ME2S*."""
|
||||
units = conn.execute(text("SELECT id, product_number FROM acquisition_unit WHERE system_id = :s"),
|
||||
{"s": system_id}).all()
|
||||
pick = units if len(units) == 1 else [u for u in units if u.product_number.startswith(("MER", "ME2S"))]
|
||||
return pick[0].id if pick else None
|
||||
|
||||
|
||||
def update_ranges(folder, end, tomorrow):
|
||||
"""Without --from: (start, end, replace) to pump, from the days of displacements already pumped. The missing days
|
||||
between the first and the last (ranges pumped separately, failed day) are filled without touching the others; the
|
||||
last day is pumped again (pumped during the day, it was incomplete) up to now; a station with no pumped
|
||||
displacements gets the last 30 days."""
|
||||
days = {datetime.strptime(p.stem.rsplit("_", 1)[1], "%Y%m%d").date()
|
||||
for p in (folder / "displacement").glob(f"{folder.name}_advanced_*.csv")}
|
||||
if not days:
|
||||
return [(tomorrow - timedelta(days=NEW_STATION_DAYS + 1), end, True)]
|
||||
# a failed metrics day has no file at all (one query for all of them): it is a gap too; a station without any
|
||||
# metric never has one, its days are not gaps
|
||||
metric_days = {datetime.strptime(p.name.split(" to ")[0][-16:-6], "%Y_%m_%d").date()
|
||||
for p in (folder / "metrics").glob(f"{folder.name}_*_00_00 to *.csv")}
|
||||
first, last = min(days), max(days)
|
||||
gaps = [first + timedelta(days=k) for k in range((last - first).days)
|
||||
if first + timedelta(days=k) not in days or (metric_days and first + timedelta(days=k) not in metric_days)]
|
||||
return ([(gaps[0], last, False)] if gaps else []) + [(last, end, True)]
|
||||
|
||||
|
||||
def pump_metrics(conn, system_id, station, start, end, out, replace):
|
||||
"""Like the download of the System page (Dashboard/System.py, get_metrics(..., bucketing="1 hour")): per hour,
|
||||
mean * transform_factor + transform_offset, or the list of values of a text metric; the labelled series of one
|
||||
metric merged; an empty row for an hour without measurement. get_metrics counts in hours before now: its query is
|
||||
redone here on a fixed day, one query per day for all the metrics."""
|
||||
names = sorted(conn.execute(text("SELECT DISTINCT dt.name FROM datatype dt JOIN metric_series se "
|
||||
"ON se.datatype_id = dt.id WHERE se.system_id = :s"), {"s": system_id}).scalars())
|
||||
conn.rollback()
|
||||
for k in range((end - start).days):
|
||||
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
|
||||
paths = {n: out / station / "metrics"
|
||||
/ f"{station}_{n.replace(os.sep, '-')}_{a:%Y_%m_%d}_00_00 to {a:%Y_%m_%d}_23_00.csv" for n in names}
|
||||
if not replace and all(p.exists() for p in paths.values()):
|
||||
continue
|
||||
t0 = time.time()
|
||||
try:
|
||||
# date_trunc('hour'): same bucketing as time_bucket('1 hour')
|
||||
df = query(conn, """
|
||||
SELECT date_trunc('hour', ms.time_utc) AS hour, dt.name AS name,
|
||||
avg(ms.value_num) * any_value(dt.transform_factor) + any_value(dt.transform_offset) AS num,
|
||||
jsonb_agg(ms.value_text) FILTER (WHERE ms.value_text IS NOT NULL) AS txt
|
||||
FROM metric_sample ms
|
||||
JOIN metric_series se ON ms.series_id = se.id
|
||||
JOIN datatype dt ON se.datatype_id = dt.id
|
||||
WHERE se.system_id = :s AND ms.time_utc >= :a AND ms.time_utc < :b
|
||||
GROUP BY 1, dt.name""", s=system_id, a=a, b=a + timedelta(days=1))
|
||||
except Exception as e: # ponytail: like the displacements, a failed day is skipped, a new run picks it up
|
||||
print(f"{station} {a:%Y-%m-%d} metrics: failed ({root_cause(e)}), run again")
|
||||
continue
|
||||
finally:
|
||||
conn.rollback()
|
||||
df["hour"] = pd.to_datetime(df["hour"], utc=True)
|
||||
hours = pd.to_datetime(pd.date_range(a, periods=24, freq="h"), utc=True) # same time zone as df, for reindex
|
||||
# current day: the completed hours only, otherwise the hours to come would read as a gap
|
||||
hours = hours[hours + pd.Timedelta(hours=1) <= pd.Timestamp.now(tz=UTC)]
|
||||
for n, path in paths.items():
|
||||
d = df[df["name"] == n].set_index("hour")
|
||||
col = d["num"] if d["num"].notna().any() else d["txt"] # text metric: the list of the hour
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
# CSV of the System page: ',time_utc,metric', the index as first column; an empty day is written too
|
||||
pd.DataFrame({"time_utc": hours, "metric": col.reindex(hours).to_numpy()}).to_csv(path)
|
||||
print(f"{station} {a:%Y-%m-%d} metrics: {len(names)} files, {df['name'].nunique()} with data, "
|
||||
f"{time.time() - t0:.0f} s")
|
||||
|
||||
|
||||
# --- the "advanced" export: build_advanced_frame (app/datapoints.py) ---
|
||||
|
||||
def raw_datapoints(conn, unit, a, b):
|
||||
"""AcquisitionUnit.get_datapoints: the datapoints of the unit, one type per column, the sensor properties next to
|
||||
them (prop_x / prop_y: the reference position, namesake of a type)."""
|
||||
types = ",\n".join(f"max(d.value) FILTER (WHERE dt.name = '{t}') AS {t}" for t in DATATYPES)
|
||||
names = {p: f"prop_{p}" if p in DATATYPES else p for p in PROPERTIES}
|
||||
props = ",\n".join(f"any_value(p_{p}.value) AS {names[p]}" for p in PROPERTIES)
|
||||
joins = "\n".join(f"LEFT JOIN (SELECT p.sensor_id, p.value FROM property p JOIN datatype pdt "
|
||||
f"ON p.datatype_id = pdt.id WHERE pdt.name = '{p}') p_{p} ON d.sensor_id = p_{p}.sensor_id"
|
||||
for p in PROPERTIES)
|
||||
return query(conn, f"""
|
||||
SELECT d.time_utc, d.sensor_id, any_value(d.measurement_id) AS measurement_id,
|
||||
{types},
|
||||
{props}
|
||||
FROM datapoint d
|
||||
JOIN sensor s ON d.sensor_id = s.id
|
||||
JOIN datatype dt ON d.datatype_id = dt.id
|
||||
{joins}
|
||||
WHERE s.acquisition_unit_id = :au AND d.time_utc >= :a AND d.time_utc <= :b
|
||||
GROUP BY d.time_utc, d.sensor_id
|
||||
ORDER BY d.sensor_id, d.time_utc""", au=unit, a=a, b=b)
|
||||
|
||||
|
||||
def prepare_raw(shifts):
|
||||
"""prepare_datapoints_frame: sigma0 (camera rows, without sensor_name) copied onto each sensor at its time;
|
||||
relative = x - prop_x; error -> error_value; the full schema."""
|
||||
if shifts.empty:
|
||||
return pd.DataFrame(columns=COLUMNS)
|
||||
sig = shifts.loc[shifts["sigma0"].notna(), ["time_utc", "sigma0"]]
|
||||
sigma0 = sig.groupby("time_utc")["sigma0"].median() if not sig.empty else None
|
||||
shifts = shifts.dropna(subset=["sensor_name"]).reset_index(drop=True)
|
||||
shifts["sigma0"] = shifts["time_utc"].map(sigma0) if sigma0 is not None else pd.NA
|
||||
shifts["relative_x"] = shifts["x"] - pd.to_numeric(shifts["prop_x"], errors="coerce")
|
||||
shifts["relative_y"] = shifts["y"] - pd.to_numeric(shifts["prop_y"], errors="coerce")
|
||||
shifts = shifts.drop(columns=["prop_x", "prop_y"]).rename(columns={"error": "error_value"})
|
||||
shifts["time_utc"] = pd.to_datetime(shifts["time_utc"])
|
||||
shifts["sensor_name"] = shifts["sensor_name"].astype(str)
|
||||
for c in NUMERIC:
|
||||
if c in shifts:
|
||||
shifts[c] = pd.to_numeric(shifts[c], errors="coerce")
|
||||
return shifts.reindex(columns=COLUMNS)
|
||||
|
||||
|
||||
def aggregate_states(conn, unit, a, b):
|
||||
"""fetch_aggregate_datapoints_frame: the per-minute medians of the state tables (shift_sensor_1min,
|
||||
shift_common_1min), only source for "burst" devices, in the schema of the export (prepare_aggregate_datapoints_frame).
|
||||
In the states, x_pct holds the CORRECTED x and rx_pct the raw x."""
|
||||
thr = "COALESCE((th.value #>> '{}')::double precision, CAST(:thr AS double precision))"
|
||||
sensor = query(conn, f"""
|
||||
SELECT ss.time_utc, COALESCE(sn.value #>> '{{}}', s.name) AS name,
|
||||
approx_percentile(0.5, ss.x_pct) AS corr_x, approx_percentile(0.5, ss.y_pct) AS corr_y,
|
||||
approx_percentile(0.5, ss.rx_pct) AS x, approx_percentile(0.5, ss.ry_pct) AS y,
|
||||
approx_percentile(0.5, ss.error_pct) AS error, {thr} AS error_threshold,
|
||||
approx_percentile(0.5, ss.reproj_err_pct) AS reproj_err,
|
||||
{", ".join(f"ss.{n}" for n in INLIER_BANDS)},
|
||||
{", ".join(f"approx_percentile(0.5, ss.{m}_pct) AS {m}" for m in QUALITY)}
|
||||
FROM shift_sensor_1min ss
|
||||
JOIN sensor s ON s.id = ss.sensor_id
|
||||
LEFT JOIN (SELECT p.sensor_id, any_value(p.value) AS value FROM property p JOIN datatype dt
|
||||
ON dt.id = p.datatype_id WHERE dt.name = 'sensor_name' GROUP BY p.sensor_id) sn
|
||||
ON sn.sensor_id = ss.sensor_id
|
||||
LEFT JOIN (SELECT p.sensor_id, any_value(p.value) AS value FROM property p JOIN datatype dt
|
||||
ON dt.id = p.datatype_id WHERE dt.name = 'error_threshold' GROUP BY p.sensor_id) th
|
||||
ON th.sensor_id = ss.sensor_id
|
||||
WHERE s.acquisition_unit_id = :au AND ss.time_utc >= :a AND ss.time_utc <= :b
|
||||
ORDER BY ss.time_utc""", au=unit, a=a, b=b, thr=DEFAULT_ERROR_THRESHOLD)
|
||||
if sensor.empty:
|
||||
return pd.DataFrame(columns=COLUMNS)
|
||||
common = query(conn, """
|
||||
SELECT sc.time_utc, approx_percentile(0.5, sc.sigma0_pct) AS sigma0 FROM shift_common_1min sc
|
||||
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b""", au=unit, a=a, b=b)
|
||||
out = pd.DataFrame({"time_utc": pd.to_datetime(sensor["time_utc"]), "sensor_name": sensor["name"].astype(str)})
|
||||
for col, src in [("error_value", "error"), ("error_threshold", "error_threshold"), ("corr_x", "corr_x"),
|
||||
("corr_y", "corr_y"), ("relative_x", "corr_x"), ("relative_y", "corr_y"), ("x", "x"), ("y", "y"),
|
||||
("reproj_err", "reproj_err")] + [(m, m) for m in QUALITY]:
|
||||
out[col] = pd.to_numeric(sensor[src], errors="coerce")
|
||||
# inlier_code_from_bands: the majority band, a tie goes to the worst; all at 0 -> NaN (no redundancy)
|
||||
counts = sensor[list(INLIER_BANDS)].apply(pd.to_numeric, errors="coerce").fillna(0)
|
||||
code = counts[list(INLIER_BANDS)[::-1]].idxmax(axis=1).map(INLIER_BANDS)
|
||||
out["inlier"] = code.where(counts.sum(axis=1) > 0).to_numpy()
|
||||
out["pattern_type"] = None
|
||||
sig = common.loc[common["sigma0"].notna()]
|
||||
out["sigma0"] = (out["time_utc"].map(sig.assign(time_utc=pd.to_datetime(sig["time_utc"]))
|
||||
.groupby("time_utc")["sigma0"].median()) if not sig.empty else float("nan"))
|
||||
return out.reindex(columns=COLUMNS)
|
||||
|
||||
|
||||
def merge(raw, agg):
|
||||
"""merge_raw_and_aggregate_frames: the raw wins; a state is kept only where its sensor has no raw row in the same
|
||||
minute."""
|
||||
if agg.empty:
|
||||
return raw
|
||||
if raw.empty:
|
||||
return agg
|
||||
|
||||
def utc(t):
|
||||
t = pd.to_datetime(t)
|
||||
return t.dt.tz_localize("UTC") if t.dt.tz is None else t.dt.tz_convert("UTC")
|
||||
|
||||
keys = pd.MultiIndex.from_arrays([raw["sensor_name"], utc(raw["time_utc"]).dt.floor("1min")])
|
||||
agg = agg.loc[~pd.MultiIndex.from_arrays([agg["sensor_name"], utc(agg["time_utc"]).dt.floor("1min")]).isin(keys)]
|
||||
if agg.empty:
|
||||
return raw
|
||||
agg = agg.copy()
|
||||
tz = pd.to_datetime(raw["time_utc"]).dt.tz
|
||||
t = utc(agg["time_utc"])
|
||||
agg["time_utc"] = t.dt.tz_convert(tz) if tz is not None else t.dt.tz_localize(None)
|
||||
return (pd.concat([raw, agg], ignore_index=True).sort_values(["sensor_name", "time_utc"], kind="stable")
|
||||
.reset_index(drop=True).reindex(columns=COLUMNS))
|
||||
|
||||
|
||||
def advanced(conn, unit, a, b):
|
||||
return merge(prepare_raw(raw_datapoints(conn, unit, a, b)), aggregate_states(conn, unit, a, b))
|
||||
|
||||
|
||||
def camera(conn, unit, a, b):
|
||||
"""Pan, tilt, scale, rotation and sigma0 of the camera, as median and bounds (median_ / min_ / max_) per minute,
|
||||
hence one row per camera measurement (every 15 min at NORDEND, min = max): what the viewer plots
|
||||
(fetch_cam_movements_frame), without its except, which would return a failed day as empty instead of leaving it
|
||||
to run again."""
|
||||
sigs = ", ".join(f"approx_percentile(0.5, sc.{s}_pct) AS median_{s}, min_val(sc.{s}_pct) AS min_{s}, "
|
||||
f"max_val(sc.{s}_pct) AS max_{s}" for s in CAM_SIGNALS)
|
||||
return query(conn, f"""SELECT sc.time_utc, {sigs} FROM shift_common_1min sc
|
||||
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b
|
||||
ORDER BY sc.time_utc""", au=unit, a=a, b=b)
|
||||
|
||||
|
||||
DAILY_FILES = (("displacement", "advanced", advanced), ("camera", "camera", camera)) # folder, file, reader
|
||||
|
||||
|
||||
def pump_displacements(conn, unit, station, start, end, out, replace):
|
||||
"""Displacements and camera movements: one file and one query per day, bounded, under the timeout (--delay)."""
|
||||
for k in range((end - start).days):
|
||||
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
|
||||
for folder, name, reader in DAILY_FILES:
|
||||
path = out / station / folder / f"{station}_{name}_{a:%Y%m%d}.csv"
|
||||
if path.exists() and not replace:
|
||||
continue
|
||||
t0 = time.time()
|
||||
try:
|
||||
df = reader(conn, unit, a, a + timedelta(days=1) - timedelta(microseconds=1))
|
||||
except Exception as e: # ponytail: a failed day is skipped, a new run picks it up; no retry
|
||||
print(f"{station} {a:%Y-%m-%d} {name}: failed ({root_cause(e)}), run again")
|
||||
continue
|
||||
finally:
|
||||
conn.rollback() # closes the read transaction
|
||||
if "sensor_name" in df:
|
||||
df = df.drop_duplicates(["sensor_name", "time_utc"])
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
df.to_csv(path, index=False) # an empty day is written too: the next run does not ask for it again
|
||||
print(f"{station} {a:%Y-%m-%d} {name}: {len(df)} rows, {time.time() - t0:.0f} s -> {path.name}")
|
||||
|
||||
|
||||
def pump_images(conn, s3, unit, station, start, end, folder, every, out):
|
||||
mt = conn.execute(text("SELECT id, extension FROM measurement_type WHERE name = :n"), {"n": TYPES[folder]}).first()
|
||||
if mt is None:
|
||||
print(f"{station}: unknown type {TYPES[folder]}")
|
||||
return
|
||||
client, bucket_name = s3
|
||||
for k in range((end - start).days):
|
||||
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
|
||||
measurements = conn.execute(text(
|
||||
"SELECT id, time_utc, measurement_uuid FROM measurement WHERE acquisition_unit_id = :au "
|
||||
"AND measurement_type_id = :mt AND time_utc >= :a AND time_utc < :b ORDER BY time_utc"),
|
||||
{"au": unit, "mt": mt.id, "a": a, "b": a + timedelta(days=1)}).all()
|
||||
conn.rollback()
|
||||
seen, n, nbytes, no_uuid, unreturned = set(), 0, 0, 0, 0
|
||||
for m in measurements:
|
||||
# empty measurement_uuid: the image never reached S3 (unfinished multipart upload); skipped, the slot
|
||||
# takes the next image if there is one
|
||||
if m.measurement_uuid is None:
|
||||
no_uuid += 1
|
||||
continue
|
||||
t = m.time_utc.astimezone(UTC)
|
||||
bucket = (t.hour * 60 + t.minute) // every if every else m.id # one image per slot of `every` min
|
||||
path = out / station / folder / f"{t:%Y%m%dT%H%M}Z_{m.id}.{mt.extension}"
|
||||
if bucket in seen:
|
||||
continue
|
||||
seen.add(bucket) # even if already there: otherwise the next run would take the next image of the slot
|
||||
if path.exists():
|
||||
continue
|
||||
try:
|
||||
data = client.get_object(Bucket=bucket_name, Key=m.measurement_uuid)["Body"].read()
|
||||
except Exception: # like get_measurement_payload: S3 not answering, the image is counted as missing
|
||||
data = None
|
||||
if data:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
n, nbytes = n + 1, nbytes + len(data)
|
||||
else:
|
||||
unreturned += 1
|
||||
# the detail says why a day gives 0: nothing in the database, never reached S3, or S3 not returning them
|
||||
print(f"{station} {a:%Y-%m-%d} {folder}: {n} new images, {nbytes / 1e6:.0f} MB "
|
||||
f"({len(measurements)} in database, {no_uuid} never reached S3, {unreturned} not returned by S3)")
|
||||
time.sleep(0.5)
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("stations", nargs="*", help="serial number of the system, as in the URL of the System page; "
|
||||
"none: those of stations.txt")
|
||||
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD; without: the last pumped day of each "
|
||||
"station, pumped again, or the last 30 days")
|
||||
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD; without: up to now")
|
||||
p.add_argument("--images", action="store_true", help="also the OPTICAL_SHIFT-IRON images")
|
||||
p.add_argument("--diff", action="store_true", help="also the OPTICAL_SHIFT-DIFF images")
|
||||
p.add_argument("--every", type=int, default=60, help="one image per slot of N minutes, 0 = all (default 60)")
|
||||
p.add_argument("--delay", type=int, default=60, help="maximum duration of a query, in s (default 60: database "
|
||||
"shared with production); longer only for a day that failed on a timeout")
|
||||
p.add_argument("--out", type=Path, default=Path(__file__).resolve().parent.parent / "data")
|
||||
a = p.parse_args()
|
||||
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
|
||||
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
|
||||
end = min(datetime.strptime(a.end, "%Y-%m-%d").date(), tomorrow) if a.end else tomorrow # no day to come
|
||||
stations = a.stations or STATIONS.read_text().split()
|
||||
|
||||
missing = [n for n in DB_NAMES if not os.environ.get(n)]
|
||||
if missing:
|
||||
sys.exit(f"Missing variables (names only): {', '.join(missing)}")
|
||||
url = URL.create("postgresql+psycopg2", username=os.environ["AIMSIGHT_DATABASE_USER"],
|
||||
password=os.environ["AIMSIGHT_DATABASE_PSWD"], host=os.environ["AIMSIGHT_DATABASE_TURL"],
|
||||
port=int(os.environ.get("AIMSIGHT_DATABASE_PORT", "5432")),
|
||||
database=os.environ["AIMSIGHT_DATABASE_NAME"])
|
||||
engine = create_engine(url, connect_args={
|
||||
"options": f"-c timezone=utc -c default_transaction_read_only=on -c statement_timeout={a.delay}s"})
|
||||
s3 = None
|
||||
if a.images or a.diff:
|
||||
import boto3 # like the app's S3client (Garage)
|
||||
s3 = (boto3.client("s3", endpoint_url=os.environ.get("AIMSIGHT_S3_URL", "http://s3:3900"),
|
||||
aws_access_key_id=os.environ.get("AIMSIGHT_S3_ACCESS_KEY", ""),
|
||||
aws_secret_access_key=os.environ.get("AIMSIGHT_S3_SECRET_KEY", ""),
|
||||
region_name="garage", use_ssl=False),
|
||||
os.environ.get("AIMSIGHT_S3_BUCKET", "measurements-assets"))
|
||||
try:
|
||||
with engine.connect() as conn:
|
||||
for station in stations:
|
||||
system_id = conn.execute(text("SELECT id FROM system WHERE serial_number = :s"), {"s": station}).scalar()
|
||||
if system_id is None:
|
||||
print(f"{station}: unknown system")
|
||||
continue
|
||||
ranges = [(start, end, False)] if start else update_ranges(a.out / station, end, tomorrow)
|
||||
unit = acquisition_unit(conn, system_id)
|
||||
conn.rollback()
|
||||
for range_start, range_end, replace in ranges:
|
||||
print(f"{station}: from {range_start} to {range_end - timedelta(days=1)} inclusive"
|
||||
+ ("" if replace else ", missing days only"))
|
||||
pump_metrics(conn, system_id, station, range_start, range_end, a.out, replace)
|
||||
if unit is None:
|
||||
print(f"{station}: no recognised acquisition unit, no displacements, camera or images")
|
||||
continue
|
||||
pump_displacements(conn, unit, station, range_start, range_end, a.out, replace)
|
||||
for flag, folder in ((a.images, "images"), (a.diff, "images_diff")):
|
||||
if flag:
|
||||
pump_images(conn, s3, unit, station, range_start, range_end, folder, a.every, a.out)
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
# Pumping under WSL. Reads the credentials from the .env at the root of the repository (never versioned),
|
||||
# runs pump.py in a throwaway uv environment (4 packages), then the post-processing of LENZSPITZE.
|
||||
# extraction/pump.sh update of the tracked stations, images included (the scheduler)
|
||||
# extraction/pump.sh --from D --to E [SERIAL] any other call: the arguments go to pump.py as they are
|
||||
# Output appended to out/pump.log; one pumping at a time (lock out/pump.lock).
|
||||
set -u
|
||||
export PATH="$HOME/.local/bin:$PATH" # uv, missing from the PATH when the Task Scheduler starts wsl.exe
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
root=$(dirname "$here")
|
||||
mkdir -p "$root/out"
|
||||
|
||||
exec 9>"$root/out/pump.lock"
|
||||
flock -n 9 || { echo "A pumping is already running." >&2; exit 1; }
|
||||
|
||||
[ $# -eq 0 ] && set -- --images --delay 300
|
||||
{
|
||||
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) pump.py $*"
|
||||
(set -a +u && . "$root/.env" && set +a -u && # +u: an unquoted $ in .env does not stop the load
|
||||
uv run --no-project --quiet --python 3.12 \
|
||||
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
|
||||
python "$here/pump.py" "$@")
|
||||
[ -f "$root/data/LENZSPITZE/lentzpitze_patterns.json" ] && python3 "$here/lenzspitze_absolute.py"
|
||||
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end"
|
||||
} 2>&1 | tee -a "$root/out/pump.log"
|
||||
Reference in new issue
Block a user