Pumping: the weather of the stations with a position written into data/<SERIAL>/weather/ (extraction/weather.py, after pump.py); the mm (corr_x_mm, corr_y_mm by the sensor_id of each row) and targets.csv with the references (pattern_type, distance_mm, mm_per_px); --replace pumps a range again

pump.py --check passes; the mm and targets.csv parts are not yet run against the database (no VPN).
stations.txt: NIEDERHORN and NORDEND without a position (outside Switzerland).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-18 07:17:27 +01:00
1 parent 9adc38f9b9
commit ea0228a73a
8 files changed
+205 -22

No files matched your search

+2 -2
View File
@@ -2,9 +2,9 @@
The data of the AIMsight stations, in a single copy on the workstation, and their pumping.
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`…). A lost folder is pumped again.
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again.
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations).
- `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`, `hourly` for a caller: the Data page of aimsight-app); its doc in `weather/README.md`.
- `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data/<SERIAL>/weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`.
- `.env`: the credentials of the database and of S3, at the root, never versioned.
The other repositories (aimsight-app, Optimizer) read the data through a link `data -> ~/aimsight-data/data`, ignored by git. The methods that bring data in live here; the app only calls them. Pumping is done from here only: the lock `out/pump.lock` holds for this folder alone.
+5 -3
View File
@@ -2,19 +2,21 @@
`pump.py` fetches, for each station (serial number of the system) and each UTC day asked, into `data/<SERIAL>/`:
- the **system metrics** (energy, disk, 4G, temperatures…), hourly means, one CSV per metric and per day in `metrics/`;
- the **displacements** ("advanced" export), one CSV per day in `displacement/`;
- the **displacements** ("advanced" export), one CSV per day in `displacement/`, with `corr_x_mm` / `corr_y_mm` = `corr_x` / `corr_y` × `mm_per_px` of the sensor of each row (by `sensor_id`, never by name; empty without a factor); a day pumped before keeps its file: `--from D --to E --replace` pumps it again with the mm;
- the **sensors** measured, targets and references (every sensor with a `pattern_type`), in `targets.csv`: `pattern_type`, `distance_mm` (from the camera; 1000 = not filled in on the platform) and `mm_per_px` (`distance_convertion_factor`, the factor of the dashboard's exports in mm), their current values, read again at each pumping; a name not measured in the run keeps its line;
- the **camera movements** (`pan`, `tilt`, `scale`, `rotation`, `sigma0`), as median and bounds (`median_`, `min_`, `max_`) per minute, only at the times the camera was realigned, one CSV per day in `camera/`;
- the **weather** (`weather.py`, after the pumping), for a station with a position in `stations.txt`: the hourly MeteoSwiss measurements nearest to it, one CSV per day with data, in `weather/` (`weather/README.md`);
- the IRON **images** (`--images`, one per hour by default, `--every 10` or `--every 0` for more) in `images/`, and the differential ones (`--diff`) in `images_diff/`.
Formats: `~/aimsight-labeling/knowledge/displacement-columns.md`.
**Tracked stations**: `stations.txt`, one serial number per line, optionally followed by the position of the station (`lat,lon` or a Swiss place, read by the app for the weather; the app edits this list: page Stations, through `pumping.py`). Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in `data/` gets the last 30 days.
**Tracked stations**: `stations.txt`, one serial number per line, optionally followed by the position of the station (`lat,lon` or a Swiss place, for the weather pumped into `weather/`; MeteoSwiss covers Switzerland only; the app edits this list: page Stations, through `pumping.py`). Without station nor dates, the script updates them all up to now: each station first fills its missing days between the first and the last pumped day (without displacements, or without any metric), then restarts from its last day, pumped again (it was perhaps incomplete); a station without a folder in `data/` gets the last 30 days.
Resuming: with `--from`, a file already in `data/<SERIAL>/` is not asked for again; running again on a wider range reads only the days and images that are missing. The current day is pumped as it is, its metrics up to the last completed hour; the next run without `--from` replaces it.
## Running (under WSL)
`pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `absolute.py`. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump.
`pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `absolute.py`, then `weather.py` (every station with a position: its missing days, those written for another position, and the last 2 days, fetched again; a place not found or MeteoSwiss unreachable: a line in the log, the pumping goes on). `python3 extraction/weather.py [SERIAL…]` alone fills the weather of the days already pumped. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump.
Variables of the `.env`: `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT`; for the images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`.
+100 -12
View File
@@ -14,6 +14,9 @@ from the last 30 days if it has none; without `--to`: up to now.
Output (default: the repository's `data/`, next to this folder):
<STATION>/metrics/<STATION>_<metric>_<YYYY_MM_DD>_00_00 to <YYYY_MM_DD>_23_00.csv system metrics, per day
<STATION>/displacement/<STATION>_advanced_<YYYYMMDD>.csv displacements, one file per day
+ corr_x_mm, corr_y_mm: corr * mm_per_px of the sensor that measured (references included)
<STATION>/targets.csv the targets and references measured: sensor_name, pattern_type,
distance_mm, mm_per_px (those of the last day pumped)
<STATION>/camera/<STATION>_camera_<YYYYMMDD>.csv camera movements, per day
<STATION>/images/<YYYYMMDD>T<HHMM>Z_<id>.<ext> OPTICAL_SHIFT-IRON images (--images)
<STATION>/images_diff/… differential images (--diff)
@@ -175,7 +178,7 @@ def prepare_raw(shifts):
"""prepare_datapoints_frame: sigma0 (camera rows, without sensor_name) copied onto each sensor at its time;
relative = x - prop_x; error -> error_value; the full schema."""
if shifts.empty:
return pd.DataFrame(columns=COLUMNS)
return pd.DataFrame(columns=COLUMNS + ["sensor_id"])
sig = shifts.loc[shifts["sigma0"].notna(), ["time_utc", "sigma0"]]
sigma0 = sig.groupby("time_utc")["sigma0"].median() if not sig.empty else None
shifts = shifts.dropna(subset=["sensor_name"]).reset_index(drop=True)
@@ -188,7 +191,7 @@ def prepare_raw(shifts):
for c in NUMERIC:
if c in shifts:
shifts[c] = pd.to_numeric(shifts[c], errors="coerce")
return shifts.reindex(columns=COLUMNS)
return shifts.reindex(columns=COLUMNS + ["sensor_id"])
def aggregate_states(conn, unit, a, b):
@@ -197,7 +200,7 @@ def aggregate_states(conn, unit, a, b):
In the states, x_pct holds the CORRECTED x and rx_pct the raw x."""
thr = "COALESCE((th.value #>> '{}')::double precision, CAST(:thr AS double precision))"
sensor = query(conn, f"""
SELECT ss.time_utc, COALESCE(sn.value #>> '{{}}', s.name) AS name,
SELECT ss.time_utc, ss.sensor_id, COALESCE(sn.value #>> '{{}}', s.name) AS name,
approx_percentile(0.5, ss.x_pct) AS corr_x, approx_percentile(0.5, ss.y_pct) AS corr_y,
approx_percentile(0.5, ss.rx_pct) AS x, approx_percentile(0.5, ss.ry_pct) AS y,
approx_percentile(0.5, ss.error_pct) AS error, {thr} AS error_threshold,
@@ -215,11 +218,12 @@ def aggregate_states(conn, unit, a, b):
WHERE s.acquisition_unit_id = :au AND ss.time_utc >= :a AND ss.time_utc <= :b
ORDER BY ss.time_utc""", au=unit, a=a, b=b, thr=DEFAULT_ERROR_THRESHOLD)
if sensor.empty:
return pd.DataFrame(columns=COLUMNS)
return pd.DataFrame(columns=COLUMNS + ["sensor_id"])
common = query(conn, """
SELECT sc.time_utc, approx_percentile(0.5, sc.sigma0_pct) AS sigma0 FROM shift_common_1min sc
WHERE sc.acquisition_unit_id = :au AND sc.time_utc >= :a AND sc.time_utc <= :b""", au=unit, a=a, b=b)
out = pd.DataFrame({"time_utc": pd.to_datetime(sensor["time_utc"]), "sensor_name": sensor["name"].astype(str)})
out = pd.DataFrame({"time_utc": pd.to_datetime(sensor["time_utc"]), "sensor_name": sensor["name"].astype(str),
"sensor_id": sensor["sensor_id"]})
for col, src in [("error_value", "error"), ("error_threshold", "error_threshold"), ("corr_x", "corr_x"),
("corr_y", "corr_y"), ("relative_x", "corr_x"), ("relative_y", "corr_y"), ("x", "x"), ("y", "y"),
("reproj_err", "reproj_err")] + [(m, m) for m in QUALITY]:
@@ -232,7 +236,7 @@ def aggregate_states(conn, unit, a, b):
sig = common.loc[common["sigma0"].notna()]
out["sigma0"] = (out["time_utc"].map(sig.assign(time_utc=pd.to_datetime(sig["time_utc"]))
.groupby("time_utc")["sigma0"].median()) if not sig.empty else float("nan"))
return out.reindex(columns=COLUMNS)
return out.reindex(columns=COLUMNS + ["sensor_id"])
def merge(raw, agg):
@@ -256,11 +260,42 @@ def merge(raw, agg):
t = utc(agg["time_utc"])
agg["time_utc"] = t.dt.tz_convert(tz) if tz is not None else t.dt.tz_localize(None)
return (pd.concat([raw, agg], ignore_index=True).sort_values(["sensor_name", "time_utc"], kind="stable")
.reset_index(drop=True).reindex(columns=COLUMNS))
.reset_index(drop=True).reindex(columns=COLUMNS + ["sensor_id"]))
SENSOR_PROPS = {"pattern_type": "pattern_type", "distance_mm": "distance_mm", "distance_convertion_factor": "mm_per_px"}
MM = {"corr_x": "corr_x_mm", "corr_y": "corr_y_mm"}
def sensors(conn, unit):
"""The sensors of the unit that have a pattern_type (targets and references), one row per sensor_id: sensor_name,
pattern_type, distance_mm (from the camera; 1000: not filled in on the platform) and mm_per_px
(distance_convertion_factor, the factor of the dashboard's exports in mm: DeviceOutput.get_scale,
compute_output_signals, (corr - offset) * factor). A name may have several sensors (an older configuration of the
unit, another lens): the factor of a measurement is that of its sensor_id, never looked up by name."""
prop = "max(p.value #>> '{{}}') FILTER (WHERE dt.name = '{}')".format
df = query(conn, f"""
SELECT s.id AS sensor_id, COALESCE({prop('sensor_name')}, s.name) AS sensor_name,
{", ".join(f"{prop(n)} AS {c}" for n, c in SENSOR_PROPS.items())}
FROM sensor s JOIN property p ON p.sensor_id = s.id JOIN datatype dt ON dt.id = p.datatype_id
WHERE s.acquisition_unit_id = :au
GROUP BY s.id, s.name HAVING {prop('pattern_type')} IS NOT NULL""", au=unit)
for c in ("distance_mm", "mm_per_px"):
df[c] = pd.to_numeric(df[c], errors="coerce")
return df.set_index("sensor_id")
def with_mm(df, props):
"""corr_x_mm / corr_y_mm = corr_x / corr_y * mm_per_px of the sensor_id of each row; empty without a factor."""
factor = df["sensor_id"].map(props["mm_per_px"]).astype(float)
for c, mm in MM.items():
df[mm] = pd.to_numeric(df[c], errors="coerce") * factor
return df
def advanced(conn, unit, a, b):
return merge(prepare_raw(raw_datapoints(conn, unit, a, b)), aggregate_states(conn, unit, a, b))
df = merge(prepare_raw(raw_datapoints(conn, unit, a, b)), aggregate_states(conn, unit, a, b))
return with_mm(df, sensors(conn, unit))
def camera(conn, unit, a, b):
@@ -279,7 +314,9 @@ DAILY_FILES = (("displacement", "advanced", advanced), ("camera", "camera", came
def pump_displacements(conn, unit, station, start, end, out, replace):
"""Displacements and camera movements: one file and one query per day, bounded, under the timeout (--delay)."""
"""Displacements and camera movements: one file and one query per day, bounded, under the timeout (--delay)
-> {sensor_name: sensor_id} measured, the last day pumped winning (for targets.csv)."""
seen = {}
for k in range((end - start).days):
a = datetime.combine(start + timedelta(days=k), datetime.min.time(), UTC)
for folder, name, reader in DAILY_FILES:
@@ -296,9 +333,25 @@ def pump_displacements(conn, unit, station, start, end, out, replace):
conn.rollback() # closes the read transaction
if "sensor_name" in df:
df = df.drop_duplicates(["sensor_name", "time_utc"])
if "sensor_id" in df:
seen.update(df.dropna(subset=["sensor_id"]).groupby("sensor_name")["sensor_id"].last().to_dict())
df = df.drop(columns="sensor_id")
path.parent.mkdir(parents=True, exist_ok=True)
df.to_csv(path, index=False) # an empty day is written too: the next run does not ask for it again
print(f"{station} {a:%Y-%m-%d} {name}: {len(df)} rows, {time.time() - t0:.0f} s -> {path.name}")
return seen
def write_targets(folder, props, seen):
"""targets.csv: the properties of the sensors measured (seen: {name: sensor_id}); the names not measured in this
run keep their line."""
path = folder / "targets.csv"
new = props.loc[[i for i in seen.values() if i in props.index], ["sensor_name", *SENSOR_PROPS.values()]]
old = pd.read_csv(path) if path.exists() else new.iloc[:0]
t = pd.concat([old[~old["sensor_name"].astype(str).isin(new["sensor_name"])], new]).sort_values("sensor_name")
folder.mkdir(parents=True, exist_ok=True)
t.to_csv(path, index=False)
return t
def pump_images(conn, s3, unit, station, start, end, folder, every, out):
@@ -352,6 +405,8 @@ def main():
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD; without: the last pumped day of each "
"station, pumped again, or the last 30 days")
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD; without: up to now")
p.add_argument("--replace", action="store_true", help="with --from: the days already pumped are pumped again "
"(a file is replaced only once its day is read)")
p.add_argument("--images", action="store_true", help="also the OPTICAL_SHIFT-IRON images")
p.add_argument("--diff", action="store_true", help="also the OPTICAL_SHIFT-DIFF images")
p.add_argument("--every", type=int, default=60, help="one image per slot of N minutes, 0 = all (default 60)")
@@ -388,9 +443,10 @@ def main():
if system_id is None:
print(f"{station}: unknown system")
continue
ranges = [(start, end, False)] if start else update_ranges(a.out / station, end, tomorrow)
ranges = [(start, end, a.replace)] if start else update_ranges(a.out / station, end, tomorrow)
unit = acquisition_unit(conn, system_id)
conn.rollback()
seen = {}
for range_start, range_end, replace in ranges:
print(f"{station}: from {range_start} to {range_end - timedelta(days=1)} inclusive"
+ ("" if replace else ", missing days only"))
@@ -398,13 +454,45 @@ def main():
if unit is None:
print(f"{station}: no recognised acquisition unit, no displacements, camera or images")
continue
pump_displacements(conn, unit, station, range_start, range_end, a.out, replace)
seen.update(pump_displacements(conn, unit, station, range_start, range_end, a.out, replace))
for flag, folder in ((a.images, "images"), (a.diff, "images_diff")):
if flag:
pump_images(conn, s3, unit, station, range_start, range_end, folder, a.every, a.out)
if not seen:
continue
try:
props = sensors(conn, unit)
except Exception as e: # the next run writes them
print(f"{station} targets: failed ({root_cause(e)}), run again")
continue
finally:
conn.rollback()
t = write_targets(a.out / station, props, seen)
print(f"{station} targets.csv: {len(t)} sensors, {t['distance_mm'].notna().sum()} with distance_mm, "
f"{t['mm_per_px'].notna().sum()} with mm_per_px")
finally:
engine.dispose()
def check():
"""with_mm and write_targets without the database: `uv run --with pandas --with sqlalchemy python
extraction/pump.py --check`."""
import tempfile
props = pd.DataFrame({"sensor_name": ["T1", "T1", "REF", "T2"], "pattern_type": ["target", "target", "global_ref",
"target"], "distance_mm": [20000, 20000, 50000, 1000], "mm_per_px": [7.0, 2.0, 18.0, None]},
index=pd.Index([11, 12, 13, 14], name="sensor_id"))
df = pd.DataFrame({"sensor_id": [12, 13, 14, 12], "corr_x": [0.5, 0.1, 0.1, None], "corr_y": [-1, 0.2, 0.2, None]})
df = with_mm(df, props) # T1 measured by its sensor 12 (2 mm/px), not by its older namesake 11 (7 mm/px)
assert df["corr_x_mm"].tolist()[:2] == [1.0, 1.8] and df["corr_y_mm"].tolist()[:2] == [-2.0, 3.6], df
assert df[["corr_x_mm", "corr_y_mm"]].iloc[2:].isna().all().all(), df # no factor, no corr: empty
with tempfile.TemporaryDirectory() as d:
pd.DataFrame({"sensor_name": ["OLD", "T1"], "pattern_type": "target", "distance_mm": [1, 1],
"mm_per_px": [1.0, 1.0]}).to_csv(Path(d) / "targets.csv", index=False)
t = write_targets(Path(d), props, {"T1": 12, "REF": 13})
assert t.values.tolist() == [["OLD", "target", 1, 1.0], ["REF", "global_ref", 50000, 18.0],
["T1", "target", 20000, 2.0]], t
print("ok")
if __name__ == "__main__":
main()
check() if sys.argv[1:] == ["--check"] else main()
+3 -1
View File
@@ -1,6 +1,7 @@
#!/usr/bin/env bash
# Pumping under WSL. Reads the credentials from the .env at the root of the repository (never versioned),
# runs pump.py in a throwaway uv environment (4 packages), then absolute.py (x / y in absolute).
# runs pump.py in a throwaway uv environment (4 packages), then absolute.py (x / y in absolute) and weather.py (the
# MeteoSwiss weather of the stations with a position, data/<STATION>/weather/).
# extraction/pump.sh update of the tracked stations, images included (the scheduler)
# extraction/pump.sh --from D --to E [SERIAL] any other call: the arguments go to pump.py as they are
# Output appended to out/pump.log; one pumping at a time (lock out/pump.lock).
@@ -21,5 +22,6 @@ flock -n 9 || { echo "A pumping is already running." >&2; exit 1; }
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
python "$here/pump.py" "$@")
python3 "$here/absolute.py" # x / y in absolute on every station
python3 "$here/weather.py" # the weather of every station with a position: its missing and last days
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end"
} 2>&1 | tee -a "$root/out/pump.log"
+2 -2
View File
@@ -6,8 +6,8 @@ LENZSPITZE Crissier
MICHIGAN
NADELHORN Genève
NEBRASKA
NIEDERHORN Toronto
NORDEND Ouffet
NIEDERHORN
NORDEND
WILDSTRUBEL
SUSTENHORN
SNCF01
+81
View File
@@ -0,0 +1,81 @@
"""The weather of each tracked station with a position, into data/<STATION>/weather/. Run by pump.sh after each pumping.
One CSV per day that has data in the station's folder: data/<STATION>/weather/<STATION>_weather_<YYYYMMDD>.csv, the
hourly values of the MeteoSwiss station nearest to the position (weather/meteoswiss.py), UTC, columns: time_utc, position
(the line of stations.txt), place (found), station, station_name, distance_km, height_m, then the measurements
(meteoswiss.PARAMETERS). A day without MeteoSwiss values gets the header only.
Fetched again: a missing day, a day written for another position, the last 2 days (incomplete, not yet quality-checked).
A station without a position (an indoor site) or whose place is not found: no weather/ folder, a line in the log.
Usage: python3 extraction/weather.py [SERIAL...] (default: every tracked station)
"""
import csv
import functools
import re
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path[:0] = [str(ROOT / "extraction"), str(ROOT / "weather")]
import meteoswiss # noqa: E402
import pumping # noqa: E402
DATA = ROOT / "data"
DAY = re.compile(r"(?<!\d)(20\d\d)_?(\d\d)_?(\d\d)(?!\d)") # ALASKA_advanced_20261017.csv, _2026_10_17_00_00 to …, 20261029T1617Z_…
RECENT = timedelta(days=2)
# The files of MeteoSwiss (one per station, decade, recent, now) are read once per run, for every station that shares them.
meteoswiss.fetch = functools.lru_cache(maxsize=None)(meteoswiss.fetch)
def days_with_data(folder):
"""The days (UTC dates) of the files of the station, its weather aside."""
return {datetime(*map(int, m.groups()), tzinfo=timezone.utc) for p in folder.rglob("*")
if p.is_file() and "weather" not in p.relative_to(folder).parts and (m := DAY.search(p.name))}
def position_of(path):
"""The position a weather file was written for; None without a row."""
with open(path, newline="") as f:
return next(csv.DictReader(f), {}).get("position")
def station(serial, position, now):
folder = DATA / serial
wanted = sorted(d for d in days_with_data(folder) if d <= now)
out = folder / "weather"
path = lambda d: out / f"{serial}_weather_{d:%Y%m%d}.csv" # noqa: E731
todo = [d for d in wanted if not path(d).is_file() or now - d < RECENT or position_of(path(d)) not in (position, None)]
if not todo:
return
where, hours = meteoswiss.hourly(position, todo[0], todo[-1] + timedelta(days=1))
meta = [position, where["place"], where["station"], where["name"], f"{where['distance_km']:.1f}", f"{where['height_m']:.0f}"]
out.mkdir(exist_ok=True)
for d in todo:
with open(path(d), "w", newline="") as f:
w = csv.writer(f)
w.writerow(["time_utc", "position", "place", "station", "station_name", "distance_km", "height_m",
*meteoswiss.PARAMETERS.values()])
for t, values in hours.items():
if d <= t < d + timedelta(days=1):
w.writerow([t.strftime("%Y-%m-%dT%H:%MZ"), *meta, *("" if v is None else f"{v:g}" for v in values.values())])
print(f"{serial}: weather of {len(todo)} days, {where['place']} -> MeteoSwiss {where['station']} {where['name']}, "
f"{where['distance_km']:.1f} km")
def main(serials):
tracked, now = pumping.tracked(), datetime.now(timezone.utc)
for serial in serials or tracked:
if not (DATA / serial).is_dir():
continue
if not (position := tracked.get(serial)):
print(f"{serial}: no position in stations.txt, no weather")
continue
try:
station(serial, position, now)
except (LookupError, OSError) as e: # the pumping goes on without the weather of this station
print(f"{serial}: no weather: {e}")
if __name__ == "__main__":
main(sys.argv[1:])
+2 -2
View File
@@ -8,9 +8,9 @@ python3 weather/meteoswiss.py "Genève" 2026-11-15 2026-11-16 --out gve.csv
python3 weather/meteoswiss.py 46.52,6.63 2025-06-01T12:00 2025-06-01T18:00
```
A caller (the Data page of aimsight-app) calls `hourly(place, start, end)` -> `(where, {time: {column: value or None}})`, `where` the place found and the station chosen; `LookupError` for a place not found, `OSError` when MeteoSwiss or swisstopo do not answer.
The pumping writes it into `data/<SERIAL>/weather/<SERIAL>_weather_<YYYYMMDD>.csv` (`extraction/weather.py`): the columns below, `station` followed by `position` (the line of `stations.txt`), `place`, `station_name`, `distance_km`, `height_m`; the app and the agents read these files. A caller calls `hourly(place, start, end)` -> `(where, {time: {column: value or None}})`, `where` the place found and the station chosen; `LookupError` for a place not found, `OSError` when MeteoSwiss or swisstopo do not answer.
- `place`: a Swiss place name, resolved by the swisstopo search (`api3.geo.admin.ch`), or coordinates `lat,lon` (WGS84). The resolved place and the chosen station (name, distance, altitude) are written on stderr: check them, the search always returns its best match.
- `place`: a Swiss place name, resolved by the swisstopo search (`api3.geo.admin.ch`), or coordinates `lat,lon` (WGS84). The resolved place and the chosen station (name, distance, altitude) are written on stderr: check them. A best match whose label does not start with the first word asked is refused (`Toronto` gave `Trient (VS)`): a place outside Switzerland has no weather here, MeteoSwiss measures in Switzerland only.
- `start`, `end`: `YYYY-MM-DD` or `YYYY-MM-DDTHH:MM`, in UTC, `end` excluded. The range is in the past.
- The output: a CSV, one row per hour, `time_utc` (start of the hour) and `station`, then `precipitation_mm`, `temperature_c`, `humidity_pct`, `wind_speed_kmh`, `wind_gust_kmh`, `wind_direction_deg`, `pressure_hpa` (at the station, QFE), `sunshine_min`, `global_radiation_wm2`, `snow_depth_cm`. An empty cell: not measured at this station or at this hour.
+10
View File
@@ -22,6 +22,7 @@ import json
import math
import re
import sys
import unicodedata
import urllib.parse
import urllib.request
from datetime import datetime, timedelta, timezone
@@ -75,9 +76,18 @@ def locate(place):
raise LookupError(f"place not found: {place}")
attrs = results[0]["attrs"]
label = re.sub(r"<[^>]+>", "", attrs["label"]).split("\n")[0].strip()
# the search always returns its best match ("Toronto" -> "Trient (VS)"): the label must start with the first word asked
# ponytail: first word only; "lat,lon" for a place whose name differs from its label
if not plain(label).startswith(plain(place.split()[0])):
raise LookupError(f"place not found: {place} (best match: {label})")
return attrs["lat"], attrs["lon"], label
def plain(text):
"""Lower case, without accents: "Genève" -> "geneve"."""
return "".join(c for c in unicodedata.normalize("NFKD", text.lower()) if not unicodedata.combining(c))
def distance_km(lat1, lon1, lat2, lon2):
p1, p2 = math.radians(lat1), math.radians(lat2)
a = (math.sin((p2 - p1) / 2) ** 2