Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)

The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every
measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived
included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into
data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now
returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF
images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-19 09:59:50 +01:00
1 parent 1b918facd9
commit 81908ca370
9 files changed
+660 -8

No files matched your search

+12 -7
View File
@@ -7,7 +7,8 @@ hourly values of the MeteoSwiss station nearest to the position (weather/meteosw
Fetched again: a missing day, a day written for another position, the last 2 days (incomplete, not yet quality-checked).
A station without a position (an indoor site) or whose place is not found: no weather/ folder, a line in the log.
Usage: python3 extraction/weather.py [SERIAL...] (default: every tracked station)
Usage: python3 extraction/weather.py [--data DIR] [SERIAL...] (default: every tracked station, into data/; --data:
another root with the tree of data/, e.g. data_diff/, which pump_diff.py passes)
"""
import csv
import functools
@@ -40,8 +41,8 @@ def position_of(path):
return next(csv.DictReader(f), {}).get("position")
def station(serial, position, now):
folder = DATA / serial
def station(serial, position, now, data=DATA):
folder = data / serial
wanted = sorted(d for d in days_with_data(folder) if d <= now)
out = folder / "weather"
path = lambda d: out / f"{serial}_weather_{d:%Y%m%d}.csv" # noqa: E731
@@ -63,19 +64,23 @@ def station(serial, position, now):
f"{where['distance_km']:.1f} km")
def main(serials):
def main(serials, data=DATA):
tracked, now = pumping.tracked(), datetime.now(timezone.utc)
for serial in serials or tracked:
if not (DATA / serial).is_dir():
if not (Path(data) / serial).is_dir():
continue
if not (position := tracked.get(serial)):
print(f"{serial}: no position in stations.txt, no weather")
continue
try:
station(serial, position, now)
station(serial, position, now, Path(data))
except (LookupError, OSError) as e: # the pumping goes on without the weather of this station
print(f"{serial}: no weather: {e}")
if __name__ == "__main__":
main(sys.argv[1:])
args = sys.argv[1:]
if args[:1] == ["--data"]:
main(args[2:], Path(args[1]))
else:
main(args)