Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)
The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1b918facd9
commit
81908ca370
9 files changed
+660
-8
No files matched your search
Executable
+24
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
# Pumping of the differential images under WSL, into data_diff/ (never data/). Reads the credentials from the .env at
|
||||
# the root of the repository (never versioned) and runs pump_diff.py in the throwaway uv environment of pump.sh; no
|
||||
# absolute.py nor weather.py (they work on data/).
|
||||
# extraction/pump_diff.sh --from D [--to E] SERIAL... a range; the history of a station starts here
|
||||
# extraction/pump_diff.sh [SERIAL...] from the last indexed day up to now
|
||||
# Output appended to out/pump_diff.log; one at a time (lock out/pump_diff.lock), independent of pump.sh's.
|
||||
set -u
|
||||
export PATH="$HOME/.local/bin:$PATH" # uv, missing from the PATH when the Task Scheduler starts wsl.exe
|
||||
here=$(cd "$(dirname "$0")" && pwd)
|
||||
root=$(dirname "$here")
|
||||
mkdir -p "$root/out"
|
||||
|
||||
exec 9>"$root/out/pump_diff.lock"
|
||||
flock -n 9 || { echo "A pumping of the differential images is already running." >&2; exit 1; }
|
||||
|
||||
{
|
||||
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) pump_diff.py $*"
|
||||
(set -a +u && . "$root/.env" && set +a -u && # +u: an unquoted $ in .env does not stop the load
|
||||
uv run --no-project --quiet --python 3.12 \
|
||||
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
|
||||
python "$here/pump_diff.py" "$@")
|
||||
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end"
|
||||
} 2>&1 | tee -a "$root/out/pump_diff.log"
|
||||
Reference in new issue
Block a user