Files
aimsight-data/extraction/pump_diff.sh
T
maximeandClaude Opus 5.5 81908ca370 Pumping of the differential images: extraction/pump_diff.sh into data_diff/<SERIAL>/, in the tree of data/ (never data/ itself)
The OPTICAL_SHIFT-DIFF images full size, with what measuring them again needs under images_diff/ (index of every
measurement per day with its exact time and why an image is missing, sensors.json with every pattern archived
included, unit.json), and the metrics, IRON images, weather and targets.csv as pump.py and weather.py write them into
data/. Its own lock and log; a day is indexed last, and not when its metrics failed; --min-free-gb. pump_metrics now
returns whether every day succeeded; weather.py takes --data. Tried on JUNGFRAU 2026-09-01 and 2026-10-12..16: DIFF
images byte-identical to pump.py --diff, metrics, IRON and weather byte-identical to data/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 09:59:50 +01:00

25 lines
1.3 KiB
Bash
Executable File

#!/usr/bin/env bash
# Pumping of the differential images under WSL, into data_diff/ (never data/). Reads the credentials from the .env at
# the root of the repository (never versioned) and runs pump_diff.py in the throwaway uv environment of pump.sh; no
# absolute.py nor weather.py (they work on data/).
# extraction/pump_diff.sh --from D [--to E] SERIAL... a range; the history of a station starts here
# extraction/pump_diff.sh [SERIAL...] from the last indexed day up to now
# Output appended to out/pump_diff.log; one at a time (lock out/pump_diff.lock), independent of pump.sh's.
set -u
export PATH="$HOME/.local/bin:$PATH" # uv, missing from the PATH when the Task Scheduler starts wsl.exe
here=$(cd "$(dirname "$0")" && pwd)
root=$(dirname "$here")
mkdir -p "$root/out"
exec 9>"$root/out/pump_diff.lock"
flock -n 9 || { echo "A pumping of the differential images is already running." >&2; exit 1; }
{
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) pump_diff.py $*"
(set -a +u && . "$root/.env" && set +a -u && # +u: an unquoted $ in .env does not stop the load
uv run --no-project --quiet --python 3.12 \
--with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \
python "$here/pump_diff.py" "$@")
echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end"
} 2>&1 | tee -a "$root/out/pump_diff.log"