diff --git a/extraction/README.md b/extraction/README.md index 5a286a7..cbdffb0 100644 --- a/extraction/README.md +++ b/extraction/README.md @@ -14,11 +14,11 @@ Resuming: with `--from`, a file already in `data//` is not asked for aga ## Running (under WSL) -`pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `lenzspitze_absolute.py`. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump. +`pump.sh` reads the credentials from the `.env` at the root of the repository (PostgreSQL database and S3; never versioned, Claude may not read it), installs the 4 packages needed (`sqlalchemy`, `psycopg2-binary`, `pandas`, `boto3`) in a throwaway `uv` environment (Python 3.12), runs `pump.py`, then `absolute.py`. Its output is appended to `out/pump.log`; a second run during a pumping stops at once (lock `out/pump.lock`). The lock holds for this folder only: the other repositories read `data/` through a link, they do not pump. Variables of the `.env`: `AIMSIGHT_DATABASE_USER`, `_PSWD`, `_TURL`, `_NAME`, `_PORT`; for the images, `AIMSIGHT_S3_URL`, `_ACCESS_KEY`, `_SECRET_KEY`, `_BUCKET`. -1. **A trial** (1 station, 2 days, in a throwaway folder): `extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND` (6 s on 2026-11-12). `lenzspitze_absolute.py` then converts `data/LENZSPITZE`, not the `--out` folder: no effect if `data/LENZSPITZE` is already converted. +1. **A trial** (1 station, 2 days, in a throwaway folder): `extraction/pump.sh --from 2026-11-09 --to 2026-11-11 --images --out /tmp/pump-trial NORDEND` (6 s on 2026-11-12). `absolute.py` then converts `data/`, not the `--out` folder: no effect on what is already converted. 2. **Updating the tracked stations**: `extraction/pump.sh` (without argument: `--images --delay 300`), or the VS Code task **pump**. The script says, for each station, the range it pumps. 3. **A chosen range** (history, failed day), with 3 more days before (the history of the detectors): `extraction/pump.sh --from 2026-10-02 --to 2026-10-26 --images ` (`--to` excluded; several stations: ` `). Help: `extraction/pump.sh -h`. @@ -52,4 +52,4 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small - A day failed on a timeout (`QueryCanceled: … statement timeout`): run it again alone with `--delay 300`. The slowness comes from the database, not from the volume. - An hour without an image can have its measurement: the image never reached S3 (empty `measurement_uuid`, unfinished upload). The script skips it and takes the next image of the slot if there is one. - A past day, pumped before the station had finished sending its data (outage, late synchronisation), stays incomplete: delete its files in `data//` and run again. -- LENZSPITZE: its exports have relative `x` / `y` and an empty `pattern_type`; `pump.sh` then runs `lenzspitze_absolute.py` (provisional; details in `~/aimsight-labeling/knowledge/displacement-columns.md`). +- Relative `x` / `y` (LENZSPITZE, SUSTENHORN, NADELHORN `c*_mon`): after each pumping, `pump.sh` runs `absolute.py`, which converts every sensor whose `x` / `y` medians are under 100 px, from `data//*_patterns.json` and the offset of the station (only LENZSPITZE has both today); a relative sensor it cannot place is dropped (details in `~/aimsight-labeling/knowledge/displacement-columns.md`). diff --git a/extraction/absolute.py b/extraction/absolute.py new file mode 100644 index 0000000..84eaf58 --- /dev/null +++ b/extraction/absolute.py @@ -0,0 +1,92 @@ +"""x / y of the advanced exports as absolute positions in the image, on every station. Run by pump.sh after each pumping. + +A sensor of a file is relative when the medians of its x and y are both under RELATIVE_PX (relative: up to ~40 px seen, +NADELHORN c*_mon, SUSTENHORN, LENZSPITZE; absolute: 429 px and more). Rewrites in place data//displacement/*.csv: + x = x_pattern + DX + x_csv, y = y_pattern + DY + y_csv, where (x_pattern, y_pattern) is the centre of the pattern in + data//*_patterns.json and (DX, DY) the offset between the frame of the JSON and the IRON images (OFFSETS); + an empty pattern_type is filled from the JSON. +A relative sensor without a position (no JSON, no offset, missing from the JSON) is dropped: its rows are removed. +A file with no relative sensor is left as it is (already converted). Pumping again restores the originals. + +Usage: python3 extraction/absolute.py [DATA] (default: the repository's data/) + python3 extraction/absolute.py --check (a check of the conversion, without touching data/) +""" +import csv +import glob +import json +import os +import statistics +import sys + +RELATIVE_PX = 100 +# Measured on 2026-11-04 on LENZSPITZE: median (spot − pattern of the JSON) over 9 night images from 17.10 to 03.11, 25 +# to 34 matched targets each, stable within 0.5 px. Includes the constant offset spot − (x, y) already seen on NORDEND +# (displacement-columns.md). A station with a JSON but no offset here: its relative sensors are dropped. +OFFSETS = {"LENZSPITZE": (330.5, 15.8)} + + +def relative_sensors(rows): + xy = {} + for r in rows: + if r["x"] and r["y"]: + xy.setdefault(r["sensor_name"], []).append((float(r["x"]), float(r["y"]))) + return {s for s, v in xy.items() + if max(abs(statistics.median(p[0] for p in v)), abs(statistics.median(p[1] for p in v))) < RELATIVE_PX} + + +def convert(rows, patterns, offset): + """-> (rows with absolute x / y, relative sensors dropped); None if nothing is relative.""" + relative = relative_sensors(rows) + if not relative: + return None + out = [] + for r in rows: + if r["sensor_name"] in relative: + m = patterns.get(r["sensor_name"].strip()) + if m is None or offset is None: + continue + r["pattern_type"] = r["pattern_type"] or m.get("pattern_type", "") + if r["x"]: + r["x"] = repr(m["x"] + offset[0] + float(r["x"])) + if r["y"]: + r["y"] = repr(m["y"] + offset[1] + float(r["y"])) + out.append(r) + return out + + +def station(folder): + name = os.path.basename(folder) + patterns = {} + for path in glob.glob(os.path.join(folder, "*_patterns.json")): + patterns.update({str(v["sensor_name"]): v for v in json.load(open(path)).values()}) + offset = OFFSETS.get(name) if patterns else None + for path in sorted(glob.glob(os.path.join(folder, "displacement", "*.csv"))): + with open(path, newline="") as f: + reader = csv.DictReader(f) + fields, rows = reader.fieldnames, list(reader) + if not fields or "x" not in fields: + continue # an empty day + out = convert(rows, patterns, offset) + if out is None: + continue + with open(path, "w", newline="") as f: + w = csv.DictWriter(f, fieldnames=fields) + w.writeheader() + w.writerows(out) + print(f"{name} {os.path.basename(path)}: {len(out)} rows kept, {len(rows) - len(out)} relative dropped") + + +if __name__ == "__main__": + if sys.argv[1:] == ["--check"]: + def row(s, x, y, t=""): + return {"sensor_name": s, "x": x, "y": y, "pattern_type": t} + rows = [row("a", "1.5", "-2", ""), row("a", "", "", ""), row("b", "0.1", "0.2"), row("c", "800", "600", "ref")] + out = convert(rows, {"a": {"x": 100, "y": 200, "pattern_type": "target"}}, (10, 20)) + assert out == [row("a", "111.5", "218.0", "target"), row("a", "", "", "target"), row("c", "800", "600", "ref")], out + assert convert(out, {}, None) is None # already absolute: left as it is + assert convert([row("b", "0.1", "0.2")], {"b": {"x": 1, "y": 1}}, None) == [] # no offset: dropped + print("ok") + sys.exit() + data = sys.argv[1] if len(sys.argv) > 1 else os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "data") + for folder in sorted(glob.glob(os.path.join(data, "*", ""))): + station(folder.rstrip(os.sep)) diff --git a/extraction/lenzspitze_absolute.py b/extraction/lenzspitze_absolute.py deleted file mode 100644 index 00c5739..0000000 --- a/extraction/lenzspitze_absolute.py +++ /dev/null @@ -1,49 +0,0 @@ -"""Provisional, LENZSPITZE only: x / y of the advanced exports as absolute positions in the image, as on the other stations. - -On LENZSPITZE, x / y are relative to the reference position of the pattern (medians ~0 px). This script rewrites in place -data/LENZSPITZE/displacement/*.csv: - x = x_pattern + DX + x_csv, y = y_pattern + DY + y_csv, where (x_pattern, y_pattern) is the centre of the pattern in - data/LENZSPITZE/lentzpitze_patterns.json and (DX, DY) the offset between the frame of the JSON and the IRON images; - pattern_type, empty in these exports, is filled from the JSON. -The sensors missing from the JSON (1023, 8019): x / y emptied (position unknown); 8019 gets global_ref (it has reproj_err). -A file whose pattern_type is already filled is left as it is (already converted). Pumping again restores the originals. - -Usage: python3 extraction/lenzspitze_absolute.py -""" -import csv -import glob -import json -import os - -STATION = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "data", "LENZSPITZE") -# Measured on 2026-11-04: median (spot − pattern of the JSON) over 9 night images from 17.10 to 03.11, 25 to 34 matched -# targets each, stable within 0.5 px. Includes the constant offset spot − (x, y) already seen on NORDEND (displacement-columns.md). -DX, DY = 330.5, 15.8 -TYPE_WITHOUT_PATTERN = {"8019": "global_ref"} # reproj_err and inlier filled: reference of the compensation - -patterns = {str(v["sensor_name"]): v for v in json.load(open(os.path.join(STATION, "lentzpitze_patterns.json"))).values()} - -for path in sorted(glob.glob(os.path.join(STATION, "displacement", "*.csv"))): - with open(path, newline="") as f: - reader = csv.DictReader(f) - fields, rows = reader.fieldnames, list(reader) - if any(r["pattern_type"] for r in rows): - print(f"{os.path.basename(path)}: already converted") - continue - for r in rows: - s = r["sensor_name"].strip() - m = patterns.get(s) - if m is None: - r["x"] = r["y"] = "" - r["pattern_type"] = TYPE_WITHOUT_PATTERN.get(s, "") - continue - r["pattern_type"] = m["pattern_type"] - if r["x"]: - r["x"] = repr(m["x"] + DX + float(r["x"])) - if r["y"]: - r["y"] = repr(m["y"] + DY + float(r["y"])) - with open(path, "w", newline="") as f: - w = csv.DictWriter(f, fieldnames=fields) - w.writeheader() - w.writerows(rows) - print(f"{os.path.basename(path)}: {len(rows)} rows converted") diff --git a/extraction/pump.sh b/extraction/pump.sh index 0ff90c5..c855607 100755 --- a/extraction/pump.sh +++ b/extraction/pump.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Pumping under WSL. Reads the credentials from the .env at the root of the repository (never versioned), -# runs pump.py in a throwaway uv environment (4 packages), then the post-processing of LENZSPITZE. +# runs pump.py in a throwaway uv environment (4 packages), then absolute.py (x / y in absolute). # extraction/pump.sh update of the tracked stations, images included (the scheduler) # extraction/pump.sh --from D --to E [SERIAL] any other call: the arguments go to pump.py as they are # Output appended to out/pump.log; one pumping at a time (lock out/pump.lock). @@ -20,6 +20,6 @@ flock -n 9 || { echo "A pumping is already running." >&2; exit 1; } uv run --no-project --quiet --python 3.12 \ --with sqlalchemy --with psycopg2-binary --with pandas --with boto3 \ python "$here/pump.py" "$@") - [ -f "$root/data/LENZSPITZE/lentzpitze_patterns.json" ] && python3 "$here/lenzspitze_absolute.py" + python3 "$here/absolute.py" # x / y in absolute on every station echo "=== $(date -u +%Y-%m-%dT%H:%MZ) end" } 2>&1 | tee -a "$root/out/pump.log"