Runs in data_diff/<SERIAL>/runs/<run>/: new_run.sh creates one, run_config.sh <SERIAL> <run> computes it
- The user's layout: the station's root holds only what is the same for every run (images_diff/, images/, metrics/, weather/); each run's folder holds what depends on it: config.yaml, displacement/, camera/, targets.csv, patterns.json. - processing/new_run.sh <SERIAL> <run>: config.yaml of a new, empty run, the cloud's processing written in full (the patterns of sensors.json, the camera, optical's current commit); never over an existing one. - processing/run_config.sh <SERIAL> <run>: reads the run's config.yaml (patterns, camera, emulator.version, checked against $OPTICAL's commit and local changes) and writes its displacement/, camera/, targets.csv (from its patterns) and patterns.json (the patterns as given to the emulator). No configs/, no runs/<name>/ outside data_diff/. - pump_diff no longer writes targets.csv: it depends on the run. JUNGFRAU run01 (2026-09-01, 2026-10-12..16, 525 images, 18 s): 7875 rows as data/, the same rejections, corr within 0.034 px; a run created by new_run.sh and run as is gives run01 exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d3bf4b6860
commit
ca366c150e
8 files changed
+185
-123
No files matched your search
@@ -3,10 +3,8 @@
|
|||||||
The data of the AIMsight stations, in a single copy on the workstation, and their pumping.
|
The data of the AIMsight stations, in a single copy on the workstation, and their pumping.
|
||||||
|
|
||||||
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again.
|
- `data/<SERIAL>/`: filled by the pumping, **out of git** (`metrics/`, `displacement/`, `camera/`, `images/`, `weather/` for a station with a Swiss position…). A lost folder is pumped again.
|
||||||
- `data_diff/<SERIAL>/`: the differential images, full size, with their index and the patterns, to measure them again, and the metrics, IRON images, weather and `targets.csv` as in `data/` (`extraction/pump_diff.sh`), **out of git**; it never touches `data/`. Shared by every configuration. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`.
|
- `data_diff/<SERIAL>/`, **out of git**: pumped by `extraction/pump_diff.sh` (never touching `data/`): the differential images full size under `images_diff/` (with their index, the patterns `sensors.json` and the unit `unit.json`), `images/` (IRON), `metrics/`, `weather/`, as in `data/`: only what is the same for every run. Its runs in `runs/<run>/`, with everything that depends on the run: `config.yaml` (the run's configuration), `displacement/`, `camera/` and `targets.csv` (its result, the files of `data/`), `patterns.json`. Design: `docs/superpowers/specs/2026-11-19-configurations-design.md`.
|
||||||
- `configs/<name>.yaml`: a configuration, what differs from the cloud (patterns by `sensor_name`: a property, or a window redrawn from a DIFF image; the camera correction: algorithm, mode, focal distance, one measurement at a time or the series), in git; `cloud.yaml`, empty, is the cloud and reproduces `data/` (JUNGFRAU 2026-10-12..16: `corr_x/y` within 0.03 px, the same rejections, the camera pose within 5e-4 deg); `jungfrau_example.yaml`, a model.
|
- `processing/run_config.sh <SERIAL> <run> [--from D] [--to E]`: runs `data_diff/<SERIAL>/runs/<run>/config.yaml` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 18 s for 6 days of JUNGFRAU) and writes the run's `displacement/`, `camera/`, `targets.csv` (from its patterns) and `patterns.json` (the patterns exactly as given to the emulator, templates included); `--check` without data. `config.yaml` states what differs from the cloud's processing (an empty file: the cloud's): `patterns` by `sensor_name` (a property, or a window redrawn from a DIFF image), `camera` (algorithm, ransac_threshold, mode, focal_distance, pixel_size, per_epoch) and `emulator.version` (optical's commit: the run stops if `$OPTICAL` is at another one or has local changes); its keys are checked. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml` only, the cloud's processing written in full (every pattern of `sensors.json`, the camera, optical's current commit), to edit before running it; it never overwrites an existing `config.yaml`. The standard configuration reproduces `data/` (JUNGFRAU, 6 days: `corr_x/y` within 0.034 px, the same rejections). A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new). Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`.
|
||||||
- `runs/<name>/<SERIAL>/`: the result of a configuration, **out of git**, the tree of `data/` (`displacement/`, `camera/`, `targets.csv`; `images/`, `metrics/`, `weather/` linked to `data_diff/`): a repository reads it by pointing its `data` link at `~/aimsight-data/runs/<name>` instead of `~/aimsight-data/data`. Beside the stations, `run.yaml` (the configuration as resolved, optical's commit) and `diagnostics/` (the patterns used, the emulator's rows: edge margin, second spot).
|
|
||||||
- `processing/run_config.sh <config> [SERIAL...] [--from D] [--to E]`: computes `runs/<name>/` (optical's emulator and camera correction, from `$OPTICAL`, default `~/optical`; 16 s for 5 days of JUNGFRAU); `--check` without data. Design: `docs/superpowers/specs/2026-11-19-run-config-design.md`. A window redrawn on a target restarts its displacement at zero at the image it is cut from (`x_ref` is kept, the template is new): cut it from an image at the start of the period studied.
|
|
||||||
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations).
|
- `extraction/`: the pumping, read-only on the platform; how to use it: `extraction/README.md`. VS Code tasks "pump" and "pump a range". `extraction/pumping.py`: the tracked stations, launching the pumping and its state, for the callers (aimsight-app, page Stations).
|
||||||
- `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data/<SERIAL>/weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`.
|
- `weather/`: the weather measured by MeteoSwiss at a place over a past range (`weather/meteoswiss.py`), written into `data/<SERIAL>/weather/` at each pumping (`extraction/weather.py`); its doc in `weather/README.md`.
|
||||||
- `.env`: the credentials of the database and of S3, at the root, never versioned.
|
- `.env`: the credentials of the database and of S3, at the root, never versioned.
|
||||||
|
|||||||
@@ -3,6 +3,24 @@
|
|||||||
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
|
2026-11-19. Approved in conversation. Extends `2026-11-19-pump-diff-design.md` (its layout of `data_diff/` changes:
|
||||||
section "data_diff").
|
section "data_diff").
|
||||||
|
|
||||||
|
**Replaced the same day by the user's own layout** (the layout below, `configs/` and `runs/<name>/`, is not used):
|
||||||
|
|
||||||
|
```
|
||||||
|
data_diff/<SERIAL>/
|
||||||
|
images_diff/ (index/, sensors.json, unit.json) images/ metrics/ weather/ pumped, the same for every run
|
||||||
|
runs/<run>/ one run: whatever depends on it
|
||||||
|
config.yaml the run's configuration: patterns (by sensor_name), camera and emulator.version (optical's
|
||||||
|
commit, checked), over the cloud's processing
|
||||||
|
displacement/ camera/ targets.csv its result, the files of data/ (targets.csv from its patterns)
|
||||||
|
patterns.json the patterns exactly as given to the emulator, templates included
|
||||||
|
```
|
||||||
|
|
||||||
|
`processing/run_config.sh <SERIAL> <run>` reads the run's `config.yaml` and writes its `displacement/`, `camera/`,
|
||||||
|
`targets.csv` and `patterns.json`, nothing else. Rule: what is not the same for every possible run is in the run's
|
||||||
|
folder. `processing/new_run.sh <SERIAL> <run>` creates a run, empty: its `config.yaml`, the cloud's processing written
|
||||||
|
in full (patterns of `sensors.json`, camera, optical's current commit); never over an existing one. Note: the reading tools (aimsight-tools `data.find_files`) read a station folder whole, subfolders
|
||||||
|
included: pointed at `data_diff/<SERIAL>/`, they see every run's curves together.
|
||||||
|
|
||||||
## Intent
|
## Intent
|
||||||
|
|
||||||
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
|
Recompute a station's data from its differential images under configurations of our own (patterns redrawn,
|
||||||
|
|||||||
@@ -1,7 +1,15 @@
|
|||||||
# run_config: a configuration recomputed into runs/<name>/
|
# run_config: a run of a station recomputed from its differential images
|
||||||
|
|
||||||
2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`.
|
2026-11-19. Approved in conversation. Step 2 of `2026-11-19-configurations-design.md`.
|
||||||
|
|
||||||
|
**Amended the same day (the user's layout, see the configurations design):** `processing/run_config.sh <SERIAL> <run>`
|
||||||
|
reads `data_diff/<SERIAL>/runs/<run>/config.yaml` (`patterns`, `camera`, `emulator.version`: optical's commit, the run
|
||||||
|
stops if `$OPTICAL` is at another one or has local changes; no `stations` level) and writes only that run's
|
||||||
|
`displacement/`, `camera/`, `targets.csv` (from its patterns; none at the station's root any more) and `patterns.json`
|
||||||
|
(the patterns as given to the emulator); the emulator's other files go through a temporary folder. No `runs/<name>/`,
|
||||||
|
no links, no `run.yaml`. The rest below holds (patterns, correction as
|
||||||
|
the cloud, columns, acceptance).
|
||||||
|
|
||||||
## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`)
|
## What the cloud does (monitoring-dashboard `app/processing/process_shifts.py`, branch `explo`)
|
||||||
|
|
||||||
Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw
|
Per DIFF measurement, alone: the non-archived DIFF patterns (`load_pattern_from_params`) are found in the image; raw
|
||||||
|
|||||||
@@ -58,7 +58,7 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small
|
|||||||
|
|
||||||
## The differential images, into `data_diff/` (`pump_diff.sh`)
|
## The differential images, into `data_diff/` (`pump_diff.sh`)
|
||||||
|
|
||||||
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the configurations, `runs/<name>/`, read like `data/`).
|
`pump_diff.py`, through `pump_diff.sh`, fetches the `OPTICAL_SHIFT-DIFF` images of a station, full size, with what is needed to measure them again, into `data_diff/<SERIAL>/` (out of git), in the tree of `data/`, so that the other repositories read it the same way; it never touches `data/` and does not run `absolute.py` (the recomputed positions are absolute). Its own lock (`out/pump_diff.lock`) and log (`out/pump_diff.log`): a long history does not block the nightly `pump.sh`. Designs: `docs/superpowers/specs/2026-11-19-pump-diff-design.md`, `2026-11-19-configurations-design.md` (the runs, `data_diff/<SERIAL>/runs/<run>/`).
|
||||||
|
|
||||||
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is;
|
- `images_diff/<YYYYMMDD>T<HHMM>Z_<measurement_id>.png`: the S3 object as is;
|
||||||
- `images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON);
|
- `images_diff/index/<SERIAL>_diff_<YYYYMMDD>.csv`: every DIFF measurement of the day in the database, `measurement_id, time_utc` (exact), `measurement_group_id, file, status` (`ok`; `no_uuid`: never reached S3; `s3_missing`: S3 returned nothing), `properties` (JSON);
|
||||||
@@ -66,7 +66,7 @@ Claude may run `pump.sh` itself (user's agreement, 2026-11-12): read-only, small
|
|||||||
- `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm);
|
- `images_diff/unit.json`: the acquisition unit and its properties (`focal_distance`, `pixel_size`, `image_shape` when filled in; none: the platform takes 6.0 mm and 0.0024 mm);
|
||||||
- (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder)
|
- (under `images_diff/`, the reading tools do not see these files: `data.find_files` skips the folder)
|
||||||
- `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`;
|
- `metrics/`, `images/` (IRON, one per `--every` minutes, default 60), `weather/`: with the functions of `pump.py` and `weather.py` (`weather.py --data data_diff`), the same files as in `data/`;
|
||||||
- `targets.csv`: as in `data/`, for the patterns the DIFF images are measured with (not archived); `displacement/` and `camera/` come only from the recomputation of a configuration (`runs/<name>/`), never from the cloud.
|
- `displacement/`, `camera/` and `targets.csv` are not pumped: they depend on the run (its patterns, its settings), each run writes its own in `runs/<run>/` (`processing/run_config.sh`), never from the cloud; a pumping never touches `runs/`.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here
|
extraction/pump_diff.sh --from 2026-09-01 --to 2026-09-08 JUNGFRAU # a range; the history of a station starts here
|
||||||
@@ -74,6 +74,6 @@ extraction/pump_diff.sh JUNGFRAU # from its
|
|||||||
extraction/pump_diff.sh # the same for every station of data_diff/
|
extraction/pump_diff.sh # the same for every station of data_diff/
|
||||||
```
|
```
|
||||||
|
|
||||||
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`; `targets.csv` the same without the two non-DIFF sensors `target_01`, `target_02`.
|
Resuming: a day indexed before the last indexed day is not read again, the last one and later ones are; an image on disk is never downloaded again; every DIFF file is written through a temporary one; the index of a day is written last, after its metrics, IRON and DIFF images, and not at all when its metrics failed (the next run reads it again). `--min-free-gb` (default 50) stops the run before a day. Check without database: `python3 extraction/pump_diff.py --check`. Tried on 2026-11-19: JUNGFRAU 2026-09-01, 94 images (615 MB, 1 min) byte-identical to those of `pump.py --diff`; JUNGFRAU 2026-09-01 and 2026-10-12..16: metrics (168 files), IRON images (133) and weather (6) byte-identical to `data/`.
|
||||||
|
|
||||||
Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.
|
Not every station sends its differential images (2026-11-19, last day of each unit): one per measurement on BREITHORN, CLARIDEN, FLORIDA, JUNGFRAU, MICHIGAN, NADELHORN, NORDEND, SNCF01, and ALASKA / WYOMING since 2026-10-30 (new units); one per hour on LENZSPITZE, and none after 2026-10-13; none on SUSTENHORN; WILDSTRUBEL none since 2026-11-16. The script prints, per station, the range the database holds. Only the unit `pump.py` picks is read: the older units of ALASKA (until 2025-08-21) and WYOMING (until 2025-12-08) are not.
|
||||||
+5
-24
@@ -18,8 +18,9 @@ read it the same way (docs/superpowers/specs/2026-11-19-configurations-design.md
|
|||||||
time_utc, status}] (every property row as stored), rewritten at each run
|
time_utc, status}] (every property row as stored), rewritten at each run
|
||||||
<STATION>/images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties
|
<STATION>/images_diff/unit.json the acquisition unit: id, serial_number, product_number, properties
|
||||||
(under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder)
|
(under images_diff/, the tools of aimsight-tools do not see them: data.find_files skips that folder)
|
||||||
<STATION>/metrics/, images/ (IRON, one per --every minutes), targets.csv (the DIFF patterns), weather/: as pump.py
|
<STATION>/metrics/, images/ (IRON, one per --every minutes), weather/: as pump.py and weather.py write them into
|
||||||
and weather.py write them into data/, with their own functions
|
data/, with their own functions. Not targets.csv: it depends on the run (its patterns), each run writes its own
|
||||||
|
(processing/run_config.py)
|
||||||
|
|
||||||
Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and
|
Resumable: a day whose index exists and is before the last indexed day is not read again; the last indexed day and
|
||||||
the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index
|
the later ones are (pumped while running, a day is incomplete); an image on disk is never downloaded again. An index
|
||||||
@@ -138,17 +139,6 @@ def unit_json(conn, unit):
|
|||||||
"properties": [prop(*r) for r in rows]}
|
"properties": [prop(*r) for r in rows]}
|
||||||
|
|
||||||
|
|
||||||
def diff_patterns(sensors):
|
|
||||||
"""{sensor_name: sensor_id} of the patterns the DIFF images are measured with (sensors_json rows): type
|
|
||||||
OPTICAL_SHIFT-DIFF, a pattern, not archived; for targets.csv."""
|
|
||||||
out = {}
|
|
||||||
for s in sensors:
|
|
||||||
p = {r["name"]: r["value"] for r in s["properties"]}
|
|
||||||
if s["type"] == MTYPE and p.get("pattern") and not p.get("archive"):
|
|
||||||
out[p.get("sensor_name") or s["name"]] = s["id"]
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def to_json(obj):
|
def to_json(obj):
|
||||||
return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode()
|
return json.dumps(obj, indent=1, ensure_ascii=False, default=str).encode()
|
||||||
|
|
||||||
@@ -172,7 +162,7 @@ def main():
|
|||||||
from sqlalchemy import create_engine, text
|
from sqlalchemy import create_engine, text
|
||||||
from sqlalchemy.engine import URL
|
from sqlalchemy.engine import URL
|
||||||
|
|
||||||
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause, sensors, write_targets
|
from pump import DB_NAMES, acquisition_unit, pump_images, pump_metrics, root_cause
|
||||||
|
|
||||||
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
|
tomorrow = datetime.now(UTC).date() + timedelta(days=1)
|
||||||
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
|
start = datetime.strptime(a.start, "%Y-%m-%d").date() if a.start else None
|
||||||
@@ -224,11 +214,8 @@ def main():
|
|||||||
sens = sensors_json(conn, unit)
|
sens = sensors_json(conn, unit)
|
||||||
write_atomic(folder / "images_diff" / "sensors.json", to_json(sens))
|
write_atomic(folder / "images_diff" / "sensors.json", to_json(sens))
|
||||||
write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit)))
|
write_atomic(folder / "images_diff" / "unit.json", to_json(unit_json(conn, unit)))
|
||||||
t = write_targets(folder, sensors(conn, unit), diff_patterns(sens))
|
|
||||||
print(f"{station}: targets.csv, {len(t)} sensors")
|
|
||||||
except Exception as e: # the next run writes them
|
except Exception as e: # the next run writes them
|
||||||
print(f"{station} images range, sensors.json, unit.json, targets.csv: failed ({root_cause(e)}), "
|
print(f"{station} images range, sensors.json, unit.json: failed ({root_cause(e)}), run again")
|
||||||
"run again")
|
|
||||||
finally:
|
finally:
|
||||||
conn.rollback()
|
conn.rollback()
|
||||||
print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}")
|
print(f"{station}: {len(days)} days, from {days[0] if days else '-'} to {days[-1] if days else '-'}")
|
||||||
@@ -313,12 +300,6 @@ def check():
|
|||||||
assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}'
|
assert index[0]["time_utc"] == "2026-11-03T10:00:07.250000+00:00" and index[0]["properties"] == '{"e": 1}'
|
||||||
assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index
|
assert index[1]["properties"] == "" and index[2]["measurement_group_id"] == "", index
|
||||||
assert indexed_days(folder) == {d(3)}
|
assert indexed_days(folder) == {d(3)}
|
||||||
row = lambda name, typ, **p: {"name": name, "type": typ, "id": len(p), # noqa: E731
|
|
||||||
"properties": [{"name": k, "value": v} for k, v in p.items()]}
|
|
||||||
assert diff_patterns([row("u1", MTYPE, pattern="b64", sensor_name="T1"), row("u2", MTYPE, pattern="b64"),
|
|
||||||
row("u3", MTYPE, pattern="b64", sensor_name="OLD", archive="2026-01-01"),
|
|
||||||
row("u4", "camera_movement", sensor_name="CAM"), row("u5", MTYPE, sensor_name="NOPAT")]) \
|
|
||||||
== {"T1": 2, "u2": 1}, "DIFF patterns: by sensor_name, else the sensor's name; archived and others left out"
|
|
||||||
print("ok")
|
print("ok")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Executable
+6
@@ -0,0 +1,6 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Creates a new, empty run: data_diff/<SERIAL>/runs/<run>/config.yaml with the cloud's processing written in full (the
|
||||||
|
# patterns of images_diff/sensors.json, the camera, optical's current commit); no result until it is run.
|
||||||
|
# processing/new_run.sh JUNGFRAU run02
|
||||||
|
# processing/run_config.sh JUNGFRAU run02 then, once config.yaml is edited
|
||||||
|
exec "$(dirname "$0")/run_config.sh" "$@" --new
|
||||||
+139
-89
@@ -1,35 +1,42 @@
|
|||||||
"""Recomputes a configuration from the differential images into runs/<name>/, in the tree of data/, so that the other
|
"""Runs one configuration of a station: data_diff/<SERIAL>/runs/<run>/config.yaml in, the same folder's displacement/,
|
||||||
repositories read it as they read data/ (point their `data` link at runs/<name>). Never writes outside runs/<name>/.
|
camera/ and targets.csv out, in the files of data/. The station's pumped data, the same for every run (images_diff/,
|
||||||
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
|
images/, metrics/, weather/), stay at data_diff/<SERIAL>/ and are only read; whatever depends on the run is in its folder.
|
||||||
|
|
||||||
processing/run_config.sh cloud JUNGFRAU --from 2026-10-12 --to 2026-10-17
|
processing/run_config.sh JUNGFRAU run02 --from 2026-10-12 --to 2026-10-17
|
||||||
processing/run_config.sh configs/mine.yaml the stations the configuration names, all their days
|
processing/run_config.sh JUNGFRAU run02 every indexed day
|
||||||
|
|
||||||
<config>: configs/<name>.yaml (or a path); stations: those given, else those of the configuration; days: those indexed
|
Days: those indexed in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). Running a run again
|
||||||
in data_diff/<SERIAL>/images_diff/index/ (within --from / --to, --to excluded). For each station, runs/<name>/<SERIAL>/
|
replaces its displacement/, camera/, targets.csv and patterns.json; nothing else is written:
|
||||||
is replaced (the other stations of the run are kept):
|
|
||||||
displacement/<S>_advanced_<YYYYMMDD>.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator;
|
displacement/<S>_advanced_<YYYYMMDD>.csv the columns of data/: x, y, error_value, peak..snr from optical's emulator;
|
||||||
relative = x - pattern x; corr from the camera correction; reproj_err,
|
relative = x - pattern x; corr from the camera correction; reproj_err,
|
||||||
inlier of the references; sigma0 of the epoch; corr_*_mm
|
inlier of the references; sigma0 of the epoch; corr_*_mm
|
||||||
camera/<S>_camera_<YYYYMMDD>.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute
|
camera/<S>_camera_<YYYYMMDD>.csv pan, tilt, scale, rotation, sigma0: median / min / max per minute
|
||||||
targets.csv the patterns of the configuration
|
targets.csv as in data/, from the run's patterns: pattern_type, distance_mm, mm_per_px
|
||||||
images/ metrics/ weather/ relative symbolic links to data_diff/<SERIAL>/
|
patterns.json the patterns exactly as given to the emulator (templates included)
|
||||||
and, outside the station folders (the reading tools do not see them): runs/<name>/run.yaml (the configuration as
|
|
||||||
resolved per station, optical's commit, the days) and runs/<name>/diagnostics/ (the patterns used, the emulator's rows).
|
|
||||||
|
|
||||||
A configuration (YAML) states what differs from the cloud: the patterns (by sensor_name: any of their properties, or
|
config.yaml states what differs from the cloud's processing (an empty file: the cloud's), checked key by key:
|
||||||
`redraw: {image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}`, cut as the device's pattern editor does) and
|
patterns: by sensor_name, over images_diff/sensors.json: any existing property, or
|
||||||
the camera correction (`algorithm`, `ransac_threshold`, `mode`: off | rotation | rotation_scale | relative,
|
PM_G: {error_threshold: 0.7} a template redrawn from a DIFF image of images_diff/, as the
|
||||||
`focal_distance`, `pixel_size`, `per_epoch`), common or per station; configs/cloud.yaml, empty, is the cloud.
|
BL_C: {redraw: {image: <file>, corners: [[x0, y0], [x1, y1]]}} device's pattern editor does (x_ref kept)
|
||||||
|
camera: over images_diff/unit.json, over the cloud's defaults
|
||||||
|
algorithm: ransac ransac | umeyama | danish_hybrid | trusted_danish
|
||||||
|
ransac_threshold: 0.99
|
||||||
|
mode: "off" off | rotation | rotation_scale | relative (optical's refine_allpoints)
|
||||||
|
focal_distance: 6.0 pixel_size: 0.0024
|
||||||
|
per_epoch: true one measurement at a time, as the cloud; false: the whole series at once
|
||||||
|
emulator:
|
||||||
|
version: "65a4c70" optical's commit: the run stops if $OPTICAL is at another one or has local changes
|
||||||
|
Design: docs/superpowers/specs/2026-11-19-run-config-design.md.
|
||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import csv
|
import csv
|
||||||
|
import datetime
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
from datetime import datetime, timezone
|
import tempfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
# Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and
|
# Before numpy loads its BLAS, as reprocess_images.py does: the re-measurement's worker processes inherit it, and
|
||||||
@@ -54,7 +61,7 @@ CAM = ["pan", "tilt", "scale", "rotation", "sigma0"]
|
|||||||
DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0,
|
DEFAULTS = {"algorithm": "ransac", "ransac_threshold": 0.99, "mode": "off", "focal_distance": 6.0,
|
||||||
"pixel_size": 0.0024, "per_epoch": True}
|
"pixel_size": 0.0024, "per_epoch": True}
|
||||||
UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size")
|
UNIT_KEYS = ("algorithm", "ransac_threshold", "focal_distance", "pixel_size")
|
||||||
LINKED = ("images", "metrics", "weather")
|
OUTPUTS = ("displacement", "camera") # what a run writes into its folder, and replaces
|
||||||
|
|
||||||
|
|
||||||
def _keys(d, allowed, where):
|
def _keys(d, allowed, where):
|
||||||
@@ -68,11 +75,10 @@ def _keys(d, allowed, where):
|
|||||||
def load_config(path):
|
def load_config(path):
|
||||||
"""The configuration, its keys checked (a typo is an error, not a setting silently ignored)."""
|
"""The configuration, its keys checked (a typo is an error, not a setting silently ignored)."""
|
||||||
c = yaml.safe_load(Path(path).read_text()) or {}
|
c = yaml.safe_load(Path(path).read_text()) or {}
|
||||||
_keys(c, {"stations", "camera"}, "the configuration")
|
_keys(c, {"patterns", "camera", "emulator"}, str(path))
|
||||||
_keys(c.get("camera") or {}, DEFAULTS, "camera")
|
_keys(c.get("camera") or {}, DEFAULTS, f"{path}: camera")
|
||||||
for s, block in (c.get("stations") or {}).items():
|
_keys(c.get("emulator") or {}, {"version"}, f"{path}: emulator")
|
||||||
_keys(block or {}, {"patterns", "camera"}, f"stations.{s}")
|
_keys(c.get("patterns") or {}, set(c.get("patterns") or {}), f"{path}: patterns") # a mapping (names: later)
|
||||||
_keys((block or {}).get("camera") or {}, DEFAULTS, f"stations.{s}.camera")
|
|
||||||
return c
|
return c
|
||||||
|
|
||||||
|
|
||||||
@@ -81,11 +87,10 @@ def latest(rows):
|
|||||||
return {r["name"]: r["value"] for r in rows}
|
return {r["name"]: r["value"] for r in rows}
|
||||||
|
|
||||||
|
|
||||||
def camera_settings(config, station, unit_props):
|
def camera_settings(config, unit_props):
|
||||||
cam = dict(DEFAULTS)
|
cam = dict(DEFAULTS)
|
||||||
cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props})
|
cam.update({k: unit_props[k] for k in UNIT_KEYS if k in unit_props})
|
||||||
cam.update(config.get("camera") or {})
|
cam.update(config.get("camera") or {})
|
||||||
cam.update(((config.get("stations") or {}).get(station) or {}).get("camera") or {})
|
|
||||||
return cam
|
return cam
|
||||||
|
|
||||||
|
|
||||||
@@ -214,15 +219,14 @@ def correct(meas, props, cam):
|
|||||||
return adv, agg.reset_index()
|
return adv, agg.reset_index()
|
||||||
|
|
||||||
|
|
||||||
def write_station(adv, camera, props, dest, station):
|
def write_run(adv, camera, dest, station):
|
||||||
|
"""displacement/ and camera/ of a run, one file per UTC day, replacing the run's previous ones."""
|
||||||
for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")):
|
for folder, df, name in (("displacement", adv, "advanced"), ("camera", camera, "camera")):
|
||||||
(dest / folder).mkdir(parents=True, exist_ok=True)
|
if (dest / folder).exists():
|
||||||
|
shutil.rmtree(dest / folder)
|
||||||
|
(dest / folder).mkdir()
|
||||||
for day, d in df.groupby(df["time_utc"].dt.date):
|
for day, d in df.groupby(df["time_utc"].dt.date):
|
||||||
d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False)
|
d.to_csv(dest / folder / f"{station}_{name}_{day:%Y%m%d}.csv", index=False)
|
||||||
t = pd.DataFrame([{"sensor_name": q.get("sensor_name"), "pattern_type": q.get("pattern_type"),
|
|
||||||
"distance_mm": q.get("distance_mm"), "mm_per_px": q.get("distance_convertion_factor")}
|
|
||||||
for q in props.values()]).sort_values("sensor_name")
|
|
||||||
t.to_csv(dest / "targets.csv", index=False)
|
|
||||||
|
|
||||||
|
|
||||||
def optical_state():
|
def optical_state():
|
||||||
@@ -231,52 +235,107 @@ def optical_state():
|
|||||||
"dirty": bool(git("status", "--porcelain", "--untracked-files=no"))}
|
"dirty": bool(git("status", "--porcelain", "--untracked-files=no"))}
|
||||||
|
|
||||||
|
|
||||||
|
PATTERN_KEYS = ["pattern_type", "corners", "x", "y", "error_threshold", "resolution", "distance_convertion_factor"]
|
||||||
|
CAMERA_NOTES = {"algorithm": "ransac | umeyama | danish_hybrid | trusted_danish", "ransac_threshold": "px, ransac only",
|
||||||
|
"mode": "off | rotation | rotation_scale | relative", "focal_distance": "mm; pan, tilt, sigma0 only",
|
||||||
|
"pixel_size": "mm; pan, tilt, sigma0 only",
|
||||||
|
"per_epoch": "one measurement at a time, as the cloud; false: the whole series at once"}
|
||||||
|
|
||||||
|
|
||||||
|
def new_run(src, name):
|
||||||
|
"""runs/<name>/config.yaml of a new, empty run (no result yet): the cloud's processing written in full, the patterns
|
||||||
|
of images_diff/sensors.json, the camera of unit.json or the cloud's defaults, optical's current commit."""
|
||||||
|
run = src / "runs" / name
|
||||||
|
if (run / "config.yaml").exists():
|
||||||
|
raise FileExistsError(f"{run / 'config.yaml'} exists: choose another name, or edit that file")
|
||||||
|
props = sorted(cloud_properties(json.loads((src / "images_diff" / "sensors.json").read_text())).values(),
|
||||||
|
key=lambda p: (p.get("pattern_type") != "global_ref", str(p.get("sensor_name"))))
|
||||||
|
unit = latest(json.loads((src / "images_diff" / "unit.json").read_text())["properties"])
|
||||||
|
cam, state = camera_settings({}, unit), optical_state()
|
||||||
|
width = max(len(str(p.get("sensor_name"))) for p in props) + 1
|
||||||
|
line = lambda p: f" {str(p.get('sensor_name')) + ':':{width}s} {{" + ", ".join( # noqa: E731
|
||||||
|
f"{k}: {json.dumps(p[k])}" for k in PATTERN_KEYS if k in p) + "}"
|
||||||
|
refs = [line(p) for p in props if p.get("pattern_type") == "global_ref"]
|
||||||
|
targets = [line(p) for p in props if p.get("pattern_type") != "global_ref"]
|
||||||
|
text = f"""# {name}: created on {datetime.date.today()} with the cloud's processing. Change a value, then run it:
|
||||||
|
# processing/run_config.sh {src.name} {name}
|
||||||
|
# The run also writes here targets.csv (as in data/, from these patterns) and patterns.json (the patterns exactly as
|
||||||
|
# given to the emulator, templates included).
|
||||||
|
emulator:
|
||||||
|
version: "{state['commit'][:7]}" # optical's commit (branch {state['branch']}); the run stops if $OPTICAL is
|
||||||
|
# at another one or has local changes
|
||||||
|
patterns:
|
||||||
|
# The {len(props)} patterns of the DIFF images, as in images_diff/sensors.json on {datetime.date.today()} (frozen here: a
|
||||||
|
# later pumping does not change them). The template stays in sensors.json; to change it:
|
||||||
|
# <name>: {{redraw: {{image: <file of images_diff/>, corners: [[x0, y0], [x1, y1]]}}}}
|
||||||
|
# pattern_type: global_ref = a reference of the camera fit, anything else a target. corners and x, y move together
|
||||||
|
# (same window size as the template). resolution: 1/100 px. distance_convertion_factor: mm per px (corr_*_mm only).
|
||||||
|
# references
|
||||||
|
""" + "\n".join(refs) + "\n # targets\n" + "\n".join(targets) + "\ncamera:" + "".join(
|
||||||
|
f"\n {f'{k}: {json.dumps(v)}':26s} # {CAMERA_NOTES[k]}" for k, v in cam.items()) + (
|
||||||
|
"\n # focal_distance, pixel_size: the unit has none on the platform, the cloud's values"
|
||||||
|
if not {"focal_distance", "pixel_size"} & set(unit) else "") + "\n"
|
||||||
|
run.mkdir(parents=True, exist_ok=True)
|
||||||
|
(run / "config.yaml").write_text(text)
|
||||||
|
return run / "config.yaml", state
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
p.add_argument("config", help="name of configs/<name>.yaml, or a path")
|
p.add_argument("station", help="serial number: the folder data_diff/<SERIAL>/")
|
||||||
p.add_argument("stations", nargs="*", help="serial numbers; none: those of the configuration")
|
p.add_argument("run", help="the folder data_diff/<SERIAL>/runs/<run>/, holding config.yaml")
|
||||||
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD")
|
p.add_argument("--from", dest="start", help="first UTC day, YYYY-MM-DD")
|
||||||
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD")
|
p.add_argument("--to", dest="end", help="last UTC day, excluded, YYYY-MM-DD")
|
||||||
p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)")
|
p.add_argument("--jobs", type=int, default=8, help="processes of the re-measurement (default 8; 0: one per core)")
|
||||||
p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff")
|
p.add_argument("--data-diff", type=Path, default=ROOT / "data_diff")
|
||||||
p.add_argument("--out", type=Path, default=ROOT / "runs")
|
p.add_argument("--new", action="store_true", help="create the run, empty: its config.yaml only (processing/new_run.sh)")
|
||||||
a = p.parse_args()
|
a = p.parse_args()
|
||||||
path = Path(a.config) if a.config.endswith((".yaml", ".yml")) or os.sep in a.config \
|
src = a.data_diff / a.station
|
||||||
else ROOT / "configs" / f"{a.config}.yaml"
|
run = src / "runs" / a.run
|
||||||
config, name = load_config(path), path.stem
|
if a.new:
|
||||||
stations = a.stations or list(config.get("stations") or {})
|
try:
|
||||||
if not stations:
|
path, state = new_run(src, a.run)
|
||||||
sys.exit(f"No station: name one ({path} names none)")
|
except FileExistsError as e:
|
||||||
run = a.out / name
|
sys.exit(str(e))
|
||||||
(run / "diagnostics").mkdir(parents=True, exist_ok=True)
|
print(f"{path}: written{', but optical has local changes: the run will stop until they are committed' if state['dirty'] else ''}"
|
||||||
record_path = run / "run.yaml"
|
f"\nRun it: processing/run_config.sh {a.station} {a.run}")
|
||||||
record = yaml.safe_load(record_path.read_text()) if record_path.exists() else {}
|
return
|
||||||
record.update({"configuration": name, "source": str(path), "optical": optical_state()})
|
if not (run / "config.yaml").is_file():
|
||||||
record.setdefault("stations", {})
|
sys.exit(f"No {run / 'config.yaml'}: create the run first, processing/new_run.sh {a.station} {a.run}")
|
||||||
|
config = load_config(run / "config.yaml")
|
||||||
|
version = (config.get("emulator") or {}).get("version")
|
||||||
|
if version is not None: # the code that makes the run is the one config.yaml names
|
||||||
|
state = optical_state()
|
||||||
|
want = subprocess.run(["git", "-C", str(OPTICAL), "rev-parse", "--verify", "--quiet", f"{version}^{{commit}}"],
|
||||||
|
capture_output=True, text=True).stdout.strip()
|
||||||
|
if not want:
|
||||||
|
sys.exit(f"emulator.version {version}: not a commit of {OPTICAL}")
|
||||||
|
if want != state["commit"] or state["dirty"]:
|
||||||
|
sys.exit(f"emulator.version {version}, but {OPTICAL} is at {state['commit'][:7]}"
|
||||||
|
f"{' with local changes' if state['dirty'] else ''}: check {version} out, or change the version")
|
||||||
|
|
||||||
|
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{a.station}_diff_*.csv")
|
||||||
|
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
|
||||||
|
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
|
||||||
|
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
|
||||||
|
if not rows:
|
||||||
|
sys.exit(f"{a.station}: no differential image indexed in the range, nothing done")
|
||||||
|
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
|
||||||
|
unit = json.loads((src / "images_diff" / "unit.json").read_text())
|
||||||
|
cam = camera_settings(config, latest(unit["properties"]))
|
||||||
|
props = apply_patterns(cloud_properties(sensors), config.get("patterns"), src / "images_diff")
|
||||||
|
print(f"{a.station} {a.run}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} "
|
||||||
|
f"patterns, camera {cam}, optical {optical_state()}")
|
||||||
|
|
||||||
import reprocess_images
|
import reprocess_images
|
||||||
|
|
||||||
for station in stations:
|
sp = run / "patterns.json" # kept: the target configuration the run was made with, templates included
|
||||||
src = a.data_diff / station
|
sp.write_text(json.dumps(props, indent=1))
|
||||||
days = sorted(f for f in (src / "images_diff" / "index").glob(f"{station}_diff_*.csv")
|
pd.DataFrame([{"sensor_name": p.get("sensor_name"), "pattern_type": p.get("pattern_type"), # pump.py's columns
|
||||||
if (not a.start or f.stem[-8:] >= a.start.replace("-", ""))
|
"distance_mm": p.get("distance_mm"), "mm_per_px": p.get("distance_convertion_factor")}
|
||||||
and (not a.end or f.stem[-8:] < a.end.replace("-", "")))
|
for p in props.values()]).sort_values("sensor_name").to_csv(run / "targets.csv", index=False)
|
||||||
rows = [r for f in days for r in csv.DictReader(open(f, newline="")) if r["file"]]
|
with tempfile.TemporaryDirectory() as tmp: # the emulator's other files: not kept
|
||||||
if not rows:
|
times, remeasured = Path(tmp) / "times.csv", Path(tmp) / "rows.csv"
|
||||||
print(f"{station}: no differential image indexed in the range, nothing done")
|
|
||||||
continue
|
|
||||||
sensors = json.loads((src / "images_diff" / "sensors.json").read_text())
|
|
||||||
unit = json.loads((src / "images_diff" / "unit.json").read_text())
|
|
||||||
cam = camera_settings(config, station, latest(unit["properties"]))
|
|
||||||
overrides = ((config.get("stations") or {}).get(station) or {}).get("patterns")
|
|
||||||
props = apply_patterns(cloud_properties(sensors), overrides, src / "images_diff")
|
|
||||||
print(f"{station}: {len(rows)} images, {days[0].stem[-8:]} to {days[-1].stem[-8:]}, {len(props)} patterns, "
|
|
||||||
f"camera {cam}")
|
|
||||||
|
|
||||||
diag = run / "diagnostics"
|
|
||||||
sp, times, remeasured = (diag / f"{station}_sensor_properties.json", diag / f"{station}_times.csv",
|
|
||||||
diag / f"{station}_remeasured.csv")
|
|
||||||
sp.write_text(json.dumps(props))
|
|
||||||
with open(times, "w", newline="") as fp:
|
with open(times, "w", newline="") as fp:
|
||||||
w = csv.writer(fp)
|
w = csv.writer(fp)
|
||||||
w.writerow(["file", "time_utc"])
|
w.writerow(["file", "time_utc"])
|
||||||
@@ -284,42 +343,33 @@ def main():
|
|||||||
reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured),
|
reprocess_images.main(["--sensor-properties", str(sp), "--times", str(times), "--out", str(remeasured),
|
||||||
"--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows])
|
"--jobs", str(a.jobs)] + [str(src / "images_diff" / r["file"]) for r in rows])
|
||||||
adv, camera = correct(pd.read_csv(remeasured), props, cam)
|
adv, camera = correct(pd.read_csv(remeasured), props, cam)
|
||||||
|
write_run(adv, camera, run, a.station)
|
||||||
dest = run / station
|
print(f"{a.station} {a.run}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows "
|
||||||
if dest.exists():
|
f"-> {run}")
|
||||||
assert dest.resolve().parent == run.resolve(), dest # only runs/<name>/<SERIAL> is ever removed
|
|
||||||
shutil.rmtree(dest)
|
|
||||||
write_station(adv, camera, props, dest, station)
|
|
||||||
for sub in LINKED:
|
|
||||||
if (src / sub).exists():
|
|
||||||
os.symlink(os.path.relpath(src / sub, dest), dest / sub)
|
|
||||||
record["stations"][station] = {
|
|
||||||
"days": [days[0].stem[-8:], days[-1].stem[-8:]], "images": len(rows), "camera": cam,
|
|
||||||
"patterns": overrides or {}, "date": datetime.now(timezone.utc).isoformat(timespec="seconds")}
|
|
||||||
print(f"{station}: {len(adv)} rows, {adv['corr_x'].notna().sum()} corrected, {len(camera)} camera rows -> {dest}")
|
|
||||||
record_path.write_text(yaml.safe_dump(record, sort_keys=False, allow_unicode=True))
|
|
||||||
|
|
||||||
|
|
||||||
def check():
|
def check():
|
||||||
"""`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical."""
|
"""`processing/run_config.sh --check`: the configuration rules and a synthetic correction through optical."""
|
||||||
import tempfile
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as tmp:
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
f = Path(tmp) / "c.yaml"
|
f = Path(tmp) / "config.yaml"
|
||||||
for text in ("camera: {algoritm: ransac}", "stations: {S: {pattern: {}}}", "stations: {S: {camera: {mod: x}}}",
|
for text in ("camera: {algoritm: ransac}", "pattern: {}", "stations: {S: {}}", "patterns: [PM_G]",
|
||||||
"other: 1"):
|
"emulator: {versio: 65a4c70}"):
|
||||||
f.write_text(text)
|
f.write_text(text)
|
||||||
try:
|
try:
|
||||||
load_config(f)
|
load_config(f)
|
||||||
raise AssertionError(f"accepted: {text}")
|
raise AssertionError(f"accepted: {text}")
|
||||||
except ValueError:
|
except ValueError:
|
||||||
pass
|
pass
|
||||||
f.write_text("camera: {mode: rotation, focal_distance: 6.1}\nstations: {S: {camera: {focal_distance: 6.3}}}")
|
f.write_text("")
|
||||||
|
assert load_config(f) == {} # an empty file: the cloud's processing
|
||||||
|
f.write_text("camera: {mode: rotation, focal_distance: 6.3}\npatterns: {T1: {error_threshold: 0.9}}\n"
|
||||||
|
"emulator: {version: '65a4c70'}")
|
||||||
c = load_config(f)
|
c = load_config(f)
|
||||||
cam = camera_settings(c, "S", {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
|
assert c["emulator"]["version"] == "65a4c70"
|
||||||
|
cam = camera_settings(c, {"focal_distance": 6.2, "pixel_size": 0.003, "other": 1})
|
||||||
assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation",
|
assert (cam["focal_distance"], cam["pixel_size"], cam["mode"], cam["algorithm"]) == (6.3, 0.003, "rotation",
|
||||||
"ransac"), cam
|
"ransac"), cam
|
||||||
assert camera_settings({}, "S", {})["focal_distance"] == 6.0
|
assert camera_settings({}, {})["focal_distance"] == 6.0
|
||||||
|
|
||||||
pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)}
|
pos = {"R1": (100.0, 100.0), "R2": (900.0, 120.0), "R3": (500.0, 800.0), "T1": (300.0, 400.0), "T2": (700.0, 500.0)}
|
||||||
props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y,
|
props = {str(i): {"sensor_name": n, "pattern_type": "global_ref" if n[0] == "R" else "target", "x": x, "y": y,
|
||||||
|
|||||||
@@ -1,7 +1,8 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Recomputes a configuration into runs/<name>/ (processing/run_config.py), in a throwaway uv environment with what
|
# Runs data_diff/<SERIAL>/runs/<run>/config.yaml (processing/run_config.py), in a throwaway uv environment with what
|
||||||
# optical's emulator and camera correction need; optical's code from $OPTICAL (default ~/optical). No credentials.
|
# optical's emulator and camera correction need; optical's code from $OPTICAL (default ~/optical). No credentials.
|
||||||
# processing/run_config.sh <config> [SERIAL...] [--from D] [--to E] [--jobs N]
|
# processing/run_config.sh <SERIAL> <run> [--from D] [--to E] [--jobs N]
|
||||||
|
# processing/run_config.sh <SERIAL> <run> --new (processing/new_run.sh): the run created, empty
|
||||||
# processing/run_config.sh --check
|
# processing/run_config.sh --check
|
||||||
set -u
|
set -u
|
||||||
export PATH="$HOME/.local/bin:$PATH"
|
export PATH="$HOME/.local/bin:$PATH"
|
||||||
|
|||||||
Reference in new issue
Block a user