Compare commits
21
Commits
c1719cbd1d
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9560502dc5 | ||
|
|
a9948d3404 | ||
|
|
fc305697f7 | ||
|
|
e03d911433 | ||
|
|
f57e6ea045 | ||
|
|
0677c3c341 | ||
|
|
af6cf23f94 | ||
|
|
72ff92126c | ||
|
|
f5ff3192ae | ||
|
|
9900604002 | ||
|
|
7ada30a1cc | ||
|
|
84cfd6925b | ||
|
|
e17202d596 | ||
|
|
342728286f | ||
|
|
7b2fb8fbfc | ||
|
|
7bb0f4e8d1 | ||
|
|
e2c34c711c | ||
|
|
b6f19b17b6 | ||
|
|
4fc86e68f5 | ||
|
|
0705e71655 | ||
|
|
6ed4c420e1 |
No files matched your search
@@ -1,6 +1,6 @@
|
|||||||
# CLAUDE.md
|
# CLAUDE.md
|
||||||
|
|
||||||
tools: the reading harness of the AIMsight agents (`README.md`), and nothing else. The user guide is `GUIDE.md`; a new tool or a new option goes into it in the same change. `DATA.md` (what the data are) is maintained by hand by the user; `GUIDE.md` is the upgrader's (`agents` repository, run by Optimizer's `optimize` skill).
|
tools: the reading harness of the AIMsight agents (`README.md`), and nothing else. The user guide is `GUIDE.md`; a new tool or a new option goes into it in the same change, a removed one leaves it in the same change. `DATA.md` (what the data are) is maintained by hand by the user; `GUIDE.md` is the upgrader's (`agents` repository, run by Optimizer's `optimize` skill).
|
||||||
|
|
||||||
- **Checking a change**: rerun the touched tool from a calling repository (aimsight-app: `python3 tools/scripts/<tool>.py data/<STATION> --start … --end …`), and the commands of the guide that use it; `python3 tools/scripts/quality.py --self-test` (from the calling repository).
|
- **Checking a change**: rerun the touched tool from a calling repository (aimsight-app: `python3 tools/scripts/<tool>.py data/<STATION> --start … --end …`), and the commands of the guide that use it; `python3 tools/scripts/quality.py --self-test` (from the calling repository).
|
||||||
|
|
||||||
@@ -10,9 +10,10 @@ Rules:
|
|||||||
- No import from the detectors (`detectors/`).
|
- No import from the detectors (`detectors/`).
|
||||||
- Standard library; matplotlib, through the caller's `.venv/bin/python`, for the figures only.
|
- Standard library; matplotlib, through the caller's `.venv/bin/python`, for the figures only.
|
||||||
- Never a value from `metrics/` that is an identifier (`SIM_CCID`, `ip*`); read-only on `data/`; never read `.env`.
|
- Never a value from `metrics/` that is an identifier (`SIM_CCID`, `ip*`); read-only on `data/`; never read `.env`.
|
||||||
- Break nothing: existing options and outputs stay, the analyses' reports cite commands that must keep working.
|
- Break nothing by accident: the analyses' reports cite commands, so a change keeps the existing options and outputs working; only a removal (below) may break one, and its report gives the command that replaces it.
|
||||||
- A change to improve the tools touches `scripts/` and `GUIDE.md` only; `README.md`, `CLAUDE.md` and `DATA.md` (the user's) are not part of it.
|
- A change to improve the tools touches `scripts/` and `GUIDE.md` only; `README.md`, `CLAUDE.md` and `DATA.md` (the user's) are not part of it.
|
||||||
- Remove nothing: no tool, no option, no output, no sentence of the guide; a tool that did not help on one range may help on another, and a single analysis is not enough to clean the list. Optimise or add.
|
- Add, change or remove. A sentence of the guide is changed or removed freely. A tool (or one of its options or outputs) is removed only for a good reason, said in the change's report: it is superseded (another tool, or an option of one, now shows the same: nothing is lost), or it is wrong (false values, a judgement in its output) and fixing it costs more than it is worth. Not a reason: it did not help on one range; it may help on another, and a single analysis is not enough to clean the list. Never removed by a change to improve the tools, only proposed to the user: `data.py`, `common.py`, a tool named outside this repository (`README.md`, aimsight-app's `instructions/analyst.md`).
|
||||||
- The simplest first: a sentence of the guide or a better default; otherwise an option of an existing tool; a new tool as a last resort. A new script has a docstring at the top, like the others, and enters the table of `GUIDE.md` in the same change.
|
- The simplest first: a sentence of the guide or a better default; otherwise an option of an existing tool; a new tool as a last resort. A new script has a docstring at the top, like the others, and enters the table of `GUIDE.md` in the same change.
|
||||||
- `scripts/data.py` is the common reader of the app and the tools: touch it only to fix a reading error. The app passes these measurements to the detectors (the events of the Data page), which otherwise read with `detectors/scripts/data.py` (benchmark, Optimizer): a reading fix here is carried over there, otherwise the page and the benchmark diverge silently. A change to `data.py` is said at the top of the change's report; the same fix in `detectors/scripts/data.py` is left to the user, named in that report.
|
- `scripts/data.py` is the one reader of the station data, shared by the app, the tools and the detectors (which import it: benchmark, Optimizer): touch it only to fix a reading error, never remove or rename a public name. A change to it changes the detectors' benchmark: rerun `python3 detectors/bench/regression.py --labels labeling/auto --data data` from aimsight-app, and say the change at the top of the change's report. Standard library only, no import from the other scripts.
|
||||||
|
- `scripts/common.py`: what the tools share beyond reading (columns, steps, noise, sensor list, runs, file names); standard library, no matplotlib (some tools run with `python3`). A tool imports `data` and `common`, never another tool.
|
||||||
- Every time is in UTC. Commits by path, never `git add -A`. Language: English.
|
- Every time is in UTC. Commits by path, never `git add -A`. Language: English.
|
||||||
@@ -13,6 +13,7 @@ What the files of a station folder are, and what each column means. How to read
|
|||||||
| `x`, `y` | raw position in the image [px]. Not the centroid of the spot: the offset centroid − (`x`, `y`) is constant per sensor by day and differs between sensors; it changes at night, when the spot is saturated. Probably the position of the reference pattern |
|
| `x`, `y` | raw position in the image [px]. Not the centroid of the spot: the offset centroid − (`x`, `y`) is constant per sensor by day and differs between sensors; it changes at night, when the spot is saturated. Probably the position of the reference pattern |
|
||||||
| `relative_x`, `relative_y` | relative displacement before the camera compensation [px]; not `x − x0` |
|
| `relative_x`, `relative_y` | relative displacement before the camera compensation [px]; not `x − x0` |
|
||||||
| `corr_x`, `corr_y` | the position of the target once the estimated camera movement is removed: **the curves that are labelled**. In px, not mm (below) |
|
| `corr_x`, `corr_y` | the position of the target once the estimated camera movement is removed: **the curves that are labelled**. In px, not mm (below) |
|
||||||
|
| `corr_x_mm`, `corr_y_mm` | `corr_x`, `corr_y` in mm: × the sensor's `mm_per_px` (`targets.csv`, below), computed by `data.load`; empty without a sure factor. The same curves in another unit: the noise threshold of a label is read in noises, whatever the unit; mm say how far a target moves, for the reader |
|
||||||
| `error_value` | recognition error of the pattern, not an uncertainty |
|
| `error_value` | recognition error of the pattern, not an uncertainty |
|
||||||
| `error_threshold` | specific to each sensor (0.3 to 0.7). Above it, the measurement is rejected: the row and its `error_value` stay, `corr_x` and `corr_y` are empty, not interpolated |
|
| `error_threshold` | specific to each sensor (0.3 to 0.7). Above it, the measurement is rejected: the row and its `error_value` stay, `corr_x` and `corr_y` are empty, not interpolated |
|
||||||
| `reproj_err`, `inlier` | references only: residual and contribution of each reference in the compensation |
|
| `reproj_err`, `inlier` | references only: residual and contribution of each reference in the compensation |
|
||||||
@@ -24,6 +25,14 @@ What the files of a station folder are, and what each column means. How to read
|
|||||||
- **LENZSPITZE**: its export gives `x`, `y` relative to the pattern; the extraction rewrites them in image px and fills `pattern_type`. Sensors 1023 and 8019 have no pattern: their `x`, `y` are empty. The tools' crosses may sit slightly off the spots there.
|
- **LENZSPITZE**: its export gives `x`, `y` relative to the pattern; the extraction rewrites them in image px and fills `pattern_type`. Sensors 1023 and 8019 have no pattern: their `x`, `y` are empty. The tools' crosses may sit slightly off the spots there.
|
||||||
- `inventory.py` says, on a batch, which columns carry values.
|
- `inventory.py` says, on a batch, which columns carry values.
|
||||||
|
|
||||||
|
## The targets
|
||||||
|
|
||||||
|
`targets.csv`, from the pumping: one line per sensor measured (targets and references), `sensor_name`, `pattern_type`, `distance_mm` (from the camera) and `mm_per_px` (`distance_convertion_factor`, the factor of the dashboard's exports in mm), their current values. Read by `data.load_targets`, shown by `inventory.py` (sensor table).
|
||||||
|
- The references are in it too (`pattern_type` `global_ref`…), with their distance when it is filled in; they are assumed fixed.
|
||||||
|
- `distance_mm` 1000: not filled in on the platform; the line is ignored.
|
||||||
|
- A name may have several lines (a target counted twice, an older configuration of the unit, another lens): when they disagree, `data.load_targets` keeps no value (`inventory.py`: `?(n)`) and that target has no mm; the distance stays when the lines agree on it.
|
||||||
|
- The distance says which targets are near and far. A far target moves few px for many mm: targets are compared with each other in mm, or in px with their distance. A camera that turns shifts every target by about the same px (the mm grow with the distance); a movement of the structure of the same mm gives px that shrink with the distance; a parallax shows between targets at different distances.
|
||||||
|
|
||||||
## The camera pose
|
## The camera pose
|
||||||
|
|
||||||
`camera/<Station>_camera_<YYYYMMDD>.csv`, from the pumping only: the pose at each registration, `time_utc` in UTC, `median_|min_|max_` of `pan`, `tilt`, `rotation` [deg], `scale`, `sigma0`, one row per measurement (min = max). A measurement instant without a row: the camera was not registered.
|
`camera/<Station>_camera_<YYYYMMDD>.csv`, from the pumping only: the pose at each registration, `time_utc` in UTC, `median_|min_|max_` of `pan`, `tilt`, `rotation` [deg], `scale`, `sigma0`, one row per measurement (min = max). A measurement instant without a row: the camera was not registered.
|
||||||
@@ -32,6 +41,10 @@ What the files of a station folder are, and what each column means. How to read
|
|||||||
- **A positive `tilt` means the camera is nosing down** (the expert).
|
- **A positive `tilt` means the camera is nosing down** (the expert).
|
||||||
- Read by `overview.py` and `inventory.py`.
|
- Read by `overview.py` and `inventory.py`.
|
||||||
|
|
||||||
|
## The weather
|
||||||
|
|
||||||
|
`weather/<Station>_weather_<YYYYMMDD>.csv`, from the pumping, for a station with a Swiss position only: the hourly measurements of the nearest MeteoSwiss station, `time_utc` in UTC (start of the hour); `position`, `place`, `station`, `station_name`, `distance_km`, `height_m` say where; then `precipitation_mm`, `temperature_c`, `humidity_pct`, `wind_speed_kmh`, `wind_gust_kmh`, `wind_direction_deg`, `pressure_hpa`, `sunshine_min`, `global_radiation_wm2`, `snow_depth_cm` (empty: not measured there). The weather a few km away, not at the camera: what the images show prevails. Read by `data.load_weather`.
|
||||||
|
|
||||||
## The hourly export
|
## The hourly export
|
||||||
|
|
||||||
`<Station>_<start>_to_<end>_1hour.csv`: one row per hour, targets only; per target `horizontal_displacement`, `vertical_displacement`, their `_min` / `_max`, `_error` (mean `error_value` of the hour), `_count_val` (measurements of the hour, rejected ones included).
|
`<Station>_<start>_to_<end>_1hour.csv`: one row per hour, targets only; per target `horizontal_displacement`, `vertical_displacement`, their `_min` / `_max`, `_error` (mean `error_value` of the hour), `_count_val` (measurements of the hour, rejected ones included).
|
||||||
|
|||||||
@@ -7,28 +7,201 @@ A camera tracks targets (reflectors) fixed on a structure; its own movement is c
|
|||||||
## Read first
|
## Read first
|
||||||
|
|
||||||
- `tools/DATA.md`: the files of a station and the meaning of each column, the image quality metrics, the camera images (**first**);
|
- `tools/DATA.md`: the files of a station and the meaning of each column, the image quality metrics, the camera images (**first**);
|
||||||
- `labeling/reading-rules.md`: what the data mean (the measurement chain, gaps, reading rules, false friends).
|
- `labeling/reading-rules.md`: what the data mean (the measurement chain, gaps, reading rules); the false friends and the phenomena: the cases (`labeling/scripts/cases.py --list`).
|
||||||
|
|
||||||
## The tools
|
## The tools
|
||||||
|
|
||||||
Run them from the root of the calling repository, where `tools/`, `data/` and `labeling/` are links. They return values, never verdicts: you judge.
|
Run them from the root of the calling repository, where `tools/`, `data/` and `labeling/` are links. They return values, never verdicts: you judge. `--out <folder>`: where whoever calls you stores your figures; they give it to you. The method (below) says in which order to run them; each section says what a tool returns and how to read it.
|
||||||
|
|
||||||
| Command | What it returns |
|
| Tool | The question it answers |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `python3 tools/scripts/inventory.py <folder> [--start T] [--end T] [--files]` | Files by type (beyond 5 per type: the count, the first and the last; `--files` for the full list, to be used only to look for a specific file), **what each source contains** over the window (columns of the export, of the pose and of `quality/`, with the number of rows that have a value, "empty" otherwise; names of the system metrics of `metrics/`; images and difference images; names and counts, never values), images: by runs of equal gaps between two images (within 5 min, over the window if given; one line per change of gap: a hole is dated by its two images), camera poses (count, period, measurement instants without registration), period, cadence, missing measurements, sensors (reference or target, error threshold, rejected measurements): these counts cover the **whole period**. With `--start`/`--end` (on a batch: those of the batch), one more section for the window: the missing steps, dated (for all the sensors, or which ones are missing), by sensor, the rejected measurements, dated (runs of consecutive steps), then the **number of references retained by the compensation** (`inlier` = 1) at each instant and the dated runs where some references are not, which ones and why (`r` rejected, `0` discarded by the compensation, `empty`). A measurement is rejected when its `error_value` exceeds the sensor's threshold, **references included**. When `pattern_type` is empty, a sensor is a reference if it carries `reproj_err` or `inlier` (the "pattern_type empty for N sensors" line says so); a column of the export that not all sensors carry says how many carry it. When `inlier` is empty over the window, the retained references are read from the filled `reproj_err` (the title says so); a mark other than `r` or `empty` is the value of `inlier` (e.g. `-1`). "Duplicate readings": an export (often named after the customer's project) whose readings are already in the others; with 0 different values, it changes nothing to the reading, otherwise the tools keep the one of the last file in name order. A run that ends the same day writes only the time of its end. The "Export image quality (peak to snr)" line says whether the six quality metrics are filled (then `quality.py`) or empty. Over a month, the output with `--start`/`--end` exceeds what a call displays: read each section through a pipe to `grep` (e.g. `\| grep -A40 '^Retained references'`, `\| grep '^ PR04 '`) |
|
| [`inventory.py`](#inventory) | What is there, what is missing: files, contents of each source, images, poses, cadence, sensors, rejections |
|
||||||
| `python3 tools/scripts/summary.py <folder> [--column C] [--stat median\|min\|max] [--step 1h] [--level period\|window] [--sensors A,B] [--start T] [--end T] [--instants T1,T2] [--mark]` | Usual level of each sensor (median, MAD, min, max) over the whole period, or over the `--start`/`--end` window with `--level window` (for a batch: a level change within the batch is not compared with weeks of other measurements), then a table by time step. Columns: `error_value`, `corr_x`, `corr_y`, `relative_x`, `relative_y`, `x`, `y`, `sigma0`, `reproj_err`, and the six image quality metrics `peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr` (raw values; against a calm reference, `quality.py`). The sensor header is repeated every 24 lines. `--mark`: `*` after the value when a measurement of the step is rejected (`x`, `y`, `error_value` of a rejected measurement stay filled), `o` when a measurement of the step has inlier = 0. `--instants T1,T2`: only the steps that contain these instants (the values of all the sensors at a few chosen times). With `--step` equal to the cadence, `--level window --mark`, it is the per-instant, per-sensor table of one column: one column per run |
|
| [`summary.py`](#summary) | The usual level of each sensor, then one column by time step |
|
||||||
| `.venv/bin/python tools/scripts/plot.py <folder> [--sensors A,B] [--columns C1,C2] [--start T] [--end T] [--zone T1/T2 ...] [--out D]` | A PNG of the series; each `--zone` is greyed; it prints its path. Columns: those of `summary.py`, quality metrics included (raw values). **Look at it with Read**: you read a chart better than a table |
|
| [`plot.py`](#plot) | The series of chosen sensors and columns, as a PNG |
|
||||||
| `.venv/bin/python tools/scripts/overview.py <folder> [--columns x,y,sigma0] [--column error_value] [--step 15min] [--k 5] [--min 5] [--residuals corr_x,corr_y,error_value\|no] [--outside T1/T2] [--others] [--rigid BASE [--without A,B] [--follow A,B]] [--table T1/T2\|no] [--sensors A,B] [--start T] [--end T] [--out D]` | All the sensors together, each against its median and its MAD over the window. Two PNGs: **common** (median across sensors of `x`, `y`, `sigma0`: with the dispersion across sensors as a grey band; on the `x`, `y` panels, in addition, **each global reference separately** (one line each) and the **median of the targets alone** (black line); and, at each step, the number of sensors whose `--column` exceeds its median by more than `--k` MAD; at the top, if the folder has `camera/`, the **camera pose**, raw values, one point per registration, the line cut at measurement instants without registration) and **multiples** (a small chart per sensor, one scale each: a step of a single sensor shows there). It also prints the **common median table** (one line per `--step`: the time and the deviation from the window median of `x`, `y`, `sigma0`, then `x_targets`, `y_targets` (median of the targets alone), then the raw camera pose, columns `cam_pan`, `cam_tilt`, `cam_rotation`, `cam_scale`, `cam_sigma0`, empty without registration) with the minimum and the maximum of each column and their instant, then those of `x`, `y` of each global reference (marked `(rejected)` when the measurement is, with the number of rejected measurements of the reference), then the measurement instants without pose (dated runs), then the instants where at least `--min` sensors exceed together, with their names. Three more PNGs, **residuals-corr_x**, **residuals-corr_y**, **residuals-error_value** (`--residuals`): a small chart per sensor of the column minus the sensor's own level (its median over the window; with `--outside T1/T2`, that of the rest of the window, outside the moment read), one scale each, zero dotted: a deviation of a few hundredths on `corr`, hidden in the parallel bands of `plot.py`, shows there; for each sensor it prints the level, the own noise (median of the differences between two measurements) and the min/max deviation per step with their instant, in units and in noises; `--residuals no`: none of these PNGs nor these tables. With `--others`, more **residuals-…-others** PNGs: each sensor minus its own median **and minus the median of the other sensors at the same instant** (what all share, the remainder of the compensation of a common movement, cancels; what is specific remains), and, per sensor, the slope and the correlation r of this residual on the common median of `x`, `y`, `sigma0`: a dip that remains after `--others` is not a remainder shared by all; a high slope with r close to ±1 on common `y` says it is proportional to the common movement (gain specific to the sensor), without r or slope it is independent of the movement. With `--rigid BASE`, one more **rigid** PNG: at each measurement, the overall movement of the targets since the instant `BASE` (translation, rotation, scale, estimated by medians: a target that jumps alone does not pull it; `--without A,B` removes targets from it), and, for each sensor, the **residual** `dx`, `dy` (raw position minus its position at `BASE` carried by this movement, in px); panels: `dx`, `dy` of each global reference against the median and the 10th-90th percentile of the targets, `|residual|` of the targets (median, 90th percentile), fitted rotation and scale, `reproj_err` of the references; it prints the table by step (`rot_deg`, `scale`, `tx`, `ty`, `tgt_med`, `tgt_max`, `dx`, `dy` of each reference, and of each sensor of `--follow A,B`: the step-by-step sequence of a target) and, per sensor, min, max and value at the last instant of `dx`, `dy`. Rejected measurements enter neither the fit, nor the band of the targets, `tgt_med`, `tgt_max`, nor the columns of the table: over several days, parked cars or night rejections no longer drown the rest; on the PNG, a rejected reference is a cross (the scale is that of the accepted measurements, a far cross is out of frame); the per-sensor summary gives them separately (`; N rejected: min …, max …`, and `(rejected)` after a last value that is rejected). Unlike the deviation from the median, this residual does not depend on the sensor's place in the image when the scene rotates or changes scale. The tables by step (common median, `--rigid`) repeat their header every 24 lines; `--table T1/T2` prints them only between T1 and T2 (`no`: not at all), levels, min, max, PNGs and the rest remaining over the whole window: over a one-month batch, first read with `--table no`, then rerun with `--table` on the day of a moment (same levels, which a shorter `--start`/`--end` window would not give). The ticks of the "common" PNG are every 1 to 12 h depending on the window, every n days beyond 20 days. The default step is 15 min, tLine truncated
|
| [`overview.py`](#overview) | All the sensors together, against the camera pose: common movement, single sensors, residuals, rigid movement |
|
||||||
| `python3 tools/scripts/ranges.py <folder> [--columns error_value,corr_x,corr_y] [--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N\|DURATION] [--instants N] [--sensors A,B] [--start T] [--end T]` | By column and by sensor, at the measurements (native step, never averaged): level (median of the window), own noise, **largest jump** from one measurement to the next (value, in noises, the two instants); then the **ranges** where the sensor deviates from its level by more than `--k` noises: start, end, number of measurements, extreme (signed, in units and in noises) and its instant; two ranges of the same sign less than `--merge` apart make only one, rejected measurements do not cut a range; column `rejected`: how many of the range's measurements are (and `rejected extreme`), `(rejected: …)` after a largest jump of which one end is. Used to date the start and the end of a sensor's moment without rerunning ten tables. A noise of `0` (and "-" in noises, no range): the values are constant to rounding (deviations below 1e-9), like the `corr` of the references when the compensation retains only two (see Conventions); the same in `overview.py --residuals` and `days.py`. When a long episode skews the window median (the sensor spends half the batch there), `--outside T1/T2` computes level and noise outside the episode; a range that touches the edge of the window is dated at the edge. `--min-measurements N` prints only the ranges of at least N measurements (and says how many out of the total): over a batch of several days, the one-step ranges number in the hundreds and drown the long steps; first read with `--min-measurements 6`, then without. `--rolling N` (N measurements on each side) or `--rolling 1d` (the measurements within a duration): the level of each measurement is the median of its neighbours, without it; a long drift or a step follows this local level, only the deviations shorter than the window remain (the level column of the table and the noise do not change; with `--outside`, only the noise is computed outside the episode). `--instants N`: at the end, the instants where at least N sensors are out of level on one of the columns, with each sensor and its signed deviation in noises (`cx` corr_x, `cy` corr_y, `ev` error_value), the sensors rejected at that instant (among `--sensors`: removing the always-rejected sensors lightens the list), and each reference: `error_value` (`r` if rejected) / inlier |
|
| [`ranges.py`](#ranges) | Dated ranges where each sensor leaves its level, at the native step |
|
||||||
| `.venv/bin/python tools/scripts/days.py <folder> [--columns corr_x,corr_y,error_value] [--night 20-3] [--k 5] [--days 2] [--step 15min] [--sensors A,B] [--bounds [--smoothing N] [--fraction F]] [--start T] [--end T] [--out D]` | What comes back from one day to the next, on accepted measurements only. 1. **Level of each night** (hours `--night` H1-H2 UTC, straddling midnight, dated from the day it starts), by sensor and by column: median, MAD, number of measurements, and the own noise; then, from one night to the next, the **step** (difference of the medians, in units and in noises) and **where it falls** ("between A and B": where the sequence of night measurements of the two nights splits best between the two levels; B at the start of the following night: the cut falls by day, between the two nights). A step of a few noises, a drift from one night to the next, and two columns that step at the same instant read there, even when the sensor has several levels in the batch. 2. **Superimposed days**: PNG **byhour-<column>**, a small chart per sensor, hour of the day on the x axis, one curve per day, each measurement minus the level of the last night started (night greyed, frame from the 1st to the 99th percentile, the title says how many measurements fall outside the frame); table: per `--step` slot of the hour of the day, the sensors whose deviation exceeds `--k` noises, on the same side, at least `--days` days, with the number of days and the median of these deviations. What comes back every day at the same hour, on one sensor or several, groups there, even when it lasts only one to three measurements. 3. **Bounds of each day**, with `--bounds`: for the median of the targets of each column and for the camera pose (`pan`, `tilt`, `rotation`, `scale`, `sigma0`, if `camera/` exists), the deviation from the level of the night that precedes each day (a day goes from the end of one night to the end of the next; with `--sensors`, the median of these sensors, with a single one, its own). Per day: number of measurements, the first and the last; first and last instant beyond `--k` noises, with the measurement just before and just after, at the native step; signed max and min with their instant; the sequence of measurements beyond `--k` noises that surrounds the larger of the two; the following night minus the reference night. A day without measurement says so; a reference night that is not the previous one is named. Each day line starts with the short name of its series (`corr_x`, `pose_tilt`…): over a month, `\| grep '^pose_tilt '` reads one, `\| grep ' 12/10 '` one day. `--smoothing N`: the deviation of each measurement is the median of the N measurements of the day centred on it (noise and `--k` unchanged; PNG: smoothed in blue, raw in grey): an isolated point no longer cuts the sequence around the extreme and no longer makes a first or a last exceedance. `--fraction F`: the threshold of each day is F times the largest |deviation| of that day (printed on its line, `threshold …`), instead of `--k` noises: when the noise at the native step is unrelated to the amplitude of the cycle (very noisy or very smooth pose). PNG **bounds**: the deviation over the batch, first → last exceedance light grey, sequence around the extreme dark grey, max and min marked. The choice of `--night`: hours where the day cycle (`error_value`, camera pose) has subsided. **Look at the PNGs with Read** |
|
| [`days.py`](#days) | What comes back from one day to the next: night levels, days superimposed by hour, bounds of each day |
|
||||||
| `.venv/bin/python tools/scripts/metrics.py <folder> [--names A,B] [--step 1h] [--stat median\|min\|max] [--against temperature] [--smoothing 24h] [--metrics-only] [--start T] [--end T] [--out D]` | The system metrics of `metrics/` (one value per hour), never the identifiers (`SIM_CCID`, `ip*`). Prints the available metrics with their number of values over the window ("empty": none), then a table per `--step`: `temperature` (median of the `sensor_*_deg_c` components, the one the detectors read), each metric of `--names` (default: the `*_deg_c` and `BusVoltage`), then, unless `--metrics-only`, `tgt_corr_x`, `tgt_corr_y` (median over the targets of `corr` minus each target's own level, accepted measurements) and the raw camera pose (`cam_pan` … `cam_sigma0`); an empty cell is skipped, a text metric (`link`, `BatteryState`, `last_sync`) is returned as it is. Then, per column: filled steps, min and max with their instant; the correlation r and the slope of each column on `--against` (with `--smoothing 24h`, on its rolling mean, column `<against>_smooth`); the dated runs of steps without any metric. A PNG: temperatures on one panel (median in black), each other metric on its own, then `tgt_corr_x`, `tgt_corr_y` and the pose, line cut on a gap. **Look at it with Read** |
|
| [`metrics.py`](#metrics) | The system metrics (temperature, voltage…) against the targets and the pose |
|
||||||
| `.venv/bin/python tools/scripts/stations.py <folder> <folder> [...] [--columns temperature,cam_tilt,cam_rotation,cam_scale,tgt_corr_x,tgt_corr_y] [--step 1h] [--stat median\|min\|max] [--table T1/T2\|no] [--start T] [--end T] [--out D]` | **Several stations on the same timeline** (neighbouring stations, same building). Possible columns: `temperature` (as in `metrics.py`), `cam_pan` … `cam_sigma0` (raw pose), `tgt_corr_x`, `tgt_corr_y` (as in `metrics.py`), `rej_targets`, `rej_refs` (number of rejected targets, of rejected references, the largest of the step). Prints per station the period, the cadence and the number of instants; then the **gaps**: the runs of steps without measurement or without temperature, with the stations where they are missing (a gap common to several stations on a single line); then the table by step (one `STATION:column` column each, tabs; `--table` as in `overview.py`); then, per column and per pair of stations, r of the values and r of the steps from one value to the next (a common movement shows on the steps, a common trend on the values), with n. PNG **stations**: a panel per column, a line per station (raw temperature, the rest minus its median), and a band per station of the steps without measurement. Take `--step` equal to the slowest cadence of the stations; over a month at `1h`, first read with `--table no`. **Look at it with Read** |
|
| [`stations.py`](#stations) | Several stations on the same timeline |
|
||||||
| `python3 tools/scripts/quality.py <folder> [--calm T1/T2,T3/T4] [--common] [--table M] [--step 15min] [--threshold 0.5] [--sensors A,B] [--start T] [--end T]` ; `... --labels F [--numbers 3,7] [--margin 60] [--edges]` ; `... --labels F --outside [--min 1] [--except "Thermal displacement"]` ; `.venv/bin/python tools/scripts/quality.py <folder> --figure [--metrics flux,snr,dominance] [--labels F] [--zone T1/T2 ...] [--out D]` ; `python3 tools/scripts/quality.py --self-test` | The six **image quality metrics** (`peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr`; definitions: `tools/DATA.md`), on the stations that have them (otherwise: "No quality metrics", and nothing else). Each measurement becomes a **ratio** to the **calm reference** of its sensor: the median of the metric over its measurements in the `--calm` windows, at the same UTC hour of the day (without `--calm`: the whole folder); 1 = like the reference, < 1 lower, > 1 higher; for `fwhm` and `dominance`, it is the rise that says a degraded image; rejected measurements included. Without options: the folder's maximal `peak`, then per sensor (`r` = reference) the rejected measurements, `pk/max` (median peak over the window divided by this maximum: at 1, the spot touches the image ceiling, its `peak` no longer moves and its `sharpness` follows the exposure: read `flux`, `snr` instead), and for each metric the median ratio and the extreme with its instant; then the median of the targets and of the references. `--common`: per step, the median of the ratios of the targets / of the references for each metric (what affects the whole scene, visibility or turbulence; `--step 1d` to see on one screen on which days the reference holds). `--table M`: per step, the ratio of `M` of each sensor (`*`: a rejected measurement), their median and how many pass `--threshold` (below, or above 1/`--threshold` for `fwhm`, `dominance`): a target that goes out and comes back. `--labels F` (one JSON per line, `start`, `end`, `family`): per numbered label, the median ratio of the targets and of the references over the hour before, the range and the hour after (`tgt 0.96\|0.26\|0.79`, `--margin` minutes), then the sensors **affected** over the range (flux or snr below `--threshold`, or dominance above 1/`--threshold`), with their worst flux, from lowest to highest; `--edges`: per instant of the widened range, the number of affected sensors and the median of the targets of flux, snr, dominance (`<` within the range), to date the first and the last degraded instant. `--outside`: the runs of instants that no label covers (outside `--except` families) where at least `--min` sensors are affected, with the sensors and their worst flux: candidates for forgotten labels, to check on `corr_x`, `corr_y`. `--figure` (instead of the tables): PNG **quality-common** (a panel per metric, log scale: median of the targets in black, of the references in red, grey band from the 10th to the 90th percentile of the targets: a disturbance of the whole scene moves the median, an occultation of a few targets moves only the bottom of the band) and a PNG **quality-<metric>** per `--metrics` (a small chart per sensor, references in red); `--labels` greys each label there in one colour per family, `--zone` ranges. **Look at the PNGs with Read** |
|
| [`quality.py`](#quality) | The six image quality metrics against a calm reference; labels; degraded instants no label covers |
|
||||||
| `.venv/bin/python tools/scripts/thumbnails.py <folder> --sensors A,B --times T1,T2 [--radius 100] [--contrast [MIN,MAX]] [--fixed] [--zone X,Y,W,H] [--out D]` | A sheet of full-resolution thumbnails: one row per sensor, one column per time, centred on the sensor's raw position (`x`, `y`) at each time (the centre of the frame is written under each thumbnail: when `x`, `y` jump, the frame moves by as much, and a spot that enters the frame seems to appear); `--fixed` keeps for each sensor the frame of the first time and marks with a red cross the `x`, `y` of each time; unretouched pixels, the brightness can be compared from one thumbnail to the next; prints for each thumbnail the background (median), the 99th percentile and the peak (grey levels 0 to 255); without an image at the requested time, takes the nearest and prints the gap (the actual time is in the title). A faint or deformed spot is invisible as it is: `--contrast MIN,MAX` puts all the thumbnails in grey with the same stretch (without value: from the 1st percentile to the sheet's maximum, crushed by a saturated pixel; then choose MIN,MAX from the printed background and peak, e.g. `10,40`), `--radius 30` to `50` magnifies; compare the shape and the peak before, during and after an episode, at the same hour of the day. **Look at it with Read**: a whole image read by Read is reduced, a small reflector becomes illegible in it. A frame that the image edge cuts is flagged ("frame cut by the image edge (W × H px)"): on some stations, `x`, `y` are not pixels (values close to 0) and the frame falls in the corner; then look at the scene through `--zone`. `--zone X,Y,W,H` (pixels of the original image: top left corner, width, height, as in `gif.py`): the same zone at each time, side by side (up to 4 per row), same stretch with `--contrast`; `--sensors` becomes optional, and those that are given are marked with a cross and their name. For the scene around the targets (foreground object, headlights, lighting), prefer it to a large `--radius`. Each thumbnail is rendered pixel for pixel up to 1000 px a side, but Read reduces a wide PNG: with a large `--radius` or a large `--zone`, three or four times per sheet |
|
| [`thumbnails.py`](#thumbnails) | Full-resolution thumbnails of the camera images, around sensors or of a zone |
|
||||||
| `.venv/bin/python tools/scripts/gif.py <folder> --start T --end T [--width 720] [--duration 700] [--max 150] [--zone X,Y,W,H] [--out D]` | A GIF of the camera images between `--start` and `--end` inclusive, cropped to `--zone` if needed, set to `--width` px, UTC date and time written at the bottom right; beyond `--max` images, `--max` at regular intervals; prints the time of each image, then the path and the size of the GIF. Used to attach the scene to a reply to the customer; Read does not read a GIF: to look, `thumbnails.py --zone` |
|
| [`gif.py`](#gif) | A GIF of the camera images over a range, for a reply to the customer |
|
||||||
| `ls <folder>/images/ \| grep <YYYYMMDD>T<HHMM>Z`, then Read on the file found (Read takes no pattern) | The whole camera image at that time (UTC), for the scene (occultation, lighting, weather) |
|
| [`ls … \| grep`](#whole-image) | The whole camera image at a time |
|
||||||
|
|
||||||
`--out <folder>`: where whoever calls you stores your figures; they give it to you.
|
### inventory
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 tools/scripts/inventory.py <folder> [--start T] [--end T] [--files]
|
||||||
|
```
|
||||||
|
|
||||||
|
Names and counts, never values. Over the **whole period**:
|
||||||
|
- files by type: beyond 5 per type, the count, the first and the last (`--files`: the full list, only to look for a specific file);
|
||||||
|
- **what each source contains** over the window: the columns of the export, of the pose, of `quality/` and of `weather/` (the MeteoSwiss weather, hourly; a station without a Swiss position has none), with the number of rows that have a value ("empty" otherwise; an export column that not all sensors carry says how many carry it); the names of the system metrics of `metrics/`; the images and the difference images. The "Export image quality (peak to snr)" line says whether the six quality metrics are filled (then `quality.py`) or empty;
|
||||||
|
- images by runs of equal gaps between two images (within 5 min, over the window if given): one line per change of gap, a hole is dated by its two images;
|
||||||
|
- camera poses (count, period, measurement instants without registration), period, cadence, missing measurements, sensors (reference or target, error threshold, rejected measurements, distance from the camera and mm per px of `targets.csv`). When `pattern_type` is empty, a sensor is a reference if it carries `reproj_err` or `inlier` (the "pattern_type empty for N sensors" line says so);
|
||||||
|
- "Duplicate readings": an export (often named after the customer's project) whose readings are already in the others; with 0 different values, it changes nothing to the reading, otherwise the tools keep the one of the last file in name order.
|
||||||
|
|
||||||
|
With `--start`/`--end` (on a batch: those of the batch), a section for the window: the missing steps, dated (for all the sensors, or which ones are missing); by sensor, the rejected measurements, dated (runs of consecutive steps; rejected: `error_value` above the sensor's threshold, **references included**); then the **number of references retained by the compensation** (`inlier` = 1) at each instant and the dated runs where some are not, which ones and why (`r` rejected, `0` discarded by the compensation, `empty`; another mark is the value of `inlier`, e.g. `-1`). When `inlier` is empty over the window, the retained references are read from the filled `reproj_err` (the title says so).
|
||||||
|
|
||||||
|
A run that ends the same day writes only the time of its end. Over a month, the output with `--start`/`--end` exceeds what a call displays: read each section through `grep` (`| grep -A40 '^Retained references'`, `| grep '^ PR04 '`).
|
||||||
|
|
||||||
|
### summary
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 tools/scripts/summary.py <folder> [--column C] [--stat median|min|max] [--step 1h] [--level period|window] [--sensors A,B] [--start T] [--end T] [--instants T1,T2] [--mark]
|
||||||
|
```
|
||||||
|
|
||||||
|
The usual level of each sensor (median, MAD, min, max) over the whole period, or over the `--start`/`--end` window with `--level window` (for a batch: a level change within the batch is not compared with weeks of other measurements); then a table by time step, the sensor header repeated every 24 lines.
|
||||||
|
- Columns: `error_value`, `corr_x`, `corr_y`, `relative_x`, `relative_y`, `x`, `y`, `sigma0`, `reproj_err`, `corr_x_mm`, `corr_y_mm` (the `corr` in mm, empty without a sure factor: `tools/DATA.md`, "The targets"), and the six image quality metrics `peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr` (raw values; against a calm reference, `quality.py`).
|
||||||
|
- `--mark`: `*` after the value when a measurement of the step is rejected (`x`, `y`, `error_value` of a rejected measurement stay filled), `o` when one has inlier = 0.
|
||||||
|
- `--instants T1,T2`: only the steps that contain these instants (the values of all the sensors at a few chosen times).
|
||||||
|
- With `--step` equal to the cadence, `--level window --mark`: the per-instant, per-sensor table of one column, one column per run.
|
||||||
|
|
||||||
|
### plot
|
||||||
|
|
||||||
|
```
|
||||||
|
.venv/bin/python tools/scripts/plot.py <folder> [--sensors A,B] [--columns C1,C2] [--start T] [--end T] [--zone T1/T2 ...] [--out D]
|
||||||
|
```
|
||||||
|
|
||||||
|
A PNG of the series, each `--zone` greyed; prints its path. Columns: those of `summary.py`, quality metrics included (raw values).
|
||||||
|
|
||||||
|
### overview
|
||||||
|
|
||||||
|
```
|
||||||
|
.venv/bin/python tools/scripts/overview.py <folder> [--columns x,y,sigma0] [--column error_value] [--step 15min] [--k 5] [--min 5] [--residuals corr_x,corr_y,error_value|no] [--outside T1/T2] [--others] [--rigid BASE [--without A,B] [--follow A,B]] [--table T1/T2|no] [--sensors A,B] [--start T] [--end T] [--out D]
|
||||||
|
```
|
||||||
|
|
||||||
|
All the sensors together, each against its median and its MAD over the window.
|
||||||
|
|
||||||
|
PNGs:
|
||||||
|
- **common**: at the top, if the folder has `camera/`, the **camera pose** (raw values, one point per registration, the line cut at measurement instants without registration); the median across sensors of `x`, `y`, `sigma0`, the dispersion across sensors as a grey band; on the `x`, `y` panels, also **each global reference separately** (one line each) and the **median of the targets alone** (black line); at each step, the number of sensors whose `--column` exceeds its median by more than `--k` MAD. Ticks every 1 to 12 h depending on the window, every n days beyond 20 days.
|
||||||
|
- **multiples**: a small chart per sensor, one scale each: a step of a single sensor shows there.
|
||||||
|
- **residuals-corr_x**, **-corr_y**, **-error_value** (`--residuals`; `no`: none of these PNGs nor their tables): a small chart per sensor of the column minus the sensor's own level (its median over the window; with `--outside T1/T2`, that of the rest of the window, outside the moment read), one scale each, zero dotted: a deviation of a few hundredths on `corr`, hidden in the parallel bands of `plot.py`, shows there.
|
||||||
|
- **residuals-…-others** (`--others`): each sensor minus its own median **and minus the median of the other sensors at the same instant**: what all share (the remainder of the compensation of a common movement) cancels, what is specific remains.
|
||||||
|
- **rigid** (`--rigid BASE`): at each measurement, the overall movement of the targets since the instant `BASE` (translation, rotation, scale, estimated by medians: a target that jumps alone does not pull it; `--without A,B` removes targets from it) and, for each sensor, the **residual** `dx`, `dy` (raw position minus its position at `BASE` carried by this movement, in px). Panels: `dx`, `dy` of each global reference against the median and the 10th-90th percentile of the targets, `|residual|` of the targets (median, 90th percentile), fitted rotation and scale, `reproj_err` of the references; a rejected reference is a cross (the scale is that of the accepted measurements, a far cross is out of frame).
|
||||||
|
|
||||||
|
Printed:
|
||||||
|
- the **common median table**, one line per `--step`: the time, the deviation from the window median of `x`, `y`, `sigma0`, then `x_targets`, `y_targets` (median of the targets alone), then the raw camera pose `cam_pan`, `cam_tilt`, `cam_rotation`, `cam_scale`, `cam_sigma0` (empty without registration); the minimum and the maximum of each column with their instant; then those of `x`, `y` of each global reference (`(rejected)` when the measurement is, with the reference's number of rejected measurements); the measurement instants without pose (dated runs); the instants where at least `--min` sensors exceed together, with their names (`--min 1`: also a single sensor that leaves its level, absent from the default list of 5 sensors);
|
||||||
|
- `--residuals`: per sensor, the level, the own noise (median of the differences between two measurements), the min/max deviation per step with their instant, in units and in noises;
|
||||||
|
- `--others`: per sensor, the slope and the correlation r of the residual on the common median of `x`, `y`, `sigma0`;
|
||||||
|
- `--rigid`: the table by step (`rot_deg`, `scale`, `tx`, `ty`, `tgt_med`, `tgt_max`, `dx`, `dy` of each reference and of each sensor of `--follow A,B`: the step-by-step sequence of a target) and, per sensor, min, max and value at the last instant of `dx`, `dy`. Rejected measurements enter neither the fit, nor the band of the targets, `tgt_med`, `tgt_max`, nor the columns of the table (over several days, parked cars or night rejections no longer drown the rest); the per-sensor summary gives them separately (`; N rejected: min …, max …`, and `(rejected)` after a last value that is rejected).
|
||||||
|
|
||||||
|
The tables by step (common median, `--rigid`) repeat their header every 24 lines; `--table T1/T2` prints them only between T1 and T2 (`no`: not at all), levels, min, max, PNGs and the rest remaining over the whole window: over a one-month batch, first `--table no`, then `--table` on the day of a moment (same levels, which a shorter `--start`/`--end` window would not give). The default step is 15 min, the cadence of the measurements: a one-hour step flattens a peak of one to four steps, dates to the hour and erases an agitation of `x`, `y`; do not widen it to read a batch.
|
||||||
|
|
||||||
|
Reading, in this order:
|
||||||
|
1. **The camera pose first** (top panels, `cam_` columns), dated at the native step: when it rotates (`pan`, `tilt`, `rotation`), changes scale, when its registration degrades (`sigma0`) or is absent (line cut, instants without pose). Each moment of the sensors is read against it: during a movement or a registration gap, or camera still.
|
||||||
|
2. **Who moves in the image.** The pose is computed from the global references: a change of pose (or a rise of `sigma0`) is a camera movement only if **the targets also move in the image** (on the `x`, `y` panels, the lines of the references against the black line of the targets alone; `x_targets`, `y_targets`). Only references move, targets still: these references moved, and the pose and the `corr` of all the targets are skewed without raw movement of the targets; say which references and by how much. References and targets together: the camera or the scene; then look whether the `corr` come back.
|
||||||
|
3. **Rotation or scale: `--rigid`.** Then the deviation from the median no longer says who moves alone (each sensor deviates according to its place in the image; the rigid residual does not). Before writing "everything moves together", rerun with `--rigid BASE`, `BASE` a calm instant before the first movement: a reference whose residual stays far from the 10th-90th percentile of the targets, or a `reproj_err` of the references that does not come back, means the compensation remains skewed by these references and the steps of `corr` partly come from it (say which, by how much, until when); a rising `|residual|` of the targets says they do not move as a block.
|
||||||
|
4. **Then `x`, `y`, `sigma0`, before `corr`**: the median of `x`, `y` (raw position, references included), the references one by one, the targets alone and `sigma0` say what moves in the image. Date the start, the end, the return of a movement, or a level that does not come back, on the printed table (time, value), not on the PNG axis. A dip of `corr` after a movement of the raw median is the remainder of the compensation, not a movement of these sensors. The multiples of `error_value` show a lasting step of a sensor, the count the instants where many rise together.
|
||||||
|
5. **The residuals, on every batch, without waiting for a suspicion**: a single sensor that rises or falls, a `corr` that drifts by a few hundredths, a step specific to one sensor under a common movement show there; the table says by how much (in noises) and when. On a target whose `corr` goes down under a strong common movement, `--others` separates a remainder of the compensation from a movement of its own: a dip that remains is not a remainder shared by all; a high slope with r close to ±1 on common `y` says it is proportional to the common movement (gain specific to the sensor); without r or slope, it is independent of the movement.
|
||||||
|
|
||||||
|
### ranges
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 tools/scripts/ranges.py <folder> [--columns error_value,corr_x,corr_y] [--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N] [--labels F [--margin 6h] [--top 5]] [--sensors A,B] [--start T] [--end T]
|
||||||
|
```
|
||||||
|
|
||||||
|
Dates the start and the end of a sensor's moment without rerunning ten tables. By column and by sensor, at the measurements (native step, never averaged): level (median of the window), own noise, **largest jump** from one measurement to the next (value, in noises, the two instants; `(rejected: …)` when one end is); then the **ranges** where the sensor deviates from its level by more than `--k` noises: start, end, number of measurements, extreme (signed, in units and in noises) and its instant. Two ranges of the same sign less than `--merge` apart make one; rejected measurements do not cut a range; column `rejected`: how many of the range's measurements are (and `rejected extreme`). A range that touches the edge of the window is dated at the edge.
|
||||||
|
- `--outside T1/T2`: level and noise computed outside an episode that skews the window median (the sensor spends half the batch there). Rerun with it around a long episode to date its return to the level: do not conclude "does not come back" without having read it.
|
||||||
|
- `--min-measurements N`: only the ranges of at least N measurements (and how many out of the total). Over several days, the one-step ranges number in the hundreds and drown the long steps: first `--min-measurements 6`, then without.
|
||||||
|
- `--rolling N` (N measurements on each side) or `--rolling 1d` (the measurements within a duration): the level of each measurement is the median of its neighbours, without it; a long drift or a step follows this local level, only the deviations shorter than the window remain (the level column and the noise do not change; with `--outside`, only the noise is computed outside the episode).
|
||||||
|
- `--instants N`: at the end, the instants where at least N sensors are out of level on one of the columns, with each sensor and its signed deviation in noises (`cx` corr_x, `cy` corr_y, `ev` error_value), the sensors rejected at that instant (among `--sensors`: removing the always-rejected sensors lightens the list), and each reference: `error_value` (`r` if rejected) / inlier.
|
||||||
|
- `--labels F` (one JSON per line, `start`, `end`, `family`, `id`; numbered in the order of the file): each line of `--instants` ends with the numbers of the labels that cover the instant (`labels: none`); then, per label, the measurements strictly inside it (a measurement at a bound is the point before or after), the largest deviation inside from the level (sensor, column, in noises, instant) and the number of instants inside beyond `--k` noises; under it, the **size of the label**, the `--top` sensor-columns (default 5) whose extreme inside deviates most from the **level before the label** (median of the `--margin` before its start, default 6h): `before` (measurements in that margin), the extreme (signed, in units and in noises) and its instant, `net` (median of the `--margin` after the end minus the level before, in units and in noises) and `after` (its measurements). A large extreme with a small `net` comes back within the label; a `net` close to the extreme keeps its new level. To measure existing labels (keep, correct, remove), take `--columns corr_x_mm,corr_y_mm` (or `corr_x,corr_y`) and widen `--start`/`--end` by the margin. A label without a measurement inside (an outage, a label shorter than one step) gives `net` alone, largest first: what changed across it.
|
||||||
|
- When the batch has a long drift or several steps (the ranges number in the hundreds, or one half of the batch is "out of level"), rerun with `--rolling` (a few measurements, or a few hours to a day depending on the cadence) and `--instants 2`: the brief deviations under the drift come out alone, and the list per instant says which sensors deviate together, which are rejected and what the references do; a deviation of all the targets with the references at the same instant is read against the camera pose.
|
||||||
|
|
||||||
|
A noise of `0` (and "-" in noises, no range): the values are constant to rounding (deviations below 1e-9), like the `corr` of the references when the compensation retains only two (see Conventions); the same in `overview.py --residuals` and `days.py`.
|
||||||
|
|
||||||
|
### days
|
||||||
|
|
||||||
|
```
|
||||||
|
.venv/bin/python tools/scripts/days.py <folder> [--columns corr_x,corr_y,error_value] [--night 20-3] [--k 5] [--days 2] [--step 15min] [--sensors A,B] [--bounds [--smoothing N] [--fraction F]] [--start T] [--end T] [--out D]
|
||||||
|
```
|
||||||
|
|
||||||
|
What comes back from one day to the next, on accepted measurements only. A night: the hours `--night` H1-H2 UTC, straddling midnight, dated from the day it starts; choose hours where the day cycle (`error_value`, camera pose) has subsided (**byhour** figure).
|
||||||
|
|
||||||
|
1. **Level of each night**, by sensor and by column: median, MAD, number of measurements, own noise; from one night to the next, the **step** (difference of the medians, in units and in noises) and **where it falls** ("between A and B": where the night measurements of the two nights split best between the two levels; B at the start of the following night: the cut falls by day). Read it for any sensor that has already made a step or for which `--outside` is not enough (several levels in the batch): a step of a few noises is dated by its "between A and B", a step of `error_value` and one of `corr` at the same instant are linked, a slow drift reads from one night to the next.
|
||||||
|
2. **Superimposed days**: PNG **byhour-<column>**, a small chart per sensor, hour of the day on the x axis, one curve per day, each measurement minus the level of the last night started (night greyed, frame from the 1st to the 99th percentile, the title says how many measurements fall outside it). Table: per `--step` slot of the hour of the day, the sensors whose deviation exceeds `--k` noises on the same side at least `--days` days, with the number of days and the median of these deviations. What comes back every day at the same hour, on one or several sensors, groups there, even when it lasts one to three measurements: one moment per hour of the day (say which sensors, how many days out of how many, the amplitude, what the camera pose does at that hour), not one moment per day, nor nothing because each step is short.
|
||||||
|
3. **Bounds of each day** (`--bounds`), for what comes back every day and is labelled day by day (thermal cycle, warming and cooling), rather than fixing the same hours for all the days: for the median of the targets of each column (with `--sensors`, the median of these sensors; with one, its own) and for the camera pose (`pan`, `tilt`, `rotation`, `scale`, `sigma0`, if `camera/` exists), the deviation from the level of the night that precedes each day (a day goes from the end of one night to the end of the next). Per day: number of measurements, the first and the last; first and last instant beyond `--k` noises, with the measurement just before and just after, at the native step; signed max and min with their instant; the sequence of measurements beyond `--k` noises around the larger of the two; the following night minus the reference night. A day without measurement says so; a reference night that is not the previous one is named. Each day line starts with the short name of its series (`corr_x`, `pose_tilt`…): `| grep '^pose_tilt '` reads one, `| grep ' 12/10 '` one day. PNG **bounds**: the deviation over the batch, first → last exceedance light grey, sequence around the extreme dark grey, max and min marked.
|
||||||
|
- `--smoothing N`: the deviation of each measurement is the median of the N measurements of the day centred on it (noise and `--k` unchanged; PNG: smoothed in blue, raw in grey): an isolated point no longer cuts the sequence around the extreme nor makes a first or a last exceedance.
|
||||||
|
- `--fraction F`: the threshold of each day is F times its largest |deviation| (printed on its line, `threshold …`), instead of `--k` noises: when the noise at the native step is unrelated to the amplitude of the cycle (very noisy or very smooth pose).
|
||||||
|
- Reading: the sequence around the extreme dates the cycle, the max and the min a go and a return (morning and afternoon); "before" and "after" give the bounds at the native step (at a cadence of 1 h or 4 h, an event goes from the measurement before to the one after). First exceedance at the start of the day ("before -") and last at its end every day: the night is not calm; change `--night`, or `--sensors` the targets that carry the cycle. Camera drifting all night: a short night just before the morning (`--night 3-5`: the level of the rest that precedes the start). Noisy series or isolated points: `--smoothing 5`. Threshold in noises that does not bite (never exceeded, or exceeded all day): `--fraction 0.3`, then read the sequence around the extreme, not the first and the last exceedance. A "1 measurements" line is a peak, not a cycle.
|
||||||
|
|
||||||
|
### metrics
|
||||||
|
|
||||||
|
```
|
||||||
|
.venv/bin/python tools/scripts/metrics.py <folder> [--names A,B] [--step 1h] [--stat median|min|max] [--against temperature] [--smoothing 24h] [--metrics-only] [--start T] [--end T] [--out D]
|
||||||
|
```
|
||||||
|
|
||||||
|
The system metrics of `metrics/` (one value per hour), never the identifiers (`SIM_CCID`, `ip*`):
|
||||||
|
- the available metrics with their number of values over the window ("empty": none; all empty is an absent datum, not a stable temperature);
|
||||||
|
- a table per `--step`: `temperature` (median of the `sensor_*_deg_c` components, the one the detectors read), each metric of `--names` (default: the `*_deg_c` and `BusVoltage`), then, unless `--metrics-only`, `tgt_corr_x`, `tgt_corr_y` (median over the targets of `corr` minus each target's own level, accepted measurements) and the raw camera pose (`cam_pan` … `cam_sigma0`); an empty cell is skipped, a text metric (`link`, `BatteryState`, `last_sync`) is returned as it is. `--step 1d --stat min` (or `max`): the minimum or the maximum of each day;
|
||||||
|
- per column: filled steps, min and max with their instant; the correlation r and the slope of each column on `--against` (with `--smoothing 24h`, on its rolling mean, column `<against>_smooth`: for a station where the temperature varies little and fast, indoors); the dated runs of steps without any metric;
|
||||||
|
- a PNG: temperatures on one panel (median in black), each other metric on its own, then `tgt_corr_x`, `tgt_corr_y` and the pose, line cut on a gap.
|
||||||
|
|
||||||
|
An r does not say a cause: say it as a value, with the window where it is computed.
|
||||||
|
|
||||||
|
### stations
|
||||||
|
|
||||||
|
```
|
||||||
|
.venv/bin/python tools/scripts/stations.py <folder> <folder> [...] [--columns temperature,cam_tilt,cam_rotation,cam_scale,tgt_corr_x,tgt_corr_y] [--step 1h] [--stat median|min|max] [--table T1/T2|no] [--start T] [--end T] [--out D]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Several stations on the same timeline** (neighbouring stations, same building). Columns: `temperature` (as in `metrics.py`), `cam_pan` … `cam_sigma0` (raw pose), `tgt_corr_x`, `tgt_corr_y` (as in `metrics.py`), `rej_targets`, `rej_refs` (number of rejected targets, of rejected references, the largest of the step).
|
||||||
|
- Prints per station the period, the cadence and the number of instants; the **gaps**: the runs of steps without measurement or without temperature, with the stations where they are missing (a gap common to several stations on a single line); the table by step (one `STATION:column` column each, tabs; `--table` as in `overview.py`); per column and per pair of stations, r of the values and r of the steps from one value to the next (a common movement shows on the steps, a common trend on the values), with n.
|
||||||
|
- PNG **stations**: a panel per column, a line per station (raw temperature, the rest minus its median), and a band per station of the steps without measurement.
|
||||||
|
- Take `--step` equal to the slowest cadence of the stations; over a month at `1h`, first `--table no`.
|
||||||
|
|
||||||
|
### quality
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 tools/scripts/quality.py <folder> [--calm T1/T2,T3/T4] [--common] [--table M] [--step 15min] [--threshold 0.5] [--sensors A,B] [--start T] [--end T]
|
||||||
|
... --labels F [--numbers 3,7] [--margin 60] [--edges]
|
||||||
|
... --labels F --outside [--min 1] [--except "Thermal displacement"]
|
||||||
|
.venv/bin/python tools/scripts/quality.py <folder> --figure [--metrics flux,snr,dominance] [--labels F] [--zone T1/T2 ...] [--out D]
|
||||||
|
python3 tools/scripts/quality.py --self-test
|
||||||
|
```
|
||||||
|
|
||||||
|
The six **image quality metrics** (`peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr`; definitions: `tools/DATA.md`), on the stations that have them (otherwise: "No quality metrics", and nothing else). Each measurement becomes a **ratio** to the **calm reference** of its sensor: the median of the metric over its measurements in the `--calm` windows, at the same UTC hour of the day (without `--calm`: the whole folder); 1 = like the reference, < 1 lower, > 1 higher; for `fwhm` and `dominance`, the rise says a degraded image; rejected measurements included.
|
||||||
|
- Without options: the folder's maximal `peak`, then per sensor (`r` = reference) the rejected measurements, `pk/max` (median peak over the window divided by this maximum: at 1, the spot touches the image ceiling, its `peak` no longer moves and its `sharpness` follows the exposure: read `flux`, `snr` instead), and for each metric the median ratio and the extreme with its instant; then the median of the targets and of the references.
|
||||||
|
- `--common`: per step, the median of the ratios of the targets / of the references for each metric (what affects the whole scene, visibility or turbulence; `--step 1d` shows on one screen on which days the reference holds).
|
||||||
|
- `--table M`: per step, the ratio of `M` of each sensor (`*`: a rejected measurement), their median and how many pass `--threshold` (below, or above 1/`--threshold` for `fwhm`, `dominance`): a target that goes out and comes back; on a moment, `--table flux --sensors …` says which targets are hidden (flux cut) or veiled, from which instant to which instant: write them by their name.
|
||||||
|
- `--labels F` (one JSON per line, `start`, `end`, `family`): per numbered label, the median ratio of the targets and of the references over the hour before, the range and the hour after (`tgt 0.96|0.26|0.79`, `--margin` minutes), then the sensors **affected** over the range (flux or snr below `--threshold`, or dominance above 1/`--threshold`), with their worst flux, from lowest to highest. `--edges`: per instant of the widened range, the number of affected sensors and the median of the targets of flux, snr, dominance (`<` within the range), to date the first and the last degraded instant.
|
||||||
|
- `--outside`: the runs of instants that no label covers (outside `--except` families) where at least `--min` sensors are affected, with the sensors and their worst flux: candidates for forgotten labels, to check on `corr_x`, `corr_y`.
|
||||||
|
- `--figure` (instead of the tables): PNG **quality-common** (a panel per metric, log scale: median of the targets in black, of the references in red, grey band from the 10th to the 90th percentile of the targets: a disturbance of the whole scene moves the median, an occultation of a few targets only the bottom of the band) and a PNG **quality-<metric>** per `--metrics` (a small chart per sensor, references in red); `--labels` greys each label in one colour per family, `--zone` ranges.
|
||||||
|
|
||||||
|
Reading: the metrics say **why** a measurement degrades, where `error_value` says only that it degrades; they prove a cause and name the affected targets. What they showed on LENZSPITZE (to be confirmed on each station): `flux` cut (close to 0) on a group of targets, the others intact, for one to a few measurements: these targets are hidden (the list is read on the flux each time, not on the usual list; a hidden target can stay under its error threshold, and its point is wrong); `flux` down over the whole scene, stronger on the distant targets: reduced visibility; `sharpness` down and `dominance` up, `flux` whole: the image shakes (turbulence); nothing moves: neither the image nor the view, to be looked for in the movement. A whole flux refutes an occultation. `fwhm`, truncated to the disc, moves little; on a spot at the ceiling (`pk/max` at 1), it goes down when the spot darkens.
|
||||||
|
|
||||||
|
**The calm reference decides everything**: take `--calm` on calm hours of the **same image regime** as the range read. When the lighting or the exposure changes (after a gap, an intervention), a reference on the other side skews the ratios by a factor of two or three: `--common --step 1d` over the batch shows this change (medians of the targets and of the references far from 1 for whole days, then at 1); then take a reference on each side and read each side with its own. Calm days are often days of thermal cycle: a reading of the thermal against them is circular.
|
||||||
|
|
||||||
|
### thumbnails
|
||||||
|
|
||||||
|
```
|
||||||
|
.venv/bin/python tools/scripts/thumbnails.py <folder> --sensors A,B --times T1,T2 [--radius 100] [--contrast [MIN,MAX]] [--fixed] [--zone X,Y,W,H] [--out D]
|
||||||
|
```
|
||||||
|
|
||||||
|
A sheet of full-resolution thumbnails: one row per sensor, one column per time, centred on the sensor's raw position (`x`, `y`) at each time. Look at it with Read: a whole image read by Read is reduced, a small reflector becomes illegible in it.
|
||||||
|
- The centre of the frame is written under each thumbnail: when `x`, `y` jump, the frame moves by as much, and a spot that enters the frame seems to appear; `--fixed` keeps for each sensor the frame of the first time and marks with a red cross the `x`, `y` of each time.
|
||||||
|
- Unretouched pixels: the brightness compares from one thumbnail to the next; prints for each thumbnail the background (median), the 99th percentile and the peak (grey levels 0 to 255). Without an image at the requested time, takes the nearest and prints the gap (the actual time is in the title).
|
||||||
|
- A faint or deformed spot is invisible as it is: `--contrast MIN,MAX` puts all the thumbnails in grey with the same stretch (without value: from the 1st percentile to the sheet's maximum, crushed by a saturated pixel; then choose MIN,MAX from the printed background and peak, e.g. `10,40`); `--radius 30` to `50` magnifies. Compare the shape and the peak before, during and after an episode, at the same hour of the day.
|
||||||
|
- A frame that the image edge cuts is flagged ("frame cut by the image edge (W × H px)"): on some stations, `x`, `y` are not pixels (values close to 0) and the frame falls in the corner; then look at the scene through `--zone`.
|
||||||
|
- `--zone X,Y,W,H` (pixels of the original image: top left corner, width, height, as in `gif.py`): the same zone at each time, side by side (up to 4 per row), same stretch with `--contrast`; `--sensors` becomes optional, those given are marked with a cross and their name. For the scene around the targets (foreground object, headlights, lighting), prefer it to a large `--radius`.
|
||||||
|
- Each thumbnail is rendered pixel for pixel up to 1000 px a side, but Read reduces a wide PNG: with a large `--radius` or a large `--zone`, three or four times per sheet.
|
||||||
|
|
||||||
|
### gif
|
||||||
|
|
||||||
|
```
|
||||||
|
.venv/bin/python tools/scripts/gif.py <folder> --start T --end T [--width 720] [--duration 700] [--max 150] [--zone X,Y,W,H] [--out D]
|
||||||
|
```
|
||||||
|
|
||||||
|
A GIF of the camera images between `--start` and `--end` inclusive, cropped to `--zone` if needed, set to `--width` px, UTC date and time at the bottom right; beyond `--max` images, `--max` at regular intervals; prints the time of each image, then the path and the size of the GIF. To attach the scene to a reply to the customer; Read does not read a GIF: to look, `thumbnails.py --zone`.
|
||||||
|
|
||||||
|
### whole image
|
||||||
|
|
||||||
|
```
|
||||||
|
ls <folder>/images/ | grep <YYYYMMDD>T<HHMM>Z
|
||||||
|
```
|
||||||
|
|
||||||
|
Then Read on the file found (Read takes no pattern): the whole camera image at that time (UTC), for the scene (occultation, lighting, weather).
|
||||||
|
|
||||||
## Conventions
|
## Conventions
|
||||||
|
|
||||||
@@ -36,17 +209,22 @@ Run them from the root of the calling repository, where `tools/`, `data/` and `l
|
|||||||
- Bash is used only to run these tools, `ls`, `grep` and `date`, one command per call, without a `for` loop (a pipe to `grep` is allowed): permissions are granted command by command. If the current directory is not the root of the calling repository, `cd <root> && <command>` counts as one command. Write no other script. Whoever calls you may add commands to this list.
|
- Bash is used only to run these tools, `ls`, `grep` and `date`, one command per call, without a `for` loop (a pipe to `grep` is allowed): permissions are granted command by command. If the current directory is not the root of the calling repository, `cd <root> && <command>` counts as one command. Write no other script. Whoever calls you may add commands to this list.
|
||||||
- **An output too long for one call** (beyond a few hundred lines, cut at display): shorten it first with the tool's options (`--table no` then `--table T1/T2`, `--min-measurements`, `--sensors`, `--columns`, a shorter window), or read a section of it through a pipe to `grep` (`| grep -A40 '^Title'`, `| grep '^pose_tilt '`). If you have to reread it several times, store it in the output folder in the same call (`<command> > <out>/<name>.txt`) and read that file with Read (`offset`, `limit`) or with `grep`: no `head`, `tail`, `sed`, `cut`. The tools create their output folder themselves: no `mkdir`.
|
- **An output too long for one call** (beyond a few hundred lines, cut at display): shorten it first with the tool's options (`--table no` then `--table T1/T2`, `--min-measurements`, `--sensors`, `--columns`, a shorter window), or read a section of it through a pipe to `grep` (`| grep -A40 '^Title'`, `| grep '^pose_tilt '`). If you have to reread it several times, store it in the output folder in the same call (`<command> > <out>/<name>.txt`) and read that file with Read (`offset`, `limit`) or with `grep`: no `head`, `tail`, `sed`, `cut`. The tools create their output folder themselves: no `mkdir`.
|
||||||
- A sensor name that carries an apostrophe (`COLH20'''`, `REFP''21`) goes in double quotes in the shell: `--sensors "COLH20'''",COLI22`.
|
- A sensor name that carries an apostrophe (`COLH20'''`, `REFP''21`) goes in double quotes in the shell: `--sensors "COLH20'''",COLI22`.
|
||||||
- Each PNG is looked at with Read: a PNG you have not looked at has shown nothing.
|
- **Each PNG is looked at with Read**: you read a chart better than a table, and a PNG you have not looked at has shown nothing.
|
||||||
- **Image quality metrics** (`quality.py`): they say **why** a measurement degrades, where `error_value` says only that it degrades; they prove a cause and name the affected targets. What they showed on LENZSPITZE (to be confirmed on each station): `flux` cut (close to 0) on a group of targets, the others intact, for one to a few measurements: these targets are hidden (the list is read on the flux each time, not on the usual list; a hidden target can stay under its error threshold, and its point is wrong); `flux` down over the whole scene, stronger on the distant targets: reduced visibility; `sharpness` down and `dominance` up, `flux` whole: the image shakes (turbulence); nothing moves: neither the image nor the view, to be looked for in the movement. A whole flux refutes an occultation. `fwhm`, truncated to the disc, moves little; on a spot at the ceiling (`pk/max` at 1), it goes down when the spot darkens. **The calm reference decides everything**: take `--calm` on calm hours of the **same image regime** as the range read. When the lighting or the exposure changes (after a gap, an intervention), a reference on the other side skews the ratios by a factor of two or three: `--common --step 1d` over the batch shows this change (medians of the targets and of the references far from 1 for whole days, then at 1); then take a reference on each side and read each side with its own. Calm days are often days of thermal cycle: a reading of the thermal against them is circular.
|
|
||||||
- A rejected measurement (`error_value` above the threshold) is not a measurement: the station empties the `corr` of a rejected target, but leaves those of a rejected **reference**, often absurd; the tools empty them (`corr`, ranges, residuals, jumps cover only accepted measurements). When the compensation retains only **two references**, their `corr` is 0 (to 1e-13): it passes exactly through them; their noise prints as `0` and their deviations in noises as "-"; read nothing into these values, read the references through `x`, `y` (`overview.py --rigid`) and `error_value`. Its `x`, `y` remain: before citing a jump of `x`, `y`, check whether it is marked `(rejected)`; a rejected measurement is not cited as a movement.
|
- A rejected measurement (`error_value` above the threshold) is not a measurement: the station empties the `corr` of a rejected target, but leaves those of a rejected **reference**, often absurd; the tools empty them (`corr`, ranges, residuals, jumps cover only accepted measurements). When the compensation retains only **two references**, their `corr` is 0 (to 1e-13): it passes exactly through them; their noise prints as `0` and their deviations in noises as "-"; read nothing into these values, read the references through `x`, `y` (`overview.py --rigid`) and `error_value`. Its `x`, `y` remain: before citing a jump of `x`, `y`, check whether it is marked `(rejected)`; a rejected measurement is not cited as a movement.
|
||||||
|
|
||||||
## The method
|
## The method
|
||||||
|
|
||||||
1. **Inventory**: what is available, what is missing (files, what each source contains, images, camera poses, period, cadence, sensors). A filled column that the tools do not show or a source that no tool reads (`images_diff/`) remains data: know that it exists. The six image quality columns (`peak` to `snr`), filled on recent cameras, are read through `quality.py`. `metrics/` is read through `metrics.py`: a cut of the measurements is read against the steps without metric and the voltage (`--names BusVoltage,BusCurrent,Power,BatteryPercent --metrics-only` on the cut); metrics that are all empty ("empty" on each name) are an absent datum, not a stable temperature. On a batch, rerun it with the batch's `--start`/`--end`: the absent steps and the rejections of the batch, dated; those of the whole period say nothing about the batch. Read the rejections of the **references** as much as those of the targets: a sensor rejected at the same hours every day makes a moment on its own (say its share of rejected measurements and its hours, day by day); when it is a reference, say with how many references the compensation then runs (the list of retained references). When several targets jump together by one step, or the compensation drops (no pose), look first in that list at which references were lost at that instant.
|
1. **Inventory** (`inventory.py`): what is available, what is missing. A filled column that the tools do not show or a source that no tool reads (`images_diff/`) remains data: know that it exists. The image quality columns (`peak` to `snr`, recent cameras) are read through `quality.py`; `metrics/` through `metrics.py`: a cut of the measurements is read against the steps without metric and the voltage (`--names BusVoltage,BusCurrent,Power,BatteryPercent --metrics-only` on the cut). On a batch, rerun it with the batch's `--start`/`--end`: the absent steps and the rejections of the batch, dated; those of the whole period say nothing about the batch. Read the rejections of the **references** as much as those of the targets: a sensor rejected at the same hours every day makes a moment on its own (say its share of rejected measurements and its hours, day by day); when it is a reference, say with how many references the compensation then runs (the list of retained references). When several targets jump together by one step, or the compensation drops (no pose), look first in that list at which references were lost at that instant.
|
||||||
2. **Wide first**: `overview.py` on the batch, first (on a batch of more than a few days, with `--table no`, then `--table T1/T2` on the days to read); **the camera pose first** (top panels, `cam_` columns of the table): when the camera rotates (`pan`, `tilt`, `rotation`), changes scale (`scale`), when its registration degrades (`sigma0`) and when it is not registered (line cut, instants without pose), dated at the native step; each moment of the sensors is then read against it: during a movement or a registration gap, or camera still. The pose is computed from the global references: a change of pose (or a rise of `sigma0`) reads as a camera movement only if **the targets also move in the image**; look, on the `x`, `y` panels, at the lines of the references against the black line of the targets alone (and the `x_targets`, `y_targets` column of the table). If only references move in the image and the targets stay still, it is these references that moved: the pose and the `corr` of all the targets are skewed by it, without raw movement of the targets; say which references and by how much. If everything moves together, references and targets, it is the camera or the scene; then look whether the `corr` come back. As soon as the pose rotates or changes scale, the deviation from the median no longer says who moves alone (each sensor deviates from it according to its place in the image): before writing "everything moves together", rerun with `--rigid BASE`, `BASE` a calm instant before the first movement, and read the **rigid** PNG and its table: a reference whose residual stays far from the 10th-90th percentile of the targets, or a `reproj_err` of the references that does not come back, means that the compensation remains skewed by these references and that the steps of `corr` partly come from it; say which ones, by how much, until when; a `|residual|` of the targets that rises says they do not move as a block. Then the median of `x`, `y` (raw position, references included), the references one by one, the targets alone and `sigma0` say what moves in the image, before reading `corr`; read the printed table (time, value) to date the start, the end and the return of the movement and to see a level that does not come back, and do not guess them on the PNG axis: a dip of `corr` that follows a movement of the raw median is the remainder of the compensation, not a movement of the affected sensors themselves; the small multiples of `error_value` show a lasting step of a sensor, the count shows the instants where many sensors rise together. On every batch, without waiting for a suspicion, look at the **residuals** PNGs (`corr_x`, `corr_y`, `error_value`): each sensor against its own level, that is where a single sensor that rises or falls, a `corr` that drifts by a few hundredths, a step specific to one sensor under a common movement are seen; the table says by how much (in noises) and when. To separate, on a target whose `corr` goes down under a strong common movement, a remainder of the compensation from a movement of its own, rerun with `--others` (the shared remainder disappears, the slope on common `x`, `y` says whether it is proportional). On a batch of more than two days, then run `days.py` (before reading the isolated steps of `ranges.py`): what comes back every day at the same hour is read first on its **byhour** PNGs and its table by slot, and makes one moment per hour of the day (say which sensors, how many days out of how many, the amplitude, what the camera pose does at that hour), not one moment per day nor nothing at all because each step is short; then read its **night levels** table, for any sensor that has already made a step or for which `--outside` is not enough (several levels in the batch): each step from one night to the next is dated by its "between A and B", and a step of `error_value` and one of `corr` at the same instant are linked; a slow drift reads from one night to the next. For what comes back every day and is labelled day by day (thermal cycle, warming and cooling), rerun `days.py --bounds` rather than fixing the same hours for all the days: each day is dated on its line, the sequence around the extreme for the cycle, the max and the min for a go and a return (morning and afternoon); the "before" and "after" measurements give the bounds at the native step (at a cadence of 1 h or 4 h, an event goes from the measurement before to the one after). Take `--night` on hours where the pose and `error_value` have subsided (**byhour** figure): if the first exceedance falls at the start of the day ("before -") and the last at its end every day, the chosen night is not calm; change `--night` or switch to `--sensors` for the targets that carry the cycle. When the camera drifts all night, take a short night just before the morning (`--night 3-5`: the level is that of the rest that precedes the start); on a noisy series or one spiked with isolated points, add `--smoothing 5`; when the threshold in noiseLine truncated
|
2. **Wide first**:
|
||||||
|
- `overview.py` on the batch, first (on a batch of more than a few days, `--table no`, then `--table T1/T2` on the days to read), read in the order of its Reading section: the pose, who moves in the image (`--rigid` as soon as the pose rotates or changes scale), `x`, `y`, `sigma0` before `corr`, the residuals on every batch.
|
||||||
|
- On a batch of more than two days, then `days.py`, before reading the isolated steps of `ranges.py`: first its **byhour** PNGs and table by slot, then its **night levels**; `days.py --bounds` for what comes back every day and is labelled day by day.
|
||||||
|
- Several neighbouring stations given by whoever calls you: `stations.py` (temperatures, poses, `corr` of the targets, common rejections and gaps, at the same hours) before comparing their moments.
|
||||||
|
- When the pose or the `corr` of the targets drift over several days, or move without day/night cycle: `metrics.py` on the batch, the temperature against `tgt_corr_x`, `tgt_corr_y` and the pose, on the PNG, then r and the slope.
|
||||||
|
- On a station that has the image quality metrics: `quality.py --common --step 1d` on the batch (choice of the calm reference), then `quality.py --figure` with this reference: the median `flux` of the targets and of the references dates the hours of reduced visibility, the grey band that dives alone the occultations; with a labels file, `--labels F --outside`.
|
||||||
|
- Then `plot.py` on all the sensors, `error_value` then `corr_x`, `corr_y` (or `relative_x`, `relative_y`), references and targets; `summary.py` on these columns with a wide step (6h or 1d).
|
||||||
3. **Then zoom**: `summary.py` at 15min (`--stat max` for the real peak of a sensor), and `plot.py` on the windows and the sensors that move.
|
3. **Then zoom**: `summary.py` at 15min (`--stat max` for the real peak of a sensor), and `plot.py` on the windows and the sensors that move.
|
||||||
4. **Each sensor against its own usual level** (its median and its MAD), never against an absolute threshold; on a batch, the level is computed on the batch (`--level window`, `overview.py`), not on the whole period.
|
4. **Each sensor against its own usual level** (its median and its MAD), never against an absolute threshold; on a batch, the level is computed on the batch (`--level window`, `overview.py`), not on the whole period.
|
||||||
5. **Each retained moment is dated and measured at the native 15 min step**: start, end and maximum read on the 15 min table (`overview.py` by default, `summary.py --step 15min`), never on an hourly table; also read `overview.py --min 1` for single sensors, and `ranges.py` on the batch (first with `--min-measurements 6`) to get, per sensor, start, end, extreme and largest jump at the native step (rerun it with `--outside` around a long episode to date its return to the level, do not conclude "does not come back" without having read it; when the batch has a long drift or several steps (the ranges number in the hundreds, or one half of the batch is "out of level"), rerun it with `--rolling` (a few measurements, or a duration of a few hours to a day depending on the cadence) and `--instants 2`: the brief deviations under the drift come out alone, and the list per instant says which sensors deviate together, which are rejected and what the references do at that instant; a deviation of all the targets with the references at the same instant is read against the camera pose), and the tables of `x`, `y` on the windows where they stir. On a station that has the quality metrics, for each moment: `quality.py --table flux --sensors …` (or `--labels F --numbers N --edges` on a label) says which targets are hidden (flux cut) or veiled, from which instant to which instant; write them by their name.
|
5. **Each retained moment is dated and measured at the native 15 min step**: start, end and maximum read on the 15 min table (`overview.py` by default, `summary.py --step 15min`), never on an hourly table; also `overview.py --min 1` for single sensors; `ranges.py` on the batch (first `--min-measurements 6`) for, per sensor, start, end, extreme and largest jump at the native step (`--outside` around a long episode, `--rolling` and `--instants 2` under a long drift or several steps); the tables of `x`, `y` on the windows where they stir. On a station that has the quality metrics, for each moment: `quality.py --table flux --sensors …` (or `--labels F --numbers N --edges` on a label): which targets are hidden or veiled, from which instant to which instant.
|
||||||
6. **The images**: thumbnails (with `--contrast` for a faint target) spaced, one every 2 to 3 h, tighter where the curves move, always with an unaffected witness sensor, at a time before, during and after (at the same hour of the day if possible); when `x`, `y` of a sensor jump between two thumbnails (centre of the frame written under each), rerun with `--fixed` before saying that a spot appears or disappears; the whole image for the scene (occultation, lighting, weather).
|
6. **The images**: thumbnails (with `--contrast` for a faint target) spaced, one every 2 to 3 h, tighter where the curves move, always with an unaffected witness sensor, at a time before, during and after (at the same hour of the day if possible); when `x`, `y` of a sensor jump between two thumbnails (centre of the frame written under each), rerun with `--fixed` before saying that a spot appears or disappears; the whole image for the scene (occultation, lighting, weather).
|
||||||
|
|
||||||
## What the reading returns
|
## What the reading returns
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ The reading harness: the methods given to the agent to make sense of the data of
|
|||||||
|
|
||||||
- `scripts/`: the tools, one script each; `GUIDE.md`: which tool for which question, and how to read what it returns. The agent reads it first.
|
- `scripts/`: the tools, one script each; `GUIDE.md`: which tool for which question, and how to read what it returns. The agent reads it first.
|
||||||
- `DATA.md`: the files of a station and the meaning of their columns, maintained by hand; the agent reads it with the guide.
|
- `DATA.md`: the files of a station and the meaning of their columns, maintained by hand; the agent reads it with the guide.
|
||||||
- `scripts/data.py`: the common reader of the data (also that of the aimsight-app app, which passes these measurements to the detectors); the detectors keep their own (`detectors/scripts/data.py`), to stay autonomous: a reading fix in one is carried to the other.
|
- `scripts/data.py`: the one reader of the data, shared by the aimsight-app app, the tools and the detectors (which import it); a change to it changes the detectors' benchmark (`python3 detectors/bench/regression.py --labels labeling/auto --data data`, from aimsight-app). `scripts/common.py`: the helpers the tools share.
|
||||||
- The tools: `inventory`, `summary`, `ranges`, `days`, `overview`, `metrics`, `quality`, `stations`, `plot`, `thumbnails`, `gif` (details in the guide).
|
- The tools: `inventory`, `summary`, `ranges`, `days`, `overview`, `metrics`, `quality`, `stations`, `plot`, `thumbnails`, `gif` (details in the guide).
|
||||||
|
|
||||||
## Using it
|
## Using it
|
||||||
@@ -20,6 +20,6 @@ Without `--out`, the figures go to `out/diagnostic/` of the current folder. Stan
|
|||||||
|
|
||||||
## Who changes it
|
## Who changes it
|
||||||
|
|
||||||
Only the `upgrader` (`agents` repository) writes there, run by Optimizer's `optimize` skill after an analysis: it brings the analyst's useful scripts into the tools, it optimizes or adds, it removes nothing.
|
Only the `upgrader` (`agents` repository) writes there, run by Optimizer's `optimize` skill after an analysis: it brings the analyst's useful scripts into the tools; it adds, changes or removes (a tool only for a good reason, `CLAUDE.md`).
|
||||||
|
|
||||||
Copied from aimsight-app (`agent-tools/` and the `lire-les-donnees` skill, commit dd6f1bf) on 2026-11-16. The architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`.
|
Copied from aimsight-app (`agent-tools/` and the `lire-les-donnees` skill, commit dd6f1bf) on 2026-11-16. The architecture of the repositories: `~/aimsight-app/docs/architecture.drawio`.
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
"""What the reading tools share beyond reading (data.py): columns, steps, noise, sensor list, runs, file names.
|
||||||
|
|
||||||
|
Standard library only: the tools without a figure run with python3, without matplotlib.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import re
|
||||||
|
import statistics
|
||||||
|
import sys
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
from data import filter_window, is_reference, load, parse_time
|
||||||
|
|
||||||
|
COLUMNS = ("error_value", "corr_x", "corr_y", "relative_x", "relative_y", "x", "y", "sigma0", "reproj_err",
|
||||||
|
"corr_x_mm", "corr_y_mm")
|
||||||
|
QUALITY = ("peak", "flux", "sharpness", "fwhm", "dominance", "snr") # image quality: recent cameras; read by quality.py
|
||||||
|
STATS = {"median": statistics.median, "min": min, "max": max}
|
||||||
|
ROUNDING = 1e-9 # below this, a difference between two measurements is the rounding of the station's computation, not noise
|
||||||
|
|
||||||
|
|
||||||
|
def duration(text):
|
||||||
|
"""'15min', '1h', '6h', '1d' -> timedelta."""
|
||||||
|
m = re.fullmatch(r"(\d+)(min|h|d)", text)
|
||||||
|
if not m:
|
||||||
|
raise argparse.ArgumentTypeError(f"invalid step: {text!r} (e.g. 15min, 1h, 1d)")
|
||||||
|
n, unit = int(m.group(1)), m.group(2)
|
||||||
|
return {"min": timedelta(minutes=n), "h": timedelta(hours=n), "d": timedelta(days=n)}[unit]
|
||||||
|
|
||||||
|
|
||||||
|
def own_noise(*series):
|
||||||
|
"""Median of the absolute differences between two consecutive values (within each series given); 0 when it is below
|
||||||
|
ROUNDING (values constant up to rounding: the corr of the references when the compensation goes exactly through them,
|
||||||
|
two references retained), so that a deviation is not written as billions of noises; 0 also with fewer than two values."""
|
||||||
|
d = [abs(b - a) for vs in series for a, b in zip(vs, vs[1:])]
|
||||||
|
b = statistics.median(d) if d else 0
|
||||||
|
return b if b >= ROUNDING else 0
|
||||||
|
|
||||||
|
|
||||||
|
def sensors(folder, wanted):
|
||||||
|
"""-> ({sensor: [measurement]}, the --sensors names, or all of them, references first); exits without an export."""
|
||||||
|
by = load(folder)
|
||||||
|
if not by:
|
||||||
|
sys.exit(f'No "advanced" export in {folder} (see inventory.py).')
|
||||||
|
return by, wanted.split(",") if wanted else sorted(by, key=lambda s: (not is_reference(by[s]), s))
|
||||||
|
|
||||||
|
|
||||||
|
def check(what, requested, known):
|
||||||
|
"""Exits on the names of requested that known does not have."""
|
||||||
|
if missing := [x for x in requested if x not in known]:
|
||||||
|
sys.exit(f"Unknown {what}: {', '.join(missing)}. Known: {', '.join(sorted(known))}.")
|
||||||
|
|
||||||
|
|
||||||
|
def per_step(first, step):
|
||||||
|
"""Start of the step of each time, steps aligned on the midnight of `first` (hh:00, hh:15…)."""
|
||||||
|
midnight = first.replace(hour=0, minute=0, second=0, microsecond=0)
|
||||||
|
return lambda t: midnight + (t - midnight) // step * step
|
||||||
|
|
||||||
|
|
||||||
|
def window_filter(text):
|
||||||
|
"""--table: '' everything, 'no' nothing, 'T1/T2' between T1 and T2 -> predicate on a time."""
|
||||||
|
if text in ("", "no"):
|
||||||
|
return lambda t: text == ""
|
||||||
|
limits = [parse_time(x) for x in text.split("/")]
|
||||||
|
if len(limits) != 2:
|
||||||
|
sys.exit("--table expects T1/T2 or no.")
|
||||||
|
return lambda t: limits[0] <= t <= limits[1]
|
||||||
|
|
||||||
|
|
||||||
|
def outside(text):
|
||||||
|
"""--outside T1/T2 -> predicate: is a time outside that moment (always true without --outside)."""
|
||||||
|
if not text:
|
||||||
|
return lambda t: True
|
||||||
|
limits = [parse_time(x) for x in text.split("/")]
|
||||||
|
if len(limits) != 2:
|
||||||
|
sys.exit("--outside expects T1/T2.")
|
||||||
|
return lambda t: not limits[0] <= t <= limits[1]
|
||||||
|
|
||||||
|
|
||||||
|
def group(items, close):
|
||||||
|
"""Sorted items -> runs (lists) of consecutive items where close(previous, next)."""
|
||||||
|
out = []
|
||||||
|
for x in items:
|
||||||
|
if out and close(out[-1][-1], x):
|
||||||
|
out[-1].append(x)
|
||||||
|
else:
|
||||||
|
out.append([x])
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def without_pose(instants, poses):
|
||||||
|
"""Runs (start, end, n) of consecutive measurement instants without a camera pose (no registration), to the minute."""
|
||||||
|
posed = {p["t"].replace(second=0, microsecond=0) for p in poses}
|
||||||
|
lacking = [(i, t) for i, t in enumerate(instants) if t.replace(second=0, microsecond=0) not in posed]
|
||||||
|
return [(g[0][1], g[-1][1], len(g)) for g in group(lacking, lambda a, b: b[0] == a[0] + 1)]
|
||||||
|
|
||||||
|
|
||||||
|
def extremes(d):
|
||||||
|
"""{t: value} -> ((t, value) of the minimum, (t, value) of the maximum)."""
|
||||||
|
return min(d.items(), key=lambda x: x[1]), max(d.items(), key=lambda x: x[1])
|
||||||
|
|
||||||
|
|
||||||
|
def targets(by_sensor, col, start, end):
|
||||||
|
"""At each measurement, median over the targets of col minus the target's own level (median over the window)."""
|
||||||
|
deviations = {}
|
||||||
|
for s, ms in by_sensor.items():
|
||||||
|
if is_reference(ms):
|
||||||
|
continue
|
||||||
|
ms = [m for m in filter_window(ms, start, end) if m[col] is not None]
|
||||||
|
if ms:
|
||||||
|
level = statistics.median(m[col] for m in ms)
|
||||||
|
for m in ms:
|
||||||
|
deviations.setdefault(m["t"], []).append(m[col] - level)
|
||||||
|
return [{"t": t, "v": statistics.median(v)} for t, v in sorted(deviations.items())]
|
||||||
|
|
||||||
|
|
||||||
|
def cut(series):
|
||||||
|
"""Times and numeric values, NaN inserted over a gap of more than 1.5 times the usual step (the line breaks)."""
|
||||||
|
pts = [(p["t"], p["v"]) for p in series if isinstance(p["v"], float)]
|
||||||
|
if len(pts) < 3:
|
||||||
|
return [t for t, _ in pts], [v for _, v in pts]
|
||||||
|
step = statistics.median(b[0] - a[0] for a, b in zip(pts, pts[1:]))
|
||||||
|
ts, vs = [pts[0][0]], [pts[0][1]]
|
||||||
|
for (a, _), (b, v) in zip(pts, pts[1:]):
|
||||||
|
if b - a > 1.5 * step:
|
||||||
|
ts.append(a + (b - a) / 2)
|
||||||
|
vs.append(float("nan"))
|
||||||
|
ts.append(b)
|
||||||
|
vs.append(v)
|
||||||
|
return ts, vs
|
||||||
|
|
||||||
|
|
||||||
|
def bound(t, default):
|
||||||
|
"""--start/--end in a file name."""
|
||||||
|
return t.strftime("%Y%m%dT%H%MZ") if t else default
|
||||||
+55
-5
@@ -1,4 +1,4 @@
|
|||||||
"""Reading a station's data for the diagnostic agent (standard library).
|
"""Reading a station folder (standard library only): the one reader of the app, the reading tools and the detectors.
|
||||||
|
|
||||||
Formats: tools/DATA.md. Every time is UTC.
|
Formats: tools/DATA.md. Every time is UTC.
|
||||||
"""
|
"""
|
||||||
@@ -12,6 +12,7 @@ from datetime import datetime, timezone
|
|||||||
# Numeric columns of the "advanced" export; an empty cell becomes None.
|
# Numeric columns of the "advanced" export; an empty cell becomes None.
|
||||||
NUM = ("x", "y", "error_value", "error_threshold", "relative_x", "relative_y",
|
NUM = ("x", "y", "error_value", "error_threshold", "relative_x", "relative_y",
|
||||||
"corr_x", "corr_y", "reproj_err", "inlier", "sigma0",
|
"corr_x", "corr_y", "reproj_err", "inlier", "sigma0",
|
||||||
|
"corr_x_mm", "corr_y_mm", # corr × mm_per_px of the sensor (load_targets); None without a sure factor
|
||||||
"peak", "flux", "sharpness", "fwhm", "dominance", "snr") # image quality: recent cameras only (LENZSPITZE)
|
"peak", "flux", "sharpness", "fwhm", "dominance", "snr") # image quality: recent cameras only (LENZSPITZE)
|
||||||
# Camera pose (camera/, pumping): pan, tilt, rotation [deg], scale, sigma0 (registration residual).
|
# Camera pose (camera/, pumping): pan, tilt, rotation [deg], scale, sigma0 (registration residual).
|
||||||
POSE = ("pan", "tilt", "rotation", "scale", "sigma0")
|
POSE = ("pan", "tilt", "rotation", "scale", "sigma0")
|
||||||
@@ -51,11 +52,14 @@ def _number(v):
|
|||||||
def load(folder):
|
def load(folder):
|
||||||
""""advanced" exports of the folder -> {sensor: [measurement]}, sorted by t, one reading per instant.
|
""""advanced" exports of the folder -> {sensor: [measurement]}, sorted by t, one reading per instant.
|
||||||
|
|
||||||
Measurement: dict with t (UTC datetime), pattern_type, and the NUM columns (float or None).
|
Measurement: dict with t (UTC datetime), pattern_type, and the NUM columns (float or None). corr_x_mm, corr_y_mm
|
||||||
|
are corr_x, corr_y × the sensor's mm_per_px of load_targets, whatever the file holds (a factor of the pumping may
|
||||||
|
belong to a namesake sensor): None without a sure factor.
|
||||||
A rejected measurement (see rejected) has empty corr_x, corr_y, as the column docs say: the station empties them
|
A rejected measurement (see rejected) has empty corr_x, corr_y, as the column docs say: the station empties them
|
||||||
for a target but leaves some filled (often absurd) for a reference; they are emptied here.
|
for a target but leaves some filled (often absurd) for a reference; they are emptied here, with their mm.
|
||||||
"""
|
"""
|
||||||
by = {}
|
by = {}
|
||||||
|
factor = {s: t["mm_per_px"] for s, t in load_targets(folder).items()}
|
||||||
for path in find_files(folder)["advanced"]:
|
for path in find_files(folder)["advanced"]:
|
||||||
with open(path, newline="") as f:
|
with open(path, newline="") as f:
|
||||||
for r in csv.DictReader(f):
|
for r in csv.DictReader(f):
|
||||||
@@ -64,10 +68,39 @@ def load(folder):
|
|||||||
m["pattern_type"] = r["pattern_type"]
|
m["pattern_type"] = r["pattern_type"]
|
||||||
if rejected(m):
|
if rejected(m):
|
||||||
m["corr_x"] = m["corr_y"] = None
|
m["corr_x"] = m["corr_y"] = None
|
||||||
by.setdefault(r["sensor_name"].strip(), {})[m["t"]] = m # some exports end the name with a newline
|
name = r["sensor_name"].strip() # some exports end the name with a newline
|
||||||
|
k = factor.get(name)
|
||||||
|
for c in ("corr_x", "corr_y"):
|
||||||
|
m[c + "_mm"] = None if k is None or m[c] is None else m[c] * k
|
||||||
|
by.setdefault(name, {})[m["t"]] = m
|
||||||
return {s: [d[t] for t in sorted(d)] for s, d in by.items()}
|
return {s: [d[t] for t in sorted(d)] for s, d in by.items()}
|
||||||
|
|
||||||
|
|
||||||
|
UNFILLED_DISTANCE_MM = 1000 # the distance_mm the platform holds when nobody filled it in
|
||||||
|
|
||||||
|
|
||||||
|
def load_targets(folder):
|
||||||
|
"""targets.csv (the pumping: the properties of the targets, and of the references once pumped again) -> {sensor:
|
||||||
|
{"pattern_type", "distance_mm", "mm_per_px", "lines"}}; {} without the file. A line at distance_mm 1000 is not
|
||||||
|
filled in on the platform: ignored. A name may have several lines (several sensors of that name, an older
|
||||||
|
configuration of the unit, another lens): a value on which its remaining lines disagree is None, never guessed;
|
||||||
|
"lines" says how many lines the name has."""
|
||||||
|
path = os.path.join(folder, "targets.csv")
|
||||||
|
if not os.path.exists(path):
|
||||||
|
return {}
|
||||||
|
by = {}
|
||||||
|
with open(path, newline="") as f:
|
||||||
|
for r in csv.DictReader(f):
|
||||||
|
by.setdefault(r["sensor_name"].strip(), []).append(r)
|
||||||
|
out = {}
|
||||||
|
for s, rows in by.items():
|
||||||
|
kept = [r for r in rows if _number(r.get("distance_mm") or "") != UNFILLED_DISTANCE_MM] or rows[:0]
|
||||||
|
one = lambda c, cast: (lambda v: v.pop() if len(v) == 1 else None)({cast(r.get(c) or "") for r in kept}) # noqa: E731
|
||||||
|
out[s] = {"pattern_type": one("pattern_type", str.strip), "distance_mm": one("distance_mm", _number),
|
||||||
|
"mm_per_px": one("mm_per_px", _number), "lines": len(rows)}
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
def load_temperature(folder):
|
def load_temperature(folder):
|
||||||
"""Station board temperatures (metrics/, sensor_*_deg_c) -> [(t, °C)] sorted: the median of the components at each
|
"""Station board temperatures (metrics/, sensor_*_deg_c) -> [(t, °C)] sorted: the median of the components at each
|
||||||
hour; [] without these metrics. The components rise and fall together (the labeler reads "the temperatures of all
|
hour; [] without these metrics. The components rise and fall together (the labeler reads "the temperatures of all
|
||||||
@@ -97,8 +130,25 @@ def load_camera(folder):
|
|||||||
return [by[t] for t in sorted(by)]
|
return [by[t] for t in sorted(by)]
|
||||||
|
|
||||||
|
|
||||||
|
WHERE = ("position", "place", "station", "station_name", "distance_km", "height_m") # text columns of weather/
|
||||||
|
|
||||||
|
|
||||||
|
def load_weather(folder):
|
||||||
|
"""The MeteoSwiss weather of the station (weather/, written by the pumping, one CSV per day) -> [hour] sorted by t:
|
||||||
|
t, where it was measured (WHERE, text), the measurements (precipitation_mm, temperature_c…: float or None); [] without
|
||||||
|
weather (no position in stations.txt, or a place outside Switzerland)."""
|
||||||
|
by = {}
|
||||||
|
for path in glob.glob(os.path.join(folder, "weather", "*.csv")):
|
||||||
|
with open(path, newline="") as f:
|
||||||
|
for r in csv.DictReader(f):
|
||||||
|
h = {c: v if c in WHERE else _number(v or "") for c, v in r.items() if c != "time_utc"}
|
||||||
|
h["t"] = parse_time(r["time_utc"])
|
||||||
|
by[h["t"]] = h
|
||||||
|
return [by[t] for t in sorted(by)]
|
||||||
|
|
||||||
|
|
||||||
IDENTIFIERS = re.compile(r"^(SIM_CCID|ip(_.*)?)$") # station identifiers: never read
|
IDENTIFIERS = re.compile(r"^(SIM_CCID|ip(_.*)?)$") # station identifiers: never read
|
||||||
METRIC_RE = re.compile(r"^(.+?)_\d{4}_\d{2}_\d{2}_")
|
METRIC_RE = re.compile(r"^(.+?)_(\d{4})_(\d{2})_(\d{2})_") # name, then the date of the file (one CSV per metric and per day)
|
||||||
|
|
||||||
|
|
||||||
def load_metrics(folder):
|
def load_metrics(folder):
|
||||||
|
|||||||
+38
-46
@@ -55,9 +55,8 @@ import matplotlib
|
|||||||
matplotlib.use("Agg")
|
matplotlib.use("Agg")
|
||||||
import matplotlib.pyplot as plt # noqa: E402
|
import matplotlib.pyplot as plt # noqa: E402
|
||||||
|
|
||||||
from data import POSE, filter_window, is_reference, iso, load, load_camera, parse_time, rejected # noqa: E402
|
from common import COLUMNS, bound, check, duration, own_noise, sensors # noqa: E402
|
||||||
from plot import COLUMNS, bound # noqa: E402
|
from data import POSE, filter_window, is_reference, iso, load_camera, parse_time, rejected # noqa: E402
|
||||||
from summary import ROUNDING, duration, own_noise # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
def hours(text):
|
def hours(text):
|
||||||
@@ -73,6 +72,27 @@ def at_night(t, h1, h2):
|
|||||||
return (h1 <= t.hour or t.hour < h2) if h1 > h2 else h1 <= t.hour < h2
|
return (h1 <= t.hour or t.hour < h2) if h1 > h2 else h1 <= t.hour < h2
|
||||||
|
|
||||||
|
|
||||||
|
def nights_of(pts, h1, h2):
|
||||||
|
"""[(t, v)] -> {date of the night: [(t, v)]}: the night hours, dated by the day the night starts."""
|
||||||
|
nights = {}
|
||||||
|
for t, v in pts:
|
||||||
|
if at_night(t, h1, h2):
|
||||||
|
nights.setdefault((t - timedelta(hours=h1)).date(), []).append((t, v))
|
||||||
|
return nights
|
||||||
|
|
||||||
|
|
||||||
|
def frame(ax, values):
|
||||||
|
"""Frame from the 1st to the 99th percentile widened by half: an isolated peak does not crush the scale; -> the number
|
||||||
|
of values out of frame."""
|
||||||
|
v = sorted(values)
|
||||||
|
if not v:
|
||||||
|
return 0
|
||||||
|
q1, q2 = v[len(v) // 100], v[-1 - len(v) // 100]
|
||||||
|
margin = (q2 - q1) * 0.5 or 1
|
||||||
|
ax.set_ylim(q1 - margin, q2 + margin)
|
||||||
|
return sum(not q1 - margin <= e <= q2 + margin for e in v)
|
||||||
|
|
||||||
|
|
||||||
def cut_index(pts, la, lb):
|
def cut_index(pts, la, lb):
|
||||||
"""Index j (1..n-1) where sum|v-la| before j + sum|v-lb| from j on is smallest."""
|
"""Index j (1..n-1) where sum|v-la| before j + sum|v-lb| from j on is smallest."""
|
||||||
before, after = 0.0, sum(abs(v - lb) for _, v in pts)
|
before, after = 0.0, sum(abs(v - lb) for _, v in pts)
|
||||||
@@ -103,15 +123,10 @@ def main():
|
|||||||
a.add_argument("--out", default=os.path.join("out", "diagnostic"), help="folder of the PNGs")
|
a.add_argument("--out", default=os.path.join("out", "diagnostic"), help="folder of the PNGs")
|
||||||
o = a.parse_args()
|
o = a.parse_args()
|
||||||
|
|
||||||
by_sensor = load(o.folder)
|
by_sensor, names = sensors(o.folder, o.sensors)
|
||||||
if not by_sensor:
|
|
||||||
sys.exit(f"No \"advanced\" export in {o.folder} (see inventory.py).")
|
|
||||||
names = o.sensors.split(",") if o.sensors else sorted(by_sensor, key=lambda s: (not is_reference(by_sensor[s]), s))
|
|
||||||
columns = o.columns.split(",")
|
columns = o.columns.split(",")
|
||||||
for requested, known, what in ((names, by_sensor, "sensors"), (columns, COLUMNS, "columns")):
|
check("sensors", names, by_sensor)
|
||||||
missing = [x for x in requested if x not in known]
|
check("columns", columns, COLUMNS)
|
||||||
if missing:
|
|
||||||
sys.exit(f"Unknown {what}: {', '.join(missing)}. Known: {', '.join(sorted(known))}.")
|
|
||||||
series = {s: [m for m in filter_window(by_sensor[s], o.start, o.end) if not rejected(m)] for s in names}
|
series = {s: [m for m in filter_window(by_sensor[s], o.start, o.end) if not rejected(m)] for s in names}
|
||||||
if not any(series.values()):
|
if not any(series.values()):
|
||||||
sys.exit("No accepted measurement in the requested window.")
|
sys.exit("No accepted measurement in the requested window.")
|
||||||
@@ -133,10 +148,7 @@ def main():
|
|||||||
ax.axis("off")
|
ax.axis("off")
|
||||||
continue
|
continue
|
||||||
noise = noises[s] = own_noise([v for _, v in ms])
|
noise = noises[s] = own_noise([v for _, v in ms])
|
||||||
nights = {}
|
nights = nights_of(ms, h1, h2)
|
||||||
for t, v in ms:
|
|
||||||
if at_night(t, h1, h2):
|
|
||||||
nights.setdefault((t - timedelta(hours=h1)).date(), []).append((t, v))
|
|
||||||
lvl = {d: statistics.median(v for _, v in p) for d, p in nights.items()}
|
lvl = {d: statistics.median(v for _, v in p) for d, p in nights.items()}
|
||||||
cells = []
|
cells = []
|
||||||
for d in sorted(nights):
|
for d in sorted(nights):
|
||||||
@@ -167,11 +179,7 @@ def main():
|
|||||||
for i, (d, p) in enumerate(sorted(per_day.items())):
|
for i, (d, p) in enumerate(sorted(per_day.items())):
|
||||||
ax.plot([x for x, _ in p], [e for _, e in p], "-", lw=0.8, color=f"C{i % 10}", label=f"{d:%d/%m}")
|
ax.plot([x for x, _ in p], [e for _, e in p], "-", lw=0.8, color=f"C{i % 10}", label=f"{d:%d/%m}")
|
||||||
ax.axhline(0, color="0.3", ls=":", lw=0.8)
|
ax.axhline(0, color="0.3", ls=":", lw=0.8)
|
||||||
every = sorted(e for p in per_day.values() for _, e in p)
|
out_of_frame = frame(ax, (e for p in per_day.values() for _, e in p))
|
||||||
q1, q2 = every[len(every) // 100], every[-1 - len(every) // 100] # frame from the 1st to the 99th percentile: an isolated peak does not crush the scale
|
|
||||||
margin = (q2 - q1) * 0.5 or 1
|
|
||||||
ax.set_ylim(q1 - margin, q2 + margin)
|
|
||||||
out_of_frame = sum(not q1 - margin <= e <= q2 + margin for e in every)
|
|
||||||
for x0, x1 in ((h1, 24), (0, h2)) if h1 > h2 else ((h1, h2),):
|
for x0, x1 in ((h1, 24), (0, h2)) if h1 > h2 else ((h1, h2),):
|
||||||
ax.axvspan(x0, x1, color="0.9", lw=0)
|
ax.axvspan(x0, x1, color="0.9", lw=0)
|
||||||
ax.set_title(s + (" (ref.)" if is_reference(series[s]) else "") + f" noise {noise:.2g}"
|
ax.set_title(s + (" (ref.)" if is_reference(series[s]) else "") + f" noise {noise:.2g}"
|
||||||
@@ -217,15 +225,6 @@ def main():
|
|||||||
day_bounds(o, names, series, columns, station)
|
day_bounds(o, names, series, columns, station)
|
||||||
|
|
||||||
|
|
||||||
def night_levels(pts, h1, h2):
|
|
||||||
"""[(t, v)] -> {date of the night: median}."""
|
|
||||||
nights = {}
|
|
||||||
for t, v in pts:
|
|
||||||
if at_night(t, h1, h2):
|
|
||||||
nights.setdefault((t - timedelta(hours=h1)).date(), []).append(v)
|
|
||||||
return {d: statistics.median(v) for d, v in nights.items()}
|
|
||||||
|
|
||||||
|
|
||||||
def day_bounds(o, names, series, columns, station):
|
def day_bounds(o, names, series, columns, station):
|
||||||
"""Section 3: per day, the deviation from the level of the night that precedes it, for the target median and the pose."""
|
"""Section 3: per day, the deviation from the level of the night that precedes it, for the target median and the pose."""
|
||||||
h1, h2 = o.night
|
h1, h2 = o.night
|
||||||
@@ -240,7 +239,10 @@ def day_bounds(o, names, series, columns, station):
|
|||||||
t0, t1 = min(every), max(every)
|
t0, t1 = min(every), max(every)
|
||||||
night_of = lambda t: (t - night_length - timedelta(hours=h1)).date() # night that ends before t (the day of t starts at its end)
|
night_of = lambda t: (t - night_length - timedelta(hours=h1)).date() # night that ends before t (the day of t starts at its end)
|
||||||
days = [night_of(t0) + timedelta(days=i) for i in range((night_of(t1) - night_of(t0)).days + 1)]
|
days = [night_of(t0) + timedelta(days=i) for i in range((night_of(t1) - night_of(t0)).days + 1)]
|
||||||
hm = lambda t: f"{t:%d/%m %H:%M}"
|
hm = lambda t: f"{t:%d/%m %H:%M}" # noqa: E731
|
||||||
|
span = lambda p, i, j: (f"from {hm(p[i][0])} (before {hm(p[i - 1][0]) if i else '-'}) " # noqa: E731
|
||||||
|
f"to {hm(p[j][0])} (after {hm(p[j + 1][0]) if j + 1 < len(p) else '-'})")
|
||||||
|
exceed = "threshold" if o.fraction else f"{o.k:g} noises"
|
||||||
threshold_txt = f"threshold {o.fraction:g} times the largest |deviation| of the day" if o.fraction else f"threshold {o.k:g} noises"
|
threshold_txt = f"threshold {o.fraction:g} times the largest |deviation| of the day" if o.fraction else f"threshold {o.k:g} noises"
|
||||||
print(f"\n== Bounds of each day: deviation from the level of the night that precedes it ({h1} h → {h2} h UTC), {threshold_txt}"
|
print(f"\n== Bounds of each day: deviation from the level of the night that precedes it ({h1} h → {h2} h UTC), {threshold_txt}"
|
||||||
+ (f", deviation smoothed over {o.smoothing} measurements" if o.smoothing > 1 else "") + " ==")
|
+ (f", deviation smoothed over {o.smoothing} measurements" if o.smoothing > 1 else "") + " ==")
|
||||||
@@ -248,7 +250,7 @@ def day_bounds(o, names, series, columns, station):
|
|||||||
"exceedance, just after the last; next night: its level minus that of the reference night")
|
"exceedance, just after the last; next night: its level minus that of the reference night")
|
||||||
fig, axes = plt.subplots(len(seqs), 1, sharex=True, figsize=(18, 2.2 * len(seqs)), squeeze=False)
|
fig, axes = plt.subplots(len(seqs), 1, sharex=True, figsize=(18, 2.2 * len(seqs)), squeeze=False)
|
||||||
for ax, (name, short, per_sensor) in zip(axes.flat, seqs):
|
for ax, (name, short, per_sensor) in zip(axes.flat, seqs):
|
||||||
lvl = {s: night_levels(p, h1, h2) for s, p in per_sensor.items()}
|
lvl = {s: {d: statistics.median(v for _, v in n) for d, n in nights_of(p, h1, h2).items()} for s, p in per_sensor.items()}
|
||||||
nights = sorted({d for n in lvl.values() for d in n})
|
nights = sorted({d for n in lvl.values() for d in n})
|
||||||
devs = [] # (day, ref. night, [(t, deviation)], deviation of the next night)
|
devs = [] # (day, ref. night, [(t, deviation)], deviation of the next night)
|
||||||
for d in days:
|
for d in days:
|
||||||
@@ -264,9 +266,7 @@ def day_bounds(o, names, series, columns, station):
|
|||||||
next_n = [lvl[s][d + timedelta(days=1)] - lvl[s][ref] for s in lvl if ref in lvl[s] and d + timedelta(days=1) in lvl[s]]
|
next_n = [lvl[s][d + timedelta(days=1)] - lvl[s][ref] for s in lvl if ref in lvl[s] and d + timedelta(days=1) in lvl[s]]
|
||||||
devs.append((start.date(), d, ref, [(t, statistics.median(e)) for t, e in sorted(per_t.items())],
|
devs.append((start.date(), d, ref, [(t, statistics.median(e)) for t, e in sorted(per_t.items())],
|
||||||
statistics.median(next_n) if next_n else None))
|
statistics.median(next_n) if next_n else None))
|
||||||
diffs = [abs(b[1] - a_[1]) for *_, p, _ in devs for a_, b in zip(p, p[1:])]
|
noise = own_noise(*[[e for _, e in p] for *_, p, _ in devs])
|
||||||
noise = statistics.median(diffs) if diffs else 0
|
|
||||||
noise = noise if noise >= ROUNDING else 0
|
|
||||||
print(f"\n{name}: noise {noise:.3g}")
|
print(f"\n{name}: noise {noise:.3g}")
|
||||||
in_noises = lambda e: f"{e:+.4g} ({e / noise:+.1f})" if noise else f"{e:+.4g}"
|
in_noises = lambda e: f"{e:+.4g} ({e / noise:+.1f})" if noise else f"{e:+.4g}"
|
||||||
h = o.smoothing // 2
|
h = o.smoothing // 2
|
||||||
@@ -281,11 +281,10 @@ def day_bounds(o, names, series, columns, station):
|
|||||||
over = [i for i, (_, e) in enumerate(p) if threshold and abs(e) > threshold]
|
over = [i for i, (_, e) in enumerate(p) if threshold and abs(e) > threshold]
|
||||||
if over:
|
if over:
|
||||||
i, j = over[0], over[-1]
|
i, j = over[0], over[-1]
|
||||||
line += (f"> {'threshold' if o.fraction else f'{o.k:g} noises'}: {len(over)} measurements, from {hm(p[i][0])} (before {hm(p[i - 1][0]) if i else '-'}) "
|
line += f"> {exceed}: {len(over)} measurements, {span(p, i, j)}"
|
||||||
f"to {hm(p[j][0])} (after {hm(p[j + 1][0]) if j + 1 < len(p) else '-'})")
|
|
||||||
ax.axvspan(p[i][0], p[j][0], color="0.85", lw=0)
|
ax.axvspan(p[i][0], p[j][0], color="0.85", lw=0)
|
||||||
else:
|
else:
|
||||||
line += f"never > {'threshold' if o.fraction else f'{o.k:g} noises'}"
|
line += f"never > {exceed}"
|
||||||
top, bottom = max(p, key=lambda x: x[1]), min(p, key=lambda x: x[1])
|
top, bottom = max(p, key=lambda x: x[1]), min(p, key=lambda x: x[1])
|
||||||
line += f" | max {in_noises(top[1])} at {hm(top[0])}, min {in_noises(bottom[1])} at {hm(bottom[0])}"
|
line += f" | max {in_noises(top[1])} at {hm(top[0])}, min {in_noises(bottom[1])} at {hm(bottom[0])}"
|
||||||
x = max(range(len(p)), key=lambda i: abs(p[i][1]))
|
x = max(range(len(p)), key=lambda i: abs(p[i][1]))
|
||||||
@@ -296,8 +295,7 @@ def day_bounds(o, names, series, columns, station):
|
|||||||
i -= 1
|
i -= 1
|
||||||
while j + 1 < len(p) and sign * p[j + 1][1] > threshold:
|
while j + 1 < len(p) and sign * p[j + 1][1] > threshold:
|
||||||
j += 1
|
j += 1
|
||||||
line += (f" | around the {'max' if sign > 0 else 'min'}: {j - i + 1} measurements, from {hm(p[i][0])} (before {hm(p[i - 1][0]) if i else '-'}) "
|
line += f" | around the {'max' if sign > 0 else 'min'}: {j - i + 1} measurements, {span(p, i, j)}"
|
||||||
f"to {hm(p[j][0])} (after {hm(p[j + 1][0]) if j + 1 < len(p) else '-'})")
|
|
||||||
ax.axvspan(p[i][0], p[j][0], color="0.7", lw=0)
|
ax.axvspan(p[i][0], p[j][0], color="0.7", lw=0)
|
||||||
line += f" | next night {in_noises(nxt)}" if nxt is not None else " | next night not measured"
|
line += f" | next night {in_noises(nxt)}" if nxt is not None else " | next night not measured"
|
||||||
print(line)
|
print(line)
|
||||||
@@ -309,13 +307,7 @@ def day_bounds(o, names, series, columns, station):
|
|||||||
for y in (o.k * noise, -o.k * noise):
|
for y in (o.k * noise, -o.k * noise):
|
||||||
ax.axhline(y, color="0.4", ls=":", lw=0.8)
|
ax.axhline(y, color="0.4", ls=":", lw=0.8)
|
||||||
ax.axhline(0, color="0.3", lw=0.5)
|
ax.axhline(0, color="0.3", lw=0.5)
|
||||||
all_e = sorted(e for *_, p, _ in devs for _, e in p)
|
out_of_frame = frame(ax, (e for *_, p, _ in devs for _, e in p))
|
||||||
out_of_frame = 0
|
|
||||||
if all_e: # frame from the 1st to the 99th percentile widened by half, as in byhour: an isolated peak does not crush the scale
|
|
||||||
q1, q2 = all_e[len(all_e) // 100], all_e[-1 - len(all_e) // 100]
|
|
||||||
margin = (q2 - q1) * 0.5 or 1
|
|
||||||
ax.set_ylim(q1 - margin, q2 + margin)
|
|
||||||
out_of_frame = sum(not q1 - margin <= e <= q2 + margin for e in all_e)
|
|
||||||
ax.set_title(f"{name} noise {noise:.2g}" + (f" ({out_of_frame} out of frame)" if out_of_frame else ""), fontsize=9)
|
ax.set_title(f"{name} noise {noise:.2g}" + (f" ({out_of_frame} out of frame)" if out_of_frame else ""), fontsize=9)
|
||||||
ax.grid(alpha=0.3)
|
ax.grid(alpha=0.3)
|
||||||
ax.tick_params(labelsize=7)
|
ax.tick_params(labelsize=7)
|
||||||
|
|||||||
@@ -17,7 +17,6 @@ from PIL import Image, ImageDraw, ImageFont
|
|||||||
from data import find_files, image_time, iso, parse_time
|
from data import find_files, image_time, iso, parse_time
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
a = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
a = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
a.add_argument("folder")
|
a.add_argument("folder")
|
||||||
|
|||||||
+61
-53
@@ -2,13 +2,14 @@
|
|||||||
|
|
||||||
Usage: python3 tools/scripts/inventory.py data/PS_DES_COUDRES/ [--start T] [--end T] [--files]
|
Usage: python3 tools/scripts/inventory.py data/PS_DES_COUDRES/ [--start T] [--end T] [--files]
|
||||||
The counts cover the whole period. With --start and/or --end, one more section lists, over this window,
|
The counts cover the whole period. With --start and/or --end, one more section lists, over this window,
|
||||||
the absent steps (dated, with the sensors that have no measurement) and, per sensor, the rejected measurements (dated,
|
the absent steps (dated, with the sensors that have no measurement) and, per sensor, the rejected measurements (out of
|
||||||
in runs of consecutive steps; rejected: error_value above the threshold, references included), then the number of
|
its measurements in the window; dated, in runs of consecutive steps; then the share rejected per UTC day, and per UTC hour of the day; rejected:
|
||||||
|
error_value above the threshold, references included), then the number of
|
||||||
references retained by the compensation (inlier = 1; inlier empty over the window: reproj_err filled) at each instant
|
references retained by the compensation (inlier = 1; inlier empty over the window: reproj_err filled) at each instant
|
||||||
and, dated, the runs of instants where references are not (which ones, and why: rejected, inlier value,
|
and, dated, the runs of instants where references are not (which ones, and why: rejected, inlier value,
|
||||||
empty). Camera movements (camera/): number of poses, period, and the measurement instants
|
empty). Camera movements (camera/): number of poses, period, and the measurement instants
|
||||||
without a pose (no registration), over the whole period then over the window. Before the counts, what each source
|
without a pose (no registration), over the whole period then over the window. Before the counts, what each source
|
||||||
contains (over the window, otherwise the whole period): the columns of the export, of the pose and of quality/, filled or empty (with,
|
contains (over the window, otherwise the whole period): the columns of the export, of the pose, of quality/ and of weather/, filled or empty (with,
|
||||||
for an export column that not all sensors carry, how many carry it), the names of the system metrics
|
for an export column that not all sensors carry, how many carry it), the names of the system metrics
|
||||||
of metrics/, the images and the differential images. Duplicate readings (same sensor, same instant in two
|
of metrics/, the images and the differential images. Duplicate readings (same sensor, same instant in two
|
||||||
exports, or twice in one) are counted per export. Images: per run of equal gaps between two images (within 5 min;
|
exports, or twice in one) are counted per export. Images: per run of equal gaps between two images (within 5 min;
|
||||||
@@ -20,14 +21,12 @@ import collections
|
|||||||
import csv
|
import csv
|
||||||
import glob
|
import glob
|
||||||
import os
|
import os
|
||||||
import re
|
|
||||||
import statistics
|
import statistics
|
||||||
from datetime import datetime, timedelta, timezone
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
from data import filter_window, find_files, image_time, is_reference, iso, load, load_camera, parse_time, rejected
|
from common import QUALITY, group, without_pose
|
||||||
from summary import QUALITY
|
from data import (METRIC_RE, filter_window, find_files, image_time, is_reference, iso, load, load_camera, load_targets,
|
||||||
|
parse_time, rejected)
|
||||||
METRIC_RE = re.compile(r"^(.+?)_(\d{4})_(\d{2})_(\d{2})_\d{2}_\d{2}") # NIEDERHORN_BusCurrent_2026_09_22_00_00 to ….csv
|
|
||||||
|
|
||||||
|
|
||||||
def span(a, b, n):
|
def span(a, b, n):
|
||||||
@@ -115,7 +114,9 @@ def contents(folder, f, start=None, end=None):
|
|||||||
for p in f["others"]:
|
for p in f["others"]:
|
||||||
others[os.path.relpath(p, folder).split(os.sep)[0]].append(p)
|
others[os.path.relpath(p, folder).split(os.sep)[0]].append(p)
|
||||||
for title, paths in (('"advanced" export', f["advanced"]), ("Camera pose (camera/)", f["camera"]),
|
for title, paths in (('"advanced" export', f["advanced"]), ("Camera pose (camera/)", f["camera"]),
|
||||||
("Image quality (quality/)", others.pop("quality", []))):
|
("Image quality (quality/)", others.pop("quality", [])),
|
||||||
|
("Weather (weather/, nearest MeteoSwiss station, hourly; tools/DATA.md; metrics.py --weather)",
|
||||||
|
others.pop("weather", []))):
|
||||||
if paths:
|
if paths:
|
||||||
n, count, who = filled(paths, start, end)
|
n, count, who = filled(paths, start, end)
|
||||||
everyone = set().union(*who.values())
|
everyone = set().union(*who.values())
|
||||||
@@ -158,20 +159,20 @@ def images(paths, start=None, end=None):
|
|||||||
return
|
return
|
||||||
# Runs of images at a regular gap (rounded to 5 min, which absorbs the few seconds of delay of a shot):
|
# Runs of images at a regular gap (rounded to 5 min, which absorbs the few seconds of delay of a shot):
|
||||||
# one line per change of gap, without losing any of the times covered.
|
# one line per change of gap, without losing any of the times covered.
|
||||||
runs = [] # [first, last, gap in min, number of gaps]
|
pairs = [(a, b, round((b - a).total_seconds() / 300) * 5) for a, b in zip(times, times[1:])]
|
||||||
for a, b in zip(times, times[1:]):
|
|
||||||
e = round((b - a).total_seconds() / 300) * 5
|
|
||||||
if runs and runs[-1][2] == e:
|
|
||||||
runs[-1][1], runs[-1][3] = b, runs[-1][3] + 1
|
|
||||||
else:
|
|
||||||
runs.append([a, b, e, 1])
|
|
||||||
what = "over the window" if start or end else "between the first and the last"
|
what = "over the window" if start or end else "between the first and the last"
|
||||||
print(f" gap between two images {what}, per run of equal gaps (within 5 min):")
|
print(f" gap between two images {what}, per run of equal gaps (within 5 min):")
|
||||||
for a, b, e, n in runs:
|
for g in group(pairs, lambda p, q: p[2] == q[2]):
|
||||||
print(f" {iso(a)} → {iso(b)}: " + (f"{n + 1} images, every {e} min" if n > 1 else f"one gap of {e} min"))
|
n, e = len(g), g[0][2]
|
||||||
|
print(f" {iso(g[0][0])} → {iso(g[-1][1])}: " + (f"{n + 1} images, every {e} min" if n > 1 else f"one gap of {e} min"))
|
||||||
|
|
||||||
|
|
||||||
def measurements(by):
|
def missing(instants, cadence):
|
||||||
|
"""Gaps between two present instants: [(before, after, number of absent steps)]."""
|
||||||
|
return [(a, b, n) for a, b in zip(instants, instants[1:]) if (n := round((b - a) / cadence) - 1) > 0]
|
||||||
|
|
||||||
|
|
||||||
|
def measurements(by, targets=None):
|
||||||
if not by:
|
if not by:
|
||||||
print('\nNo measurement: no "advanced" export in this folder.')
|
print('\nNo measurement: no "advanced" export in this folder.')
|
||||||
return
|
return
|
||||||
@@ -179,8 +180,7 @@ def measurements(by):
|
|||||||
cadence = statistics.median(b - a for a, b in zip(instants, instants[1:]))
|
cadence = statistics.median(b - a for a, b in zip(instants, instants[1:]))
|
||||||
print(f"\nMeasurement period: {iso(instants[0])} → {iso(instants[-1])}, {len(instants)} instants")
|
print(f"\nMeasurement period: {iso(instants[0])} → {iso(instants[-1])}, {len(instants)} instants")
|
||||||
print(f"Cadence (median step): {cadence.total_seconds() / 60:.0f} min")
|
print(f"Cadence (median step): {cadence.total_seconds() / 60:.0f} min")
|
||||||
gaps = [(a, b, round((b - a) / cadence) - 1) for a, b in zip(instants, instants[1:])]
|
gaps = missing(instants, cadence)
|
||||||
gaps = [x for x in gaps if x[2] > 0]
|
|
||||||
print(f"Missing measurements (instants absent from the cadence): {sum(n for *_, n in gaps)}")
|
print(f"Missing measurements (instants absent from the cadence): {sum(n for *_, n in gaps)}")
|
||||||
for a, b, n in gaps:
|
for a, b, n in gaps:
|
||||||
print(f" between {iso(a)} and {iso(b)}: {n}")
|
print(f" between {iso(a)} and {iso(b)}: {n}")
|
||||||
@@ -191,12 +191,23 @@ def measurements(by):
|
|||||||
if empty:
|
if empty:
|
||||||
print(f" pattern_type empty for {len(empty)} sensors: reference when reproj_err or inlier is filled "
|
print(f" pattern_type empty for {len(empty)} sensors: reference when reproj_err or inlier is filled "
|
||||||
"(references only, tools/DATA.md)")
|
"(references only, tools/DATA.md)")
|
||||||
print(f"{'sensor':10} {'type':10} {'pattern':11} {'threshold':>9} {'measurements':>12} {'rejected':>8}")
|
targets = targets or {}
|
||||||
|
if targets:
|
||||||
|
print(" distance (m) and mm/px: targets.csv (data.load_targets); \"-\": not in it, or not filled in "
|
||||||
|
"(distance 1000 mm); \"?\": its lines (n) disagree, no corr_x_mm / corr_y_mm")
|
||||||
|
else:
|
||||||
|
print(" no targets.csv: no distance, no corr_x_mm / corr_y_mm")
|
||||||
|
print(f"{'sensor':10} {'type':10} {'pattern':11} {'threshold':>9} {'measurements':>12} {'rejected':>8}"
|
||||||
|
f" {'distance':>9} {'mm/px':>8}")
|
||||||
for s in names:
|
for s in names:
|
||||||
ms = by[s]
|
ms = by[s]
|
||||||
threshold = "/".join(f"{x:g}" for x in sorted({m['error_threshold'] for m in ms}))
|
threshold = "/".join(f"{x:g}" for x in sorted({m['error_threshold'] for m in ms}))
|
||||||
|
t = targets.get(s, {})
|
||||||
|
mark = f"?({t['lines']})" if t.get("lines", 0) > 1 else "-"
|
||||||
|
distance = mark if t.get("distance_mm") is None else f"{t['distance_mm'] / 1000:g}"
|
||||||
|
factor = mark if t.get("mm_per_px") is None else f"{t['mm_per_px']:.3f}"
|
||||||
print(f"{s:10} {'reference' if is_reference(ms) else 'target':10} {ms[0]['pattern_type']:11} "
|
print(f"{s:10} {'reference' if is_reference(ms) else 'target':10} {ms[0]['pattern_type']:11} "
|
||||||
f"{threshold:>9} {len(ms):12} {sum(map(rejected, ms)):8}")
|
f"{threshold:>9} {len(ms):12} {sum(map(rejected, ms)):8} {distance:>9} {factor:>8}")
|
||||||
|
|
||||||
|
|
||||||
def camera(folder, by, start=None, end=None):
|
def camera(folder, by, start=None, end=None):
|
||||||
@@ -207,19 +218,11 @@ def camera(folder, by, start=None, end=None):
|
|||||||
if not poses:
|
if not poses:
|
||||||
print(f"\nCamera pose {what}: none")
|
print(f"\nCamera pose {what}: none")
|
||||||
return
|
return
|
||||||
with_pose = {p["t"].replace(second=0, microsecond=0) for p in poses}
|
|
||||||
print(f"\nCamera pose {what}: {len(poses)} registrations, {iso(poses[0]['t'])} → {iso(poses[-1]['t'])}")
|
print(f"\nCamera pose {what}: {len(poses)} registrations, {iso(poses[0]['t'])} → {iso(poses[-1]['t'])}")
|
||||||
runs = [] # [start, end, count, rank] of the consecutive measurement instants without a pose
|
runs = without_pose(instants, poses)
|
||||||
for i, t in enumerate(instants):
|
|
||||||
if t.replace(second=0, microsecond=0) in with_pose:
|
|
||||||
continue
|
|
||||||
if runs and runs[-1][3] == i - 1:
|
|
||||||
runs[-1][1], runs[-1][2], runs[-1][3] = t, runs[-1][2] + 1, i
|
|
||||||
else:
|
|
||||||
runs.append([t, t, 1, i])
|
|
||||||
print(f"Measurement instants without a pose (no registration): {sum(x[2] for x in runs)} of {len(instants)}")
|
print(f"Measurement instants without a pose (no registration): {sum(x[2] for x in runs)} of {len(instants)}")
|
||||||
if start or end:
|
if start or end:
|
||||||
for a, b, n, _ in runs:
|
for a, b, n in runs:
|
||||||
print(f" {span(a, b, n)}")
|
print(f" {span(a, b, n)}")
|
||||||
|
|
||||||
|
|
||||||
@@ -232,8 +235,7 @@ def window(by, start, end):
|
|||||||
print(f"\nWindow {iso(inside[0])} → {iso(inside[-1])}: {len(inside)} instants present")
|
print(f"\nWindow {iso(inside[0])} → {iso(inside[-1])}: {len(inside)} instants present")
|
||||||
names = sorted(by, key=lambda s: (not is_reference(by[s]), s))
|
names = sorted(by, key=lambda s: (not is_reference(by[s]), s))
|
||||||
print(f"Steps absent for all sensors, between two present instants (cadence of {cadence.total_seconds() / 60:.0f} min):")
|
print(f"Steps absent for all sensors, between two present instants (cadence of {cadence.total_seconds() / 60:.0f} min):")
|
||||||
gaps = [(a, b, round((b - a) / cadence) - 1) for a, b in zip(inside, inside[1:])]
|
gaps = missing(inside, cadence)
|
||||||
gaps = [x for x in gaps if x[2] > 0]
|
|
||||||
for a, b, n in gaps:
|
for a, b, n in gaps:
|
||||||
print(f" between {iso(a)} and {iso(b)}: {n}")
|
print(f" between {iso(a)} and {iso(b)}: {n}")
|
||||||
if not gaps:
|
if not gaps:
|
||||||
@@ -245,17 +247,27 @@ def window(by, start, end):
|
|||||||
print(f" {iso(t)}: {len(lacking)} sensors: {','.join(lacking)}")
|
print(f" {iso(t)}: {len(lacking)} sensors: {','.join(lacking)}")
|
||||||
if not any(m for _, m in partial):
|
if not any(m for _, m in partial):
|
||||||
print(" (none)")
|
print(" (none)")
|
||||||
print("Rejected measurements in the window, per sensor (start → end of each run of consecutive steps, count):")
|
print("Rejected measurements in the window, per sensor (rejected of measurements; start → end of each run of "
|
||||||
|
"consecutive steps, count), then the share rejected per UTC day (MM-DD:%), then the rejected per UTC hour of the "
|
||||||
|
"day, all days together:")
|
||||||
for s in names:
|
for s in names:
|
||||||
rej = [t for t in sorted(series[s]) if rejected(series[s][t])]
|
rej = [t for t in sorted(series[s]) if rejected(series[s][t])]
|
||||||
runs = []
|
|
||||||
for t in rej:
|
|
||||||
if runs and t - runs[-1][1] <= cadence * 1.5:
|
|
||||||
runs[-1][1], runs[-1][2] = t, runs[-1][2] + 1
|
|
||||||
else:
|
|
||||||
runs.append([t, t, 1])
|
|
||||||
if rej:
|
if rej:
|
||||||
print(f" {s} ({len(rej)}): " + "; ".join(span(a, b, n) for a, b, n in runs))
|
n = len(series[s])
|
||||||
|
print(f" {s} ({len(rej)} of {n}, {100 * len(rej) / n:.1f} %): "
|
||||||
|
+ "; ".join(span(g[0], g[-1], len(g)) for g in group(rej, lambda a, b: b - a <= cadence * 1.5)))
|
||||||
|
days = collections.defaultdict(lambda: [0, 0])
|
||||||
|
for t, m in series[s].items():
|
||||||
|
days[t.date()][0] += 1
|
||||||
|
days[t.date()][1] += rejected(m)
|
||||||
|
print(f" {s} per day, {sum(1 for v in days.values() if v[1])} of {len(days)} days with a rejection: "
|
||||||
|
+ " ".join(f"{d:%m-%d}:{100 * r / k:.0f}" for d, (k, r) in sorted(days.items())))
|
||||||
|
hours = collections.defaultdict(lambda: [0, 0])
|
||||||
|
for t, m in series[s].items():
|
||||||
|
hours[t.hour][0] += 1
|
||||||
|
hours[t.hour][1] += rejected(m)
|
||||||
|
print(f" {s} per UTC hour of the day, all days together (HH:rejected/measurements): "
|
||||||
|
+ " ".join(f"{h:02d}:{r}/{k}" for h, (k, r) in sorted(hours.items())))
|
||||||
if not any(rejected(m) for d in series.values() for m in d.values()):
|
if not any(rejected(m) for d in series.values() for m in d.values()):
|
||||||
print(" (none)")
|
print(" (none)")
|
||||||
refs = [s for s in names if is_reference(by[s])]
|
refs = [s for s in names if is_reference(by[s])]
|
||||||
@@ -276,14 +288,10 @@ def window(by, start, end):
|
|||||||
+ ", ".join(f"{k}: {count[k]}" for k in sorted(count, reverse=True)))
|
+ ", ".join(f"{k}: {count[k]}" for k in sorted(count, reverse=True)))
|
||||||
print("Instants where references are not retained (runs of consecutive steps with the same references out; "
|
print("Instants where references are not retained (runs of consecutive steps with the same references out; "
|
||||||
f"r = rejected, empty = {col} empty, otherwise the inlier value):")
|
f"r = rejected, empty = {col} empty, otherwise the inlier value):")
|
||||||
runs = []
|
runs = group([(i, t, h) for i, (t, h) in enumerate(out) if h], lambda a, b: b[0] == a[0] + 1 and b[2] == a[2])
|
||||||
for i, (t, h) in enumerate(out):
|
for g in runs:
|
||||||
if h and runs and runs[-1][3] == h and runs[-1][4] == i - 1:
|
h = g[0][2]
|
||||||
runs[-1][1], runs[-1][2], runs[-1][4] = t, runs[-1][2] + 1, i
|
print(f" {span(g[0][1], g[-1][1], len(g))}: {len(refs) - len(h)} retained; out: "
|
||||||
elif h:
|
|
||||||
runs.append([t, t, 1, h, i])
|
|
||||||
for a, b, n, h, _ in runs:
|
|
||||||
print(f" {span(a, b, n)}: {len(refs) - len(h)} retained; out: "
|
|
||||||
+ ", ".join(f"{s} ({q})" for s, q in h))
|
+ ", ".join(f"{s} ({q})" for s, q in h))
|
||||||
if not runs:
|
if not runs:
|
||||||
print(" (none)")
|
print(" (none)")
|
||||||
@@ -296,7 +304,7 @@ def main(folder, start=None, end=None, full=False):
|
|||||||
contents(folder, f, start, end)
|
contents(folder, f, start, end)
|
||||||
images(f["images"], start, end)
|
images(f["images"], start, end)
|
||||||
by = load(folder)
|
by = load(folder)
|
||||||
measurements(by)
|
measurements(by, load_targets(folder))
|
||||||
if by:
|
if by:
|
||||||
camera(folder, by)
|
camera(folder, by)
|
||||||
if by and (start or end):
|
if by and (start or end):
|
||||||
|
|||||||
+42
-51
@@ -3,13 +3,17 @@ against the target median of corr_x, corr_y and the camera pose.
|
|||||||
|
|
||||||
Usage: .venv/bin/python tools/scripts/metrics.py data/PS_DES_COUDRES/
|
Usage: .venv/bin/python tools/scripts/metrics.py data/PS_DES_COUDRES/
|
||||||
[--names sensor_A0_deg_c,BusVoltage] [--step 1h] [--stat median|min|max] [--against temperature] [--smoothing 24h]
|
[--names sensor_A0_deg_c,BusVoltage] [--step 1h] [--stat median|min|max] [--against temperature] [--smoothing 24h]
|
||||||
[--metrics-only] [--start ...] [--end ...] [--out FOLDER]
|
[--metrics-only] [--weather [precipitation_mm,temperature_c,...]] [--sensors A,B] [--start ...] [--end ...]
|
||||||
|
[--out FOLDER]
|
||||||
|
|
||||||
Columns: `temperature` (median of the sensor_*_deg_c components at each hour, as the detectors read it,
|
Columns: `temperature` (median of the sensor_*_deg_c components at each hour, as the detectors read it,
|
||||||
data.load_temperature), then each metric of --names (default: the *_deg_c and BusVoltage present), then,
|
data.load_temperature), then each metric of --names (default: the *_deg_c and BusVoltage present), then,
|
||||||
unless --metrics-only, tgt_corr_x, tgt_corr_y (at each measurement, median over the targets of corr minus the target's
|
unless --metrics-only, tgt_corr_x, tgt_corr_y (at each measurement, median over the targets of corr minus the target's
|
||||||
own level, its median over the window; accepted measurements only) and the raw camera pose (cam_pan,
|
own level, its median over the window; accepted measurements only), with --sensors A,B the same for each of these
|
||||||
cam_tilt, cam_rotation, cam_scale, cam_sigma0). An empty cell of a metrics/ CSV is skipped, not read as zero.
|
sensors alone (A_corr_x, A_corr_y: one sensor that follows the temperature where the median does not), and the raw camera pose (cam_pan,
|
||||||
|
cam_tilt, cam_rotation, cam_scale, cam_sigma0); with --weather, the hourly MeteoSwiss weather of weather/
|
||||||
|
(data.load_weather; the hour starting at time_utc), one w_<column> column each (default WEATHER), and where it was
|
||||||
|
measured is printed first. An empty cell of a metrics/ CSV is skipped, not read as zero.
|
||||||
A text metric (BatteryState, link, last_sync…) is given as it is (the last one of the step).
|
A text metric (BatteryState, link, last_sync…) is given as it is (the last one of the step).
|
||||||
Never an identifier: SIM_CCID and ip* are neither read nor listed.
|
Never an identifier: SIM_CCID and ip* are neither read nor listed.
|
||||||
Prints: the available metrics; the table, one line per step of --step (start of the step, UTC; --stat of the values of the
|
Prints: the available metrics; the table, one line per step of --step (start of the step, UTC; --stat of the values of the
|
||||||
@@ -34,10 +38,11 @@ matplotlib.use("Agg")
|
|||||||
import matplotlib.dates as mdates # noqa: E402
|
import matplotlib.dates as mdates # noqa: E402
|
||||||
import matplotlib.pyplot as plt # noqa: E402
|
import matplotlib.pyplot as plt # noqa: E402
|
||||||
|
|
||||||
from data import POSE, filter_window, is_reference, iso, load, load_camera, load_temperature, parse_time # noqa: E402
|
from common import STATS, bound, cut, duration, group, targets # noqa: E402
|
||||||
from inventory import METRIC_RE # noqa: E402
|
from data import (METRIC_RE, POSE, WHERE, filter_window, iso, load, load_camera, load_temperature, # noqa: E402
|
||||||
from plot import bound # noqa: E402
|
load_weather, parse_time)
|
||||||
from summary import STATS, duration # noqa: E402
|
|
||||||
|
WEATHER = "precipitation_mm,temperature_c,humidity_pct,wind_speed_kmh,sunshine_min" # default of --weather
|
||||||
|
|
||||||
|
|
||||||
def is_identifier(name):
|
def is_identifier(name):
|
||||||
@@ -76,20 +81,6 @@ def read_metrics(folder, start, end):
|
|||||||
return {n: filter_window([{"t": t, "v": s[t]} for t in sorted(s)], start, end) for n, s in by_name.items()}
|
return {n: filter_window([{"t": t, "v": s[t]} for t in sorted(s)], start, end) for n, s in by_name.items()}
|
||||||
|
|
||||||
|
|
||||||
def targets(by_sensor, col, start, end):
|
|
||||||
"""At each measurement, median over the targets of col minus the target's own level (median over the window)."""
|
|
||||||
deviations = {}
|
|
||||||
for s, ms in by_sensor.items():
|
|
||||||
if is_reference(ms):
|
|
||||||
continue
|
|
||||||
ms = [m for m in filter_window(ms, start, end) if m[col] is not None]
|
|
||||||
if ms:
|
|
||||||
level = statistics.median(m[col] for m in ms)
|
|
||||||
for m in ms:
|
|
||||||
deviations.setdefault(m["t"], []).append(m[col] - level)
|
|
||||||
return [{"t": t, "v": statistics.median(v)} for t, v in sorted(deviations.items())]
|
|
||||||
|
|
||||||
|
|
||||||
def smooth(series, width):
|
def smooth(series, width):
|
||||||
"""Moving average centred over width (numeric values)."""
|
"""Moving average centred over width (numeric values)."""
|
||||||
pts = [p for p in series if isinstance(p["v"], float)]
|
pts = [p for p in series if isinstance(p["v"], float)]
|
||||||
@@ -101,33 +92,6 @@ def fmt(v):
|
|||||||
return "-" if v is None else f"{v:.6g}" if isinstance(v, float) else str(v)
|
return "-" if v is None else f"{v:.6g}" if isinstance(v, float) else str(v)
|
||||||
|
|
||||||
|
|
||||||
def runs(empty_steps, step):
|
|
||||||
"""[t] sorted -> [(start, end, n)] of consecutive steps."""
|
|
||||||
out = []
|
|
||||||
for t in empty_steps:
|
|
||||||
if out and t - out[-1][1] == step:
|
|
||||||
out[-1] = (out[-1][0], t, out[-1][2] + 1)
|
|
||||||
else:
|
|
||||||
out.append((t, t, 1))
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def cut(series):
|
|
||||||
"""Times and numeric values, NaN inserted over a gap of more than 1.5 times the usual step (the line breaks)."""
|
|
||||||
pts = [(p["t"], p["v"]) for p in series if isinstance(p["v"], float)]
|
|
||||||
if len(pts) < 3:
|
|
||||||
return [t for t, _ in pts], [v for _, v in pts]
|
|
||||||
step = statistics.median(b[0] - a[0] for a, b in zip(pts, pts[1:]))
|
|
||||||
ts, vs = [pts[0][0]], [pts[0][1]]
|
|
||||||
for (a, _), (b, v) in zip(pts, pts[1:]):
|
|
||||||
if b - a > 1.5 * step:
|
|
||||||
ts.append(a + (b - a) / 2)
|
|
||||||
vs.append(float("nan"))
|
|
||||||
ts.append(b)
|
|
||||||
vs.append(v)
|
|
||||||
return ts, vs
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
a = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
a = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
a.add_argument("folder")
|
a.add_argument("folder")
|
||||||
@@ -137,6 +101,9 @@ def main():
|
|||||||
a.add_argument("--against", default="temperature", help="column on which r and the slope are computed")
|
a.add_argument("--against", default="temperature", help="column on which r and the slope are computed")
|
||||||
a.add_argument("--smoothing", type=duration, help="centred moving average of --against, e.g. 24h")
|
a.add_argument("--smoothing", type=duration, help="centred moving average of --against, e.g. 24h")
|
||||||
a.add_argument("--metrics-only", action="store_true", help="without tgt_corr_x, tgt_corr_y or the pose")
|
a.add_argument("--metrics-only", action="store_true", help="without tgt_corr_x, tgt_corr_y or the pose")
|
||||||
|
a.add_argument("--weather", nargs="?", const=WEATHER, metavar="COLUMNS",
|
||||||
|
help=f"add the weather/ columns, comma-separated (without a value: {WEATHER})")
|
||||||
|
a.add_argument("--sensors", help="sensors, comma-separated: their own corr_x, corr_y minus their level, as columns")
|
||||||
a.add_argument("--start", type=parse_time)
|
a.add_argument("--start", type=parse_time)
|
||||||
a.add_argument("--end", type=parse_time)
|
a.add_argument("--end", type=parse_time)
|
||||||
a.add_argument("--out", default=os.path.join("out", "diagnostic"), help="folder of the PNG")
|
a.add_argument("--out", default=os.path.join("out", "diagnostic"), help="folder of the PNG")
|
||||||
@@ -164,10 +131,33 @@ def main():
|
|||||||
sys.exit(f"--against {o.against}: not a column ({', '.join(series)}).")
|
sys.exit(f"--against {o.against}: not a column ({', '.join(series)}).")
|
||||||
series[o.against + "_smooth"] = smooth(series[o.against], o.smoothing)
|
series[o.against + "_smooth"] = smooth(series[o.against], o.smoothing)
|
||||||
metric_cols = list(series)
|
metric_cols = list(series)
|
||||||
|
sensor_cols = set() # columns of --sensors
|
||||||
|
if o.weather:
|
||||||
|
hours = load_weather(o.folder)
|
||||||
|
if not hours:
|
||||||
|
print(f"No weather/ in {o.folder}.")
|
||||||
|
else:
|
||||||
|
known = [c for c in hours[0] if c not in WHERE and c != "t"]
|
||||||
|
if missing := [c for c in o.weather.split(",") if c not in known]:
|
||||||
|
sys.exit(f"Unknown weather columns: {', '.join(missing)}. Known: {', '.join(known)}.")
|
||||||
|
h = hours[-1]
|
||||||
|
print(f"Weather: {h.get('station_name')} ({h.get('place')}), {h.get('distance_km')} km, {h.get('height_m')} m; "
|
||||||
|
"hourly, the hour starting at the time given")
|
||||||
|
hours = filter_window(hours, o.start, o.end)
|
||||||
|
for c in o.weather.split(","):
|
||||||
|
series["w_" + c] = [{"t": p["t"], "v": p[c]} for p in hours if p[c] is not None]
|
||||||
if not o.metrics_only:
|
if not o.metrics_only:
|
||||||
by_sensor = load(o.folder)
|
by_sensor = load(o.folder)
|
||||||
for c in ("corr_x", "corr_y"):
|
for c in ("corr_x", "corr_y"):
|
||||||
series["tgt_" + c] = targets(by_sensor, c, o.start, o.end)
|
series["tgt_" + c] = targets(by_sensor, c, o.start, o.end)
|
||||||
|
for s in o.sensors.split(",") if o.sensors else []:
|
||||||
|
if s not in by_sensor:
|
||||||
|
sys.exit(f"Unknown sensor: {s}. Known: {', '.join(by_sensor)}.")
|
||||||
|
for c in ("corr_x", "corr_y"):
|
||||||
|
ms = [m for m in filter_window(by_sensor[s], o.start, o.end) if m[c] is not None]
|
||||||
|
level = statistics.median(m[c] for m in ms) if ms else 0
|
||||||
|
series[f"{s}_{c}"] = [{"t": m["t"], "v": m[c] - level} for m in ms]
|
||||||
|
sensor_cols.add(f"{s}_{c}")
|
||||||
poses = filter_window(load_camera(o.folder), o.start, o.end)
|
poses = filter_window(load_camera(o.folder), o.start, o.end)
|
||||||
for c in POSE:
|
for c in POSE:
|
||||||
series["cam_" + c] = [{"t": p["t"], "v": p[c]} for p in poses if p[c] is not None]
|
series["cam_" + c] = [{"t": p["t"], "v": p[c]} for p in poses if p[c] is not None]
|
||||||
@@ -229,13 +219,14 @@ def main():
|
|||||||
|
|
||||||
empty = [k for k in grid if all(k not in table.get(c, {}) for c in metric_cols)]
|
empty = [k for k in grid if all(k not in table.get(c, {}) for c in metric_cols)]
|
||||||
print(f"\nSteps without any metric ({', '.join(metric_cols)}): {len(empty)} of {len(grid)}")
|
print(f"\nSteps without any metric ({', '.join(metric_cols)}): {len(empty)} of {len(grid)}")
|
||||||
for d, f, n in runs(empty, step):
|
for g in group(empty, lambda a, b: b - a == step):
|
||||||
print(f" {iso(d)} → {iso(f)} ({n} steps)")
|
print(f" {iso(g[0])} → {iso(g[-1])} ({len(g)} steps)")
|
||||||
|
|
||||||
panels = {} # title -> columns
|
panels = {} # title -> columns
|
||||||
for c in columns:
|
for c in columns:
|
||||||
if any(isinstance(p["v"], float) for p in series[c]):
|
if any(isinstance(p["v"], float) for p in series[c]):
|
||||||
key = "°C" if c.startswith("temperature") or "_deg_c" in c else c
|
key = ("°C" if c.startswith("temperature") or "_deg_c" in c
|
||||||
|
else "tgt_" + c[-6:] if c in sensor_cols else c) # a sensor's corr on the panel of the targets'
|
||||||
panels.setdefault(key, []).append(c)
|
panels.setdefault(key, []).append(c)
|
||||||
if not panels:
|
if not panels:
|
||||||
print("\nNothing to plot.")
|
print("\nNothing to plot.")
|
||||||
|
|||||||
+75
-111
@@ -62,9 +62,8 @@ matplotlib.use("Agg")
|
|||||||
import matplotlib.dates as mdates # noqa: E402
|
import matplotlib.dates as mdates # noqa: E402
|
||||||
import matplotlib.pyplot as plt # noqa: E402
|
import matplotlib.pyplot as plt # noqa: E402
|
||||||
|
|
||||||
from data import POSE, filter_window, is_reference, iso, load, load_camera, parse_time, rejected # noqa: E402
|
from common import COLUMNS, bound, check, duration, extremes, outside, own_noise, per_step, sensors, window_filter, without_pose # noqa: E402
|
||||||
from plot import COLUMNS, bound # noqa: E402
|
from data import POSE, filter_window, is_reference, iso, load_camera, parse_time, rejected # noqa: E402
|
||||||
from summary import duration, own_noise # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
def level(ms, col):
|
def level(ms, col):
|
||||||
@@ -166,11 +165,9 @@ def rigid(o, names, series, times, base, station):
|
|||||||
+ f".\nResidual = raw position minus the position at {iso(t_b)} carried by this movement [px]; rejected measurements left out of the fit, "
|
+ f".\nResidual = raw position minus the position at {iso(t_b)} carried by this movement [px]; rejected measurements left out of the fit, "
|
||||||
f"of tgt_med, tgt_max and of the sensor columns (separately in the per-sensor summary); median per step of {int(o.step.total_seconds() // 60)} min{' (table: ' + o.table + ')' if o.table else ''}:")
|
f"of tgt_med, tgt_max and of the sensor columns (separately in the per-sensor summary); median per step of {int(o.step.total_seconds() // 60)} min{' (table: ' + o.table + ')' if o.table else ''}:")
|
||||||
cols = ["rot_deg", "scale", "tx", "ty", "tgt_med", "tgt_max"] + [f"{s}_{c}" for s in refs + followed for c in ("dx", "dy")]
|
cols = ["rot_deg", "scale", "tx", "ty", "tgt_med", "tgt_max"] + [f"{s}_{c}" for s in refs + followed for c in ("dx", "dy")]
|
||||||
midnight = tf[0].replace(hour=0, minute=0, second=0, microsecond=0)
|
key, rows = per_step(tf[0], o.step), {}
|
||||||
t0 = midnight + ((tf[0] - midnight) // o.step) * o.step
|
|
||||||
per_step = {}
|
|
||||||
for t in tf:
|
for t in tf:
|
||||||
line = per_step.setdefault(t0 + ((t - t0) // o.step) * o.step, {})
|
line = rows.setdefault(key(t), {})
|
||||||
for c, v in zip(cols, fit[t]):
|
for c, v in zip(cols, fit[t]):
|
||||||
line.setdefault(c, []).append(v)
|
line.setdefault(c, []).append(v)
|
||||||
for s in refs + followed:
|
for s in refs + followed:
|
||||||
@@ -178,10 +175,10 @@ def rigid(o, names, series, times, base, station):
|
|||||||
line.setdefault(f"{s}_dx", []).append(res[s][t][0]), line.setdefault(f"{s}_dy", []).append(res[s][t][1])
|
line.setdefault(f"{s}_dx", []).append(res[s][t][0]), line.setdefault(f"{s}_dy", []).append(res[s][t][1])
|
||||||
w = max(10, max(len(c) for c in cols) + 1)
|
w = max(10, max(len(c) for c in cols) + 1)
|
||||||
header = f"{'time':17}" + "".join(f"{c:>{w}}" for c in cols)
|
header = f"{'time':17}" + "".join(f"{c:>{w}}" for c in cols)
|
||||||
for i, t in enumerate(t for t in sorted(per_step) if o.in_table(t)):
|
for i, t in enumerate(t for t in sorted(rows) if o.in_table(t)):
|
||||||
if i % 24 == 0:
|
if i % 24 == 0:
|
||||||
print(header)
|
print(header)
|
||||||
print(f"{iso(t):17}" + "".join(f"{statistics.median(per_step[t][c]):{w}.{7 if c == 'scale' else 3}g}" if c in per_step[t] else " " * w for c in cols))
|
print(f"{iso(t):17}" + "".join(f"{statistics.median(rows[t][c]):{w}.{7 if c == 'scale' else 3}g}" if c in rows[t] else " " * w for c in cols))
|
||||||
print("\nPer sensor, accepted measurements: min and max of dx, dy, value at the last time; then the rejected ones separately:")
|
print("\nPer sensor, accepted measurements: min and max of dx, dy, value at the last time; then the rejected ones separately:")
|
||||||
for s in names:
|
for s in names:
|
||||||
good = {t: v for t, v in res[s].items() if t in acc[s]}
|
good = {t: v for t, v in res[s].items() if t in acc[s]}
|
||||||
@@ -189,15 +186,15 @@ def rigid(o, names, series, times, base, station):
|
|||||||
for k, c in ((0, "dx"), (1, "dy")):
|
for k, c in ((0, "dx"), (1, "dy")):
|
||||||
line = f"{s:9}{' (ref.)' if s in refs else ' '} {c}"
|
line = f"{s:9}{' (ref.)' if s in refs else ' '} {c}"
|
||||||
if good:
|
if good:
|
||||||
lo, hi = min(good.items(), key=lambda x: x[1][k]), max(good.items(), key=lambda x: x[1][k])
|
lo, hi = extremes({t: v[k] for t, v in good.items()})
|
||||||
last = res[s].get(tf[-1])
|
last = res[s].get(tf[-1])
|
||||||
line += (f" min {lo[1][k]:+.3g} at {iso(lo[0])}, max {hi[1][k]:+.3g} at {iso(hi[0])}, at the last time "
|
line += (f" min {lo[1]:+.3g} at {iso(lo[0])}, max {hi[1]:+.3g} at {iso(hi[0])}, at the last time "
|
||||||
f"{last[k] if last else math.nan:+.3g}{' (rejected)' if last and tf[-1] not in acc[s] else ''}")
|
f"{last[k] if last else math.nan:+.3g}{' (rejected)' if last and tf[-1] not in acc[s] else ''}")
|
||||||
else:
|
else:
|
||||||
line += " no accepted measurement"
|
line += " no accepted measurement"
|
||||||
if rej:
|
if rej:
|
||||||
lo, hi = min(rej.items(), key=lambda x: x[1][k]), max(rej.items(), key=lambda x: x[1][k])
|
lo, hi = extremes({t: v[k] for t, v in rej.items()})
|
||||||
line += f"; {len(rej)} rejected: min {lo[1][k]:+.3g} at {iso(lo[0])}, max {hi[1][k]:+.3g} at {iso(hi[0])}"
|
line += f"; {len(rej)} rejected: min {lo[1]:+.3g} at {iso(lo[0])}, max {hi[1]:+.3g} at {iso(hi[0])}"
|
||||||
if good or rej:
|
if good or rej:
|
||||||
print(line)
|
print(line)
|
||||||
|
|
||||||
@@ -224,26 +221,14 @@ def main():
|
|||||||
a.add_argument("--out", default=os.path.join("out", "diagnostic"), help="folder of the PNGs")
|
a.add_argument("--out", default=os.path.join("out", "diagnostic"), help="folder of the PNGs")
|
||||||
o = a.parse_args()
|
o = a.parse_args()
|
||||||
|
|
||||||
by_sensor = load(o.folder)
|
by_sensor, names = sensors(o.folder, o.sensors)
|
||||||
if not by_sensor:
|
|
||||||
sys.exit(f"No \"advanced\" export in {o.folder} (see inventory.py).")
|
|
||||||
names = o.sensors.split(",") if o.sensors else sorted(by_sensor, key=lambda s: (not is_reference(by_sensor[s]), s))
|
|
||||||
columns = o.columns.split(",")
|
columns = o.columns.split(",")
|
||||||
residuals = [c for c in o.residuals.split(",") if c and c != "no"] # '' or "no": none, as for --table
|
residuals = [c for c in o.residuals.split(",") if c and c != "no"] # '' or "no": none, as for --table
|
||||||
outside = tuple(parse_time(x) for x in o.outside.split("/")) if o.outside else None
|
kept = outside(o.outside)
|
||||||
if outside and len(outside) != 2:
|
o.in_table = window_filter(o.table)
|
||||||
sys.exit("--outside expects T1/T2.")
|
check("sensors", names, by_sensor)
|
||||||
if o.table in ("", "no"):
|
check("sensors of --follow (taken from --sensors)", [s for s in o.follow.split(",") if s], names)
|
||||||
o.in_table = lambda t: o.table == ""
|
check("columns", columns + residuals + [o.column], COLUMNS)
|
||||||
else:
|
|
||||||
limits = [parse_time(x) for x in o.table.split("/")]
|
|
||||||
if len(limits) != 2:
|
|
||||||
sys.exit("--table expects T1/T2 or no.")
|
|
||||||
o.in_table = lambda t: limits[0] <= t <= limits[1]
|
|
||||||
for requested, known, what in ((names, by_sensor, "sensors"), ([s for s in o.follow.split(",") if s], names, "sensors of --follow (taken from --sensors)"), (columns + residuals + [o.column], COLUMNS, "columns")):
|
|
||||||
missing = [x for x in requested if x not in known]
|
|
||||||
if missing:
|
|
||||||
sys.exit(f"Unknown {what}: {', '.join(missing)}. Known: {', '.join(sorted(known))}.")
|
|
||||||
series = {s: filter_window(by_sensor[s], o.start, o.end) for s in names}
|
series = {s: filter_window(by_sensor[s], o.start, o.end) for s in names}
|
||||||
if not any(series.values()):
|
if not any(series.values()):
|
||||||
sys.exit("No measurement in the requested window.")
|
sys.exit("No measurement in the requested window.")
|
||||||
@@ -274,7 +259,7 @@ def main():
|
|||||||
ax.plot(cam_t, [pose_t.get(t, math.nan) for t in cam_t], ".-", ms=3, lw=0.5, color="C2", label="camera pose (one point per registration)")
|
ax.plot(cam_t, [pose_t.get(t, math.nan) for t in cam_t], ".-", ms=3, lw=0.5, color="C2", label="camera pose (one point per registration)")
|
||||||
ax.set_ylabel(f"camera: {col}" + (" [deg]" if col in ("pan", "tilt", "rotation") else ""))
|
ax.set_ylabel(f"camera: {col}" + (" [deg]" if col in ("pan", "tilt", "rotation") else ""))
|
||||||
ax.legend(loc="upper left", fontsize=8)
|
ax.legend(loc="upper left", fontsize=8)
|
||||||
commons, refs_per_col, ref_devs = {}, {}, {}
|
commons, refs_per_col, ref_devs = {}, {}, {} # median per measurement of each column; references per column, their deviations
|
||||||
for ax, col in zip(axes[len(pose_cols):, 0], columns):
|
for ax, col in zip(axes[len(pose_cols):, 0], columns):
|
||||||
e, _ = deviations(col)
|
e, _ = deviations(col)
|
||||||
med, low, high, tt = [], [], [], []
|
med, low, high, tt = [], [], [], []
|
||||||
@@ -329,19 +314,13 @@ def main():
|
|||||||
print(common)
|
print(common)
|
||||||
|
|
||||||
# table of the common median, per step
|
# table of the common median, per step
|
||||||
midnight = times[0].replace(hour=0, minute=0, second=0, microsecond=0)
|
key = per_step(times[0], o.step) # steps aligned on the clock (hh:00, hh:15…), not on the 1st measurement
|
||||||
t0 = midnight + ((times[0] - midnight) // o.step) * o.step # steps aligned on the clock (hh:00, hh:15…), not on the 1st measurement
|
|
||||||
rows = {}
|
|
||||||
for col in columns:
|
|
||||||
for t, v in commons[col].items():
|
|
||||||
rows.setdefault(t0 + ((t - t0) // o.step) * o.step, {}).setdefault(col, []).append(v)
|
|
||||||
for col in pose_cols:
|
for col in pose_cols:
|
||||||
commons["cam_" + col] = {p["t"]: p[col] for p in poses if p[col] is not None}
|
commons["cam_" + col] = {p["t"]: p[col] for p in poses if p[col] is not None}
|
||||||
for t, v in commons["cam_" + col].items():
|
rows = {}
|
||||||
rows.setdefault(t0 + ((t - t0) // o.step) * o.step, {}).setdefault("cam_" + col, []).append(v)
|
for col in columns + ["cam_" + c for c in pose_cols] + [c + "_targets" for c in refs_per_col]:
|
||||||
for col in refs_per_col:
|
for t, v in commons[col].items():
|
||||||
for t, v in commons[col + "_targets"].items():
|
rows.setdefault(key(t), {}).setdefault(col, []).append(v)
|
||||||
rows.setdefault(t0 + ((t - t0) // o.step) * o.step, {}).setdefault(col + "_targets", []).append(v)
|
|
||||||
table_cols = columns + [c + "_targets" for c in refs_per_col] + ["cam_" + c for c in pose_cols]
|
table_cols = columns + [c + "_targets" for c in refs_per_col] + ["cam_" + c for c in pose_cols]
|
||||||
print(f"\nCommon median (deviation from the median of the window){' and raw camera pose (cam_, empty without registration)' if pose_cols else ''}, "
|
print(f"\nCommon median (deviation from the median of the window){' and raw camera pose (cam_, empty without registration)' if pose_cols else ''}, "
|
||||||
f"median per step of {int(o.step.total_seconds() // 60)} min{' (table: ' + o.table + ')' if o.table else ''}:")
|
f"median per step of {int(o.step.total_seconds() // 60)} min{' (table: ' + o.table + ')' if o.table else ''}:")
|
||||||
@@ -352,15 +331,14 @@ def main():
|
|||||||
print(f"{iso(t):17}" + "".join(f"{statistics.median(rows[t][c]):13.{6 if c.startswith('cam_') else 4}g}" if c in rows[t] else " " * 13 for c in table_cols))
|
print(f"{iso(t):17}" + "".join(f"{statistics.median(rows[t][c]):13.{6 if c.startswith('cam_') else 4}g}" if c in rows[t] else " " * 13 for c in table_cols))
|
||||||
for col in table_cols:
|
for col in table_cols:
|
||||||
if commons[col]:
|
if commons[col]:
|
||||||
lo, hi = min(commons[col].items(), key=lambda x: x[1]), max(commons[col].items(), key=lambda x: x[1])
|
lo, hi = extremes(commons[col])
|
||||||
g = 6 if col.startswith("cam_") else 4
|
g = 6 if col.startswith("cam_") else 4
|
||||||
print(f"{col} (measurements): min {lo[1]:.{g}g} at {iso(lo[0])}, max {hi[1]:.{g}g} at {iso(hi[0])}")
|
print(f"{col} (measurements): min {lo[1]:.{g}g} at {iso(lo[0])}, max {hi[1]:.{g}g} at {iso(hi[0])}")
|
||||||
|
|
||||||
for col, refs in refs_per_col.items():
|
for col, refs in refs_per_col.items():
|
||||||
print(f"\n{col} raw of the global references, deviation from their median of the window (measurements); {col}_targets: median of the targets alone:")
|
print(f"\n{col} raw of the global references, deviation from their median of the window (measurements); {col}_targets: median of the targets alone:")
|
||||||
for s in refs:
|
for s in refs:
|
||||||
d = ref_devs[col][s]
|
lo, hi = extremes(ref_devs[col][s])
|
||||||
lo, hi = min(d.items(), key=lambda x: x[1]), max(d.items(), key=lambda x: x[1])
|
|
||||||
rej = {m["t"] for m in series[s] if rejected(m)}
|
rej = {m["t"] for m in series[s] if rejected(m)}
|
||||||
r = lambda t: " (rejected)" if t in rej else ""
|
r = lambda t: " (rejected)" if t in rej else ""
|
||||||
print(f"{s:9} min {lo[1]:+.3g} at {iso(lo[0])}{r(lo[0])}, max {hi[1]:+.3g} at {iso(hi[0])}{r(hi[0])}"
|
print(f"{s:9} min {lo[1]:+.3g} at {iso(lo[0])}{r(lo[0])}, max {hi[1]:+.3g} at {iso(hi[0])}{r(hi[0])}"
|
||||||
@@ -368,66 +346,71 @@ def main():
|
|||||||
if not pose_cols:
|
if not pose_cols:
|
||||||
print("Camera pose: none in the window (no camera/ in the folder, or no registration).")
|
print("Camera pose: none in the window (no camera/ in the folder, or no registration).")
|
||||||
else:
|
else:
|
||||||
posed = {p["t"].replace(second=0, microsecond=0) for p in poses}
|
runs = without_pose(times, poses)
|
||||||
runs = [] # runs of consecutive measurement times without a pose: [start, end, number]
|
|
||||||
for i, t in enumerate(times):
|
|
||||||
if t.replace(second=0, microsecond=0) in posed:
|
|
||||||
continue
|
|
||||||
if runs and runs[-1][3] == i - 1:
|
|
||||||
runs[-1][1], runs[-1][2], runs[-1][3] = t, runs[-1][2] + 1, i
|
|
||||||
else:
|
|
||||||
runs.append([t, t, 1, i])
|
|
||||||
print(f"Measurement times without a camera pose (no registration): {sum(x[2] for x in runs)} of {len(times)}"
|
print(f"Measurement times without a camera pose (no registration): {sum(x[2] for x in runs)} of {len(times)}"
|
||||||
+ (": " + "; ".join(iso(a) if n == 1 else f"{iso(a)} → {iso(b)} ({n})" for a, b, n, _ in runs) if runs else ""))
|
+ (": " + "; ".join(iso(a) if n == 1 else f"{iso(a)} → {iso(b)} ({n})" for a, b, n in runs) if runs else ""))
|
||||||
|
|
||||||
# 2. small multiples
|
# 2. small multiples and 3. residuals: one small plot per sensor, one scale each
|
||||||
col = o.column
|
mins = int(o.step.total_seconds() // 60)
|
||||||
n = len(names)
|
nrows = math.ceil(len(names) / 5)
|
||||||
nrows = math.ceil(n / 5)
|
|
||||||
fig, axes = plt.subplots(nrows, 5, sharex=True, figsize=(18, 2.5 * nrows), squeeze=False)
|
def sheet():
|
||||||
for ax, s in zip(axes.flat, names):
|
return plt.subplots(nrows, 5, sharex=True, figsize=(18, 2.5 * nrows), squeeze=False)
|
||||||
ms = [m for m in series[s] if m[col] is not None]
|
|
||||||
ax.plot([m["t"] for m in ms], [m[col] for m in ms], "-", lw=0.4, color="0.7")
|
def small(ax, pts, line, title):
|
||||||
|
"""Measurements {t: v} in grey, their median per step in blue (returned), a dotted line at `line`."""
|
||||||
steps = {}
|
steps = {}
|
||||||
for m in ms:
|
for t, v in pts.items():
|
||||||
steps.setdefault(t0 + ((m["t"] - t0) // o.step) * o.step, []).append(m[col])
|
steps.setdefault(key(t), []).append(v)
|
||||||
ax.plot(list(steps), [statistics.median(v) for v in steps.values()], "-", lw=1.1, color="C0")
|
pm = {t: statistics.median(v) for t, v in steps.items()}
|
||||||
lv = level(series[s], col)
|
ax.plot(list(pts), list(pts.values()), "-", lw=0.4, color="0.7")
|
||||||
if lv:
|
ax.plot(list(pm), list(pm.values()), "-", lw=1.1, color="C0")
|
||||||
ax.axhline(lv[0], color="C3", ls=":", lw=0.8)
|
if line is not None:
|
||||||
ax.set_title(s + (" (ref.)" if series[s] and is_reference(series[s]) else ""), fontsize=9)
|
ax.axhline(line, color="C3", ls=":", lw=0.8)
|
||||||
|
ax.set_title(title, fontsize=9)
|
||||||
ax.grid(alpha=0.3)
|
ax.grid(alpha=0.3)
|
||||||
ax.tick_params(labelsize=7)
|
ax.tick_params(labelsize=7)
|
||||||
ax.xaxis.set_major_locator(mdates.AutoDateLocator(minticks=2, maxticks=5))
|
ax.xaxis.set_major_locator(mdates.AutoDateLocator(minticks=2, maxticks=5))
|
||||||
ax.xaxis.set_major_formatter(mdates.DateFormatter("%d/%m\n%Hh"))
|
ax.xaxis.set_major_formatter(mdates.DateFormatter("%d/%m\n%Hh"))
|
||||||
for ax in list(axes.flat)[n:]:
|
return pm
|
||||||
ax.axis("off")
|
|
||||||
fig.suptitle(f"{station}: {col}, median per step of {int(o.step.total_seconds() // 60)} min (blue), measurements (grey), "
|
def save(fig, axes, title, name):
|
||||||
f"median of the window (dotted); one scale per sensor (UTC times)")
|
for ax in list(axes.flat)[len(names):]:
|
||||||
fig.tight_layout()
|
ax.axis("off")
|
||||||
multiples = base.format(f"multiples-{col}")
|
fig.suptitle(title)
|
||||||
fig.savefig(multiples, dpi=100, bbox_inches="tight")
|
fig.tight_layout()
|
||||||
print(multiples)
|
path = base.format(name)
|
||||||
|
fig.savefig(path, dpi=100, bbox_inches="tight")
|
||||||
|
plt.close(fig)
|
||||||
|
print(path)
|
||||||
|
|
||||||
|
col = o.column
|
||||||
|
fig, axes = sheet()
|
||||||
|
for ax, s in zip(axes.flat, names):
|
||||||
|
lv = level(series[s], col)
|
||||||
|
small(ax, {m["t"]: m[col] for m in series[s] if m[col] is not None}, lv and lv[0],
|
||||||
|
s + (" (ref.)" if series[s] and is_reference(series[s]) else ""))
|
||||||
|
save(fig, axes, f"{station}: {col}, median per step of {mins} min (blue), measurements (grey), "
|
||||||
|
f"median of the window (dotted); one scale per sensor (UTC times)", f"multiples-{col}")
|
||||||
|
|
||||||
|
def own_level(ms, col): # (median, MAD) outside --outside, or over the window when nothing is left
|
||||||
|
return level([m for m in ms if kept(m["t"])] or ms, col)
|
||||||
|
|
||||||
# 3. residuals: each sensor minus its own level
|
|
||||||
for col, others in [(c, False) for c in residuals] + [(c, True) for c in residuals if o.others]:
|
for col, others in [(c, False) for c in residuals] + [(c, True) for c in residuals if o.others]:
|
||||||
fig, axes = plt.subplots(nrows, 5, sharex=True, figsize=(18, 2.5 * nrows), squeeze=False)
|
fig, axes = sheet()
|
||||||
dev = {}
|
dev = {}
|
||||||
if others: # {sensor: {t: value - own level}} for the median of the others
|
if others: # {sensor: {t: value - own level}} for the median of the others
|
||||||
for q in names:
|
for q in names:
|
||||||
mq = [m for m in series[q] if m[col] is not None]
|
mq = [m for m in series[q] if m[col] is not None]
|
||||||
rq = [m for m in mq if not outside or not outside[0] <= m["t"] <= outside[1]] or mq
|
if nq := own_level(mq, col):
|
||||||
nq = level(rq, col)
|
|
||||||
if nq:
|
|
||||||
dev[q] = {m["t"]: m[col] - nq[0] for m in mq}
|
dev[q] = {m["t"]: m[col] - nq[0] for m in mq}
|
||||||
print(f"\n{col} minus the own level{' and minus the median of the other sensors at the same time' if others else ''} ({'outside ' + o.outside if outside else 'median of the window'}), median per step of "
|
print(f"\n{col} minus the own level{' and minus the median of the other sensors at the same time' if others else ''} ({'outside ' + o.outside if o.outside else 'median of the window'}), median per step of "
|
||||||
f"{int(o.step.total_seconds() // 60)} min:")
|
f"{mins} min:")
|
||||||
slopes = []
|
slopes = []
|
||||||
print(f"{'sensor':9}{'level':>10}{'noise':>10}{'min dev.':>11} {'at':<17}{'(noises)':>9}{'max dev.':>11} {'at':<17}{'(noises)':>9}")
|
print(f"{'sensor':9}{'level':>10}{'noise':>10}{'min dev.':>11} {'at':<17}{'(noises)':>9}{'max dev.':>11} {'at':<17}{'(noises)':>9}")
|
||||||
for ax, s in zip(axes.flat, names):
|
for ax, s in zip(axes.flat, names):
|
||||||
ms = [m for m in series[s] if m[col] is not None]
|
ms = [m for m in series[s] if m[col] is not None]
|
||||||
ref = [m for m in ms if not outside or not outside[0] <= m["t"] <= outside[1]] or ms
|
lv = own_level(ms, col)
|
||||||
lv = level(ref, col)
|
|
||||||
if not lv:
|
if not lv:
|
||||||
ax.axis("off")
|
ax.axis("off")
|
||||||
continue
|
continue
|
||||||
@@ -443,20 +426,8 @@ def main():
|
|||||||
continue
|
continue
|
||||||
else:
|
else:
|
||||||
res = {m["t"]: m[col] - lv[0] for m in ms}
|
res = {m["t"]: m[col] - lv[0] for m in ms}
|
||||||
steps = {}
|
lo, hi = extremes(small(ax, res, 0, s + (" (ref.)" if is_reference(ms) else "") + f" noise {noise:.2g}"))
|
||||||
for t, v in res.items():
|
f = lambda v: f"{v / noise:9.1f}" if noise else f"{'-':>9}" # noqa: E731
|
||||||
steps.setdefault(t0 + ((t - t0) // o.step) * o.step, []).append(v)
|
|
||||||
pm = {t: statistics.median(v) for t, v in steps.items()}
|
|
||||||
ax.plot(list(res), list(res.values()), "-", lw=0.4, color="0.7")
|
|
||||||
ax.plot(list(pm), list(pm.values()), "-", lw=1.1, color="C0")
|
|
||||||
ax.axhline(0, color="C3", ls=":", lw=0.8)
|
|
||||||
ax.set_title(s + (" (ref.)" if is_reference(ms) else "") + f" noise {noise:.2g}", fontsize=9)
|
|
||||||
ax.grid(alpha=0.3)
|
|
||||||
ax.tick_params(labelsize=7)
|
|
||||||
ax.xaxis.set_major_locator(mdates.AutoDateLocator(minticks=2, maxticks=5))
|
|
||||||
ax.xaxis.set_major_formatter(mdates.DateFormatter("%d/%m\n%Hh"))
|
|
||||||
lo, hi = min(pm.items(), key=lambda x: x[1]), max(pm.items(), key=lambda x: x[1])
|
|
||||||
f = lambda v: f"{v / noise:9.1f}" if noise else f"{'-':>9}"
|
|
||||||
print(f"{s:9}{lv[0]:10.4g}{noise:10.3g}{lo[1]:11.3g} {iso(lo[0]):<17}{f(lo[1])}{hi[1]:11.3g} {iso(hi[0]):<17}{f(hi[1])}")
|
print(f"{s:9}{lv[0]:10.4g}{noise:10.3g}{lo[1]:11.3g} {iso(lo[0]):<17}{f(lo[1])}{hi[1]:11.3g} {iso(hi[0]):<17}{f(hi[1])}")
|
||||||
if others:
|
if others:
|
||||||
for c in columns:
|
for c in columns:
|
||||||
@@ -465,17 +436,10 @@ def main():
|
|||||||
xs, ys = [commons[c][t] for t in tt], [res[t] for t in tt]
|
xs, ys = [commons[c][t] for t in tt], [res[t] for t in tt]
|
||||||
slope = statistics.linear_regression(xs, ys).slope
|
slope = statistics.linear_regression(xs, ys).slope
|
||||||
slopes.append(f"{s:9}slope on common {c} {slope:+.4g} per unit, r = {statistics.correlation(xs, ys):+.2f}")
|
slopes.append(f"{s:9}slope on common {c} {slope:+.4g} per unit, r = {statistics.correlation(xs, ys):+.2f}")
|
||||||
for ax in list(axes.flat)[n:]:
|
|
||||||
ax.axis("off")
|
|
||||||
if slopes:
|
if slopes:
|
||||||
print("\n".join(slopes))
|
print("\n".join(slopes))
|
||||||
fig.suptitle(f"{station}: {col} minus the own level{' and the median of the others' if others else ''} of each sensor, median per step of "
|
save(fig, axes, f"{station}: {col} minus the own level{' and the median of the others' if others else ''} of each sensor, median per step of "
|
||||||
f"{int(o.step.total_seconds() // 60)} min (blue), measurements (grey); one scale per sensor (UTC times)")
|
f"{mins} min (blue), measurements (grey); one scale per sensor (UTC times)", f"residuals-{col}" + ("-others" if others else ""))
|
||||||
fig.tight_layout()
|
|
||||||
path = base.format(f"residuals-{col}" + ("-others" if others else ""))
|
|
||||||
fig.savefig(path, dpi=100, bbox_inches="tight")
|
|
||||||
plt.close(fig)
|
|
||||||
print(path)
|
|
||||||
|
|
||||||
if o.rigid:
|
if o.rigid:
|
||||||
rigid(o, names, series, times, base, station)
|
rigid(o, names, series, times, base, station)
|
||||||
|
|||||||
+5
-16
@@ -17,10 +17,8 @@ matplotlib.use("Agg")
|
|||||||
import matplotlib.dates as mdates # noqa: E402
|
import matplotlib.dates as mdates # noqa: E402
|
||||||
import matplotlib.pyplot as plt # noqa: E402
|
import matplotlib.pyplot as plt # noqa: E402
|
||||||
|
|
||||||
from data import filter_window, is_reference, load, parse_time # noqa: E402
|
from common import COLUMNS, QUALITY, bound, check, sensors # noqa: E402
|
||||||
from summary import QUALITY # noqa: E402
|
from data import filter_window, parse_time # noqa: E402
|
||||||
|
|
||||||
COLUMNS = ("error_value", "corr_x", "corr_y", "relative_x", "relative_y", "x", "y", "sigma0", "reproj_err")
|
|
||||||
|
|
||||||
|
|
||||||
def zone(text):
|
def zone(text):
|
||||||
@@ -31,10 +29,6 @@ def zone(text):
|
|||||||
return parse_time(start), parse_time(end)
|
return parse_time(start), parse_time(end)
|
||||||
|
|
||||||
|
|
||||||
def bound(t, default):
|
|
||||||
return t.strftime("%Y%m%dT%H%MZ") if t else default
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
a = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
a = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
a.add_argument("folder")
|
a.add_argument("folder")
|
||||||
@@ -46,15 +40,10 @@ def main():
|
|||||||
a.add_argument("--out", default=os.path.join("out", "diagnostic"), help="folder of the PNG")
|
a.add_argument("--out", default=os.path.join("out", "diagnostic"), help="folder of the PNG")
|
||||||
o = a.parse_args()
|
o = a.parse_args()
|
||||||
|
|
||||||
by = load(o.folder)
|
by, names = sensors(o.folder, o.sensors)
|
||||||
if not by:
|
|
||||||
sys.exit(f'No "advanced" export in {o.folder} (see inventory.py).')
|
|
||||||
names = o.sensors.split(",") if o.sensors else sorted(by, key=lambda s: (not is_reference(by[s]), s))
|
|
||||||
columns = o.columns.split(",")
|
columns = o.columns.split(",")
|
||||||
for requested, known, what in ((names, by, "sensors"), (columns, COLUMNS + QUALITY, "columns")):
|
check("sensors", names, by)
|
||||||
missing = [x for x in requested if x not in known]
|
check("columns", columns, COLUMNS + QUALITY)
|
||||||
if missing:
|
|
||||||
sys.exit(f"Unknown {what}: {', '.join(missing)}. Known: {', '.join(sorted(known))}.")
|
|
||||||
series = {s: filter_window(by[s], o.start, o.end) for s in names}
|
series = {s: filter_window(by[s], o.start, o.end) for s in names}
|
||||||
if not any(series.values()):
|
if not any(series.values()):
|
||||||
sys.exit("No measurement in the requested window.")
|
sys.exit("No measurement in the requested window.")
|
||||||
|
|||||||
+13
-22
@@ -46,11 +46,11 @@ import sys
|
|||||||
from datetime import timedelta
|
from datetime import timedelta
|
||||||
|
|
||||||
import data
|
import data
|
||||||
from summary import QUALITY as METRICS, duration
|
from common import QUALITY as METRICS, duration, group, per_step, sensors
|
||||||
|
|
||||||
HIGH_BAD = ("fwhm", "dominance") # for these two, it is the rise that says a degraded image
|
HIGH_BAD = ("fwhm", "dominance") # for these two, it is the rise that says a degraded image
|
||||||
COLOURS = {"Visibility": "tab:blue", "Obstruction": "tab:orange", "Thermal displacement": "tab:red",
|
COLOURS = {"Visibility": "tab:blue", "Obstruction": "tab:orange", "Thermal displacement": "tab:red",
|
||||||
"Displacement": "tab:purple", "Target issue": "tab:brown", "No data": "black"}
|
"Displacement": "tab:purple", "Perturbation": "tab:cyan", "Target issue": "tab:brown", "No data": "black"}
|
||||||
|
|
||||||
|
|
||||||
def windows(text):
|
def windows(text):
|
||||||
@@ -89,6 +89,11 @@ def affected(r, c, threshold):
|
|||||||
return r is not None and ((c in ("flux", "snr") and r < threshold) or (c == "dominance" and r > 1 / threshold))
|
return r is not None and ((c in ("flux", "snr") and r < threshold) or (c == "dominance" and r > 1 / threshold))
|
||||||
|
|
||||||
|
|
||||||
|
def hit(ref, s, m, threshold):
|
||||||
|
"""The measurement says a degraded image: flux or snr below threshold, or dominance above 1/threshold."""
|
||||||
|
return any(affected(ratio(ref, s, c, m), c, threshold) for c in ("flux", "snr", "dominance"))
|
||||||
|
|
||||||
|
|
||||||
def med(v):
|
def med(v):
|
||||||
return f"{statistics.median(v):.2f}" if v else "-"
|
return f"{statistics.median(v):.2f}" if v else "-"
|
||||||
|
|
||||||
@@ -123,14 +128,8 @@ def summary(by_sensor, ref, names, refs, pmax):
|
|||||||
for c in METRICS))
|
for c in METRICS))
|
||||||
|
|
||||||
|
|
||||||
def per_step(by_sensor, names, step):
|
|
||||||
step = duration(step)
|
|
||||||
start = min(m["t"] for s in names for m in by_sensor[s]).replace(hour=0, minute=0, second=0, microsecond=0)
|
|
||||||
return lambda t: start + (t - start) // step * step
|
|
||||||
|
|
||||||
|
|
||||||
def table(by_sensor, ref, names, c, step, threshold):
|
def table(by_sensor, ref, names, c, step, threshold):
|
||||||
key, cells = per_step(by_sensor, names, step), {}
|
key, cells = per_step(min(m["t"] for s in names for m in by_sensor[s]), duration(step)), {}
|
||||||
for s in names:
|
for s in names:
|
||||||
for m in by_sensor[s]:
|
for m in by_sensor[s]:
|
||||||
r = ratio(ref, s, c, m)
|
r = ratio(ref, s, c, m)
|
||||||
@@ -158,7 +157,7 @@ def table(by_sensor, ref, names, c, step, threshold):
|
|||||||
|
|
||||||
def common(by_sensor, ref, names, refs, step):
|
def common(by_sensor, ref, names, refs, step):
|
||||||
"""Per step: median of the ratios of the targets, then of the references, for each metric."""
|
"""Per step: median of the ratios of the targets, then of the references, for each metric."""
|
||||||
key, cells = per_step(by_sensor, names, step), {}
|
key, cells = per_step(min(m["t"] for s in names for m in by_sensor[s]), duration(step)), {}
|
||||||
for s in names:
|
for s in names:
|
||||||
for m in by_sensor[s]:
|
for m in by_sensor[s]:
|
||||||
d = cells.setdefault(key(m["t"]), {c: ([], []) for c in METRICS})
|
d = cells.setdefault(key(m["t"]), {c: ([], []) for c in METRICS})
|
||||||
@@ -204,7 +203,7 @@ def labels(a, every, ref, refs, names, start, end):
|
|||||||
worst = []
|
worst = []
|
||||||
for s in names:
|
for s in names:
|
||||||
ms = [m for m in win[s] if t0 <= m["t"] <= t1]
|
ms = [m for m in win[s] if t0 <= m["t"] <= t1]
|
||||||
if any(affected(ratio(ref, s, c, m), c, a.threshold) for m in ms for c in ("flux", "snr", "dominance")):
|
if any(hit(ref, s, m, a.threshold) for m in ms):
|
||||||
fl = [x for x in (ratio(ref, s, "flux", m) for m in ms) if x is not None]
|
fl = [x for x in (ratio(ref, s, "flux", m) for m in ms) if x is not None]
|
||||||
worst.append((min(fl) if fl else 1, s + ("r" if s in refs else "")))
|
worst.append((min(fl) if fl else 1, s + ("r" if s in refs else "")))
|
||||||
worst.sort()
|
worst.sort()
|
||||||
@@ -236,14 +235,9 @@ def outside(a, every, ref, refs, names, start, end):
|
|||||||
for m in data.filter_window(every[s], start, end):
|
for m in data.filter_window(every[s], start, end):
|
||||||
if within(m["t"], covered):
|
if within(m["t"], covered):
|
||||||
continue
|
continue
|
||||||
if any(affected(ratio(ref, s, c, m), c, a.threshold) for c in ("flux", "snr", "dominance")):
|
if hit(ref, s, m, a.threshold):
|
||||||
inst.setdefault(m["t"], {})[s + ("r" if s in refs else "")] = ratio(ref, s, "flux", m) or 0
|
inst.setdefault(m["t"], {})[s + ("r" if s in refs else "")] = ratio(ref, s, "flux", m) or 0
|
||||||
runs = []
|
runs = group(sorted(t for t in inst if len(inst[t]) >= a.min), lambda x, y: y - x <= timedelta(minutes=15))
|
||||||
for t in sorted(t for t in inst if len(inst[t]) >= a.min):
|
|
||||||
if runs and t - runs[-1][-1] <= timedelta(minutes=15):
|
|
||||||
runs[-1].append(t)
|
|
||||||
else:
|
|
||||||
runs.append([t])
|
|
||||||
print(f"Outside the labels (except {a.except_}): times when at least {a.min} sensors are affected; {len(runs)} runs")
|
print(f"Outside the labels (except {a.except_}): times when at least {a.min} sensors are affected; {len(runs)} runs")
|
||||||
for ru in runs:
|
for ru in runs:
|
||||||
worst = {}
|
worst = {}
|
||||||
@@ -369,13 +363,10 @@ def main():
|
|||||||
if unknown:
|
if unknown:
|
||||||
p.error(f"unknown metrics: {', '.join(unknown)}; known: {', '.join(METRICS)}")
|
p.error(f"unknown metrics: {', '.join(unknown)}; known: {', '.join(METRICS)}")
|
||||||
|
|
||||||
every = data.load(a.folder)
|
every, names = sensors(a.folder, a.sensors)
|
||||||
if not every:
|
|
||||||
sys.exit(f"No \"advanced\" export in {a.folder} (see inventory.py).")
|
|
||||||
if not any(m[c] is not None for ms in every.values() for m in ms for c in METRICS):
|
if not any(m[c] is not None for ms in every.values() for m in ms for c in METRICS):
|
||||||
sys.exit(f"No quality metrics in {a.folder}: peak to snr empty (old camera; "
|
sys.exit(f"No quality metrics in {a.folder}: peak to snr empty (old camera; "
|
||||||
"tools/DATA.md).")
|
"tools/DATA.md).")
|
||||||
names = a.sensors.split(",") if a.sensors else sorted(every, key=lambda s: (not data.is_reference(every[s]), s))
|
|
||||||
missing = [s for s in names if s not in every]
|
missing = [s for s in names if s not in every]
|
||||||
if missing:
|
if missing:
|
||||||
sys.exit(f"Missing sensors: {', '.join(missing)}. Present: {', '.join(sorted(every))}.")
|
sys.exit(f"Missing sensors: {', '.join(missing)}. Present: {', '.join(sorted(every))}.")
|
||||||
|
|||||||
+101
-24
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
Usage: python3 tools/scripts/ranges.py data/PS_DES_COUDRES/ [--columns error_value,corr_x,corr_y]
|
Usage: python3 tools/scripts/ranges.py data/PS_DES_COUDRES/ [--columns error_value,corr_x,corr_y]
|
||||||
[--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N]
|
[--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N]
|
||||||
[--sensors A,B] [--start ...] [--end ...]
|
[--labels F [--margin 6h] [--top 5] [--smoothing N] [--shorter-out]] [--sensors A,B] [--start ...] [--end ...]
|
||||||
|
|
||||||
For each column, each sensor is compared with its own level over the --start/--end window (median of the
|
For each column, each sensor is compared with its own level over the --start/--end window (median of the
|
||||||
non-empty values; with --outside T1/T2, that of the rest of the window, outside the moment being read: useful when a
|
non-empty values; with --outside T1/T2, that of the rest of the window, outside the moment being read: useful when a
|
||||||
@@ -27,15 +27,34 @@ measurements, outside --outside).
|
|||||||
each, the sensors and the deviation in noises (cx corr_x, cy corr_y, ev error_value, otherwise the column name),
|
each, the sensors and the deviation in noises (cx corr_x, cy corr_y, ev error_value, otherwise the column name),
|
||||||
the sensors rejected at that instant (among --sensors: removing the sensors that are always rejected shortens the
|
the sensors rejected at that instant (among --sensors: removing the sensors that are always rejected shortens the
|
||||||
list), and each reference of the station, in --sensors or not: error_value ("r" if rejected) / inlier ("-" empty).
|
list), and each reference of the station, in --sensors or not: error_value ("r" if rejected) / inlier ("-" empty).
|
||||||
|
4. With --labels F (one JSON per line, start, end, family, id if any; numbered in the order of the file): each line of
|
||||||
|
--instants ends with the numbers of the labels that cover the instant ("labels: none" otherwise); then, per label,
|
||||||
|
the measurements strictly inside it (start < instant < end, the instant to the minute: a measurement at a bound is
|
||||||
|
the point before or after), the largest deviation inside from the level (sensor, column, signed, in noises, instant)
|
||||||
|
and the number of instants inside where a sensor is more than --k noises from it, and "as far around": how many
|
||||||
|
measurements of that sensor-column within --margin before the start and after the end (outside the label) are at
|
||||||
|
least as far from the level, of those (0: the label stands out from its surroundings; many: the sensor already
|
||||||
|
moves that much around it; the same whatever window the noise comes from); then the size of the label, up
|
||||||
|
to --top sensor-columns (default 5), largest first: the extreme inside from the level before the label (median of
|
||||||
|
the --margin before its start, default 6h; signed, in units and in noises, instant) and the net change (median of
|
||||||
|
the --margin after its end minus that level), with the number of measurements before and after, and how many of
|
||||||
|
the measurements inside are more than --k noises from the level before (one: a single point; many: sustained); a label without
|
||||||
|
a measurement inside (an outage) gives the net change alone, largest first. --smoothing N: in this size, each
|
||||||
|
measurement inside is the median of the N centred on it (a long label is not sized by a one-point jump);
|
||||||
|
--shorter-out: this size leaves out the measurements inside the shorter labels of the file that overlap the label,
|
||||||
|
before, inside and after (a long label is not sized by a short event inside it or by one under its level before;
|
||||||
|
a label that only touches it, like the previous day, is kept). Labels near the
|
||||||
|
edge of --start/--end: widen the window by --margin, or before/after lack measurements.
|
||||||
Each column is read at the measurements, never averaged per step. Gives values, not a verdict.
|
Each column is read at the measurements, never averaged per step. Gives values, not a verdict.
|
||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import bisect
|
import bisect
|
||||||
|
import json
|
||||||
import statistics
|
import statistics
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
from data import filter_window, is_reference, iso, load, parse_time, rejected
|
from common import COLUMNS, check, duration, outside, own_noise, sensors
|
||||||
from summary import COLUMNS, duration, own_noise
|
from data import filter_window, is_reference, iso, parse_time, rejected
|
||||||
|
|
||||||
ABBREV = {"corr_x": "cx", "corr_y": "cy", "error_value": "ev"}
|
ABBREV = {"corr_x": "cx", "corr_y": "cy", "error_value": "ev"}
|
||||||
|
|
||||||
@@ -63,42 +82,51 @@ def main():
|
|||||||
a.add_argument("--rolling", type=lambda x: int(x) if x.isdigit() else duration(x),
|
a.add_argument("--rolling", type=lambda x: int(x) if x.isdigit() else duration(x),
|
||||||
help="N (measurements on either side) or duration (6h, 1d): local level, median of the neighbours without the measurement")
|
help="N (measurements on either side) or duration (6h, 1d): local level, median of the neighbours without the measurement")
|
||||||
a.add_argument("--instants", type=int, help="N: instants where at least N sensors are off level, with rejections and references")
|
a.add_argument("--instants", type=int, help="N: instants where at least N sensors are off level, with rejections and references")
|
||||||
|
a.add_argument("--labels", help="label file (one JSON per line): labels covering each instant, deviation inside each label")
|
||||||
|
a.add_argument("--margin", default="6h", type=duration, help="with --labels: the level before (and after) each label is the median of this duration before its start (after its end)")
|
||||||
|
a.add_argument("--top", type=int, default=5, help="with --labels: sensor-columns listed per label, largest deviation from the level before first")
|
||||||
|
a.add_argument("--smoothing", type=int, default=1, metavar="N", help="with --labels: in the size of a label, each measurement inside is the median of the N centred on it (default 1: none)")
|
||||||
|
a.add_argument("--shorter-out", action="store_true", help="with --labels: the size of a label leaves out the measurements inside the shorter labels of the file that overlap it (before, inside, after)")
|
||||||
a.add_argument("--sensors", help="comma-separated list (default: all)")
|
a.add_argument("--sensors", help="comma-separated list (default: all)")
|
||||||
a.add_argument("--start", type=parse_time)
|
a.add_argument("--start", type=parse_time)
|
||||||
a.add_argument("--end", type=parse_time)
|
a.add_argument("--end", type=parse_time)
|
||||||
o = a.parse_args()
|
o = a.parse_args()
|
||||||
|
labels = []
|
||||||
|
if o.labels:
|
||||||
|
for line in open(o.labels):
|
||||||
|
if line.strip():
|
||||||
|
e = json.loads(line)
|
||||||
|
labels.append((parse_time(e["start"]), parse_time(e["end"]), e))
|
||||||
|
minute = lambda t: t.replace(second=0, microsecond=0)
|
||||||
|
covering = lambda t: [str(n) for n, (t0, t1, _) in enumerate(labels, 1) if t0 < minute(t) < t1]
|
||||||
|
|
||||||
by = load(o.folder)
|
by, names = sensors(o.folder, o.sensors)
|
||||||
if not by:
|
|
||||||
sys.exit(f'No "advanced" export in {o.folder} (see inventory.py).')
|
|
||||||
names = o.sensors.split(",") if o.sensors else sorted(by, key=lambda s: (not is_reference(by[s]), s))
|
|
||||||
columns = o.columns.split(",")
|
columns = o.columns.split(",")
|
||||||
outside = tuple(parse_time(x) for x in o.outside.split("/")) if o.outside else None
|
kept = outside(o.outside)
|
||||||
if outside and len(outside) != 2:
|
check("sensors", names, by)
|
||||||
sys.exit("--outside expects T1/T2.")
|
check("columns", columns, COLUMNS)
|
||||||
for requested, known, what in ((names, by, "sensors"), (columns, COLUMNS, "columns")):
|
|
||||||
missing = [x for x in requested if x not in known]
|
|
||||||
if missing:
|
|
||||||
sys.exit(f"Unknown {what}: {', '.join(missing)}. Known: {', '.join(sorted(known))}.")
|
|
||||||
series = {s: filter_window(by[s], o.start, o.end) for s in names}
|
series = {s: filter_window(by[s], o.start, o.end) for s in names}
|
||||||
|
w = max(9, max(len(s) for s in names) + 1) # sensor column: a long name never runs into the next column
|
||||||
if not any(series.values()):
|
if not any(series.values()):
|
||||||
sys.exit("No measurement in the requested window.")
|
sys.exit("No measurement in the requested window.")
|
||||||
roll = (f"{o.rolling} measurements" if isinstance(o.rolling, int) else f"{int(o.rolling.total_seconds() // 60)} min") if o.rolling is not None else ""
|
roll = (f"{o.rolling} measurements" if isinstance(o.rolling, int) else f"{int(o.rolling.total_seconds() // 60)} min") if o.rolling is not None else ""
|
||||||
by_instant = {} # iso(t) -> ["sensor.col+deviation"] (--instants)
|
by_instant = {} # iso(t) -> ["sensor.col+deviation"] (--instants)
|
||||||
|
deviations = [] # (t, sensor, col, deviation in noises) of every measurement (--labels)
|
||||||
|
values = {} # (sensor, col) -> (measurements (t, v), noise) (--labels: size against the level before each label)
|
||||||
|
|
||||||
for col in columns:
|
for col in columns:
|
||||||
print(f"\n== {col}: level, noise, largest jump from one measurement to the next ==")
|
print(f"\n== {col}: level, noise, largest jump from one measurement to the next ==")
|
||||||
print(f"{'sensor':9}{'n':>5}{'level':>10}{'noise':>10}{'jump':>10}{'(noises)':>9} between")
|
print(f"{'sensor':{w}}{'n':>5}{'level':>11}{'noise':>11}{'jump':>11}{'(noises)':>9} between")
|
||||||
ranges = []
|
ranges = []
|
||||||
for s in names:
|
for s in names:
|
||||||
ms = [(m["t"], m[col]) for m in series[s] if m[col] is not None]
|
ms = [(m["t"], m[col]) for m in series[s] if m[col] is not None]
|
||||||
rej = {m["t"] for m in series[s] if rejected(m)}
|
rej = {m["t"] for m in series[s] if rejected(m)}
|
||||||
if len(ms) < 2:
|
if len(ms) < 2:
|
||||||
print(f"{s:9}{len(ms):5} (fewer than 2 values)")
|
print(f"{s:{w}}{len(ms):5} (fewer than 2 values)")
|
||||||
continue
|
continue
|
||||||
ref = [x for x in ms if not outside or not outside[0] <= x[0] <= outside[1]]
|
ref = [x for x in ms if kept(x[0])]
|
||||||
if len(ref) < 2:
|
if len(ref) < 2:
|
||||||
print(f"{s:9}{len(ms):5} (fewer than 2 values outside --outside)")
|
print(f"{s:{w}}{len(ms):5} (fewer than 2 values outside --outside)")
|
||||||
continue
|
continue
|
||||||
lvl = statistics.median(v for _, v in ref)
|
lvl = statistics.median(v for _, v in ref)
|
||||||
jumps = [(b[1] - a_[1], a_[0], b[0]) for a_, b in zip(ms, ms[1:])]
|
jumps = [(b[1] - a_[1], a_[0], b[0]) for a_, b in zip(ms, ms[1:])]
|
||||||
@@ -106,13 +134,16 @@ def main():
|
|||||||
d, t1, t2 = max(jumps, key=lambda x: abs(x[0]))
|
d, t1, t2 = max(jumps, key=lambda x: abs(x[0]))
|
||||||
in_noises = f"{d / noise:9.1f}" if noise else f"{'-':>9}"
|
in_noises = f"{d / noise:9.1f}" if noise else f"{'-':>9}"
|
||||||
flagged = [iso(x) for x in (t1, t2) if x in rej]
|
flagged = [iso(x) for x in (t1, t2) if x in rej]
|
||||||
print(f"{s:9}{len(ms):5}{lvl:10.4g}{noise:10.3g}{d:+10.4g}{in_noises} {iso(t1)} and {iso(t2)}"
|
print(f"{s:{w}}{len(ms):5}{lvl:11.4g}{noise:11.3g}{d:+11.4g}{in_noises} {iso(t1)} and {iso(t2)}"
|
||||||
+ (f" (rejected: {', '.join(flagged)})" if flagged else ""))
|
+ (f" (rejected: {', '.join(flagged)})" if flagged else ""))
|
||||||
if not noise:
|
if not noise:
|
||||||
continue
|
continue
|
||||||
|
values[s, col] = (ms, noise)
|
||||||
lv = local_levels(ms, o.rolling) if o.rolling is not None else [lvl] * len(ms)
|
lv = local_levels(ms, o.rolling) if o.rolling is not None else [lvl] * len(ms)
|
||||||
group = [] # (t, deviation from the level)
|
group = [] # (t, deviation from the level)
|
||||||
for (t, v), n in zip(ms + [(None, 0)], lv + [None]):
|
for (t, v), n in zip(ms + [(None, 0)], lv + [None]):
|
||||||
|
if labels and t is not None and n is not None:
|
||||||
|
deviations.append((t, s, col, (v - n) / noise))
|
||||||
if t is not None and n is not None and abs(v - n) > o.k * noise:
|
if t is not None and n is not None and abs(v - n) > o.k * noise:
|
||||||
if group and (t - group[-1][0] > o.merge or (v > n) != (group[-1][1] > 0)): # same direction only
|
if group and (t - group[-1][0] > o.merge or (v > n) != (group[-1][1] > 0)): # same direction only
|
||||||
ranges.append((s, group, noise, rej))
|
ranges.append((s, group, noise, rej))
|
||||||
@@ -123,12 +154,12 @@ def main():
|
|||||||
ranges.append((s, group, noise, rej))
|
ranges.append((s, group, noise, rej))
|
||||||
total = len(ranges)
|
total = len(ranges)
|
||||||
ranges = [p for p in ranges if len(p[1]) >= o.min_measurements]
|
ranges = [p for p in ranges if len(p[1]) >= o.min_measurements]
|
||||||
print(f"\n{col}: ranges more than {o.k:g} noises from the level{' rolling ±' + roll + ' (median of the neighbours)' if roll else ''}{' (level outside ' + o.outside + ')' if outside and not roll else ''}{' (noise outside ' + o.outside + ')' if outside and roll else ''} (merge {int(o.merge.total_seconds() // 60)} min)"
|
print(f"\n{col}: ranges more than {o.k:g} noises from the level{' rolling ±' + roll + ' (median of the neighbours)' if roll else ''}{' (level outside ' + o.outside + ')' if o.outside and not roll else ''}{' (noise outside ' + o.outside + ')' if o.outside and roll else ''} (merge {int(o.merge.total_seconds() // 60)} min)"
|
||||||
+ (f"; {len(ranges)} of at least {o.min_measurements} measurements out of {total}" if o.min_measurements > 1 else ""))
|
+ (f"; {len(ranges)} of at least {o.min_measurements} measurements out of {total}" if o.min_measurements > 1 else ""))
|
||||||
print(f"{'sensor':9}{'start':<18}{'end':<18}{'measurements':>13}{'extreme':>10}{'(noises)':>9} {'at':<17}{'rejected':>9}")
|
print(f"{'sensor':{w}}{'start':<18}{'end':<18}{'measurements':>13}{'extreme':>11}{'(noises)':>9} {'at':<17}{'rejected':>9}")
|
||||||
for s, g, noise, rej in ranges:
|
for s, g, noise, rej in ranges:
|
||||||
t, d = max(g, key=lambda x: abs(x[1]))
|
t, d = max(g, key=lambda x: abs(x[1]))
|
||||||
print(f"{s:9}{iso(g[0][0]):<18}{iso(g[-1][0]):<18}{len(g):13}{d:+10.4g}{d / noise:9.1f} {iso(t):<17}"
|
print(f"{s:{w}}{iso(g[0][0]):<18}{iso(g[-1][0]):<18}{len(g):13}{d:+11.4g}{d / noise:9.1f} {iso(t):<17}"
|
||||||
f"{sum(x[0] in rej for x in g):9}{' rejected extreme' if t in rej else ''}")
|
f"{sum(x[0] in rej for x in g):9}{' rejected extreme' if t in rej else ''}")
|
||||||
if not ranges:
|
if not ranges:
|
||||||
print("(none)")
|
print("(none)")
|
||||||
@@ -146,12 +177,58 @@ def main():
|
|||||||
rows = [(t, c) for t, c in sorted(by_instant.items()) if len(c) >= o.instants]
|
rows = [(t, c) for t, c in sorted(by_instant.items()) if len(c) >= o.instants]
|
||||||
print(f"\nInstants where at least {o.instants} sensors are more than {o.k:g} noises from the level{' rolling ±' + roll if roll else ''} "
|
print(f"\nInstants where at least {o.instants} sensors are more than {o.k:g} noises from the level{' rolling ±' + roll if roll else ''} "
|
||||||
f"({', '.join(columns)}): {len(rows)} of {len(by_instant)} instants with at least one sensor off level")
|
f"({', '.join(columns)}): {len(rows)} of {len(by_instant)} instants with at least one sensor off level")
|
||||||
print("instant n sensors off level (deviation in noises) | rejected | references: error_value (r rejected) / inlier")
|
print("instant n sensors off level (deviation in noises) | rejected | references: error_value (r rejected) / inlier"
|
||||||
|
+ (" | labels covering" if labels else ""))
|
||||||
for t, c in rows:
|
for t, c in rows:
|
||||||
r = " ".join(f"{s} {f(ref_t[s][t]['error_value'], '.3g')}{'r' if rejected(ref_t[s][t]) else ''}/{f(ref_t[s][t]['inlier'], 'g')}"
|
r = " ".join(f"{s} {f(ref_t[s][t]['error_value'], '.3g')}{'r' if rejected(ref_t[s][t]) else ''}/{f(ref_t[s][t]['inlier'], 'g')}"
|
||||||
for s in refs if t in ref_t[s])
|
for s in refs if t in ref_t[s])
|
||||||
print(f"{t} {len(c):3} {' '.join(s + '.' + ','.join(v) for s, v in c.items())} | {','.join(rej_t.get(t, [])) or '-'} | {r or '-'}")
|
cov = f" | labels: {','.join(covering(parse_time(t))) or 'none'}" if labels else ""
|
||||||
|
print(f"{t} {len(c):3} {' '.join(s + '.' + ','.join(v) for s, v in c.items())} | {','.join(rej_t.get(t, [])) or '-'} | {r or '-'}{cov}")
|
||||||
|
|
||||||
|
if labels:
|
||||||
|
print(f"\nPer label ({o.labels}): measurements strictly inside, largest deviation from the level"
|
||||||
|
f"{' rolling ±' + roll if roll else ''} ({', '.join(columns)}), instants with a sensor more than {o.k:g} noises; "
|
||||||
|
f"as far around: the measurements of that sensor-column within the {int(o.margin.total_seconds() // 60)} min before and after, outside the label, at least as far from the level, of those")
|
||||||
|
print(f"{'n':>4} {'id':<16}{'start':<18}{'end':<18}{'family':<22}{'inside':>7} largest deviation (noises, at) instants > k as far around")
|
||||||
|
m = int(o.margin.total_seconds() // 60)
|
||||||
|
print(f"{'':6}{'sensor.column':<22}{'before':>7}{'extreme':>11}{'(n)':>8} {'at':<17}{'net':>11}{'(n)':>8}{'after':>6}{'>k':>9}"
|
||||||
|
f" (then, up to {o.top} per label: extreme inside from the median of the {m} min before the start;"
|
||||||
|
f" net: median of the {m} min after the end minus it, alone when no measurement is inside; before, after: their measurements; >k: measurements inside more than {o.k:g} noises from the level before, of those inside; n: noises"
|
||||||
|
+ (f"; inside smoothed by the median of {o.smoothing} measurements" if o.smoothing > 1 else "")
|
||||||
|
+ ("; the measurements inside shorter labels left out" if o.shorter_out else "") + ")")
|
||||||
|
for n, (t0, t1, e) in enumerate(labels, 1):
|
||||||
|
ins = [d for d in deviations if t0 < minute(d[0]) < t1]
|
||||||
|
head = f"{n:4} {str(e.get('id', '-')):<16}{iso(t0):<18}{iso(t1):<18}{e.get('family', '-')[:21]:<22}"
|
||||||
|
if not ins:
|
||||||
|
print(head + f"{0:7} (no measurement inside)")
|
||||||
|
else:
|
||||||
|
t, s, col, d = max(ins, key=lambda x: abs(x[3]))
|
||||||
|
over = len({minute(x[0]) for x in ins if abs(x[3]) > o.k})
|
||||||
|
near = [abs(x[3]) for x in deviations if x[1] == s and x[2] == col # same sensor-column, within --margin, outside the label
|
||||||
|
and t0 - o.margin <= minute(x[0]) <= t1 + o.margin and not t0 < minute(x[0]) < t1]
|
||||||
|
print(head + f"{len({minute(x[0]) for x in ins}):7} {s + '.' + ABBREV.get(col, col):<14}{d:+8.1f} {iso(t):<17}{over:>5}"
|
||||||
|
f"{f'{sum(v >= abs(d) for v in near)}/{len(near)}':>12}")
|
||||||
|
rows = [] # size against the level before the label, and the level after it (net alone without a measurement inside)
|
||||||
|
shorter = [(a_, b) for a_, b, _ in labels if b - a_ < t1 - t0 and a_ < t1 and b > t0] if o.shorter_out else []
|
||||||
|
for (s, col), (ms, noise) in values.items():
|
||||||
|
ms = [(t, v) for t, v in ms if t0 - o.margin <= minute(t) <= t1 + o.margin
|
||||||
|
and not any(a_ < minute(t) < b for a_, b in shorter)]
|
||||||
|
pre = [v for t, v in ms if t0 - o.margin <= minute(t) <= t0]
|
||||||
|
h = o.smoothing // 2 # inside: each measurement is the median of the o.smoothing centred on it
|
||||||
|
inside = [(t, statistics.median(v for _, v in ms[max(0, i - h):i + h + 1]) if h else v)
|
||||||
|
for i, (t, v) in enumerate(ms) if t0 < minute(t) < t1]
|
||||||
|
post = [v for t, v in ms if t1 <= minute(t) <= t1 + o.margin]
|
||||||
|
if not pre or not (inside if ins else post):
|
||||||
|
continue
|
||||||
|
before = statistics.median(pre)
|
||||||
|
t, d = max(((t, v - before) for t, v in inside), key=lambda x: abs(x[1])) if inside else (None, None)
|
||||||
|
net = statistics.median(post) - before if post else None
|
||||||
|
beyond = f"{sum(abs(v - before) > o.k * noise for _, v in inside)}/{len(inside)}" if inside else "-"
|
||||||
|
rows.append((abs(d if ins else net) / noise, s, col, d, noise, t, len(pre), net, len(post), beyond))
|
||||||
|
for _, s, col, d, noise, t, n_pre, net, n_post, beyond in sorted(rows, key=lambda r: -r[0])[:o.top]:
|
||||||
|
ext = f"{d:+11.4g}{d / noise:+8.1f} {iso(t):<17}" if d is not None else f"{'-':>11}{'-':>8} {'-':<17}"
|
||||||
|
after = f"{net:+11.4g}{net / noise:+8.1f}{n_post:6}" if net is not None else f"{'-':>11}{'-':>8}{0:6}"
|
||||||
|
print(f"{'':6}{s + '.' + col:<22}{n_pre:7}{ext}{after}{beyond:>9}")
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
main()
|
main()
|
||||||
+3
-14
@@ -33,10 +33,8 @@ matplotlib.use("Agg")
|
|||||||
import matplotlib.dates as mdates # noqa: E402
|
import matplotlib.dates as mdates # noqa: E402
|
||||||
import matplotlib.pyplot as plt # noqa: E402
|
import matplotlib.pyplot as plt # noqa: E402
|
||||||
|
|
||||||
|
from common import STATS, bound, cut, duration, group, targets, window_filter # noqa: E402
|
||||||
from data import POSE, filter_window, is_reference, iso, load, load_camera, load_temperature, parse_time, rejected # noqa: E402
|
from data import POSE, filter_window, is_reference, iso, load, load_camera, load_temperature, parse_time, rejected # noqa: E402
|
||||||
from metrics import cut, runs, targets # noqa: E402
|
|
||||||
from plot import bound # noqa: E402
|
|
||||||
from summary import STATS, duration # noqa: E402
|
|
||||||
|
|
||||||
COLUMNS = ("temperature",) + tuple("cam_" + c for c in POSE) + ("tgt_corr_x", "tgt_corr_y", "rej_targets", "rej_refs")
|
COLUMNS = ("temperature",) + tuple("cam_" + c for c in POSE) + ("tgt_corr_x", "tgt_corr_y", "rej_targets", "rej_refs")
|
||||||
|
|
||||||
@@ -83,13 +81,7 @@ def main():
|
|||||||
columns = o.columns.split(",")
|
columns = o.columns.split(",")
|
||||||
if missing := [c for c in columns if c not in COLUMNS]:
|
if missing := [c for c in columns if c not in COLUMNS]:
|
||||||
sys.exit(f"Unknown columns: {', '.join(missing)}. Known: {', '.join(COLUMNS)}.")
|
sys.exit(f"Unknown columns: {', '.join(missing)}. Known: {', '.join(COLUMNS)}.")
|
||||||
if o.table in ("", "no"):
|
in_table = window_filter(o.table)
|
||||||
in_table = lambda t: o.table == "" # noqa: E731
|
|
||||||
else:
|
|
||||||
limits = [parse_time(x) for x in o.table.split("/")]
|
|
||||||
if len(limits) != 2:
|
|
||||||
sys.exit("--table expects T1/T2 or no.")
|
|
||||||
in_table = lambda t: limits[0] <= t <= limits[1] # noqa: E731
|
|
||||||
|
|
||||||
names = [os.path.basename(os.path.normpath(d)) for d in o.folders]
|
names = [os.path.basename(os.path.normpath(d)) for d in o.folders]
|
||||||
by_station, measured = {}, {}
|
by_station, measured = {}, {}
|
||||||
@@ -123,10 +115,7 @@ def main():
|
|||||||
t = tuple(n for n in names if k not in with_temp[n])
|
t = tuple(n for n in names if k not in with_temp[n])
|
||||||
if m or t:
|
if m or t:
|
||||||
keys.setdefault((m, t), []).append(k)
|
keys.setdefault((m, t), []).append(k)
|
||||||
lines = []
|
lines = [(g[0], g[-1], len(g), m, t) for (m, t), ks in keys.items() for g in group(ks, lambda a, b: b - a == o.step)]
|
||||||
for (m, t), ks in keys.items():
|
|
||||||
for d, f, nb in runs(ks, o.step):
|
|
||||||
lines.append((d, f, nb, m, t))
|
|
||||||
for d, f, nb, m, t in sorted(lines):
|
for d, f, nb, m, t in sorted(lines):
|
||||||
print(f" {iso(d)} → {iso(f)} ({nb} steps): no measurement {','.join(m) or '-'}; no temperature {','.join(t) or '-'}")
|
print(f" {iso(d)} → {iso(f)} ({nb} steps): no measurement {','.join(m) or '-'}; no temperature {','.join(t) or '-'}")
|
||||||
if not lines:
|
if not lines:
|
||||||
|
|||||||
+6
-36
@@ -18,37 +18,11 @@ Usage: python3 tools/scripts/summary.py data/PS_DES_COUDRES/ [--column error_val
|
|||||||
Gives values, not a verdict.
|
Gives values, not a verdict.
|
||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import re
|
|
||||||
import statistics
|
import statistics
|
||||||
import sys
|
import sys
|
||||||
from datetime import timedelta
|
|
||||||
|
|
||||||
from data import filter_window, is_reference, iso, load, parse_time, rejected
|
from common import COLUMNS, QUALITY, STATS, duration, per_step, sensors
|
||||||
|
from data import filter_window, iso, parse_time, rejected
|
||||||
COLUMNS = ("error_value", "corr_x", "corr_y", "relative_x", "relative_y", "x", "y", "sigma0", "reproj_err")
|
|
||||||
QUALITY = ("peak", "flux", "sharpness", "fwhm", "dominance", "snr") # image quality: recent cameras; read by quality.py
|
|
||||||
STATS = {"median": statistics.median, "min": min, "max": max}
|
|
||||||
|
|
||||||
|
|
||||||
def duration(text):
|
|
||||||
"""'15min', '1h', '6h', '1d' -> timedelta."""
|
|
||||||
m = re.fullmatch(r"(\d+)(min|h|d)", text)
|
|
||||||
if not m:
|
|
||||||
raise argparse.ArgumentTypeError(f"invalid step: {text!r} (e.g. 15min, 1h, 1d)")
|
|
||||||
n, unit = int(m.group(1)), m.group(2)
|
|
||||||
return {"min": timedelta(minutes=n), "h": timedelta(hours=n), "d": timedelta(days=n)}[unit]
|
|
||||||
|
|
||||||
|
|
||||||
ROUNDING = 1e-9 # below this, a difference between two measurements is the rounding of the station's computation, not noise
|
|
||||||
|
|
||||||
|
|
||||||
def own_noise(vs):
|
|
||||||
"""Median of the absolute differences between two consecutive values; 0 when it is below ROUNDING (values constant
|
|
||||||
up to rounding: the corr of the references when the compensation goes exactly through them, two references
|
|
||||||
retained), so that a deviation is not written as billions of noises; 0 also with fewer than two values."""
|
|
||||||
d = [abs(b - a) for a, b in zip(vs, vs[1:])]
|
|
||||||
b = statistics.median(d) if d else 0
|
|
||||||
return b if b >= ROUNDING else 0
|
|
||||||
|
|
||||||
|
|
||||||
def values(ms, col):
|
def values(ms, col):
|
||||||
@@ -70,20 +44,16 @@ def main():
|
|||||||
a.add_argument("--end", type=parse_time)
|
a.add_argument("--end", type=parse_time)
|
||||||
o = a.parse_args()
|
o = a.parse_args()
|
||||||
|
|
||||||
by = load(o.folder)
|
by, names = sensors(o.folder, o.sensors)
|
||||||
if not by:
|
|
||||||
sys.exit(f'No "advanced" export in {o.folder} (see inventory.py).')
|
|
||||||
names = o.sensors.split(",") if o.sensors else sorted(by, key=lambda s: (not is_reference(by[s]), s))
|
|
||||||
unknown = [s for s in names if s not in by]
|
unknown = [s for s in names if s not in by]
|
||||||
if unknown:
|
if unknown:
|
||||||
sys.exit(f"Missing sensors: {', '.join(unknown)}. Present: {', '.join(sorted(by))}.")
|
sys.exit(f"Missing sensors: {', '.join(unknown)}. Present: {', '.join(sorted(by))}.")
|
||||||
|
|
||||||
t0 = min(m["t"] for s in names for m in by[s]).replace(hour=0, minute=0, second=0, microsecond=0)
|
key = per_step(min(m["t"] for s in names for m in by[s]), o.step)
|
||||||
steps = {}
|
steps = {}
|
||||||
for s in names:
|
for s in names:
|
||||||
for m in filter_window(by[s], o.start, o.end):
|
for m in filter_window(by[s], o.start, o.end):
|
||||||
step_start = t0 + ((m["t"] - t0) // o.step) * o.step
|
steps.setdefault(key(m["t"]), {}).setdefault(s, []).append(m)
|
||||||
steps.setdefault(step_start, {}).setdefault(s, []).append(m)
|
|
||||||
if not steps:
|
if not steps:
|
||||||
sys.exit("No measurement in the requested window.")
|
sys.exit("No measurement in the requested window.")
|
||||||
|
|
||||||
@@ -104,7 +74,7 @@ def main():
|
|||||||
if o.mark:
|
if o.mark:
|
||||||
print("*: at least one measurement of the step is rejected; o: at least one measurement of the step has inlier = 0")
|
print("*: at least one measurement of the step is rejected; o: at least one measurement of the step has inlier = 0")
|
||||||
header = f"{'start':17} " + " ".join(f"{s:>9}" for s in names)
|
header = f"{'start':17} " + " ".join(f"{s:>9}" for s in names)
|
||||||
wanted = {t0 + ((parse_time(x) - t0) // o.step) * o.step for x in o.instants.split(",")} if o.instants else None
|
wanted = {key(parse_time(x)) for x in o.instants.split(",")} if o.instants else None
|
||||||
for i, k in enumerate(k for k in sorted(steps) if wanted is None or k in wanted):
|
for i, k in enumerate(k for k in sorted(steps) if wanted is None or k in wanted):
|
||||||
if i % 24 == 0:
|
if i % 24 == 0:
|
||||||
print(header)
|
print(header)
|
||||||
|
|||||||
+10
-8
@@ -15,7 +15,8 @@ The frame centre (px) is written under each thumbnail: when the x, y of the expo
|
|||||||
by as much. --fixed keeps, on each row, the frame centred on the position at the first requested time, and marks
|
by as much. --fixed keeps, on each row, the frame centred on the position at the first requested time, and marks
|
||||||
the x, y of the export at each time with a red cross: a spot that enters or leaves the frame is not confused with
|
the x, y of the export at each time with a red cross: a spot that enters or leaves the frame is not confused with
|
||||||
a spot that appears. A frame cut by the edge of the image is flagged (image size): on some
|
a spot that appears. A frame cut by the edge of the image is flagged (image size): on some
|
||||||
exports, x, y are not pixels (values close to 0), and the frame falls in the corner of the image.
|
exports, x, y are not pixels (values close to 0), and the frame falls in the corner of the image. A sensor without
|
||||||
|
any x, y in the export is left out, and said so.
|
||||||
--zone X,Y,W,H: the same zone of the image (pixels of the original image: top left corner, width, height, as
|
--zone X,Y,W,H: the same zone of the image (pixels of the original image: top left corner, width, height, as
|
||||||
gif.py) at each time, side by side (up to 4 per row, fewer when the zone is wide), same stretch with --contrast; --sensors becomes optional:
|
gif.py) at each time, side by side (up to 4 per row, fewer when the zone is wide), same stretch with --contrast; --sensors becomes optional:
|
||||||
those given are marked with a red cross and their name at their x, y at each time. For the scene around
|
those given are marked with a red cross and their name at their x, y at each time. For the scene around
|
||||||
@@ -32,18 +33,16 @@ import matplotlib
|
|||||||
matplotlib.use("Agg")
|
matplotlib.use("Agg")
|
||||||
import matplotlib.pyplot as plt # noqa: E402
|
import matplotlib.pyplot as plt # noqa: E402
|
||||||
|
|
||||||
|
from common import check # noqa: E402
|
||||||
from data import find_files, image_time, iso, load, parse_time # noqa: E402
|
from data import find_files, image_time, iso, load, parse_time # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def percentile(grey, p):
|
def percentile(grey, p):
|
||||||
"""Percentile p (0 to 100) of the pixels of a thumbnail, without interpolation."""
|
"""Percentile p (0 to 100) of the pixels of a thumbnail, without interpolation."""
|
||||||
v = sorted(grey.ravel().tolist())
|
v = sorted(grey.ravel().tolist())
|
||||||
return v[min(int(p / 100 * len(v)), len(v) - 1)]
|
return v[min(int(p / 100 * len(v)), len(v) - 1)]
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
a = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
a = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
a.add_argument("folder")
|
a.add_argument("folder")
|
||||||
@@ -62,9 +61,12 @@ def main():
|
|||||||
zone = tuple(map(int, o.zone.split(","))) if o.zone else None
|
zone = tuple(map(int, o.zone.split(","))) if o.zone else None
|
||||||
if not names and not zone:
|
if not names and not zone:
|
||||||
sys.exit("Give --sensors, or --zone X,Y,W,H.")
|
sys.exit("Give --sensors, or --zone X,Y,W,H.")
|
||||||
missing = [s for s in names if s not in by]
|
check("sensors", names, by)
|
||||||
if missing:
|
if without := [s for s in names if not any(m["x"] is not None and m["y"] is not None for m in by[s])]:
|
||||||
sys.exit(f"Unknown sensors: {', '.join(missing)}. Known: {', '.join(sorted(by))}.")
|
print(f"No x, y for {', '.join(without)} in the export: not placed (--zone X,Y,W,H looks at the scene around them).")
|
||||||
|
names = [s for s in names if s not in without]
|
||||||
|
if not names and not zone:
|
||||||
|
sys.exit("No sensor left to centre a thumbnail on.")
|
||||||
known =sorted(h for h in images if h)
|
known =sorted(h for h in images if h)
|
||||||
if not known:
|
if not known:
|
||||||
sys.exit(f"No image in {o.folder}.")
|
sys.exit(f"No image in {o.folder}.")
|
||||||
@@ -78,7 +80,7 @@ def main():
|
|||||||
return grey * 255 if grey.max() <= 1 and c.dtype.kind == "f" else grey
|
return grey * 255 if grey.max() <= 1 and c.dtype.kind == "f" else grey
|
||||||
|
|
||||||
def measurement(s, t):
|
def measurement(s, t):
|
||||||
return min((m for m in by[s] if m["x"] is not None), key=lambda m: abs(m["t"] - t))
|
return min((m for m in by[s] if m["x"] is not None and m["y"] is not None), key=lambda m: abs(m["t"] - t))
|
||||||
|
|
||||||
r = o.radius
|
r = o.radius
|
||||||
crops, centres, crosses = {}, {}, {}
|
crops, centres, crosses = {}, {}, {}
|
||||||
|
|||||||
Reference in new issue
Block a user