From 4bfbad4db843a497805aa4524fe6b39ee4bc4999 Mon Sep 17 00:00:00 2001 From: maxime Date: Mon, 16 Nov 2026 12:41:44 +0100 Subject: [PATCH] DATA.md: what the data are, from labeling/knowledge (formats, columns, quality metrics, images) Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 2 +- DATA.md | 61 ++++++++++++++++++++++++++++++++++++++++++++ GUIDE.md | 7 +++-- README.md | 1 + scripts/data.py | 4 +-- scripts/inventory.py | 6 ++--- scripts/quality.py | 4 +-- 7 files changed, 73 insertions(+), 12 deletions(-) create mode 100644 DATA.md diff --git a/CLAUDE.md b/CLAUDE.md index 30fcb94..9a55c91 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,6 +1,6 @@ # CLAUDE.md -tools: the reading harness of the AIMsight agents (`README.md`), and nothing else. The user guide is `GUIDE.md`; a new tool or a new option goes into it in the same change. +tools: the reading harness of the AIMsight agents (`README.md`), and nothing else. The user guide is `GUIDE.md`; a new tool or a new option goes into it in the same change. `DATA.md` (what the data are) is maintained by hand by the user; `GUIDE.md` is the tool-builder's. - **Checking a change**: rerun the touched tool from a calling repository (Diagnostic-App: `python3 tools/scripts/.py data/ --start … --end …`), and the commands of the guide that use it; `python3 scripts/quality.py --self-test`. diff --git a/DATA.md b/DATA.md new file mode 100644 index 0000000..454dd57 --- /dev/null +++ b/DATA.md @@ -0,0 +1,61 @@ +# The data of a station + +What the files of a station folder are, and what each column means. How to read them with the tools: `GUIDE.md`. What they mean for a label (causes, reading rules, false friends): `labeling/reading-rules.md`. Maintained by hand; `GUIDE.md` is the tool-builder's. + +## The displacements: the "advanced" export + +`displacement/_advanced__.csv`, the source of the analyses: one row per sensor and per measurement, at the native rate of the station (10 or 15 min), `time_utc` in explicit UTC. + +| Column | Meaning | +|---|---| +| `sensor_name` | the sensor (target or reference) | +| `pattern_type` | `global_ref`: a reference of the compensation; `ref`: a *local reference*, treated as a target, outside the compensation, without `reproj_err`; empty: a target | +| `x`, `y` | raw position in the image [px]. Not the centroid of the spot: the offset centroid − (`x`, `y`) is constant per sensor by day and differs between sensors; it changes at night, when the spot is saturated. Probably the position of the reference pattern | +| `relative_x`, `relative_y` | relative displacement before the camera compensation [px]; not `x − x0` | +| `corr_x`, `corr_y` | the position of the target once the estimated camera movement is removed: **the curves that are labelled**. In px, not mm (below) | +| `error_value` | recognition error of the pattern, not an uncertainty | +| `error_threshold` | specific to each sensor (0.3 to 0.7). Above it, the measurement is rejected: the row and its `error_value` stay, `corr_x` and `corr_y` are empty, not interpolated | +| `reproj_err`, `inlier` | references only: residual and contribution of each reference in the compensation | +| `sigma0` | residual of the fit on the references: one value per measurement, repeated on every row | +| `peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr` | image quality of each sensor, recent cameras only (below) | + +- "Rejected" is a judgment of the AIMsight platform, not of the camera: the camera delivers all its measurements, and a client's platform that integrates the data receives them all. +- **Unit of `corr_x`, `corr_y`**: px. The hourly export's `horizontal_displacement` (mm) is a factor × the hourly mean of `corr_x` + a constant (likewise `vertical` and `corr_y`, correlation ≥ 0.99); the factor (mm per px) depends on the pattern and on the station. +- **LENZSPITZE**: its export gives `x`, `y` relative to the pattern; the extraction rewrites them in image px and fills `pattern_type`. Sensors 1023 and 8019 have no pattern: their `x`, `y` are empty. The tools' crosses may sit slightly off the spots there. +- `inventory.py` says, on a batch, which columns carry values. + +## The camera pose + +`camera/_camera_.csv`, from the pumping only: the pose at each registration, `time_utc` in UTC, `median_|min_|max_` of `pan`, `tilt`, `rotation` [deg], `scale`, `sigma0`, one row per measurement (min = max). A measurement instant without a row: the camera was not registered. + +- Orders of magnitude: `pan`, `tilt` of a few hundredths of a degree, `scale` within 10⁻³ of 1; some stations show several degrees, not explained. +- **A positive `tilt` means the camera is nosing down** (the expert). +- Read by `overview.py` and `inventory.py`. + +## The hourly export + +`__to__1hour.csv`: one row per hour, targets only; per target `horizontal_displacement`, `vertical_displacement`, their `_min` / `_max`, `_error` (mean `error_value` of the hour), `_count_val` (measurements of the hour, rejected ones included). + +- **`timestamp` has no time zone: Swiss local time** (UTC+2 in summer), at the start of the hour. + +## Image quality of a target + +Six columns of the advanced export, computed by the GPU of recent cameras on the sensor's window in the DIFF image, in units of that image. Filled only where `inventory.py` says so ("Export image quality"); for older cameras, `~/optical_emulator` can recompute them from the DIFF images. "Disc": the neighbourhood of the peak, radius `min(H, W)/6` (at least 3 px); "background": the rest of the window. + +| Metric | Definition | What it measures | Reading | +|---|---|---|---| +| `peak` | `max(I)` | the brightness of the return | high, saturation; low, underexposure or missed flash. All targets dropping: the lighting; a single one: that target (fouling, misalignment) | +| `flux` | flux of the disc minus the background | the light of the target, background removed | reflectivity of the target (fouling, ageing, flash), not ambient light; drops well before `error_value` exceeds its threshold | +| `sharpness` | `Σ(I²) / (ΣI)²` | concentration of the energy (higher = sharper) | drops: drifting focus, turbulence, degrading target. Independent of the gain, not of the background: follow it over time on one target, do not compare cameras | +| `fwhm` | width at half height around the peak, background removed [px] | width of the spot | rises: defocus, turbulence, motion blur. Truncated to the disc: under strong defocus it flattens instead of rising | +| `dominance` | `(second peak − background) / (peak − background)`, 0 (clean) toward 1 (ambiguous) | does a single spot dominate the window? | high: a comparable second reflection, a stray bright point, or the target has gone and something else is the brightest: the correlation can latch onto the wrong pattern, which `error_value` does not see. One hot pixel outside the disc is enough | +| `snr` | `(peak − background) / bg_sigma` | the signal above the background noise | low: marginal return, even when `peak` looks normal. Independent of exposure and gain. The emulator marks `empty` below 8 | + +`peak` and `flux` give the brightness, `sharpness` and `fwhm` the sharpness, `dominance` and `snr` the validity: together they say **why** a measurement degrades, where `error_value` only says that it does. They describe the image of **one target**, not the health of the reference network. + +## Camera images: IRON and DIFF + +- **IRON** (`images/`): the ordinary image of the scene, JPG. The cause of an event is seen on it (condensation, rain, train, snow). +- **DIFF** (`images_diff/`): several captures combined to raise the contrast of the reflectors; **the image the displacement algorithms measure**, by default. Not a difference between two instants. 16-bit PNG. +- Acquisition modes (station manual): "Single with infrared light", one image per measurement; "Differential", several combined (`Stacking` 20 to 40 when light is difficult); option "Work on IRON images", to follow a texture without reflector. +- Not every station has DIFF images; the pumping keeps about one image per hour. The tools look only at IRON images; `inventory.py` counts the DIFF ones, which your own scripts may read. What is very close to the lens (a web) may stand out better on DIFF. diff --git a/GUIDE.md b/GUIDE.md index 1bb6bda..c2a0a70 100644 --- a/GUIDE.md +++ b/GUIDE.md @@ -6,9 +6,8 @@ A camera tracks targets (reflectors) fixed on a structure; its own movement is c ## Read first -- `labeling/knowledge/displacement-columns.md`: formats of the exports and meaning of the columns (**first**); -- `labeling/knowledge/gaps.md`; -- on a station whose export carries the image quality metrics (`peak` to `snr`: the "Export image quality" line of `inventory.py`), `labeling/knowledge/quality-metrics.md`. +- `tools/DATA.md`: the files of a station and the meaning of each column, the image quality metrics, the camera images (**first**); +- `labeling/reading-rules.md`: what the data mean (the measurement chain, gaps, reading rules, false friends). ## The tools @@ -24,7 +23,7 @@ Run them from the root of the calling repository, where `tools/`, `data/` and `l | `.venv/bin/python tools/scripts/days.py [--columns corr_x,corr_y,error_value] [--night 20-3] [--k 5] [--days 2] [--step 15min] [--sensors A,B] [--bounds [--smoothing N] [--fraction F]] [--start T] [--end T] [--out D]` | What comes back from one day to the next, on accepted measurements only. 1. **Level of each night** (hours `--night` H1-H2 UTC, straddling midnight, dated from the day it starts), by sensor and by column: median, MAD, number of measurements, and the own noise; then, from one night to the next, the **step** (difference of the medians, in units and in noises) and **where it falls** ("between A and B": where the sequence of night measurements of the two nights splits best between the two levels; B at the start of the following night: the cut falls by day, between the two nights). A step of a few noises, a drift from one night to the next, and two columns that step at the same instant read there, even when the sensor has several levels in the batch. 2. **Superimposed days**: PNG **byhour-**, a small chart per sensor, hour of the day on the x axis, one curve per day, each measurement minus the level of the last night started (night greyed, frame from the 1st to the 99th percentile, the title says how many measurements fall outside the frame); table: per `--step` slot of the hour of the day, the sensors whose deviation exceeds `--k` noises, on the same side, at least `--days` days, with the number of days and the median of these deviations. What comes back every day at the same hour, on one sensor or several, groups there, even when it lasts only one to three measurements. 3. **Bounds of each day**, with `--bounds`: for the median of the targets of each column and for the camera pose (`pan`, `tilt`, `rotation`, `scale`, `sigma0`, if `camera/` exists), the deviation from the level of the night that precedes each day (a day goes from the end of one night to the end of the next; with `--sensors`, the median of these sensors, with a single one, its own). Per day: number of measurements, the first and the last; first and last instant beyond `--k` noises, with the measurement just before and just after, at the native step; signed max and min with their instant; the sequence of measurements beyond `--k` noises that surrounds the larger of the two; the following night minus the reference night. A day without measurement says so; a reference night that is not the previous one is named. Each day line starts with the short name of its series (`corr_x`, `pose_tilt`…): over a month, `\| grep '^pose_tilt '` reads one, `\| grep ' 12/10 '` one day. `--smoothing N`: the deviation of each measurement is the median of the N measurements of the day centred on it (noise and `--k` unchanged; PNG: smoothed in blue, raw in grey): an isolated point no longer cuts the sequence around the extreme and no longer makes a first or a last exceedance. `--fraction F`: the threshold of each day is F times the largest |deviation| of that day (printed on its line, `threshold …`), instead of `--k` noises: when the noise at the native step is unrelated to the amplitude of the cycle (very noisy or very smooth pose). PNG **bounds**: the deviation over the batch, first → last exceedance light grey, sequence around the extreme dark grey, max and min marked. The choice of `--night`: hours where the day cycle (`error_value`, camera pose) has subsided. **Look at the PNGs with Read** | | `.venv/bin/python tools/scripts/metrics.py [--names A,B] [--step 1h] [--stat median\|min\|max] [--against temperature] [--smoothing 24h] [--metrics-only] [--start T] [--end T] [--out D]` | The system metrics of `metrics/` (one value per hour), never the identifiers (`SIM_CCID`, `ip*`). Prints the available metrics with their number of values over the window ("empty": none), then a table per `--step`: `temperature` (median of the `sensor_*_deg_c` components, the one the detectors read), each metric of `--names` (default: the `*_deg_c` and `BusVoltage`), then, unless `--metrics-only`, `tgt_corr_x`, `tgt_corr_y` (median over the targets of `corr` minus each target's own level, accepted measurements) and the raw camera pose (`cam_pan` … `cam_sigma0`); an empty cell is skipped, a text metric (`link`, `BatteryState`, `last_sync`) is returned as it is. Then, per column: filled steps, min and max with their instant; the correlation r and the slope of each column on `--against` (with `--smoothing 24h`, on its rolling mean, column `_smooth`); the dated runs of steps without any metric. A PNG: temperatures on one panel (median in black), each other metric on its own, then `tgt_corr_x`, `tgt_corr_y` and the pose, line cut on a gap. **Look at it with Read** | | `.venv/bin/python tools/scripts/stations.py [...] [--columns temperature,cam_tilt,cam_rotation,cam_scale,tgt_corr_x,tgt_corr_y] [--step 1h] [--stat median\|min\|max] [--table T1/T2\|no] [--start T] [--end T] [--out D]` | **Several stations on the same timeline** (neighbouring stations, same building). Possible columns: `temperature` (as in `metrics.py`), `cam_pan` … `cam_sigma0` (raw pose), `tgt_corr_x`, `tgt_corr_y` (as in `metrics.py`), `rej_targets`, `rej_refs` (number of rejected targets, of rejected references, the largest of the step). Prints per station the period, the cadence and the number of instants; then the **gaps**: the runs of steps without measurement or without temperature, with the stations where they are missing (a gap common to several stations on a single line); then the table by step (one `STATION:column` column each, tabs; `--table` as in `overview.py`); then, per column and per pair of stations, r of the values and r of the steps from one value to the next (a common movement shows on the steps, a common trend on the values), with n. PNG **stations**: a panel per column, a line per station (raw temperature, the rest minus its median), and a band per station of the steps without measurement. Take `--step` equal to the slowest cadence of the stations; over a month at `1h`, first read with `--table no`. **Look at it with Read** | -| `python3 tools/scripts/quality.py [--calm T1/T2,T3/T4] [--common] [--table M] [--step 15min] [--threshold 0.5] [--sensors A,B] [--start T] [--end T]` ; `... --labels F [--numbers 3,7] [--margin 60] [--edges]` ; `... --labels F --outside [--min 1] [--except "Thermal displacement"]` ; `.venv/bin/python tools/scripts/quality.py --figure [--metrics flux,snr,dominance] [--labels F] [--zone T1/T2 ...] [--out D]` ; `python3 tools/scripts/quality.py --self-test` | The six **image quality metrics** (`peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr`; definitions: `labeling/knowledge/quality-metrics.md`), on the stations that have them (otherwise: "No quality metrics", and nothing else). Each measurement becomes a **ratio** to the **calm reference** of its sensor: the median of the metric over its measurements in the `--calm` windows, at the same UTC hour of the day (without `--calm`: the whole folder); 1 = like the reference, < 1 lower, > 1 higher; for `fwhm` and `dominance`, it is the rise that says a degraded image; rejected measurements included. Without options: the folder's maximal `peak`, then per sensor (`r` = reference) the rejected measurements, `pk/max` (median peak over the window divided by this maximum: at 1, the spot touches the image ceiling, its `peak` no longer moves and its `sharpness` follows the exposure: read `flux`, `snr` instead), and for each metric the median ratio and the extreme with its instant; then the median of the targets and of the references. `--common`: per step, the median of the ratios of the targets / of the references for each metric (what affects the whole scene, visibility or turbulence; `--step 1d` to see on one screen on which days the reference holds). `--table M`: per step, the ratio of `M` of each sensor (`*`: a rejected measurement), their median and how many pass `--threshold` (below, or above 1/`--threshold` for `fwhm`, `dominance`): a target that goes out and comes back. `--labels F` (one JSON per line, `start`, `end`, `family`): per numbered label, the median ratio of the targets and of the references over the hour before, the range and the hour after (`tgt 0.96\|0.26\|0.79`, `--margin` minutes), then the sensors **affected** over the range (flux or snr below `--threshold`, or dominance above 1/`--threshold`), with their worst flux, from lowest to highest; `--edges`: per instant of the widened range, the number of affected sensors and the median of the targets of flux, snr, dominance (`<` within the range), to date the first and the last degraded instant. `--outside`: the runs of instants that no label covers (outside `--except` families) where at least `--min` sensors are affected, with the sensors and their worst flux: candidates for forgotten labels, to check on `corr_x`, `corr_y`. `--figure` (instead of the tables): PNG **quality-common** (a panel per metric, log scale: median of the targets in black, of the references in red, grey band from the 10th to the 90th percentile of the targets: a disturbance of the whole scene moves the median, an occultation of a few targets moves only the bottom of the band) and a PNG **quality-** per `--metrics` (a small chart per sensor, references in red); `--labels` greys each label there in one colour per family, `--zone` ranges. **Look at the PNGs with Read** | +| `python3 tools/scripts/quality.py [--calm T1/T2,T3/T4] [--common] [--table M] [--step 15min] [--threshold 0.5] [--sensors A,B] [--start T] [--end T]` ; `... --labels F [--numbers 3,7] [--margin 60] [--edges]` ; `... --labels F --outside [--min 1] [--except "Thermal displacement"]` ; `.venv/bin/python tools/scripts/quality.py --figure [--metrics flux,snr,dominance] [--labels F] [--zone T1/T2 ...] [--out D]` ; `python3 tools/scripts/quality.py --self-test` | The six **image quality metrics** (`peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr`; definitions: `tools/DATA.md`), on the stations that have them (otherwise: "No quality metrics", and nothing else). Each measurement becomes a **ratio** to the **calm reference** of its sensor: the median of the metric over its measurements in the `--calm` windows, at the same UTC hour of the day (without `--calm`: the whole folder); 1 = like the reference, < 1 lower, > 1 higher; for `fwhm` and `dominance`, it is the rise that says a degraded image; rejected measurements included. Without options: the folder's maximal `peak`, then per sensor (`r` = reference) the rejected measurements, `pk/max` (median peak over the window divided by this maximum: at 1, the spot touches the image ceiling, its `peak` no longer moves and its `sharpness` follows the exposure: read `flux`, `snr` instead), and for each metric the median ratio and the extreme with its instant; then the median of the targets and of the references. `--common`: per step, the median of the ratios of the targets / of the references for each metric (what affects the whole scene, visibility or turbulence; `--step 1d` to see on one screen on which days the reference holds). `--table M`: per step, the ratio of `M` of each sensor (`*`: a rejected measurement), their median and how many pass `--threshold` (below, or above 1/`--threshold` for `fwhm`, `dominance`): a target that goes out and comes back. `--labels F` (one JSON per line, `start`, `end`, `family`): per numbered label, the median ratio of the targets and of the references over the hour before, the range and the hour after (`tgt 0.96\|0.26\|0.79`, `--margin` minutes), then the sensors **affected** over the range (flux or snr below `--threshold`, or dominance above 1/`--threshold`), with their worst flux, from lowest to highest; `--edges`: per instant of the widened range, the number of affected sensors and the median of the targets of flux, snr, dominance (`<` within the range), to date the first and the last degraded instant. `--outside`: the runs of instants that no label covers (outside `--except` families) where at least `--min` sensors are affected, with the sensors and their worst flux: candidates for forgotten labels, to check on `corr_x`, `corr_y`. `--figure` (instead of the tables): PNG **quality-common** (a panel per metric, log scale: median of the targets in black, of the references in red, grey band from the 10th to the 90th percentile of the targets: a disturbance of the whole scene moves the median, an occultation of a few targets moves only the bottom of the band) and a PNG **quality-** per `--metrics` (a small chart per sensor, references in red); `--labels` greys each label there in one colour per family, `--zone` ranges. **Look at the PNGs with Read** | | `.venv/bin/python tools/scripts/thumbnails.py --sensors A,B --times T1,T2 [--radius 100] [--contrast [MIN,MAX]] [--fixed] [--zone X,Y,W,H] [--out D]` | A sheet of full-resolution thumbnails: one row per sensor, one column per time, centred on the sensor's raw position (`x`, `y`) at each time (the centre of the frame is written under each thumbnail: when `x`, `y` jump, the frame moves by as much, and a spot that enters the frame seems to appear); `--fixed` keeps for each sensor the frame of the first time and marks with a red cross the `x`, `y` of each time; unretouched pixels, the brightness can be compared from one thumbnail to the next; prints for each thumbnail the background (median), the 99th percentile and the peak (grey levels 0 to 255); without an image at the requested time, takes the nearest and prints the gap (the actual time is in the title). A faint or deformed spot is invisible as it is: `--contrast MIN,MAX` puts all the thumbnails in grey with the same stretch (without value: from the 1st percentile to the sheet's maximum, crushed by a saturated pixel; then choose MIN,MAX from the printed background and peak, e.g. `10,40`), `--radius 30` to `50` magnifies; compare the shape and the peak before, during and after an episode, at the same hour of the day. **Look at it with Read**: a whole image read by Read is reduced, a small reflector becomes illegible in it. A frame that the image edge cuts is flagged ("frame cut by the image edge (W × H px)"): on some stations, `x`, `y` are not pixels (values close to 0) and the frame falls in the corner; then look at the scene through `--zone`. `--zone X,Y,W,H` (pixels of the original image: top left corner, width, height, as in `gif.py`): the same zone at each time, side by side (up to 4 per row), same stretch with `--contrast`; `--sensors` becomes optional, and those that are given are marked with a cross and their name. For the scene around the targets (foreground object, headlights, lighting), prefer it to a large `--radius`. Each thumbnail is rendered pixel for pixel up to 1000 px a side, but Read reduces a wide PNG: with a large `--radius` or a large `--zone`, three or four times per sheet | | `.venv/bin/python tools/scripts/gif.py --start T --end T [--width 720] [--duration 700] [--max 150] [--zone X,Y,W,H] [--out D]` | A GIF of the camera images between `--start` and `--end` inclusive, cropped to `--zone` if needed, set to `--width` px, UTC date and time written at the bottom right; beyond `--max` images, `--max` at regular intervals; prints the time of each image, then the path and the size of the GIF. Used to attach the scene to a reply to the customer; Read does not read a GIF: to look, `thumbnails.py --zone` | | `ls /images/ \| grep TZ`, then Read on the file found (Read takes no pattern) | The whole camera image at that time (UTC), for the scene (occultation, lighting, weather) | diff --git a/README.md b/README.md index c06afca..846045f 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,7 @@ The reading harness: the methods given to the agent to make sense of the data of an AIMsight station. Each tool answers a question the analyst asks and returns values, dated ranges, a figure or thumbnails; it shows, it does not judge. - `scripts/`: the tools, one script each; `GUIDE.md`: which tool for which question, and how to read what it returns. The agent reads it first. +- `DATA.md`: the files of a station and the meaning of their columns, maintained by hand; the agent reads it with the guide. - `scripts/data.py`: the common reader of the data (also that of the Diagnostic-App app, which passes these measurements to the detectors); the detectors keep their own (`detectors/scripts/data.py`), to stay autonomous: a reading fix in one is carried to the other. - The tools: `inventory`, `summary`, `ranges`, `days`, `overview`, `metrics`, `quality`, `stations`, `plot`, `thumbnails`, `gif` (details in the guide). diff --git a/scripts/data.py b/scripts/data.py index b806fd0..dc4bd7e 100644 --- a/scripts/data.py +++ b/scripts/data.py @@ -1,6 +1,6 @@ """Reading a station's data for the diagnostic agent (standard library). -Formats: labeling/knowledge/displacement-columns.md. Every time is UTC. +Formats: tools/DATA.md. Every time is UTC. """ import csv import glob @@ -126,7 +126,7 @@ def load_metrics(folder): def is_reference(measurements): """Reference of the compensation network (pattern_type global_ref), otherwise a target. Some exports have an empty pattern_type, over the whole period or at the start: the first filled pattern_type is read then and, if there is - none, filled reproj_err or inlier (references only: labeling/knowledge/displacement-columns.md).""" + none, filled reproj_err or inlier (references only: tools/DATA.md).""" t = next((m["pattern_type"] for m in measurements if m["pattern_type"]), None) if t is not None: return t == "global_ref" diff --git a/scripts/inventory.py b/scripts/inventory.py index af18d08..1a8b7f4 100644 --- a/scripts/inventory.py +++ b/scripts/inventory.py @@ -76,7 +76,7 @@ def files(f, full=False): list_paths(f["advanced"], full) duplicates(f["advanced"]) print(f"Hourly exports: {len(f['hourly'])} (not read by the tools: hourly averages, " - "local time without a zone; see labeling/knowledge/displacement-columns.md)") + "local time without a zone; see tools/DATA.md)") list_paths(f["hourly"], full) print(f"Camera movements (pose at each registration): {len(f['camera'])}") list_paths(f["camera"], full) @@ -125,7 +125,7 @@ def contents(folder, f, start=None, end=None): if paths is f["advanced"]: q = [c for c in QUALITY if count.get(c)] print(f" Export image quality (peak to snr): " + (f"filled ({', '.join(q)}); read by quality.py, " - "labeling/knowledge/quality-metrics.md" if q else "empty")) + "tools/DATA.md" if q else "empty")) days = collections.Counter() # metric -> days (one CSV per metric and per day) in the window for p in others.pop("metrics", []): m = METRIC_RE.match(os.path.basename(p)) @@ -190,7 +190,7 @@ def measurements(by): empty = [s for s in names if not any(m["pattern_type"] for m in by[s])] if empty: print(f" pattern_type empty for {len(empty)} sensors: reference when reproj_err or inlier is filled " - "(references only, labeling/knowledge/displacement-columns.md)") + "(references only, tools/DATA.md)") print(f"{'sensor':10} {'type':10} {'pattern':11} {'threshold':>9} {'measurements':>12} {'rejected':>8}") for s in names: ms = by[s] diff --git a/scripts/quality.py b/scripts/quality.py index f36736c..644c1d0 100644 --- a/scripts/quality.py +++ b/scripts/quality.py @@ -8,7 +8,7 @@ Usage: python3 tools/scripts/quality.py [--start T] [--end T] [--sensor [--metrics flux,snr,dominance] [--labels F] [--zone T1/T2 ...] [--out D] python3 tools/scripts/quality.py --self-test (check of the ratio computation) -Definitions: labeling/knowledge/quality-metrics.md. Columns filled on recent cameras only; on a folder where +Definitions: tools/DATA.md. Columns filled on recent cameras only; on a folder where they are empty, the tool says so and stops. Reference of a sensor: the median of each metric over its measurements in the --calm windows, at the same UTC hour of the day (24 slots; a slot without a calm measurement takes the median of all calm hours). Without --calm: the whole folder. @@ -374,7 +374,7 @@ def main(): sys.exit(f"No \"advanced\" export in {a.folder} (see inventory.py).") if not any(m[c] is not None for ms in every.values() for m in ms for c in METRICS): sys.exit(f"No quality metrics in {a.folder}: peak to snr empty (old camera; " - "labeling/knowledge/quality-metrics.md).") + "tools/DATA.md).") names = a.sensors.split(",") if a.sensors else sorted(every, key=lambda s: (not data.is_reference(every[s]), s)) missing = [s for s in names if s not in every] if missing: