DATA.md: what the data are, from labeling/knowledge (formats, columns, quality metrics, images)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-16 12:41:44 +01:00
1 parent f28bd19e04
commit 4bfbad4db8
7 files changed
+73 -12

No files matched your search

+3 -4
View File
@@ -6,9 +6,8 @@ A camera tracks targets (reflectors) fixed on a structure; its own movement is c
## Read first
- `labeling/knowledge/displacement-columns.md`: formats of the exports and meaning of the columns (**first**);
- `labeling/knowledge/gaps.md`;
- on a station whose export carries the image quality metrics (`peak` to `snr`: the "Export image quality" line of `inventory.py`), `labeling/knowledge/quality-metrics.md`.
- `tools/DATA.md`: the files of a station and the meaning of each column, the image quality metrics, the camera images (**first**);
- `labeling/reading-rules.md`: what the data mean (the measurement chain, gaps, reading rules, false friends).
## The tools
@@ -24,7 +23,7 @@ Run them from the root of the calling repository, where `tools/`, `data/` and `l
| `.venv/bin/python tools/scripts/days.py <folder> [--columns corr_x,corr_y,error_value] [--night 20-3] [--k 5] [--days 2] [--step 15min] [--sensors A,B] [--bounds [--smoothing N] [--fraction F]] [--start T] [--end T] [--out D]` | What comes back from one day to the next, on accepted measurements only. 1. **Level of each night** (hours `--night` H1-H2 UTC, straddling midnight, dated from the day it starts), by sensor and by column: median, MAD, number of measurements, and the own noise; then, from one night to the next, the **step** (difference of the medians, in units and in noises) and **where it falls** ("between A and B": where the sequence of night measurements of the two nights splits best between the two levels; B at the start of the following night: the cut falls by day, between the two nights). A step of a few noises, a drift from one night to the next, and two columns that step at the same instant read there, even when the sensor has several levels in the batch. 2. **Superimposed days**: PNG **byhour-<column>**, a small chart per sensor, hour of the day on the x axis, one curve per day, each measurement minus the level of the last night started (night greyed, frame from the 1st to the 99th percentile, the title says how many measurements fall outside the frame); table: per `--step` slot of the hour of the day, the sensors whose deviation exceeds `--k` noises, on the same side, at least `--days` days, with the number of days and the median of these deviations. What comes back every day at the same hour, on one sensor or several, groups there, even when it lasts only one to three measurements. 3. **Bounds of each day**, with `--bounds`: for the median of the targets of each column and for the camera pose (`pan`, `tilt`, `rotation`, `scale`, `sigma0`, if `camera/` exists), the deviation from the level of the night that precedes each day (a day goes from the end of one night to the end of the next; with `--sensors`, the median of these sensors, with a single one, its own). Per day: number of measurements, the first and the last; first and last instant beyond `--k` noises, with the measurement just before and just after, at the native step; signed max and min with their instant; the sequence of measurements beyond `--k` noises that surrounds the larger of the two; the following night minus the reference night. A day without measurement says so; a reference night that is not the previous one is named. Each day line starts with the short name of its series (`corr_x`, `pose_tilt`…): over a month, `\| grep '^pose_tilt '` reads one, `\| grep ' 12/10 '` one day. `--smoothing N`: the deviation of each measurement is the median of the N measurements of the day centred on it (noise and `--k` unchanged; PNG: smoothed in blue, raw in grey): an isolated point no longer cuts the sequence around the extreme and no longer makes a first or a last exceedance. `--fraction F`: the threshold of each day is F times the largest |deviation| of that day (printed on its line, `threshold …`), instead of `--k` noises: when the noise at the native step is unrelated to the amplitude of the cycle (very noisy or very smooth pose). PNG **bounds**: the deviation over the batch, first → last exceedance light grey, sequence around the extreme dark grey, max and min marked. The choice of `--night`: hours where the day cycle (`error_value`, camera pose) has subsided. **Look at the PNGs with Read** |
| `.venv/bin/python tools/scripts/metrics.py <folder> [--names A,B] [--step 1h] [--stat median\|min\|max] [--against temperature] [--smoothing 24h] [--metrics-only] [--start T] [--end T] [--out D]` | The system metrics of `metrics/` (one value per hour), never the identifiers (`SIM_CCID`, `ip*`). Prints the available metrics with their number of values over the window ("empty": none), then a table per `--step`: `temperature` (median of the `sensor_*_deg_c` components, the one the detectors read), each metric of `--names` (default: the `*_deg_c` and `BusVoltage`), then, unless `--metrics-only`, `tgt_corr_x`, `tgt_corr_y` (median over the targets of `corr` minus each target's own level, accepted measurements) and the raw camera pose (`cam_pan` … `cam_sigma0`); an empty cell is skipped, a text metric (`link`, `BatteryState`, `last_sync`) is returned as it is. Then, per column: filled steps, min and max with their instant; the correlation r and the slope of each column on `--against` (with `--smoothing 24h`, on its rolling mean, column `<against>_smooth`); the dated runs of steps without any metric. A PNG: temperatures on one panel (median in black), each other metric on its own, then `tgt_corr_x`, `tgt_corr_y` and the pose, line cut on a gap. **Look at it with Read** |
| `.venv/bin/python tools/scripts/stations.py <folder> <folder> [...] [--columns temperature,cam_tilt,cam_rotation,cam_scale,tgt_corr_x,tgt_corr_y] [--step 1h] [--stat median\|min\|max] [--table T1/T2\|no] [--start T] [--end T] [--out D]` | **Several stations on the same timeline** (neighbouring stations, same building). Possible columns: `temperature` (as in `metrics.py`), `cam_pan` … `cam_sigma0` (raw pose), `tgt_corr_x`, `tgt_corr_y` (as in `metrics.py`), `rej_targets`, `rej_refs` (number of rejected targets, of rejected references, the largest of the step). Prints per station the period, the cadence and the number of instants; then the **gaps**: the runs of steps without measurement or without temperature, with the stations where they are missing (a gap common to several stations on a single line); then the table by step (one `STATION:column` column each, tabs; `--table` as in `overview.py`); then, per column and per pair of stations, r of the values and r of the steps from one value to the next (a common movement shows on the steps, a common trend on the values), with n. PNG **stations**: a panel per column, a line per station (raw temperature, the rest minus its median), and a band per station of the steps without measurement. Take `--step` equal to the slowest cadence of the stations; over a month at `1h`, first read with `--table no`. **Look at it with Read** |
| `python3 tools/scripts/quality.py <folder> [--calm T1/T2,T3/T4] [--common] [--table M] [--step 15min] [--threshold 0.5] [--sensors A,B] [--start T] [--end T]` ; `... --labels F [--numbers 3,7] [--margin 60] [--edges]` ; `... --labels F --outside [--min 1] [--except "Thermal displacement"]` ; `.venv/bin/python tools/scripts/quality.py <folder> --figure [--metrics flux,snr,dominance] [--labels F] [--zone T1/T2 ...] [--out D]` ; `python3 tools/scripts/quality.py --self-test` | The six **image quality metrics** (`peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr`; definitions: `labeling/knowledge/quality-metrics.md`), on the stations that have them (otherwise: "No quality metrics", and nothing else). Each measurement becomes a **ratio** to the **calm reference** of its sensor: the median of the metric over its measurements in the `--calm` windows, at the same UTC hour of the day (without `--calm`: the whole folder); 1 = like the reference, < 1 lower, > 1 higher; for `fwhm` and `dominance`, it is the rise that says a degraded image; rejected measurements included. Without options: the folder's maximal `peak`, then per sensor (`r` = reference) the rejected measurements, `pk/max` (median peak over the window divided by this maximum: at 1, the spot touches the image ceiling, its `peak` no longer moves and its `sharpness` follows the exposure: read `flux`, `snr` instead), and for each metric the median ratio and the extreme with its instant; then the median of the targets and of the references. `--common`: per step, the median of the ratios of the targets / of the references for each metric (what affects the whole scene, visibility or turbulence; `--step 1d` to see on one screen on which days the reference holds). `--table M`: per step, the ratio of `M` of each sensor (`*`: a rejected measurement), their median and how many pass `--threshold` (below, or above 1/`--threshold` for `fwhm`, `dominance`): a target that goes out and comes back. `--labels F` (one JSON per line, `start`, `end`, `family`): per numbered label, the median ratio of the targets and of the references over the hour before, the range and the hour after (`tgt 0.96\|0.26\|0.79`, `--margin` minutes), then the sensors **affected** over the range (flux or snr below `--threshold`, or dominance above 1/`--threshold`), with their worst flux, from lowest to highest; `--edges`: per instant of the widened range, the number of affected sensors and the median of the targets of flux, snr, dominance (`<` within the range), to date the first and the last degraded instant. `--outside`: the runs of instants that no label covers (outside `--except` families) where at least `--min` sensors are affected, with the sensors and their worst flux: candidates for forgotten labels, to check on `corr_x`, `corr_y`. `--figure` (instead of the tables): PNG **quality-common** (a panel per metric, log scale: median of the targets in black, of the references in red, grey band from the 10th to the 90th percentile of the targets: a disturbance of the whole scene moves the median, an occultation of a few targets moves only the bottom of the band) and a PNG **quality-<metric>** per `--metrics` (a small chart per sensor, references in red); `--labels` greys each label there in one colour per family, `--zone` ranges. **Look at the PNGs with Read** |
| `python3 tools/scripts/quality.py <folder> [--calm T1/T2,T3/T4] [--common] [--table M] [--step 15min] [--threshold 0.5] [--sensors A,B] [--start T] [--end T]` ; `... --labels F [--numbers 3,7] [--margin 60] [--edges]` ; `... --labels F --outside [--min 1] [--except "Thermal displacement"]` ; `.venv/bin/python tools/scripts/quality.py <folder> --figure [--metrics flux,snr,dominance] [--labels F] [--zone T1/T2 ...] [--out D]` ; `python3 tools/scripts/quality.py --self-test` | The six **image quality metrics** (`peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr`; definitions: `tools/DATA.md`), on the stations that have them (otherwise: "No quality metrics", and nothing else). Each measurement becomes a **ratio** to the **calm reference** of its sensor: the median of the metric over its measurements in the `--calm` windows, at the same UTC hour of the day (without `--calm`: the whole folder); 1 = like the reference, < 1 lower, > 1 higher; for `fwhm` and `dominance`, it is the rise that says a degraded image; rejected measurements included. Without options: the folder's maximal `peak`, then per sensor (`r` = reference) the rejected measurements, `pk/max` (median peak over the window divided by this maximum: at 1, the spot touches the image ceiling, its `peak` no longer moves and its `sharpness` follows the exposure: read `flux`, `snr` instead), and for each metric the median ratio and the extreme with its instant; then the median of the targets and of the references. `--common`: per step, the median of the ratios of the targets / of the references for each metric (what affects the whole scene, visibility or turbulence; `--step 1d` to see on one screen on which days the reference holds). `--table M`: per step, the ratio of `M` of each sensor (`*`: a rejected measurement), their median and how many pass `--threshold` (below, or above 1/`--threshold` for `fwhm`, `dominance`): a target that goes out and comes back. `--labels F` (one JSON per line, `start`, `end`, `family`): per numbered label, the median ratio of the targets and of the references over the hour before, the range and the hour after (`tgt 0.96\|0.26\|0.79`, `--margin` minutes), then the sensors **affected** over the range (flux or snr below `--threshold`, or dominance above 1/`--threshold`), with their worst flux, from lowest to highest; `--edges`: per instant of the widened range, the number of affected sensors and the median of the targets of flux, snr, dominance (`<` within the range), to date the first and the last degraded instant. `--outside`: the runs of instants that no label covers (outside `--except` families) where at least `--min` sensors are affected, with the sensors and their worst flux: candidates for forgotten labels, to check on `corr_x`, `corr_y`. `--figure` (instead of the tables): PNG **quality-common** (a panel per metric, log scale: median of the targets in black, of the references in red, grey band from the 10th to the 90th percentile of the targets: a disturbance of the whole scene moves the median, an occultation of a few targets moves only the bottom of the band) and a PNG **quality-<metric>** per `--metrics` (a small chart per sensor, references in red); `--labels` greys each label there in one colour per family, `--zone` ranges. **Look at the PNGs with Read** |
| `.venv/bin/python tools/scripts/thumbnails.py <folder> --sensors A,B --times T1,T2 [--radius 100] [--contrast [MIN,MAX]] [--fixed] [--zone X,Y,W,H] [--out D]` | A sheet of full-resolution thumbnails: one row per sensor, one column per time, centred on the sensor's raw position (`x`, `y`) at each time (the centre of the frame is written under each thumbnail: when `x`, `y` jump, the frame moves by as much, and a spot that enters the frame seems to appear); `--fixed` keeps for each sensor the frame of the first time and marks with a red cross the `x`, `y` of each time; unretouched pixels, the brightness can be compared from one thumbnail to the next; prints for each thumbnail the background (median), the 99th percentile and the peak (grey levels 0 to 255); without an image at the requested time, takes the nearest and prints the gap (the actual time is in the title). A faint or deformed spot is invisible as it is: `--contrast MIN,MAX` puts all the thumbnails in grey with the same stretch (without value: from the 1st percentile to the sheet's maximum, crushed by a saturated pixel; then choose MIN,MAX from the printed background and peak, e.g. `10,40`), `--radius 30` to `50` magnifies; compare the shape and the peak before, during and after an episode, at the same hour of the day. **Look at it with Read**: a whole image read by Read is reduced, a small reflector becomes illegible in it. A frame that the image edge cuts is flagged ("frame cut by the image edge (W × H px)"): on some stations, `x`, `y` are not pixels (values close to 0) and the frame falls in the corner; then look at the scene through `--zone`. `--zone X,Y,W,H` (pixels of the original image: top left corner, width, height, as in `gif.py`): the same zone at each time, side by side (up to 4 per row), same stretch with `--contrast`; `--sensors` becomes optional, and those that are given are marked with a cross and their name. For the scene around the targets (foreground object, headlights, lighting), prefer it to a large `--radius`. Each thumbnail is rendered pixel for pixel up to 1000 px a side, but Read reduces a wide PNG: with a large `--radius` or a large `--zone`, three or four times per sheet |
| `.venv/bin/python tools/scripts/gif.py <folder> --start T --end T [--width 720] [--duration 700] [--max 150] [--zone X,Y,W,H] [--out D]` | A GIF of the camera images between `--start` and `--end` inclusive, cropped to `--zone` if needed, set to `--width` px, UTC date and time written at the bottom right; beyond `--max` images, `--max` at regular intervals; prints the time of each image, then the path and the size of the GIF. Used to attach the scene to a reply to the customer; Read does not read a GIF: to look, `thumbnails.py --zone` |
| `ls <folder>/images/ \| grep <YYYYMMDD>T<HHMM>Z`, then Read on the file found (Read takes no pattern) | The whole camera image at that time (UTC), for the scene (occultation, lighting, weather) |