# Reading the data The guide to the reading harness: followed by the analyst; improved by the upgrader (agents repository), run by Optimizer's `optimize` skill, the only one that writes here. A camera tracks targets (reflectors) fixed on a structure; its own movement is compensated thanks to reference targets, assumed fixed. This movement, the **camera pose** (`pan`, `tilt`, `rotation` in degrees, `scale`, `sigma0` of the registration, in `camera/`), is read before the sensors: it explains a large part of what moves on the curves (D-073). This guide says how to read a station's data: what the curves and the images show, moment by moment. It does not say what they mean: family, known event, hypothesis and verdict are the business of whoever calls you. ## Read first - `tools/DATA.md`: the files of a station and the meaning of each column, the image quality metrics, the camera images (**first**); - `labeling/reading-rules.md`: what the data mean (the measurement chain, gaps, reading rules, false friends). ## The tools Run them from the root of the calling repository, where `tools/`, `data/` and `labeling/` are links. They return values, never verdicts: you judge. `--out `: where whoever calls you stores your figures; they give it to you. The method (below) says in which order to run them; each section says what a tool returns and how to read it. | Tool | The question it answers | |---|---| | [`inventory.py`](#inventory) | What is there, what is missing: files, contents of each source, images, poses, cadence, sensors, rejections | | [`summary.py`](#summary) | The usual level of each sensor, then one column by time step | | [`plot.py`](#plot) | The series of chosen sensors and columns, as a PNG | | [`overview.py`](#overview) | All the sensors together, against the camera pose: common movement, single sensors, residuals, rigid movement | | [`ranges.py`](#ranges) | Dated ranges where each sensor leaves its level, at the native step | | [`days.py`](#days) | What comes back from one day to the next: night levels, days superimposed by hour, bounds of each day | | [`metrics.py`](#metrics) | The system metrics (temperature, voltage…) against the targets and the pose | | [`stations.py`](#stations) | Several stations on the same timeline | | [`quality.py`](#quality) | The six image quality metrics against a calm reference; labels; degraded instants no label covers | | [`thumbnails.py`](#thumbnails) | Full-resolution thumbnails of the camera images, around sensors or of a zone | | [`gif.py`](#gif) | A GIF of the camera images over a range, for a reply to the customer | | [`ls … \| grep`](#whole-image) | The whole camera image at a time | ### inventory ``` python3 tools/scripts/inventory.py [--start T] [--end T] [--files] ``` Names and counts, never values. Over the **whole period**: - files by type: beyond 5 per type, the count, the first and the last (`--files`: the full list, only to look for a specific file); - **what each source contains** over the window: the columns of the export, of the pose, of `quality/` and of `weather/` (the MeteoSwiss weather, hourly; a station without a Swiss position has none), with the number of rows that have a value ("empty" otherwise; an export column that not all sensors carry says how many carry it); the names of the system metrics of `metrics/`; the images and the difference images. The "Export image quality (peak to snr)" line says whether the six quality metrics are filled (then `quality.py`) or empty; - images by runs of equal gaps between two images (within 5 min, over the window if given): one line per change of gap, a hole is dated by its two images; - camera poses (count, period, measurement instants without registration), period, cadence, missing measurements, sensors (reference or target, error threshold, rejected measurements, distance from the camera and mm per px of `targets.csv`). When `pattern_type` is empty, a sensor is a reference if it carries `reproj_err` or `inlier` (the "pattern_type empty for N sensors" line says so); - "Duplicate readings": an export (often named after the customer's project) whose readings are already in the others; with 0 different values, it changes nothing to the reading, otherwise the tools keep the one of the last file in name order. With `--start`/`--end` (on a batch: those of the batch), a section for the window: the missing steps, dated (for all the sensors, or which ones are missing); by sensor, the rejected measurements, dated (runs of consecutive steps; rejected: `error_value` above the sensor's threshold, **references included**); then the **number of references retained by the compensation** (`inlier` = 1) at each instant and the dated runs where some are not, which ones and why (`r` rejected, `0` discarded by the compensation, `empty`; another mark is the value of `inlier`, e.g. `-1`). When `inlier` is empty over the window, the retained references are read from the filled `reproj_err` (the title says so). A run that ends the same day writes only the time of its end. Over a month, the output with `--start`/`--end` exceeds what a call displays: read each section through `grep` (`| grep -A40 '^Retained references'`, `| grep '^ PR04 '`). ### summary ``` python3 tools/scripts/summary.py [--column C] [--stat median|min|max] [--step 1h] [--level period|window] [--sensors A,B] [--start T] [--end T] [--instants T1,T2] [--mark] ``` The usual level of each sensor (median, MAD, min, max) over the whole period, or over the `--start`/`--end` window with `--level window` (for a batch: a level change within the batch is not compared with weeks of other measurements); then a table by time step, the sensor header repeated every 24 lines. - Columns: `error_value`, `corr_x`, `corr_y`, `relative_x`, `relative_y`, `x`, `y`, `sigma0`, `reproj_err`, `corr_x_mm`, `corr_y_mm` (the `corr` in mm, empty without a sure factor: `tools/DATA.md`, "The targets"), and the six image quality metrics `peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr` (raw values; against a calm reference, `quality.py`). - `--mark`: `*` after the value when a measurement of the step is rejected (`x`, `y`, `error_value` of a rejected measurement stay filled), `o` when one has inlier = 0. - `--instants T1,T2`: only the steps that contain these instants (the values of all the sensors at a few chosen times). - With `--step` equal to the cadence, `--level window --mark`: the per-instant, per-sensor table of one column, one column per run. ### plot ``` .venv/bin/python tools/scripts/plot.py [--sensors A,B] [--columns C1,C2] [--start T] [--end T] [--zone T1/T2 ...] [--out D] ``` A PNG of the series, each `--zone` greyed; prints its path. Columns: those of `summary.py`, quality metrics included (raw values). ### overview ``` .venv/bin/python tools/scripts/overview.py [--columns x,y,sigma0] [--column error_value] [--step 15min] [--k 5] [--min 5] [--residuals corr_x,corr_y,error_value|no] [--outside T1/T2] [--others] [--rigid BASE [--without A,B] [--follow A,B]] [--table T1/T2|no] [--sensors A,B] [--start T] [--end T] [--out D] ``` All the sensors together, each against its median and its MAD over the window. PNGs: - **common**: at the top, if the folder has `camera/`, the **camera pose** (raw values, one point per registration, the line cut at measurement instants without registration); the median across sensors of `x`, `y`, `sigma0`, the dispersion across sensors as a grey band; on the `x`, `y` panels, also **each global reference separately** (one line each) and the **median of the targets alone** (black line); at each step, the number of sensors whose `--column` exceeds its median by more than `--k` MAD. Ticks every 1 to 12 h depending on the window, every n days beyond 20 days. - **multiples**: a small chart per sensor, one scale each: a step of a single sensor shows there. - **residuals-corr_x**, **-corr_y**, **-error_value** (`--residuals`; `no`: none of these PNGs nor their tables): a small chart per sensor of the column minus the sensor's own level (its median over the window; with `--outside T1/T2`, that of the rest of the window, outside the moment read), one scale each, zero dotted: a deviation of a few hundredths on `corr`, hidden in the parallel bands of `plot.py`, shows there. - **residuals-…-others** (`--others`): each sensor minus its own median **and minus the median of the other sensors at the same instant**: what all share (the remainder of the compensation of a common movement) cancels, what is specific remains. - **rigid** (`--rigid BASE`): at each measurement, the overall movement of the targets since the instant `BASE` (translation, rotation, scale, estimated by medians: a target that jumps alone does not pull it; `--without A,B` removes targets from it) and, for each sensor, the **residual** `dx`, `dy` (raw position minus its position at `BASE` carried by this movement, in px). Panels: `dx`, `dy` of each global reference against the median and the 10th-90th percentile of the targets, `|residual|` of the targets (median, 90th percentile), fitted rotation and scale, `reproj_err` of the references; a rejected reference is a cross (the scale is that of the accepted measurements, a far cross is out of frame). Printed: - the **common median table**, one line per `--step`: the time, the deviation from the window median of `x`, `y`, `sigma0`, then `x_targets`, `y_targets` (median of the targets alone), then the raw camera pose `cam_pan`, `cam_tilt`, `cam_rotation`, `cam_scale`, `cam_sigma0` (empty without registration); the minimum and the maximum of each column with their instant; then those of `x`, `y` of each global reference (`(rejected)` when the measurement is, with the reference's number of rejected measurements); the measurement instants without pose (dated runs); the instants where at least `--min` sensors exceed together, with their names (`--min 1`: also a single sensor that leaves its level, absent from the default list of 5 sensors); - `--residuals`: per sensor, the level, the own noise (median of the differences between two measurements), the min/max deviation per step with their instant, in units and in noises; - `--others`: per sensor, the slope and the correlation r of the residual on the common median of `x`, `y`, `sigma0`; - `--rigid`: the table by step (`rot_deg`, `scale`, `tx`, `ty`, `tgt_med`, `tgt_max`, `dx`, `dy` of each reference and of each sensor of `--follow A,B`: the step-by-step sequence of a target) and, per sensor, min, max and value at the last instant of `dx`, `dy`. Rejected measurements enter neither the fit, nor the band of the targets, `tgt_med`, `tgt_max`, nor the columns of the table (over several days, parked cars or night rejections no longer drown the rest); the per-sensor summary gives them separately (`; N rejected: min …, max …`, and `(rejected)` after a last value that is rejected). The tables by step (common median, `--rigid`) repeat their header every 24 lines; `--table T1/T2` prints them only between T1 and T2 (`no`: not at all), levels, min, max, PNGs and the rest remaining over the whole window: over a one-month batch, first `--table no`, then `--table` on the day of a moment (same levels, which a shorter `--start`/`--end` window would not give). The default step is 15 min, the cadence of the measurements: a one-hour step flattens a peak of one to four steps, dates to the hour and erases an agitation of `x`, `y`; do not widen it to read a batch. Reading, in this order: 1. **The camera pose first** (top panels, `cam_` columns), dated at the native step: when it rotates (`pan`, `tilt`, `rotation`), changes scale, when its registration degrades (`sigma0`) or is absent (line cut, instants without pose). Each moment of the sensors is read against it: during a movement or a registration gap, or camera still. 2. **Who moves in the image.** The pose is computed from the global references: a change of pose (or a rise of `sigma0`) is a camera movement only if **the targets also move in the image** (on the `x`, `y` panels, the lines of the references against the black line of the targets alone; `x_targets`, `y_targets`). Only references move, targets still: these references moved, and the pose and the `corr` of all the targets are skewed without raw movement of the targets; say which references and by how much. References and targets together: the camera or the scene; then look whether the `corr` come back. 3. **Rotation or scale: `--rigid`.** Then the deviation from the median no longer says who moves alone (each sensor deviates according to its place in the image; the rigid residual does not). Before writing "everything moves together", rerun with `--rigid BASE`, `BASE` a calm instant before the first movement: a reference whose residual stays far from the 10th-90th percentile of the targets, or a `reproj_err` of the references that does not come back, means the compensation remains skewed by these references and the steps of `corr` partly come from it (say which, by how much, until when); a rising `|residual|` of the targets says they do not move as a block. 4. **Then `x`, `y`, `sigma0`, before `corr`**: the median of `x`, `y` (raw position, references included), the references one by one, the targets alone and `sigma0` say what moves in the image. Date the start, the end, the return of a movement, or a level that does not come back, on the printed table (time, value), not on the PNG axis. A dip of `corr` after a movement of the raw median is the remainder of the compensation, not a movement of these sensors. The multiples of `error_value` show a lasting step of a sensor, the count the instants where many rise together. 5. **The residuals, on every batch, without waiting for a suspicion**: a single sensor that rises or falls, a `corr` that drifts by a few hundredths, a step specific to one sensor under a common movement show there; the table says by how much (in noises) and when. On a target whose `corr` goes down under a strong common movement, `--others` separates a remainder of the compensation from a movement of its own: a dip that remains is not a remainder shared by all; a high slope with r close to ±1 on common `y` says it is proportional to the common movement (gain specific to the sensor); without r or slope, it is independent of the movement. ### ranges ``` python3 tools/scripts/ranges.py [--columns error_value,corr_x,corr_y] [--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N] [--sensors A,B] [--start T] [--end T] ``` Dates the start and the end of a sensor's moment without rerunning ten tables. By column and by sensor, at the measurements (native step, never averaged): level (median of the window), own noise, **largest jump** from one measurement to the next (value, in noises, the two instants; `(rejected: …)` when one end is); then the **ranges** where the sensor deviates from its level by more than `--k` noises: start, end, number of measurements, extreme (signed, in units and in noises) and its instant. Two ranges of the same sign less than `--merge` apart make one; rejected measurements do not cut a range; column `rejected`: how many of the range's measurements are (and `rejected extreme`). A range that touches the edge of the window is dated at the edge. - `--outside T1/T2`: level and noise computed outside an episode that skews the window median (the sensor spends half the batch there). Rerun with it around a long episode to date its return to the level: do not conclude "does not come back" without having read it. - `--min-measurements N`: only the ranges of at least N measurements (and how many out of the total). Over several days, the one-step ranges number in the hundreds and drown the long steps: first `--min-measurements 6`, then without. - `--rolling N` (N measurements on each side) or `--rolling 1d` (the measurements within a duration): the level of each measurement is the median of its neighbours, without it; a long drift or a step follows this local level, only the deviations shorter than the window remain (the level column and the noise do not change; with `--outside`, only the noise is computed outside the episode). - `--instants N`: at the end, the instants where at least N sensors are out of level on one of the columns, with each sensor and its signed deviation in noises (`cx` corr_x, `cy` corr_y, `ev` error_value), the sensors rejected at that instant (among `--sensors`: removing the always-rejected sensors lightens the list), and each reference: `error_value` (`r` if rejected) / inlier. - When the batch has a long drift or several steps (the ranges number in the hundreds, or one half of the batch is "out of level"), rerun with `--rolling` (a few measurements, or a few hours to a day depending on the cadence) and `--instants 2`: the brief deviations under the drift come out alone, and the list per instant says which sensors deviate together, which are rejected and what the references do; a deviation of all the targets with the references at the same instant is read against the camera pose. A noise of `0` (and "-" in noises, no range): the values are constant to rounding (deviations below 1e-9), like the `corr` of the references when the compensation retains only two (see Conventions); the same in `overview.py --residuals` and `days.py`. ### days ``` .venv/bin/python tools/scripts/days.py [--columns corr_x,corr_y,error_value] [--night 20-3] [--k 5] [--days 2] [--step 15min] [--sensors A,B] [--bounds [--smoothing N] [--fraction F]] [--start T] [--end T] [--out D] ``` What comes back from one day to the next, on accepted measurements only. A night: the hours `--night` H1-H2 UTC, straddling midnight, dated from the day it starts; choose hours where the day cycle (`error_value`, camera pose) has subsided (**byhour** figure). 1. **Level of each night**, by sensor and by column: median, MAD, number of measurements, own noise; from one night to the next, the **step** (difference of the medians, in units and in noises) and **where it falls** ("between A and B": where the night measurements of the two nights split best between the two levels; B at the start of the following night: the cut falls by day). Read it for any sensor that has already made a step or for which `--outside` is not enough (several levels in the batch): a step of a few noises is dated by its "between A and B", a step of `error_value` and one of `corr` at the same instant are linked, a slow drift reads from one night to the next. 2. **Superimposed days**: PNG **byhour-**, a small chart per sensor, hour of the day on the x axis, one curve per day, each measurement minus the level of the last night started (night greyed, frame from the 1st to the 99th percentile, the title says how many measurements fall outside it). Table: per `--step` slot of the hour of the day, the sensors whose deviation exceeds `--k` noises on the same side at least `--days` days, with the number of days and the median of these deviations. What comes back every day at the same hour, on one or several sensors, groups there, even when it lasts one to three measurements: one moment per hour of the day (say which sensors, how many days out of how many, the amplitude, what the camera pose does at that hour), not one moment per day, nor nothing because each step is short. 3. **Bounds of each day** (`--bounds`), for what comes back every day and is labelled day by day (thermal cycle, warming and cooling), rather than fixing the same hours for all the days: for the median of the targets of each column (with `--sensors`, the median of these sensors; with one, its own) and for the camera pose (`pan`, `tilt`, `rotation`, `scale`, `sigma0`, if `camera/` exists), the deviation from the level of the night that precedes each day (a day goes from the end of one night to the end of the next). Per day: number of measurements, the first and the last; first and last instant beyond `--k` noises, with the measurement just before and just after, at the native step; signed max and min with their instant; the sequence of measurements beyond `--k` noises around the larger of the two; the following night minus the reference night. A day without measurement says so; a reference night that is not the previous one is named. Each day line starts with the short name of its series (`corr_x`, `pose_tilt`…): `| grep '^pose_tilt '` reads one, `| grep ' 12/10 '` one day. PNG **bounds**: the deviation over the batch, first → last exceedance light grey, sequence around the extreme dark grey, max and min marked. - `--smoothing N`: the deviation of each measurement is the median of the N measurements of the day centred on it (noise and `--k` unchanged; PNG: smoothed in blue, raw in grey): an isolated point no longer cuts the sequence around the extreme nor makes a first or a last exceedance. - `--fraction F`: the threshold of each day is F times its largest |deviation| (printed on its line, `threshold …`), instead of `--k` noises: when the noise at the native step is unrelated to the amplitude of the cycle (very noisy or very smooth pose). - Reading: the sequence around the extreme dates the cycle, the max and the min a go and a return (morning and afternoon); "before" and "after" give the bounds at the native step (at a cadence of 1 h or 4 h, an event goes from the measurement before to the one after). First exceedance at the start of the day ("before -") and last at its end every day: the night is not calm; change `--night`, or `--sensors` the targets that carry the cycle. Camera drifting all night: a short night just before the morning (`--night 3-5`: the level of the rest that precedes the start). Noisy series or isolated points: `--smoothing 5`. Threshold in noises that does not bite (never exceeded, or exceeded all day): `--fraction 0.3`, then read the sequence around the extreme, not the first and the last exceedance. A "1 measurements" line is a peak, not a cycle. ### metrics ``` .venv/bin/python tools/scripts/metrics.py [--names A,B] [--step 1h] [--stat median|min|max] [--against temperature] [--smoothing 24h] [--metrics-only] [--start T] [--end T] [--out D] ``` The system metrics of `metrics/` (one value per hour), never the identifiers (`SIM_CCID`, `ip*`): - the available metrics with their number of values over the window ("empty": none; all empty is an absent datum, not a stable temperature); - a table per `--step`: `temperature` (median of the `sensor_*_deg_c` components, the one the detectors read), each metric of `--names` (default: the `*_deg_c` and `BusVoltage`), then, unless `--metrics-only`, `tgt_corr_x`, `tgt_corr_y` (median over the targets of `corr` minus each target's own level, accepted measurements) and the raw camera pose (`cam_pan` … `cam_sigma0`); an empty cell is skipped, a text metric (`link`, `BatteryState`, `last_sync`) is returned as it is. `--step 1d --stat min` (or `max`): the minimum or the maximum of each day; - per column: filled steps, min and max with their instant; the correlation r and the slope of each column on `--against` (with `--smoothing 24h`, on its rolling mean, column `_smooth`: for a station where the temperature varies little and fast, indoors); the dated runs of steps without any metric; - a PNG: temperatures on one panel (median in black), each other metric on its own, then `tgt_corr_x`, `tgt_corr_y` and the pose, line cut on a gap. An r does not say a cause: say it as a value, with the window where it is computed. ### stations ``` .venv/bin/python tools/scripts/stations.py [...] [--columns temperature,cam_tilt,cam_rotation,cam_scale,tgt_corr_x,tgt_corr_y] [--step 1h] [--stat median|min|max] [--table T1/T2|no] [--start T] [--end T] [--out D] ``` **Several stations on the same timeline** (neighbouring stations, same building). Columns: `temperature` (as in `metrics.py`), `cam_pan` … `cam_sigma0` (raw pose), `tgt_corr_x`, `tgt_corr_y` (as in `metrics.py`), `rej_targets`, `rej_refs` (number of rejected targets, of rejected references, the largest of the step). - Prints per station the period, the cadence and the number of instants; the **gaps**: the runs of steps without measurement or without temperature, with the stations where they are missing (a gap common to several stations on a single line); the table by step (one `STATION:column` column each, tabs; `--table` as in `overview.py`); per column and per pair of stations, r of the values and r of the steps from one value to the next (a common movement shows on the steps, a common trend on the values), with n. - PNG **stations**: a panel per column, a line per station (raw temperature, the rest minus its median), and a band per station of the steps without measurement. - Take `--step` equal to the slowest cadence of the stations; over a month at `1h`, first `--table no`. ### quality ``` python3 tools/scripts/quality.py [--calm T1/T2,T3/T4] [--common] [--table M] [--step 15min] [--threshold 0.5] [--sensors A,B] [--start T] [--end T] ... --labels F [--numbers 3,7] [--margin 60] [--edges] ... --labels F --outside [--min 1] [--except "Thermal displacement"] .venv/bin/python tools/scripts/quality.py --figure [--metrics flux,snr,dominance] [--labels F] [--zone T1/T2 ...] [--out D] python3 tools/scripts/quality.py --self-test ``` The six **image quality metrics** (`peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr`; definitions: `tools/DATA.md`), on the stations that have them (otherwise: "No quality metrics", and nothing else). Each measurement becomes a **ratio** to the **calm reference** of its sensor: the median of the metric over its measurements in the `--calm` windows, at the same UTC hour of the day (without `--calm`: the whole folder); 1 = like the reference, < 1 lower, > 1 higher; for `fwhm` and `dominance`, the rise says a degraded image; rejected measurements included. - Without options: the folder's maximal `peak`, then per sensor (`r` = reference) the rejected measurements, `pk/max` (median peak over the window divided by this maximum: at 1, the spot touches the image ceiling, its `peak` no longer moves and its `sharpness` follows the exposure: read `flux`, `snr` instead), and for each metric the median ratio and the extreme with its instant; then the median of the targets and of the references. - `--common`: per step, the median of the ratios of the targets / of the references for each metric (what affects the whole scene, visibility or turbulence; `--step 1d` shows on one screen on which days the reference holds). - `--table M`: per step, the ratio of `M` of each sensor (`*`: a rejected measurement), their median and how many pass `--threshold` (below, or above 1/`--threshold` for `fwhm`, `dominance`): a target that goes out and comes back; on a moment, `--table flux --sensors …` says which targets are hidden (flux cut) or veiled, from which instant to which instant: write them by their name. - `--labels F` (one JSON per line, `start`, `end`, `family`): per numbered label, the median ratio of the targets and of the references over the hour before, the range and the hour after (`tgt 0.96|0.26|0.79`, `--margin` minutes), then the sensors **affected** over the range (flux or snr below `--threshold`, or dominance above 1/`--threshold`), with their worst flux, from lowest to highest. `--edges`: per instant of the widened range, the number of affected sensors and the median of the targets of flux, snr, dominance (`<` within the range), to date the first and the last degraded instant. - `--outside`: the runs of instants that no label covers (outside `--except` families) where at least `--min` sensors are affected, with the sensors and their worst flux: candidates for forgotten labels, to check on `corr_x`, `corr_y`. - `--figure` (instead of the tables): PNG **quality-common** (a panel per metric, log scale: median of the targets in black, of the references in red, grey band from the 10th to the 90th percentile of the targets: a disturbance of the whole scene moves the median, an occultation of a few targets only the bottom of the band) and a PNG **quality-** per `--metrics` (a small chart per sensor, references in red); `--labels` greys each label in one colour per family, `--zone` ranges. Reading: the metrics say **why** a measurement degrades, where `error_value` says only that it degrades; they prove a cause and name the affected targets. What they showed on LENZSPITZE (to be confirmed on each station): `flux` cut (close to 0) on a group of targets, the others intact, for one to a few measurements: these targets are hidden (the list is read on the flux each time, not on the usual list; a hidden target can stay under its error threshold, and its point is wrong); `flux` down over the whole scene, stronger on the distant targets: reduced visibility; `sharpness` down and `dominance` up, `flux` whole: the image shakes (turbulence); nothing moves: neither the image nor the view, to be looked for in the movement. A whole flux refutes an occultation. `fwhm`, truncated to the disc, moves little; on a spot at the ceiling (`pk/max` at 1), it goes down when the spot darkens. **The calm reference decides everything**: take `--calm` on calm hours of the **same image regime** as the range read. When the lighting or the exposure changes (after a gap, an intervention), a reference on the other side skews the ratios by a factor of two or three: `--common --step 1d` over the batch shows this change (medians of the targets and of the references far from 1 for whole days, then at 1); then take a reference on each side and read each side with its own. Calm days are often days of thermal cycle: a reading of the thermal against them is circular. ### thumbnails ``` .venv/bin/python tools/scripts/thumbnails.py --sensors A,B --times T1,T2 [--radius 100] [--contrast [MIN,MAX]] [--fixed] [--zone X,Y,W,H] [--out D] ``` A sheet of full-resolution thumbnails: one row per sensor, one column per time, centred on the sensor's raw position (`x`, `y`) at each time. Look at it with Read: a whole image read by Read is reduced, a small reflector becomes illegible in it. - The centre of the frame is written under each thumbnail: when `x`, `y` jump, the frame moves by as much, and a spot that enters the frame seems to appear; `--fixed` keeps for each sensor the frame of the first time and marks with a red cross the `x`, `y` of each time. - Unretouched pixels: the brightness compares from one thumbnail to the next; prints for each thumbnail the background (median), the 99th percentile and the peak (grey levels 0 to 255). Without an image at the requested time, takes the nearest and prints the gap (the actual time is in the title). - A faint or deformed spot is invisible as it is: `--contrast MIN,MAX` puts all the thumbnails in grey with the same stretch (without value: from the 1st percentile to the sheet's maximum, crushed by a saturated pixel; then choose MIN,MAX from the printed background and peak, e.g. `10,40`); `--radius 30` to `50` magnifies. Compare the shape and the peak before, during and after an episode, at the same hour of the day. - A frame that the image edge cuts is flagged ("frame cut by the image edge (W × H px)"): on some stations, `x`, `y` are not pixels (values close to 0) and the frame falls in the corner; then look at the scene through `--zone`. - `--zone X,Y,W,H` (pixels of the original image: top left corner, width, height, as in `gif.py`): the same zone at each time, side by side (up to 4 per row), same stretch with `--contrast`; `--sensors` becomes optional, those given are marked with a cross and their name. For the scene around the targets (foreground object, headlights, lighting), prefer it to a large `--radius`. - Each thumbnail is rendered pixel for pixel up to 1000 px a side, but Read reduces a wide PNG: with a large `--radius` or a large `--zone`, three or four times per sheet. ### gif ``` .venv/bin/python tools/scripts/gif.py --start T --end T [--width 720] [--duration 700] [--max 150] [--zone X,Y,W,H] [--out D] ``` A GIF of the camera images between `--start` and `--end` inclusive, cropped to `--zone` if needed, set to `--width` px, UTC date and time at the bottom right; beyond `--max` images, `--max` at regular intervals; prints the time of each image, then the path and the size of the GIF. To attach the scene to a reply to the customer; Read does not read a GIF: to look, `thumbnails.py --zone`. ### whole image ``` ls /images/ | grep TZ ``` Then Read on the file found (Read takes no pattern): the whole camera image at that time (UTC), for the scene (occultation, lighting, weather). ## Conventions - Instants: `2026-01-01T00:00Z`; a time without time zone is read as UTC; `--end` includes the minute given. On a batch, `--start` and `--end` of the batch. - Bash is used only to run these tools, `ls`, `grep` and `date`, one command per call, without a `for` loop (a pipe to `grep` is allowed): permissions are granted command by command. If the current directory is not the root of the calling repository, `cd && ` counts as one command. Write no other script. Whoever calls you may add commands to this list. - **An output too long for one call** (beyond a few hundred lines, cut at display): shorten it first with the tool's options (`--table no` then `--table T1/T2`, `--min-measurements`, `--sensors`, `--columns`, a shorter window), or read a section of it through a pipe to `grep` (`| grep -A40 '^Title'`, `| grep '^pose_tilt '`). If you have to reread it several times, store it in the output folder in the same call (` > /.txt`) and read that file with Read (`offset`, `limit`) or with `grep`: no `head`, `tail`, `sed`, `cut`. The tools create their output folder themselves: no `mkdir`. - A sensor name that carries an apostrophe (`COLH20'''`, `REFP''21`) goes in double quotes in the shell: `--sensors "COLH20'''",COLI22`. - **Each PNG is looked at with Read**: you read a chart better than a table, and a PNG you have not looked at has shown nothing. - A rejected measurement (`error_value` above the threshold) is not a measurement: the station empties the `corr` of a rejected target, but leaves those of a rejected **reference**, often absurd; the tools empty them (`corr`, ranges, residuals, jumps cover only accepted measurements). When the compensation retains only **two references**, their `corr` is 0 (to 1e-13): it passes exactly through them; their noise prints as `0` and their deviations in noises as "-"; read nothing into these values, read the references through `x`, `y` (`overview.py --rigid`) and `error_value`. Its `x`, `y` remain: before citing a jump of `x`, `y`, check whether it is marked `(rejected)`; a rejected measurement is not cited as a movement. ## The method 1. **Inventory** (`inventory.py`): what is available, what is missing. A filled column that the tools do not show or a source that no tool reads (`images_diff/`) remains data: know that it exists. The image quality columns (`peak` to `snr`, recent cameras) are read through `quality.py`; `metrics/` through `metrics.py`: a cut of the measurements is read against the steps without metric and the voltage (`--names BusVoltage,BusCurrent,Power,BatteryPercent --metrics-only` on the cut). On a batch, rerun it with the batch's `--start`/`--end`: the absent steps and the rejections of the batch, dated; those of the whole period say nothing about the batch. Read the rejections of the **references** as much as those of the targets: a sensor rejected at the same hours every day makes a moment on its own (say its share of rejected measurements and its hours, day by day); when it is a reference, say with how many references the compensation then runs (the list of retained references). When several targets jump together by one step, or the compensation drops (no pose), look first in that list at which references were lost at that instant. 2. **Wide first**: - `overview.py` on the batch, first (on a batch of more than a few days, `--table no`, then `--table T1/T2` on the days to read), read in the order of its Reading section: the pose, who moves in the image (`--rigid` as soon as the pose rotates or changes scale), `x`, `y`, `sigma0` before `corr`, the residuals on every batch. - On a batch of more than two days, then `days.py`, before reading the isolated steps of `ranges.py`: first its **byhour** PNGs and table by slot, then its **night levels**; `days.py --bounds` for what comes back every day and is labelled day by day. - Several neighbouring stations given by whoever calls you: `stations.py` (temperatures, poses, `corr` of the targets, common rejections and gaps, at the same hours) before comparing their moments. - When the pose or the `corr` of the targets drift over several days, or move without day/night cycle: `metrics.py` on the batch, the temperature against `tgt_corr_x`, `tgt_corr_y` and the pose, on the PNG, then r and the slope. - On a station that has the image quality metrics: `quality.py --common --step 1d` on the batch (choice of the calm reference), then `quality.py --figure` with this reference: the median `flux` of the targets and of the references dates the hours of reduced visibility, the grey band that dives alone the occultations; with a labels file, `--labels F --outside`. - Then `plot.py` on all the sensors, `error_value` then `corr_x`, `corr_y` (or `relative_x`, `relative_y`), references and targets; `summary.py` on these columns with a wide step (6h or 1d). 3. **Then zoom**: `summary.py` at 15min (`--stat max` for the real peak of a sensor), and `plot.py` on the windows and the sensors that move. 4. **Each sensor against its own usual level** (its median and its MAD), never against an absolute threshold; on a batch, the level is computed on the batch (`--level window`, `overview.py`), not on the whole period. 5. **Each retained moment is dated and measured at the native 15 min step**: start, end and maximum read on the 15 min table (`overview.py` by default, `summary.py --step 15min`), never on an hourly table; also `overview.py --min 1` for single sensors; `ranges.py` on the batch (first `--min-measurements 6`) for, per sensor, start, end, extreme and largest jump at the native step (`--outside` around a long episode, `--rolling` and `--instants 2` under a long drift or several steps); the tables of `x`, `y` on the windows where they stir. On a station that has the quality metrics, for each moment: `quality.py --table flux --sensors …` (or `--labels F --numbers N --edges` on a label): which targets are hidden or veiled, from which instant to which instant. 6. **The images**: thumbnails (with `--contrast` for a faint target) spaced, one every 2 to 3 h, tighter where the curves move, always with an unaffected witness sensor, at a time before, during and after (at the same hour of the day if possible); when `x`, `y` of a sensor jump between two thumbnails (centre of the frame written under each), rerun with `--fixed` before saying that a spot appears or disappears; the whole image for the scene (occultation, lighting, weather). ## What the reading returns One line per moment. A moment: a sensor or a group of sensors that leaves its usual level during a window you can date. | Start | End | Sensors | Columns | What I see | Images | |---|---|---|---|---|---| - **Sensors**: references or targets, how many, which ones. - **What I see**: the shape (one-step peak, step, slow rise, agitation), the amplitude against the usual level (in MAD or in times the median), the return to normal or not, what the other sensors do, **what the camera does** at the same moment (pose moving, by how much, or still; registration absent; "no `camera/`" if the folder has none). - **Images**: what the thumbnails and the whole image show at that moment, or "no image". Times in UTC.