diff --git a/GUIDE.md b/GUIDE.md index 64d4f89..3ce1a0e 100644 --- a/GUIDE.md +++ b/GUIDE.md @@ -98,7 +98,7 @@ Reading, in this order: ### ranges ``` -python3 tools/scripts/ranges.py [--columns error_value,corr_x,corr_y] [--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N] [--labels F] [--sensors A,B] [--start T] [--end T] +python3 tools/scripts/ranges.py [--columns error_value,corr_x,corr_y] [--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N] [--labels F [--margin 6h] [--top 5]] [--sensors A,B] [--start T] [--end T] ``` Dates the start and the end of a sensor's moment without rerunning ten tables. By column and by sensor, at the measurements (native step, never averaged): level (median of the window), own noise, **largest jump** from one measurement to the next (value, in noises, the two instants; `(rejected: …)` when one end is); then the **ranges** where the sensor deviates from its level by more than `--k` noises: start, end, number of measurements, extreme (signed, in units and in noises) and its instant. Two ranges of the same sign less than `--merge` apart make one; rejected measurements do not cut a range; column `rejected`: how many of the range's measurements are (and `rejected extreme`). A range that touches the edge of the window is dated at the edge. @@ -106,7 +106,7 @@ Dates the start and the end of a sensor's moment without rerunning ten tables. B - `--min-measurements N`: only the ranges of at least N measurements (and how many out of the total). Over several days, the one-step ranges number in the hundreds and drown the long steps: first `--min-measurements 6`, then without. - `--rolling N` (N measurements on each side) or `--rolling 1d` (the measurements within a duration): the level of each measurement is the median of its neighbours, without it; a long drift or a step follows this local level, only the deviations shorter than the window remain (the level column and the noise do not change; with `--outside`, only the noise is computed outside the episode). - `--instants N`: at the end, the instants where at least N sensors are out of level on one of the columns, with each sensor and its signed deviation in noises (`cx` corr_x, `cy` corr_y, `ev` error_value), the sensors rejected at that instant (among `--sensors`: removing the always-rejected sensors lightens the list), and each reference: `error_value` (`r` if rejected) / inlier. -- `--labels F` (one JSON per line, `start`, `end`, `family`, `id`; numbered in the order of the file): each line of `--instants` ends with the numbers of the labels that cover the instant (`labels: none`); then, per label, the measurements strictly inside it (a measurement at a bound is the point before or after), the largest deviation inside from the level (sensor, column, in noises, instant) and the number of instants inside beyond `--k` noises. +- `--labels F` (one JSON per line, `start`, `end`, `family`, `id`; numbered in the order of the file): each line of `--instants` ends with the numbers of the labels that cover the instant (`labels: none`); then, per label, the measurements strictly inside it (a measurement at a bound is the point before or after), the largest deviation inside from the level (sensor, column, in noises, instant) and the number of instants inside beyond `--k` noises; under it, the **size of the label**, the `--top` sensor-columns (default 5) whose extreme inside deviates most from the **level before the label** (median of the `--margin` before its start, default 6h): `before` (measurements in that margin), the extreme (signed, in units and in noises) and its instant, `net` (median of the `--margin` after the end minus the level before, in units and in noises) and `after` (its measurements). A large extreme with a small `net` comes back within the label; a `net` close to the extreme keeps its new level. To measure existing labels (keep, correct, remove), take `--columns corr_x_mm,corr_y_mm` (or `corr_x,corr_y`) and widen `--start`/`--end` by the margin. A label without a measurement inside (an outage, a label shorter than one step) gives `net` alone, largest first: what changed across it. - When the batch has a long drift or several steps (the ranges number in the hundreds, or one half of the batch is "out of level"), rerun with `--rolling` (a few measurements, or a few hours to a day depending on the cadence) and `--instants 2`: the brief deviations under the drift come out alone, and the list per instant says which sensors deviate together, which are rejected and what the references do; a deviation of all the targets with the references at the same instant is read against the camera pose. A noise of `0` (and "-" in noises, no range): the values are constant to rounding (deviations below 1e-9), like the `corr` of the references when the compensation retains only two (see Conventions); the same in `overview.py --residuals` and `days.py`. diff --git a/scripts/ranges.py b/scripts/ranges.py index c8999a0..4ec4758 100644 --- a/scripts/ranges.py +++ b/scripts/ranges.py @@ -2,7 +2,7 @@ Usage: python3 tools/scripts/ranges.py data/PS_DES_COUDRES/ [--columns error_value,corr_x,corr_y] [--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N] - [--labels F] [--sensors A,B] [--start ...] [--end ...] + [--labels F [--margin 6h] [--top 5]] [--sensors A,B] [--start ...] [--end ...] For each column, each sensor is compared with its own level over the --start/--end window (median of the non-empty values; with --outside T1/T2, that of the rest of the window, outside the moment being read: useful when a @@ -31,7 +31,12 @@ measurements, outside --outside). --instants ends with the numbers of the labels that cover the instant ("labels: none" otherwise); then, per label, the measurements strictly inside it (start < instant < end, the instant to the minute: a measurement at a bound is the point before or after), the largest deviation inside from the level (sensor, column, signed, in noises, instant) - and the number of instants inside where a sensor is more than --k noises from it. + and the number of instants inside where a sensor is more than --k noises from it; then the size of the label, up + to --top sensor-columns (default 5), largest first: the extreme inside from the level before the label (median of + the --margin before its start, default 6h; signed, in units and in noises, instant) and the net change (median of + the --margin after its end minus that level), with the number of measurements before and after; a label without + a measurement inside (an outage) gives the net change alone, largest first. Labels near the + edge of --start/--end: widen the window by --margin, or before/after lack measurements. Each column is read at the measurements, never averaged per step. Gives values, not a verdict. """ import argparse @@ -70,6 +75,8 @@ def main(): help="N (measurements on either side) or duration (6h, 1d): local level, median of the neighbours without the measurement") a.add_argument("--instants", type=int, help="N: instants where at least N sensors are off level, with rejections and references") a.add_argument("--labels", help="label file (one JSON per line): labels covering each instant, deviation inside each label") + a.add_argument("--margin", default="6h", type=duration, help="with --labels: the level before (and after) each label is the median of this duration before its start (after its end)") + a.add_argument("--top", type=int, default=5, help="with --labels: sensor-columns listed per label, largest deviation from the level before first") a.add_argument("--sensors", help="comma-separated list (default: all)") a.add_argument("--start", type=parse_time) a.add_argument("--end", type=parse_time) @@ -94,6 +101,7 @@ def main(): roll = (f"{o.rolling} measurements" if isinstance(o.rolling, int) else f"{int(o.rolling.total_seconds() // 60)} min") if o.rolling is not None else "" by_instant = {} # iso(t) -> ["sensor.col+deviation"] (--instants) deviations = [] # (t, sensor, col, deviation in noises) of every measurement (--labels) + values = {} # (sensor, col) -> (measurements (t, v), noise) (--labels: size against the level before each label) for col in columns: print(f"\n== {col}: level, noise, largest jump from one measurement to the next ==") @@ -119,6 +127,7 @@ def main(): + (f" (rejected: {', '.join(flagged)})" if flagged else "")) if not noise: continue + values[s, col] = (ms, noise) lv = local_levels(ms, o.rolling) if o.rolling is not None else [lvl] * len(ms) group = [] # (t, deviation from the level) for (t, v), n in zip(ms + [(None, 0)], lv + [None]): @@ -169,16 +178,34 @@ def main(): print(f"\nPer label ({o.labels}): measurements strictly inside, largest deviation from the level" f"{' rolling ±' + roll if roll else ''} ({', '.join(columns)}), instants with a sensor more than {o.k:g} noises") print(f"{'n':>4} {'id':<16}{'start':<18}{'end':<18}{'family':<22}{'inside':>7} largest deviation (noises, at) instants > k") + m = int(o.margin.total_seconds() // 60) + print(f"{'':6}{'sensor.column':<22}{'before':>7}{'extreme':>11}{'(n)':>8} {'at':<17}{'net':>11}{'(n)':>8}{'after':>6}" + f" (then, up to {o.top} per label: extreme inside from the median of the {m} min before the start;" + f" net: median of the {m} min after the end minus it, alone when no measurement is inside; before, after: their measurements; n: noises)") for n, (t0, t1, e) in enumerate(labels, 1): ins = [d for d in deviations if t0 < minute(d[0]) < t1] head = f"{n:4} {str(e.get('id', '-')):<16}{iso(t0):<18}{iso(t1):<18}{e.get('family', '-')[:21]:<22}" if not ins: print(head + f"{0:7} (no measurement inside)") - continue - t, s, col, d = max(ins, key=lambda x: abs(x[3])) - over = len({minute(x[0]) for x in ins if abs(x[3]) > o.k}) - print(head + f"{len({minute(x[0]) for x in ins}):7} {s + '.' + ABBREV.get(col, col):<14}{d:+8.1f} {iso(t):<17}{over:>5}") - + else: + t, s, col, d = max(ins, key=lambda x: abs(x[3])) + over = len({minute(x[0]) for x in ins if abs(x[3]) > o.k}) + print(head + f"{len({minute(x[0]) for x in ins}):7} {s + '.' + ABBREV.get(col, col):<14}{d:+8.1f} {iso(t):<17}{over:>5}") + rows = [] # size against the level before the label, and the level after it (net alone without a measurement inside) + for (s, col), (ms, noise) in values.items(): + pre = [v for t, v in ms if t0 - o.margin <= minute(t) <= t0] + inside = [(t, v) for t, v in ms if t0 < minute(t) < t1] + post = [v for t, v in ms if t1 <= minute(t) <= t1 + o.margin] + if not pre or not (inside if ins else post): + continue + before = statistics.median(pre) + t, d = max(((t, v - before) for t, v in inside), key=lambda x: abs(x[1])) if inside else (None, None) + net = statistics.median(post) - before if post else None + rows.append((abs(d if ins else net) / noise, s, col, d, noise, t, len(pre), net, len(post))) + for _, s, col, d, noise, t, n_pre, net, n_post in sorted(rows, key=lambda r: -r[0])[:o.top]: + ext = f"{d:+11.4g}{d / noise:+8.1f} {iso(t):<17}" if d is not None else f"{'-':>11}{'-':>8} {'-':<17}" + after = f"{net:+11.4g}{net / noise:+8.1f}{n_post:6}" if net is not None else f"{'-':>11}{'-':>8}{0:6}" + print(f"{'':6}{s + '.' + col:<22}{n_pre:7}{ext}{after}") if __name__ == "__main__": main()