improve: S2

ranges.py --labels: per label, the size of the label against the level before it (--margin, default 6h): extreme inside and net change after the end, in units and noises, for the --top sensor-columns (default 5); net alone across a label without measurement inside

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
maximeandClaude Opus 5.5 committed 2026-11-18 13:57:31 +01:00
1 parent 72ff92126c
commit af6cf23f94
2 files changed
+36 -9

No files matched your search

+2 -2
View File
@@ -98,7 +98,7 @@ Reading, in this order:
### ranges ### ranges
``` ```
python3 tools/scripts/ranges.py <folder> [--columns error_value,corr_x,corr_y] [--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N] [--labels F] [--sensors A,B] [--start T] [--end T] python3 tools/scripts/ranges.py <folder> [--columns error_value,corr_x,corr_y] [--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N] [--labels F [--margin 6h] [--top 5]] [--sensors A,B] [--start T] [--end T]
``` ```
Dates the start and the end of a sensor's moment without rerunning ten tables. By column and by sensor, at the measurements (native step, never averaged): level (median of the window), own noise, **largest jump** from one measurement to the next (value, in noises, the two instants; `(rejected: …)` when one end is); then the **ranges** where the sensor deviates from its level by more than `--k` noises: start, end, number of measurements, extreme (signed, in units and in noises) and its instant. Two ranges of the same sign less than `--merge` apart make one; rejected measurements do not cut a range; column `rejected`: how many of the range's measurements are (and `rejected extreme`). A range that touches the edge of the window is dated at the edge. Dates the start and the end of a sensor's moment without rerunning ten tables. By column and by sensor, at the measurements (native step, never averaged): level (median of the window), own noise, **largest jump** from one measurement to the next (value, in noises, the two instants; `(rejected: …)` when one end is); then the **ranges** where the sensor deviates from its level by more than `--k` noises: start, end, number of measurements, extreme (signed, in units and in noises) and its instant. Two ranges of the same sign less than `--merge` apart make one; rejected measurements do not cut a range; column `rejected`: how many of the range's measurements are (and `rejected extreme`). A range that touches the edge of the window is dated at the edge.
@@ -106,7 +106,7 @@ Dates the start and the end of a sensor's moment without rerunning ten tables. B
- `--min-measurements N`: only the ranges of at least N measurements (and how many out of the total). Over several days, the one-step ranges number in the hundreds and drown the long steps: first `--min-measurements 6`, then without. - `--min-measurements N`: only the ranges of at least N measurements (and how many out of the total). Over several days, the one-step ranges number in the hundreds and drown the long steps: first `--min-measurements 6`, then without.
- `--rolling N` (N measurements on each side) or `--rolling 1d` (the measurements within a duration): the level of each measurement is the median of its neighbours, without it; a long drift or a step follows this local level, only the deviations shorter than the window remain (the level column and the noise do not change; with `--outside`, only the noise is computed outside the episode). - `--rolling N` (N measurements on each side) or `--rolling 1d` (the measurements within a duration): the level of each measurement is the median of its neighbours, without it; a long drift or a step follows this local level, only the deviations shorter than the window remain (the level column and the noise do not change; with `--outside`, only the noise is computed outside the episode).
- `--instants N`: at the end, the instants where at least N sensors are out of level on one of the columns, with each sensor and its signed deviation in noises (`cx` corr_x, `cy` corr_y, `ev` error_value), the sensors rejected at that instant (among `--sensors`: removing the always-rejected sensors lightens the list), and each reference: `error_value` (`r` if rejected) / inlier. - `--instants N`: at the end, the instants where at least N sensors are out of level on one of the columns, with each sensor and its signed deviation in noises (`cx` corr_x, `cy` corr_y, `ev` error_value), the sensors rejected at that instant (among `--sensors`: removing the always-rejected sensors lightens the list), and each reference: `error_value` (`r` if rejected) / inlier.
- `--labels F` (one JSON per line, `start`, `end`, `family`, `id`; numbered in the order of the file): each line of `--instants` ends with the numbers of the labels that cover the instant (`labels: none`); then, per label, the measurements strictly inside it (a measurement at a bound is the point before or after), the largest deviation inside from the level (sensor, column, in noises, instant) and the number of instants inside beyond `--k` noises. - `--labels F` (one JSON per line, `start`, `end`, `family`, `id`; numbered in the order of the file): each line of `--instants` ends with the numbers of the labels that cover the instant (`labels: none`); then, per label, the measurements strictly inside it (a measurement at a bound is the point before or after), the largest deviation inside from the level (sensor, column, in noises, instant) and the number of instants inside beyond `--k` noises; under it, the **size of the label**, the `--top` sensor-columns (default 5) whose extreme inside deviates most from the **level before the label** (median of the `--margin` before its start, default 6h): `before` (measurements in that margin), the extreme (signed, in units and in noises) and its instant, `net` (median of the `--margin` after the end minus the level before, in units and in noises) and `after` (its measurements). A large extreme with a small `net` comes back within the label; a `net` close to the extreme keeps its new level. To measure existing labels (keep, correct, remove), take `--columns corr_x_mm,corr_y_mm` (or `corr_x,corr_y`) and widen `--start`/`--end` by the margin. A label without a measurement inside (an outage, a label shorter than one step) gives `net` alone, largest first: what changed across it.
- When the batch has a long drift or several steps (the ranges number in the hundreds, or one half of the batch is "out of level"), rerun with `--rolling` (a few measurements, or a few hours to a day depending on the cadence) and `--instants 2`: the brief deviations under the drift come out alone, and the list per instant says which sensors deviate together, which are rejected and what the references do; a deviation of all the targets with the references at the same instant is read against the camera pose. - When the batch has a long drift or several steps (the ranges number in the hundreds, or one half of the batch is "out of level"), rerun with `--rolling` (a few measurements, or a few hours to a day depending on the cadence) and `--instants 2`: the brief deviations under the drift come out alone, and the list per instant says which sensors deviate together, which are rejected and what the references do; a deviation of all the targets with the references at the same instant is read against the camera pose.
A noise of `0` (and "-" in noises, no range): the values are constant to rounding (deviations below 1e-9), like the `corr` of the references when the compensation retains only two (see Conventions); the same in `overview.py --residuals` and `days.py`. A noise of `0` (and "-" in noises, no range): the values are constant to rounding (deviations below 1e-9), like the `corr` of the references when the compensation retains only two (see Conventions); the same in `overview.py --residuals` and `days.py`.
+34 -7
View File
@@ -2,7 +2,7 @@
Usage: python3 tools/scripts/ranges.py data/PS_DES_COUDRES/ [--columns error_value,corr_x,corr_y] Usage: python3 tools/scripts/ranges.py data/PS_DES_COUDRES/ [--columns error_value,corr_x,corr_y]
[--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N] [--k 5] [--merge 1h] [--min-measurements 1] [--outside T1/T2] [--rolling N|DURATION] [--instants N]
[--labels F] [--sensors A,B] [--start ...] [--end ...] [--labels F [--margin 6h] [--top 5]] [--sensors A,B] [--start ...] [--end ...]
For each column, each sensor is compared with its own level over the --start/--end window (median of the For each column, each sensor is compared with its own level over the --start/--end window (median of the
non-empty values; with --outside T1/T2, that of the rest of the window, outside the moment being read: useful when a non-empty values; with --outside T1/T2, that of the rest of the window, outside the moment being read: useful when a
@@ -31,7 +31,12 @@ measurements, outside --outside).
--instants ends with the numbers of the labels that cover the instant ("labels: none" otherwise); then, per label, --instants ends with the numbers of the labels that cover the instant ("labels: none" otherwise); then, per label,
the measurements strictly inside it (start < instant < end, the instant to the minute: a measurement at a bound is the measurements strictly inside it (start < instant < end, the instant to the minute: a measurement at a bound is
the point before or after), the largest deviation inside from the level (sensor, column, signed, in noises, instant) the point before or after), the largest deviation inside from the level (sensor, column, signed, in noises, instant)
and the number of instants inside where a sensor is more than --k noises from it. and the number of instants inside where a sensor is more than --k noises from it; then the size of the label, up
to --top sensor-columns (default 5), largest first: the extreme inside from the level before the label (median of
the --margin before its start, default 6h; signed, in units and in noises, instant) and the net change (median of
the --margin after its end minus that level), with the number of measurements before and after; a label without
a measurement inside (an outage) gives the net change alone, largest first. Labels near the
edge of --start/--end: widen the window by --margin, or before/after lack measurements.
Each column is read at the measurements, never averaged per step. Gives values, not a verdict. Each column is read at the measurements, never averaged per step. Gives values, not a verdict.
""" """
import argparse import argparse
@@ -70,6 +75,8 @@ def main():
help="N (measurements on either side) or duration (6h, 1d): local level, median of the neighbours without the measurement") help="N (measurements on either side) or duration (6h, 1d): local level, median of the neighbours without the measurement")
a.add_argument("--instants", type=int, help="N: instants where at least N sensors are off level, with rejections and references") a.add_argument("--instants", type=int, help="N: instants where at least N sensors are off level, with rejections and references")
a.add_argument("--labels", help="label file (one JSON per line): labels covering each instant, deviation inside each label") a.add_argument("--labels", help="label file (one JSON per line): labels covering each instant, deviation inside each label")
a.add_argument("--margin", default="6h", type=duration, help="with --labels: the level before (and after) each label is the median of this duration before its start (after its end)")
a.add_argument("--top", type=int, default=5, help="with --labels: sensor-columns listed per label, largest deviation from the level before first")
a.add_argument("--sensors", help="comma-separated list (default: all)") a.add_argument("--sensors", help="comma-separated list (default: all)")
a.add_argument("--start", type=parse_time) a.add_argument("--start", type=parse_time)
a.add_argument("--end", type=parse_time) a.add_argument("--end", type=parse_time)
@@ -94,6 +101,7 @@ def main():
roll = (f"{o.rolling} measurements" if isinstance(o.rolling, int) else f"{int(o.rolling.total_seconds() // 60)} min") if o.rolling is not None else "" roll = (f"{o.rolling} measurements" if isinstance(o.rolling, int) else f"{int(o.rolling.total_seconds() // 60)} min") if o.rolling is not None else ""
by_instant = {} # iso(t) -> ["sensor.col+deviation"] (--instants) by_instant = {} # iso(t) -> ["sensor.col+deviation"] (--instants)
deviations = [] # (t, sensor, col, deviation in noises) of every measurement (--labels) deviations = [] # (t, sensor, col, deviation in noises) of every measurement (--labels)
values = {} # (sensor, col) -> (measurements (t, v), noise) (--labels: size against the level before each label)
for col in columns: for col in columns:
print(f"\n== {col}: level, noise, largest jump from one measurement to the next ==") print(f"\n== {col}: level, noise, largest jump from one measurement to the next ==")
@@ -119,6 +127,7 @@ def main():
+ (f" (rejected: {', '.join(flagged)})" if flagged else "")) + (f" (rejected: {', '.join(flagged)})" if flagged else ""))
if not noise: if not noise:
continue continue
values[s, col] = (ms, noise)
lv = local_levels(ms, o.rolling) if o.rolling is not None else [lvl] * len(ms) lv = local_levels(ms, o.rolling) if o.rolling is not None else [lvl] * len(ms)
group = [] # (t, deviation from the level) group = [] # (t, deviation from the level)
for (t, v), n in zip(ms + [(None, 0)], lv + [None]): for (t, v), n in zip(ms + [(None, 0)], lv + [None]):
@@ -169,16 +178,34 @@ def main():
print(f"\nPer label ({o.labels}): measurements strictly inside, largest deviation from the level" print(f"\nPer label ({o.labels}): measurements strictly inside, largest deviation from the level"
f"{' rolling ±' + roll if roll else ''} ({', '.join(columns)}), instants with a sensor more than {o.k:g} noises") f"{' rolling ±' + roll if roll else ''} ({', '.join(columns)}), instants with a sensor more than {o.k:g} noises")
print(f"{'n':>4} {'id':<16}{'start':<18}{'end':<18}{'family':<22}{'inside':>7} largest deviation (noises, at) instants > k") print(f"{'n':>4} {'id':<16}{'start':<18}{'end':<18}{'family':<22}{'inside':>7} largest deviation (noises, at) instants > k")
m = int(o.margin.total_seconds() // 60)
print(f"{'':6}{'sensor.column':<22}{'before':>7}{'extreme':>11}{'(n)':>8} {'at':<17}{'net':>11}{'(n)':>8}{'after':>6}"
f" (then, up to {o.top} per label: extreme inside from the median of the {m} min before the start;"
f" net: median of the {m} min after the end minus it, alone when no measurement is inside; before, after: their measurements; n: noises)")
for n, (t0, t1, e) in enumerate(labels, 1): for n, (t0, t1, e) in enumerate(labels, 1):
ins = [d for d in deviations if t0 < minute(d[0]) < t1] ins = [d for d in deviations if t0 < minute(d[0]) < t1]
head = f"{n:4} {str(e.get('id', '-')):<16}{iso(t0):<18}{iso(t1):<18}{e.get('family', '-')[:21]:<22}" head = f"{n:4} {str(e.get('id', '-')):<16}{iso(t0):<18}{iso(t1):<18}{e.get('family', '-')[:21]:<22}"
if not ins: if not ins:
print(head + f"{0:7} (no measurement inside)") print(head + f"{0:7} (no measurement inside)")
continue else:
t, s, col, d = max(ins, key=lambda x: abs(x[3])) t, s, col, d = max(ins, key=lambda x: abs(x[3]))
over = len({minute(x[0]) for x in ins if abs(x[3]) > o.k}) over = len({minute(x[0]) for x in ins if abs(x[3]) > o.k})
print(head + f"{len({minute(x[0]) for x in ins}):7} {s + '.' + ABBREV.get(col, col):<14}{d:+8.1f} {iso(t):<17}{over:>5}") print(head + f"{len({minute(x[0]) for x in ins}):7} {s + '.' + ABBREV.get(col, col):<14}{d:+8.1f} {iso(t):<17}{over:>5}")
rows = [] # size against the level before the label, and the level after it (net alone without a measurement inside)
for (s, col), (ms, noise) in values.items():
pre = [v for t, v in ms if t0 - o.margin <= minute(t) <= t0]
inside = [(t, v) for t, v in ms if t0 < minute(t) < t1]
post = [v for t, v in ms if t1 <= minute(t) <= t1 + o.margin]
if not pre or not (inside if ins else post):
continue
before = statistics.median(pre)
t, d = max(((t, v - before) for t, v in inside), key=lambda x: abs(x[1])) if inside else (None, None)
net = statistics.median(post) - before if post else None
rows.append((abs(d if ins else net) / noise, s, col, d, noise, t, len(pre), net, len(post)))
for _, s, col, d, noise, t, n_pre, net, n_post in sorted(rows, key=lambda r: -r[0])[:o.top]:
ext = f"{d:+11.4g}{d / noise:+8.1f} {iso(t):<17}" if d is not None else f"{'-':>11}{'-':>8} {'-':<17}"
after = f"{net:+11.4g}{net / noise:+8.1f}{n_post:6}" if net is not None else f"{'-':>11}{'-':>8}{0:6}"
print(f"{'':6}{s + '.' + col:<22}{n_pre:7}{ext}{after}")
if __name__ == "__main__": if __name__ == "__main__":
main() main()