Files
aimsight-tools/DATA.md
T

6.7 KiB
Raw Blame History

The data of a station

What the files of a station folder are, and what each column means. How to read them with the tools: GUIDE.md. What they mean for a label (causes, reading rules, false friends): labeling/reading-rules.md. Maintained by hand; GUIDE.md is the tool-builder's.

The displacements: the "advanced" export

displacement/<Station>_advanced_<start>_<end>.csv, the source of the analyses: one row per sensor and per measurement, at the native rate of the station (10 or 15 min), time_utc in explicit UTC.

Column Meaning
sensor_name the sensor (target or reference)
pattern_type global_ref: a reference of the compensation; ref: a local reference, treated as a target, outside the compensation, without reproj_err; empty: a target
x, y raw position in the image [px]. Not the centroid of the spot: the offset centroid − (x, y) is constant per sensor by day and differs between sensors; it changes at night, when the spot is saturated. Probably the position of the reference pattern
relative_x, relative_y relative displacement before the camera compensation [px]; not x − x0
corr_x, corr_y the position of the target once the estimated camera movement is removed: the curves that are labelled. In px, not mm (below)
error_value recognition error of the pattern, not an uncertainty
error_threshold specific to each sensor (0.3 to 0.7). Above it, the measurement is rejected: the row and its error_value stay, corr_x and corr_y are empty, not interpolated
reproj_err, inlier references only: residual and contribution of each reference in the compensation
sigma0 residual of the fit on the references: one value per measurement, repeated on every row
peak, flux, sharpness, fwhm, dominance, snr image quality of each sensor, recent cameras only (below)
  • "Rejected" is a judgment of the AIMsight platform, not of the camera: the camera delivers all its measurements, and a client's platform that integrates the data receives them all.
  • Unit of corr_x, corr_y: px. The hourly export's horizontal_displacement (mm) is a factor × the hourly mean of corr_x + a constant (likewise vertical and corr_y, correlation ≥ 0.99); the factor (mm per px) depends on the pattern and on the station.
  • LENZSPITZE: its export gives x, y relative to the pattern; the extraction rewrites them in image px and fills pattern_type. Sensors 1023 and 8019 have no pattern: their x, y are empty. The tools' crosses may sit slightly off the spots there.
  • inventory.py says, on a batch, which columns carry values.

The camera pose

camera/<Station>_camera_<YYYYMMDD>.csv, from the pumping only: the pose at each registration, time_utc in UTC, median_|min_|max_ of pan, tilt, rotation [deg], scale, sigma0, one row per measurement (min = max). A measurement instant without a row: the camera was not registered.

  • Orders of magnitude: pan, tilt of a few hundredths of a degree, scale within 10⁻³ of 1; some stations show several degrees, not explained.
  • A positive tilt means the camera is nosing down (the expert).
  • Read by overview.py and inventory.py.

The hourly export

<Station>_<start>_to_<end>_1hour.csv: one row per hour, targets only; per target horizontal_displacement, vertical_displacement, their _min / _max, _error (mean error_value of the hour), _count_val (measurements of the hour, rejected ones included).

  • timestamp has no time zone: Swiss local time (UTC+2 in summer), at the start of the hour.

Image quality of a target

Six columns of the advanced export, computed by the GPU of recent cameras on the sensor's window in the DIFF image, in units of that image. Filled only where inventory.py says so ("Export image quality"); for older cameras, ~/optical_emulator can recompute them from the DIFF images. "Disc": the neighbourhood of the peak, radius min(H, W)/6 (at least 3 px); "background": the rest of the window.

Metric Definition What it measures Reading
peak max(I) the brightness of the return high, saturation; low, underexposure or missed flash. All targets dropping: the lighting; a single one: that target (fouling, misalignment)
flux flux of the disc minus the background the light of the target, background removed reflectivity of the target (fouling, ageing, flash), not ambient light; drops well before error_value exceeds its threshold
sharpness Σ(I²) / (ΣI)² concentration of the energy (higher = sharper) drops: drifting focus, turbulence, degrading target. Independent of the gain, not of the background: follow it over time on one target, do not compare cameras
fwhm width at half height around the peak, background removed [px] width of the spot rises: defocus, turbulence, motion blur. Truncated to the disc: under strong defocus it flattens instead of rising
dominance (second peak − background) / (peak − background), 0 (clean) toward 1 (ambiguous) does a single spot dominate the window? high: a comparable second reflection, a stray bright point, or the target has gone and something else is the brightest: the correlation can latch onto the wrong pattern, which error_value does not see. One hot pixel outside the disc is enough
snr (peak − background) / bg_sigma the signal above the background noise low: marginal return, even when peak looks normal. Independent of exposure and gain. The emulator marks empty below 8

peak and flux give the brightness, sharpness and fwhm the sharpness, dominance and snr the validity: together they say why a measurement degrades, where error_value only says that it does. They describe the image of one target, not the health of the reference network.

Camera images: IRON and DIFF

  • IRON (images/): the ordinary image of the scene, JPG. The cause of an event is seen on it (condensation, rain, train, snow).
  • DIFF (images_diff/): several captures combined to raise the contrast of the reflectors; the image the displacement algorithms measure, by default. Not a difference between two instants. 16-bit PNG.
  • Acquisition modes (station manual): "Single with infrared light", one image per measurement; "Differential", several combined (Stacking 20 to 40 when light is difficult); option "Work on IRON images", to follow a texture without reflector.
  • Not every station has DIFF images; the pumping keeps about one image per hour. The tools look only at IRON images; inventory.py counts the DIFF ones, which your own scripts may read. What is very close to the lens (a web) may stand out better on DIFF.