Files
aimsight-tools/DATA.md
T

8.7 KiB
Raw Blame History

The data of a station

What the files of a station folder are, and what each column means. How to read them with the tools: GUIDE.md. What they mean for a label (causes, reading rules, false friends): labeling/reading-rules.md. Maintained by hand; GUIDE.md is the tool-builder's.

The displacements: the "advanced" export

displacement/<Station>_advanced_<start>_<end>.csv, the source of the analyses: one row per sensor and per measurement, at the native rate of the station (10 or 15 min), time_utc in explicit UTC.

Column Meaning
sensor_name the sensor (target or reference)
pattern_type global_ref: a reference of the compensation; ref: a local reference, treated as a target, outside the compensation, without reproj_err; empty: a target
x, y raw position in the image [px]. Not the centroid of the spot: the offset centroid − (x, y) is constant per sensor by day and differs between sensors; it changes at night, when the spot is saturated. Probably the position of the reference pattern
relative_x, relative_y relative displacement before the camera compensation [px]; not x − x0
corr_x, corr_y the position of the target once the estimated camera movement is removed: the curves that are labelled. In px, not mm (below)
corr_x_mm, corr_y_mm corr_x, corr_y in mm: × the sensor's mm_per_px (targets.csv, below), computed by data.load; empty without a sure factor. The same curves in another unit: the noise threshold of a label is read in noises, whatever the unit; mm say how far a target moves, for the reader
error_value recognition error of the pattern, not an uncertainty
error_threshold specific to each sensor (0.3 to 0.7). Above it, the measurement is rejected: the row and its error_value stay, corr_x and corr_y are empty, not interpolated
reproj_err, inlier references only: residual and contribution of each reference in the compensation
sigma0 residual of the fit on the references: one value per measurement, repeated on every row
peak, flux, sharpness, fwhm, dominance, snr image quality of each sensor, recent cameras only (below)
  • "Rejected" is a judgment of the AIMsight platform, not of the camera: the camera delivers all its measurements, and a client's platform that integrates the data receives them all.
  • Unit of corr_x, corr_y: px. The hourly export's horizontal_displacement (mm) is a factor × the hourly mean of corr_x + a constant (likewise vertical and corr_y, correlation ≥ 0.99); the factor (mm per px) depends on the pattern and on the station.
  • LENZSPITZE: its export gives x, y relative to the pattern; the extraction rewrites them in image px and fills pattern_type. Sensors 1023 and 8019 have no pattern: their x, y are empty. The tools' crosses may sit slightly off the spots there.
  • inventory.py says, on a batch, which columns carry values.

The targets

targets.csv, from the pumping: one line per sensor of the platform, sensor_name, pattern_type, distance_mm (from the camera) and mm_per_px (distance_convertion_factor, the factor of the dashboard's exports in mm), their current values. Read by data.load_targets, shown by inventory.py (sensor table).

  • The references are not in it yet (pumped from the next pumping on): no distance, no mm for them. They are assumed fixed.
  • distance_mm 1000: not filled in on the platform; the line is ignored.
  • A name may have several lines (a target counted twice, an older configuration of the unit, another lens): when they disagree, data.load_targets keeps no value (inventory.py: ?(n)) and that target has no mm; the distance stays when the lines agree on it.
  • The distance says which targets are near and far. A far target moves few px for many mm: targets are compared with each other in mm, or in px with their distance. A camera that turns shifts every target by about the same px (the mm grow with the distance); a movement of the structure of the same mm gives px that shrink with the distance; a parallax shows between targets at different distances.

The camera pose

camera/<Station>_camera_<YYYYMMDD>.csv, from the pumping only: the pose at each registration, time_utc in UTC, median_|min_|max_ of pan, tilt, rotation [deg], scale, sigma0, one row per measurement (min = max). A measurement instant without a row: the camera was not registered.

  • Orders of magnitude: pan, tilt of a few hundredths of a degree, scale within 10⁻³ of 1; some stations show several degrees, not explained.
  • A positive tilt means the camera is nosing down (the expert).
  • Read by overview.py and inventory.py.

The weather

weather/<Station>_weather_<YYYYMMDD>.csv, from the pumping, for a station with a Swiss position only: the hourly measurements of the nearest MeteoSwiss station, time_utc in UTC (start of the hour); position, place, station, station_name, distance_km, height_m say where; then precipitation_mm, temperature_c, humidity_pct, wind_speed_kmh, wind_gust_kmh, wind_direction_deg, pressure_hpa, sunshine_min, global_radiation_wm2, snow_depth_cm (empty: not measured there). The weather a few km away, not at the camera: what the images show prevails. Read by data.load_weather.

The hourly export

<Station>_<start>_to_<end>_1hour.csv: one row per hour, targets only; per target horizontal_displacement, vertical_displacement, their _min / _max, _error (mean error_value of the hour), _count_val (measurements of the hour, rejected ones included).

  • timestamp has no time zone: Swiss local time (UTC+2 in summer), at the start of the hour.

Image quality of a target

Six columns of the advanced export, computed by the GPU of recent cameras on the sensor's window in the DIFF image, in units of that image. Filled only where inventory.py says so ("Export image quality"); for older cameras, ~/optical_emulator can recompute them from the DIFF images. "Disc": the neighbourhood of the peak, radius min(H, W)/6 (at least 3 px); "background": the rest of the window.

Metric Definition What it measures Reading
peak max(I) the brightness of the return high, saturation; low, underexposure or missed flash. All targets dropping: the lighting; a single one: that target (fouling, misalignment)
flux flux of the disc minus the background the light of the target, background removed reflectivity of the target (fouling, ageing, flash), not ambient light; drops well before error_value exceeds its threshold
sharpness Σ(I²) / (ΣI)² concentration of the energy (higher = sharper) drops: drifting focus, turbulence, degrading target. Independent of the gain, not of the background: follow it over time on one target, do not compare cameras
fwhm width at half height around the peak, background removed [px] width of the spot rises: defocus, turbulence, motion blur. Truncated to the disc: under strong defocus it flattens instead of rising
dominance (second peak − background) / (peak − background), 0 (clean) toward 1 (ambiguous) does a single spot dominate the window? high: a comparable second reflection, a stray bright point, or the target has gone and something else is the brightest: the correlation can latch onto the wrong pattern, which error_value does not see. One hot pixel outside the disc is enough
snr (peak − background) / bg_sigma the signal above the background noise low: marginal return, even when peak looks normal. Independent of exposure and gain. The emulator marks empty below 8

peak and flux give the brightness, sharpness and fwhm the sharpness, dominance and snr the validity: together they say why a measurement degrades, where error_value only says that it does. They describe the image of one target, not the health of the reference network.

Camera images: IRON and DIFF

  • IRON (images/): the ordinary image of the scene, JPG. The cause of an event is seen on it (condensation, rain, train, snow).
  • DIFF (images_diff/): several captures combined to raise the contrast of the reflectors; the image the displacement algorithms measure, by default. Not a difference between two instants. 16-bit PNG.
  • Acquisition modes (station manual): "Single with infrared light", one image per measurement; "Differential", several combined (Stacking 20 to 40 when light is difficult); option "Work on IRON images", to follow a texture without reflector.
  • Not every station has DIFF images; the pumping keeps about one image per hour. The tools look only at IRON images; inventory.py counts the DIFF ones, which your own scripts may read. What is very close to the lens (a web) may stand out better on DIFF.