8.8 KiB
The data of a station
What the files of a station folder are, and what each column means. How to read them with the tools: GUIDE.md. What they mean for a label (causes, reading rules, false friends): labeling/reading-rules.md. Maintained by hand; GUIDE.md is the tool-builder's.
The displacements: the "advanced" export
displacement/<Station>_advanced_<start>_<end>.csv, the source of the analyses: one row per sensor and per measurement, at the native rate of the station (10 or 15 min), time_utc in explicit UTC.
| Column | Meaning |
|---|---|
sensor_name |
the sensor (target or reference) |
pattern_type |
global_ref: a reference of the compensation; ref: a local reference, treated as a target, outside the compensation, without reproj_err; empty: a target |
x, y |
raw position in the image [px]. Not the centroid of the spot: the offset centroid − (x, y) is constant per sensor by day and differs between sensors; it changes at night, when the spot is saturated. Probably the position of the reference pattern |
relative_x, relative_y |
relative displacement before the camera compensation [px]; not x − x0 |
corr_x, corr_y |
the position of the target once the estimated camera movement is removed: the curves that are labelled. In px, not mm (below) |
corr_x_mm, corr_y_mm |
corr_x, corr_y in mm: × the sensor's mm_per_px (targets.csv, below), computed by data.load; empty without a sure factor. The same curves in another unit: the noise threshold of a label is read in noises, whatever the unit; mm say how far a target moves, for the reader |
error_value |
recognition error of the pattern, not an uncertainty |
error_threshold |
specific to each sensor (0.3 to 0.7). Above it, the measurement is rejected: the row and its error_value stay, corr_x and corr_y are empty, not interpolated |
reproj_err, inlier |
references only: residual and contribution of each reference in the compensation |
sigma0 |
residual of the fit on the references: one value per measurement, repeated on every row |
peak, flux, sharpness, fwhm, dominance, snr |
image quality of each sensor, recent cameras only (below) |
- "Rejected" is a judgment of the AIMsight platform, not of the camera: the camera delivers all its measurements, and a client's platform that integrates the data receives them all.
- Unit of
corr_x,corr_y: px. The hourly export'shorizontal_displacement(mm) is a factor × the hourly mean ofcorr_x+ a constant (likewiseverticalandcorr_y, correlation ≥ 0.99); the factor (mm per px) depends on the pattern and on the station. - LENZSPITZE: its export gives
x,yrelative to the pattern; the extraction rewrites them in image px and fillspattern_type. Sensors 1023 and 8019 have no pattern: theirx,yare empty. The tools' crosses may sit slightly off the spots there. inventory.pysays, on a batch, which columns carry values.
The targets
targets.csv, from the pumping: one line per sensor measured (targets and references), sensor_name, pattern_type, distance_mm (from the camera) and mm_per_px (distance_convertion_factor, the factor of the dashboard's exports in mm), their current values. Read by data.load_targets, shown by inventory.py (sensor table).
- The references are in it too (
pattern_typeglobal_ref…), with their distance when it is filled in; they are assumed fixed. distance_mm1000: not filled in on the platform; the line is ignored.- A name may have several lines (a target counted twice, an older configuration of the unit, another lens): when they disagree,
data.load_targetskeeps no value (inventory.py:?(n)) and that target has no mm; the distance stays when the lines agree on it. - The distance says which targets are near and far. A far target moves few px for many mm: targets are compared with each other in mm, or in px with their distance. A camera that turns shifts every target by about the same px (the mm grow with the distance); a movement of the structure of the same mm gives px that shrink with the distance; a parallax shows between targets at different distances.
The camera pose
camera/<Station>_camera_<YYYYMMDD>.csv, from the pumping only: the pose at each registration, time_utc in UTC, median_|min_|max_ of pan, tilt, rotation [deg], scale, sigma0, one row per measurement (min = max). A measurement instant without a row: the camera was not registered.
- Orders of magnitude:
pan,tiltof a few hundredths of a degree,scalewithin 10⁻³ of 1; some stations show several degrees, not explained. - A positive
tiltmeans the camera is nosing down (the expert). - Read by
overview.pyandinventory.py.
The weather
weather/<Station>_weather_<YYYYMMDD>.csv, from the pumping, for a station with a Swiss position only: the hourly measurements of the nearest MeteoSwiss station, time_utc in UTC (start of the hour); position, place, station, station_name, distance_km, height_m say where; then precipitation_mm, temperature_c, humidity_pct, wind_speed_kmh, wind_gust_kmh, wind_direction_deg, pressure_hpa, sunshine_min, global_radiation_wm2, snow_depth_cm (empty: not measured there). The weather a few km away, not at the camera: what the images show prevails. Read by data.load_weather.
The hourly export
<Station>_<start>_to_<end>_1hour.csv: one row per hour, targets only; per target horizontal_displacement, vertical_displacement, their _min / _max, _error (mean error_value of the hour), _count_val (measurements of the hour, rejected ones included).
timestamphas no time zone: Swiss local time (UTC+2 in summer), at the start of the hour.
Image quality of a target
Six columns of the advanced export, computed by the GPU of recent cameras on the sensor's window in the DIFF image, in units of that image. Filled only where inventory.py says so ("Export image quality"); for older cameras, ~/optical_emulator can recompute them from the DIFF images. "Disc": the neighbourhood of the peak, radius min(H, W)/6 (at least 3 px); "background": the rest of the window.
| Metric | Definition | What it measures | Reading |
|---|---|---|---|
peak |
max(I) |
the brightness of the return | high, saturation; low, underexposure or missed flash. All targets dropping: the lighting; a single one: that target (fouling, misalignment) |
flux |
flux of the disc minus the background | the light of the target, background removed | reflectivity of the target (fouling, ageing, flash), not ambient light; drops well before error_value exceeds its threshold |
sharpness |
Σ(I²) / (ΣI)² |
concentration of the energy (higher = sharper) | drops: drifting focus, turbulence, degrading target. Independent of the gain, not of the background: follow it over time on one target, do not compare cameras |
fwhm |
width at half height around the peak, background removed [px] | width of the spot | rises: defocus, turbulence, motion blur. Truncated to the disc: under strong defocus it flattens instead of rising |
dominance |
(second peak − background) / (peak − background), 0 (clean) toward 1 (ambiguous) |
does a single spot dominate the window? | high: a comparable second reflection, a stray bright point, or the target has gone and something else is the brightest: the correlation can latch onto the wrong pattern, which error_value does not see. One hot pixel outside the disc is enough |
snr |
(peak − background) / bg_sigma |
the signal above the background noise | low: marginal return, even when peak looks normal. Independent of exposure and gain. The emulator marks empty below 8 |
peak and flux give the brightness, sharpness and fwhm the sharpness, dominance and snr the validity: together they say why a measurement degrades, where error_value only says that it does. They describe the image of one target, not the health of the reference network.
Camera images: IRON and DIFF
- IRON (
images/): the ordinary image of the scene, JPG. The cause of an event is seen on it (condensation, rain, train, snow). - DIFF (
images_diff/): several captures combined to raise the contrast of the reflectors; the image the displacement algorithms measure, by default. Not a difference between two instants. 16-bit PNG. - Acquisition modes (station manual): "Single with infrared light", one image per measurement; "Differential", several combined (
Stacking20 to 40 when light is difficult); option "Work on IRON images", to follow a texture without reflector. - Not every station has DIFF images; the pumping keeps about one image per hour. The tools look only at IRON images;
inventory.pycounts the DIFF ones, which your own scripts may read. What is very close to the lens (a web) may stand out better on DIFF.