Files

75 lines
8.8 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# The data of a station
What the files of a station folder are, and what each column means. How to read them with the tools: `GUIDE.md`. What they mean for a label (causes, reading rules, false friends): `labeling/reading-rules.md`. Maintained by hand; `GUIDE.md` is the tool-builder's.
## The displacements: the "advanced" export
`displacement/<Station>_advanced_<start>_<end>.csv`, the source of the analyses: one row per sensor and per measurement, at the native rate of the station (10 or 15 min), `time_utc` in explicit UTC.
| Column | Meaning |
|---|---|
| `sensor_name` | the sensor (target or reference) |
| `pattern_type` | `global_ref`: a reference of the compensation; `ref`: a *local reference*, treated as a target, outside the compensation, without `reproj_err`; empty: a target |
| `x`, `y` | raw position in the image [px]. Not the centroid of the spot: the offset centroid − (`x`, `y`) is constant per sensor by day and differs between sensors; it changes at night, when the spot is saturated. Probably the position of the reference pattern |
| `relative_x`, `relative_y` | relative displacement before the camera compensation [px]; not `x − x0` |
| `corr_x`, `corr_y` | the position of the target once the estimated camera movement is removed: **the curves that are labelled**. In px, not mm (below) |
| `corr_x_mm`, `corr_y_mm` | `corr_x`, `corr_y` in mm: × the sensor's `mm_per_px` (`targets.csv`, below), computed by `data.load`; empty without a sure factor. The same curves in another unit: the noise threshold of a label is read in noises, whatever the unit; mm say how far a target moves, for the reader |
| `error_value` | recognition error of the pattern, not an uncertainty |
| `error_threshold` | specific to each sensor (0.3 to 0.7). Above it, the measurement is rejected: the row and its `error_value` stay, `corr_x` and `corr_y` are empty, not interpolated |
| `reproj_err`, `inlier` | references only: residual and contribution of each reference in the compensation |
| `sigma0` | residual of the fit on the references: one value per measurement, repeated on every row |
| `peak`, `flux`, `sharpness`, `fwhm`, `dominance`, `snr` | image quality of each sensor, recent cameras only (below) |
- "Rejected" is a judgment of the AIMsight platform, not of the camera: the camera delivers all its measurements, and a client's platform that integrates the data receives them all.
- **Unit of `corr_x`, `corr_y`**: px. The hourly export's `horizontal_displacement` (mm) is a factor × the hourly mean of `corr_x` + a constant (likewise `vertical` and `corr_y`, correlation ≥ 0.99); the factor (mm per px) depends on the pattern and on the station.
- **LENZSPITZE**: its export gives `x`, `y` relative to the pattern; the extraction rewrites them in image px and fills `pattern_type`. Sensors 1023 and 8019 have no pattern: their `x`, `y` are empty. The tools' crosses may sit slightly off the spots there.
- `inventory.py` says, on a batch, which columns carry values.
## The targets
`targets.csv`, from the pumping: one line per sensor measured (targets and references), `sensor_name`, `pattern_type`, `distance_mm` (from the camera) and `mm_per_px` (`distance_convertion_factor`, the factor of the dashboard's exports in mm), their current values. Read by `data.load_targets`, shown by `inventory.py` (sensor table).
- The references are in it too (`pattern_type` `global_ref`…), with their distance when it is filled in; they are assumed fixed.
- `distance_mm` 1000: not filled in on the platform; the line is ignored.
- A name may have several lines (a target counted twice, an older configuration of the unit, another lens): when they disagree, `data.load_targets` keeps no value (`inventory.py`: `?(n)`) and that target has no mm; the distance stays when the lines agree on it.
- The distance says which targets are near and far. A far target moves few px for many mm: targets are compared with each other in mm, or in px with their distance. A camera that turns shifts every target by about the same px (the mm grow with the distance); a movement of the structure of the same mm gives px that shrink with the distance; a parallax shows between targets at different distances.
## The camera pose
`camera/<Station>_camera_<YYYYMMDD>.csv`, from the pumping only: the pose at each registration, `time_utc` in UTC, `median_|min_|max_` of `pan`, `tilt`, `rotation` [deg], `scale`, `sigma0`, one row per measurement (min = max). A measurement instant without a row: the camera was not registered.
- Orders of magnitude: `pan`, `tilt` of a few hundredths of a degree, `scale` within 10⁻³ of 1; some stations show several degrees, not explained.
- **A positive `tilt` means the camera is nosing down** (the expert).
- Read by `overview.py` and `inventory.py`.
## The weather
`weather/<Station>_weather_<YYYYMMDD>.csv`, from the pumping, for a station with a Swiss position only: the hourly measurements of the nearest MeteoSwiss station, `time_utc` in UTC (start of the hour); `position`, `place`, `station`, `station_name`, `distance_km`, `height_m` say where; then `precipitation_mm`, `temperature_c`, `humidity_pct`, `wind_speed_kmh`, `wind_gust_kmh`, `wind_direction_deg`, `pressure_hpa`, `sunshine_min`, `global_radiation_wm2`, `snow_depth_cm` (empty: not measured there). The weather a few km away, not at the camera: what the images show prevails. Read by `data.load_weather`.
## The hourly export
`<Station>_<start>_to_<end>_1hour.csv`: one row per hour, targets only; per target `horizontal_displacement`, `vertical_displacement`, their `_min` / `_max`, `_error` (mean `error_value` of the hour), `_count_val` (measurements of the hour, rejected ones included).
- **`timestamp` has no time zone: Swiss local time** (UTC+2 in summer), at the start of the hour.
## Image quality of a target
Six columns of the advanced export, computed by the GPU of recent cameras on the sensor's window in the DIFF image, in units of that image. Filled only where `inventory.py` says so ("Export image quality"); for older cameras, `~/optical_emulator` can recompute them from the DIFF images. "Disc": the neighbourhood of the peak, radius `min(H, W)/6` (at least 3 px); "background": the rest of the window.
| Metric | Definition | What it measures | Reading |
|---|---|---|---|
| `peak` | `max(I)` | the brightness of the return | high, saturation; low, underexposure or missed flash. All targets dropping: the lighting; a single one: that target (fouling, misalignment) |
| `flux` | flux of the disc minus the background | the light of the target, background removed | reflectivity of the target (fouling, ageing, flash), not ambient light; drops well before `error_value` exceeds its threshold |
| `sharpness` | `Σ(I²) / (ΣI)²` | concentration of the energy (higher = sharper) | drops: drifting focus, turbulence, degrading target. Independent of the gain, not of the background: follow it over time on one target, do not compare cameras |
| `fwhm` | width at half height around the peak, background removed [px] | width of the spot | rises: defocus, turbulence, motion blur. Truncated to the disc: under strong defocus it flattens instead of rising |
| `dominance` | `(second peak − background) / (peak − background)`, 0 (clean) toward 1 (ambiguous) | does a single spot dominate the window? | high: a comparable second reflection, a stray bright point, or the target has gone and something else is the brightest: the correlation can latch onto the wrong pattern, which `error_value` does not see. One hot pixel outside the disc is enough |
| `snr` | `(peak − background) / bg_sigma` | the signal above the background noise | low: marginal return, even when `peak` looks normal. Independent of exposure and gain. The emulator marks `empty` below 8 |
`peak` and `flux` give the brightness, `sharpness` and `fwhm` the sharpness, `dominance` and `snr` the validity: together they say **why** a measurement degrades, where `error_value` only says that it does. They describe the image of **one target**, not the health of the reference network.
## Camera images: IRON and DIFF
- **IRON** (`images/`): the ordinary image of the scene, JPG. The cause of an event is seen on it (condensation, rain, train, snow).
- **DIFF** (`images_diff/`): several captures combined to raise the contrast of the reflectors; **the image the displacement algorithms measure**, by default. Not a difference between two instants. 16-bit PNG.
- Acquisition modes (station manual): "Single with infrared light", one image per measurement; "Differential", several combined (`Stacking` 20 to 40 when light is difficult); option "Work on IRON images", to follow a texture without reflector.
- Not every station has DIFF images; the pumping keeps about one image per hour. The tools look only at IRON images; `inventory.py` counts the DIFF ones, which your own scripts may read. What is very close to the lens (a web) may stand out better on DIFF.