# AIMsight reference quality, planning assessor and event explainer ## Purpose This note merges three related development tracks: 1. **Planning / installation assessor** — predict precision and robustness from planned or actual camera, target and reference geometry. 2. **Data explainer / event diagnostics** — compute deterministic evidence for events and expose it through structured JSON for dashboards and LLM explanation. 3. **Improved automatic reference exclusion and quality alarms** — move beyond today's target-by-target Error Value rejection plus global `sigma0` threshold. These tracks should share one geometry/diagnostic layer. They are also deliberately **parallel to the replacement/improvement of the current RANSAC similarity estimator**. The quality architecture must remain valid if RANSAC's binary inlier/outlier decision is replaced by a Danish-like robust weighted similarity fit. --- # 1. Current state and main limitation Current protection is roughly: - target/pattern matching quality -> `Error Value` threshold; - RANSAC similarity fit -> binary reference inlier/outlier; - global fit quality -> `sigma0` threshold. This is useful but insufficient for: - weak reference geometry with a small residual; - high-leverage references; - repeated switching between two plausible reference solutions; - coherent motion of a subgroup of references; - depth/parallax effects after camera translation; - a transform that is mathematically valid but poorly constrained at a particular monitored point; - a solution nominally using many references but effectively dominated by one or two. The Barcelona incident is a useful regression case: the monitored targets could remain individually measurable while the reference compensation switched between competing valid-looking states. The failure mode was therefore primarily **reference-network / compensation-state instability**, not target tracking. A long-term architecture should separate: ```text point/image measurement | +--> target image-quality diagnostics | +--> robust reference estimator | | | +--> per-reference residual / contribution / influence | +--> transform + uncertainty | +--> reference-network quality | +--> model adequacy / camera-state diagnostics | +--> target-specific corrected-measurement quality | +--> deterministic alarm policy +--> event detector / explainer ``` Core principle: > A displacement value and confidence in that displacement are separate outputs. --- # 2. Shared diagnostic concepts ## 2.1 Target image quality Can an individual pattern be measured reliably? Examples: - Error Value; - visibility / occlusion; - flux / brightness; - SNR; - sharpness; - saturation; - template consistency. This is local to one target or reference image patch. ## 2.2 Reference geometric consistency Does a reference agree with the camera-motion transform inferred from the reference network? Examples: - reprojection residual; - normalized residual; - robust estimator weight; - leverage; - influence; - leave-one-reference-out effect. This must remain distinct from image-match Error Value. ## 2.3 Reference-network geometry Does the current constellation constrain the transform adequately over the monitored area? Examples: - image-space horizontal/vertical span; - convex-hull coverage; - covariance eigenvalues; - rotation/scale lever arm; - depth distribution; - effective reference count; - transform covariance / conditioning. ## 2.4 Datum stability Are the physical reference points themselves stable relative to the monitoring datum? The camera can test internal consistency. It cannot prove that a complete coherently moving reference datum is physically fixed. This is the same observability limitation as in conventional surveying. ## 2.5 Model adequacy Does the chosen transformation represent the physical camera motion sufficiently well? A 2D similarity transform models image translation, rotation and scale. A significant 3D camera translation can produce depth-dependent parallax that one global similarity cannot represent exactly. --- # 3. Workstream A — Planning / installation assessor ## 3.1 Objective Given planned or actual camera/reference/target geometry, answer: - Is the geometry compatible with a requested accuracy such as `1 mm at 2 sigma`? - What camera/reference stability is required? - Are references spatially distributed enough? - Are reference depths appropriate relative to monitored targets? - Which references carry high leverage? - How much does one reference influence target results? - Where would an extra reference improve robustness most? - What is the worst target bias that can remain while network diagnostics still look acceptable? The assessor should support: ### Planned mode Approximate camera position/orientation, FOV, point coordinates/distances, reference roles, expected pixel precision, accuracy requirement. ### Commissioned mode Actual image coordinates, `distance_mm`, mm/pixel factor, reference/target roles, baseline residuals and measured short-term noise. --- ## 3.2 Basic precision propagation For target \(i\), with conversion factor \(k_i\) [mm/px] and localization uncertainty \(\sigma_{px,i}\): \[ \sigma_{measurement,i} \approx k_i \sigma_{px,i} \] Corrected displacement must also include uncertainty of the reference transformation: \[ \Sigma_{corrected,i} = \Sigma_{measurement,i} + J_i\Sigma_TJ_i^T \] where: - \(\Sigma_T\): transform parameter covariance; - \(J_i\): sensitivity of target \(i\) to transform parameters. This gives a **target-specific** uncertainty. A single global `sigma0` cannot capture this. --- ## 3.3 Geometry metrics For the current reference contribution/weights, calculate: - horizontal span / image width; - vertical span / image height; - convex hull area / image area; - reference centroid vs target-region centroid; - weighted reference coordinate covariance; - eigenvalues \(\lambda_1 \ge \lambda_2\); - collinearity ratio: \[ G_{collinear}=\frac{\lambda_2}{\lambda_1} \] - lever arm for rotation about target region / image centre; - depth distribution; - transform design-matrix condition number. Count alone is not a geometry criterion. --- ## 3.4 Effective reference count A future weighted estimator requires a generalization of "number of RANSAC inliers": \[ N_{eff}=\frac{(\sum_i w_i)^2}{\sum_i w_i^2} \] This gives: - equal six-reference solution -> \(N_{eff}=6\); - nominal six references but one dominates -> \(N_{eff}\) approaches 1; - RANSAC is a special case with weights 0 or 1. `N_eff` should become a core network-health quantity. --- ## 3.5 Camera translation / parallax sensitivity A simple planning model is useful. Assume: - points lie approximately along one tunnel line; - camera lateral/vertical offset from that line is \(h\); - point longitudinal distance is \(z\). Viewing angle: \[ \alpha(z)=\arctan(h/z) \] Sensitivity to a small transverse camera displacement \(\delta h\): \[ \frac{\partial\alpha}{\partial h} = \frac{z}{z^2+h^2} \] A single global image transform removes the component common to the reference network. The residual at a target depends on the difference between target depth sensitivity and weighted reference sensitivity. In a small-angle pinhole approximation: \[ \Delta x_{residual,t} \approx \delta h \left( 1-z_t\sum_r\frac{\bar w_r}{z_r} \right) \] Define weighted effective reference depth: \[ z_{eff} = \frac{1} {\sum_r \bar w_r/z_r} \] Then approximately: \[ \Delta x_{residual,t} \approx \delta h \left(1-\frac{z_t}{z_{eff}}\right) \] For an allowed residual \(E_{allowed}\): \[ |\delta h|_{max} \approx \frac{E_{allowed}} {|1-z_t/z_{eff}|} \] This is a simple explanatory bound. The actual assessor should also evaluate the real 2D image geometry numerically. --- ## 3.6 Reference-datum movement sensitivity Simulate: 1. one reference moving; 2. each possible reference subgroup moving coherently; 3. all references moving coherently; 4. a smooth deformation field across the reference network; 5. perturbations aligned with weakly constrained transform modes. For each perturbation record: - target displacement bias; - reference residuals; - robust weights / RANSAC states; - `sigma0`; - geometry quality; - whether candidate alarms would detect it. A particularly useful metric is the **worst undetected target bias**: \[ E_{undetected,max} = \max_{\Delta R\in\text{accepted quality region}} \max_t |\Delta d_t| \] For a site claiming 1 mm integrity, the long-term goal should be to demonstrate: \[ E_{undetected,max}<1\text{ mm} \] under agreed perturbation assumptions. Initially this can be evaluated with Monte Carlo / adversarial simulation. --- ## 3.7 Leave-one-reference-out / influence For each reference \(r_i\): \[ T_{-i}=\operatorname{fit}(R\setminus r_i) \] Compare the resulting target corrections with nominal. Store: - maximum target change; - median target change; - transform parameter change; - reference causing maximum sensitivity. Large sensitivity reveals weak geometry or a high-leverage reference even if `sigma0` remains acceptable. With a weighted least-squares / Danish-like estimator, approximate influence diagnostics may later replace full refits for real-time use. --- ## 3.8 Planning assessor output Example schema: ```json { "requirement": {"displacement_2sigma_mm": 1.0}, "precision": {"worst_target_2sigma_mm": 0.18}, "geometry": { "configured_reference_count": 7, "effective_reference_count": 5.8, "horizontal_span_fraction": 0.71, "vertical_span_fraction": 0.46, "collinearity_ratio": 0.23, "condition_number": 12.4 }, "parallax_sensitivity": { "worst_target_mm_per_mm_camera_translation": 0.74, "camera_translation_for_1mm_residual_mm": 1.35 }, "datum_sensitivity": { "coherent_reference_shift_for_1mm_bias_mm": 1.4 }, "robustness": { "max_leave_one_out_effect_mm": 0.16, "highest_influence_reference": "PR_284" } } ``` Numbers above are illustrative schema values, not universal limits. --- # 4. Workstream B — Data explainer / diagnostic event engine The existing event-detection note already establishes the right direction: ```text measurement + diagnostic channels ↓ common deterministic feature layer ↓ candidate events ↓ structured event JSON ↙ ↘ alarm prototype LLM explanation ``` The LLM explains deterministic evidence. It does not decide measurement validity. The important extension is that the explainer should also be the **prototype environment for future integrity alarms**. --- ## 4.1 Shadow-mode development path For every new quality metric: 1. calculate it offline / in the event engine; 2. include raw value + score in the event JSON; 3. replay historical incidents; 4. compare against expert diagnosis; 5. inspect false positives/negatives; 6. formulate transparent deterministic rules; 7. run rules in live shadow mode; 8. collect statistics; 9. only then move mature rules into production validity logic. This lets us improve integrity protection without destabilizing production processing. --- ## 4.2 Event evidence families ### Displacement - amplitude; - step; - rate/trend; - robust local z-score; - persistence; - reversibility. ### Spatial response - fraction of targets affected; - sign agreement; - common-mode score; - pairwise correlation; - spatial pattern. ### Reference network - configured / eligible references; - effective reference count; - per-reference residual; - per-reference robust weight; - leverage / influence; - `sigma0` / robust residual scale; - geometry / conditioning; - transform covariance; - target-specific propagated uncertainty; - weight/reference-state switching; - leave-one-out sensitivity. ### Camera / compensation state - translation; - rotation; - scale; - parameter covariance; - change relative to baseline; - coincidence with target displacement event. ### Target image quality - Error Value; - flux; - SNR; - sharpness; - visibility; - target validity. ### Provenance Explicit epoch states such as: ```text ACQUIRED_VALID ACQUIRED_TARGET_REJECTED ACQUIRED_REFERENCE_SOLUTION_DEGRADED ACQUIRED_REFERENCE_SOLUTION_INVALID NO_IMAGE NO_ACQUISITION REPROCESSED ``` --- ## 4.3 Useful classifications ### Probable monitored-object motion ```text displacement event + stable reference weights/residuals + stable transform + good target quality + persistent response ``` ### Probable camera/reference compensation event ```text common-mode target jump + reference weight/state change + transform jump + high reversibility ``` ### Weak / unstable reference solution ```text low effective reference count OR poor geometry OR high transform uncertainty OR excessive weight concentration OR large LOO influence OR repeated competing reference states ``` ### Possible model mismatch / parallax ```text large camera translation + depth-dependent residual/displacement pattern + good target image quality + similarity residual structure correlated with depth/image position ``` ### Target-local degradation ```text one/few targets affected + poor Error Value / flux / SNR / sharpness + healthy reference solution ``` --- # 5. Workstream C — Better automatic exclusion and quality alarms ## 5.1 Keep three actions separate 1. exclude/downweight an individual observation; 2. declare the reference solution degraded/invalid; 3. generate an operator alarm. A reference can be downweighted without invalidating the solution. A solution can be invalid because of geometry even when no individual reference is bad. An event can deserve an alarm while the measurement remains usable. --- ## 5.2 Stage A — Observation eligibility Before geometric fitting, determine whether each reference/target is measurable from the image. Inputs: - Error Value; - visibility; - flux; - SNR; - sharpness; - saturation. Output: ```json { "eligible": true, "image_quality_weight": 0.92, "reason": null } ``` Hard rejection should be reserved for clearly unmeasurable observations. --- ## 5.3 Stage B — Robust geometric fit The estimator should return per reference: ```json { "residual_px": [0.12, -0.08], "residual_norm_px": 0.144, "normalized_residual": 1.7, "robust_weight": 0.74, "leverage": 0.21, "influence": 0.12 } ``` Current RANSAC adapter: ```text robust_weight = 1 for inlier robust_weight = 0 for outlier ``` Danish-like estimator: ```text 0 <= robust_weight <= 1 ``` Downstream diagnostics should consume the generic representation. --- ## 5.4 Stage C — Network validation Compute: - effective reference count; - median / percentile / maximum residual; - `sigma0` / robust residual scale; - weight concentration; - weighted image geometry; - transform condition number; - transform covariance; - target-specific propagated uncertainty; - influence / LOO; - temporal weight-state stability; - transform plausibility; - model-mismatch indicators. Possible quality states: ```text REFERENCE_SOLUTION_VALID REFERENCE_SOLUTION_DEGRADED REFERENCE_SOLUTION_INVALID MODEL_MISMATCH_SUSPECTED DATUM_STABILITY_UNVERIFIED ``` --- ## 5.5 Stage D — Target-specific corrected quality Every corrected target should combine: - local target measurement uncertainty; - uncertainty caused by reference compensation. Example: ```json { "target": "D-300-A", "corrected_displacement_mm": 0.84, "estimated_sigma_mm": 0.11, "quality_state": "valid", "contributors": { "target_measurement_sigma_mm": 0.04, "reference_transform_sigma_mm": 0.10 } } ``` This is much more informative than one station-level `sigma0`. --- # 6. Automatic exclusion logic A future hierarchy: ```text image quality failure -> hard reject observation geometric inconsistency -> robustly downweight reference very low robust weight + persistence -> effectively excluded reference network remains well constrained -> continue, possibly DEGRADED network uncertainty/geometry unacceptable -> INVALID reference compensation large transform/model-mismatch event -> quality alarm; invalidate only according to project policy ``` Avoid repeatedly forcing a reference across a binary residual threshold near a decision boundary. For weighted estimators use: - robust weight; - hysteresis; - persistence; - leverage/influence; - effect on network uncertainty; rather than one hard cutoff. --- # 7. Metrics needed for weighted estimators ## 7.1 Weight concentration Maximum normalized weight: \[ W_{max}=\max_i\bar w_i \] Effective reference count: \[ N_{eff}=\frac{(\sum_iw_i)^2}{\sum_iw_i^2} \] Weight entropy: \[ H_w=-\sum_i \bar w_i\log\bar w_i \] Low entropy / low effective count means the solution is dominated by a small subset. ## 7.2 Temporal weight-state change With RANSAC we can track Jaccard distance of accepted sets. For a weighted estimator generalize to: \[ S_{weight-change} = \frac{\|w_t-w_{t-1}\|_1} {\max(\|w_t\|_1,\epsilon)} \] Track: - per-reference weight jumps; - repeated alternation between weight configurations; - transform change coincident with weight change; - target common-mode displacement coincident with weight change. Binary RANSAC switching becomes one special case. --- # 8. Transform uncertainty as a first-class metric A weighted fit should expose approximate parameter covariance: \[ \Sigma_T \approx \sigma^2(J^TWJ)^{-1} \] Derive: - translation uncertainty; - rotation uncertainty; - scale uncertainty; - parameter correlations; - target-specific propagated displacement uncertainty. This addresses a major weakness of `sigma0` alone: > A small residual does not imply that the transform is well constrained. --- # 9. Model adequacy / parallax diagnostics A similarity fit may be statistically clean yet physically incomplete after significant 3D camera motion. Candidate diagnostics: - residual correlated with reference depth; - residual correlated with image position/radius; - target displacement correlated with target depth; - scale/rotation change far outside normal baseline; - large raw camera-state change; - significantly better diagnostic fit from a richer model. Initially this should be an **explainer / quality feature**, not a hard automatic rejection. It is closely linked with the parallel estimator work: ```text normal operation -> similarity suspicious event -> evaluate extra degrees of freedom diagnostically without automatically allowing a flexible model to absorb real structural displacement ``` --- # 10. Estimator-independent API Do not make downstream code depend on: ```text reference.inlier = True/False ``` Use a generic result: ```json { "reference_fit": { "model": "similarity", "estimator": "danish_robust_v1", "parameters": { "tx": 0.0, "ty": 0.0, "rotation": 0.0, "scale": 1.0 }, "parameter_covariance": [], "residual_scale": { "sigma0": 0.0, "robust_sigma": 0.0 }, "references": { "PR_291": { "eligible": true, "image_quality_weight": 1.0, "robust_weight": 0.93, "effective_weight": 0.93, "residual_px": [0.0, 0.0], "normalized_residual": 0.0, "leverage": 0.0, "influence": 0.0, "status": "healthy" } }, "network": { "configured_count": 8, "eligible_count": 7, "effective_count": 5.8, "geometry_score": 0.82, "condition_number": 11.3, "max_weight_fraction": 0.24 } } } ``` RANSAC can populate the same schema with binary weights. --- # 11. RANSAC -> Danish-like architecture consequences | RANSAC-centric concept | Generalized concept | |---|---| | inlier count | effective reference count | | inlier/outlier | robust contribution weight | | accepted set | weight distribution | | set switching | weight-state switching | | residual threshold | normalized residual + weight + influence | | binary reference health | continuous reference health | | geometry of inliers | weighted geometry | | leave-one-inlier-out | influence / leave-one-reference-out | | sigma0 | residual scale + covariance + conditioning | A binary `accepted` field can remain for UI/backward compatibility, but it should be a derived field, not the architectural primitive. --- # 12. Common diagnostics across the three tracks | Diagnostic | Planning | Explainer | Runtime quality | |---|---:|---:|---:| | image-space geometry | yes | yes | yes | | depth distribution | yes | yes | yes | | pixel/mm precision | yes | baseline | yes | | effective reference count | simulated | yes | yes | | sigma0 / robust residual scale | simulated | yes | yes | | per-reference residual | simulated | yes | yes | | robust weight | simulated | yes | yes | | weight concentration | simulated | yes | yes | | condition / transform covariance | yes | yes | yes | | target-specific sigma | yes | yes | yes | | weight/reference switching | n/a | yes | yes | | LOO / influence | yes | event/on-demand | periodic/event | | camera-state change | sensitivity | yes | yes | | parallax/model mismatch | sensitivity | yes | event-triggered | | common-mode target response | n/a | yes | confirmation | | persistence/reversibility | n/a | yes | post-event | | target image quality | expected/baseline | yes | yes | --- # 13. Alarm architecture Keep the separation: ```text raw metric ↓ diagnostic feature ↓ quality/event state ↓ project-specific alarm policy ``` Example: ```text PR_284 weight 0.92 -> 0.15 + effective reference count drops + transform jumps + common-mode target jump ↓ reference-compensation instability event ↓ project policy: warning / invalidate / notify / suppress export ``` Do not hard-code client-specific thresholds inside the core estimator. Suggested alarm families: ### Target measurement quality - Error Value; - visibility; - flux/SNR/sharpness; - target tracking failure. ### Reference observation quality - poor image quality; - excessive geometric residual; - persistently low robust weight; - high influence. ### Reference-network quality - low effective reference count; - weak geometry; - poor conditioning; - high transform uncertainty; - excessive `sigma0`; - weight concentration; - unstable weights; - excessive LOO sensitivity. ### Compensation/model quality - implausible transform jump; - large camera-state change; - parallax/model mismatch suspected; - diagnostic richer model explains suspicious event substantially better. ### Datum / installation - known support/reference motion; - commissioned geometry no longer valid; - coherent datum movement cannot be ruled out. ### Acquisition / communications Keep separate from geometry: - missing acquisition; - missing upload; - excessive latency; - data gap. --- # 14. Explainer as alarm prototype This should be a deliberate development workflow. For each candidate alarm: 1. implement diagnostic computation once; 2. expose raw value in event JSON; 3. replay historical data; 4. review events with deterministic evidence + LLM summary; 5. tune rule; 6. run live in shadow mode; 7. measure false positives / missed events; 8. promote mature rule to production. The LLM is useful for investigation and synthesis, but the production rule remains deterministic. --- # 15. Barcelona as a regression dataset Use the incident as an initial acceptance suite. ## Stable pre-event Expected: - stable reference contributions; - low target scatter; - low transform variability; - no compensation-instability alarm. ## Station movement Expected: - strong camera-state change; - possible parallax/model-mismatch indication; - reference diagnostics change. ## Competing reference solution / PR-284 instability Expected: - reference weight/membership change; - common-mode target jump; - transform jump; - high reversibility; - compensation-instability event. ## Stable reprocessed configuration Expected: - stable reference solution; - improved geometry; - no artificial jumps. --- # 16. Implementation phases ## Phase A — Common estimator output Expose at every epoch: - target Error Value / image quality; - per-reference residual; - per-reference robust contribution weight; - effective reference count; - `sigma0` / robust scale; - transform parameters; - transform covariance where available; - weighted geometry; - provenance state. Adapt current RANSAC into this schema first. ## Phase B — Geometry assessor v0 Input: ```text patterns.json assumed sigma_px required_accuracy_mm ``` Output: - target/reference distance ranges; - image distribution; - geometry/conditioning; - predicted target precision; - camera-translation sensitivity; - coherent-reference-shift sensitivity; - LOO sensitivity; - recommended reference improvements. ## Phase C — Extend explainer/event JSON Add: - effective count; - weight concentration; - weighted geometry; - transform uncertainty; - target-specific compensation uncertainty; - weight-state switching; - LOO/influence; - parallax/model-mismatch diagnostics. ## Phase D — Shadow alarms Start with: ```text REFERENCE_EFFECTIVE_COUNT_LOW REFERENCE_GEOMETRY_WEAK REFERENCE_STATE_UNSTABLE REFERENCE_INFLUENCE_HIGH TRANSFORM_UNCERTAINTY_HIGH TRANSFORM_CHANGE_LARGE MODEL_MISMATCH_SUSPECTED ``` ## Phase E — Production validity After validation, support project-configurable: ```text VALID DEGRADED INVALID ``` Possible consequences: - keep value but attach degraded quality; - warn operator; - suppress automatic alarm forwarding; - invalidate corrected displacement; - expose raw/no-compensation fallback; - request reconfiguration. --- # 17. Design principles 1. Do not confuse target image quality with reference-network quality. 2. Do not confuse a small residual with a well-constrained transform. 3. Do not use reference count without geometry and weight information. 4. Do not build new architecture around binary inlier/outlier. 5. Make quality target-specific where possible. 6. Treat coherent datum motion as an observability limitation, not an estimator bug. 7. Separate measurement, diagnostics, event interpretation and alarm policy. 8. Prototype integrity logic in the explainer before making it production-critical. 9. Preserve raw evidence so historical events remain reprocessable. 10. Use deterministic rules for measurement integrity; use the LLM for synthesis. 11. Keep geometry/quality logic estimator-independent. 12. Use Barcelona as a regression test, not a one-off special case. --- # 18. Long-term architecture ```text PLANNING / COMMISSIONING geometry + precision assessor | v expected quality envelope | v images/measurements -> point/image diagnostics | v robust reference estimator (RANSAC now, weighted later) | +--------------+--------------+ | | v v per-reference contribution transform + covariance residual / weight / influence camera-state parameters | | +--------------+--------------+ | v reference-network quality geometry / Neff / condition / LOO / stability | v target-specific measurement quality | +-------------+-------------+ | | v v deterministic alarms event detector | v event JSON | v LLM explanation ``` The central idea is that **planning, explanation and automatic protection are three consumers of the same geometric and diagnostic model**. That architecture should remain valid when the current RANSAC similarity estimator is replaced or extended.