Technical Summary: GroundControl – Anticipating Navigation Failures in Vision-Language Agents via Trajectory-Consistent Uncertainty Estimates

1. Problem Statement

Vision-Language Navigation (VLN) agents have achieved competitive performance on benchmarks by combining language understanding with visual grounding. However, benchmark success metrics (e.g., Success Rate, Success weighted by Path Length or SPL) are evaluated only after an episode terminates. They fail to provide a mechanism for detecting failure dynamics as they unfold.

In practice, navigation failures often emerge gradually through characteristic trajectory-level breakdowns, such as:

• Oscillation: Repeatedly moving back and forth between conflicting actions.
• Stagnation: Stalling near distractors without geometric progress.
• Inefficient Detours: Accumulating path length without proportional distance-to-goal reduction.

Existing uncertainty proxies (e.g., predictive entropy over action distributions, token-level confidence) capture ambiguity in instantaneous decisions but do not reflect whether the resulting trajectory remains consistent with geometric progress toward the goal. An agent may maintain high step-wise confidence while executing a sequence of actions that leads to failure. The paper posits that uncertainty in embodied navigation should instead reflect the trajectory-level consistency of goal-directed dynamics.

2. Methodology

A. GroundControl: Trajectory-Consistent Uncertainty Estimator

GroundControl is an interpretable estimator that models the evolution of the distance-to-goal signal ($d_t$) as a stochastic dynamical process. It assumes that successful navigation exhibits locally consistent geometric progress, whereas failure modes produce statistically detectable inconsistencies.

1. Constant-Velocity Kalman Filter (KF):
The core of GroundControl models the distance-to-goal as a dynamical signal using a state vector $x_t = [d_t, \dot{d}_t]^\top$, where $\dot{d}_t$ is the instantaneous rate of change.

• Dynamics: $x_{t+1} = Fx_t + w_t$ (Linear-Gaussian dynamics with constant velocity assumption).
• Observation: $d_t = Hx_t + v_t$.
• Innovation: The filter computes the innovation $\nu_t = d_t - H\hat{x}_{t|t-1}$, representing the discrepancy between observed and predicted distance.
• Statistical Test: The normalized innovation squared ($\nu_t^2 / S_t$) follows a $\chi^2$ distribution under nominal conditions. Persistently large values indicate significant deviation from expected goal-directed motion.

2. Per-Step Uncertainty Score ($H_t$):
GroundControl aggregates three complementary indicators of trajectory inconsistency into a per-step score:
$$H_t \triangleq \lambda_1 \log \det(\Sigma_{t|t} + \epsilon I) + \lambda_2 H(\alpha_t) + \lambda_3 \frac{\nu_t^2}{S_t}$$

• Posterior Belief Dispersion: $\log \det(\Sigma_{t|t})$ measures uncertainty in the filter's state estimate.
• Action-Level Instability: $H(\alpha_t)$ is the Bernoulli entropy of the policy's confidence in the executed action.
• Statistical Deviation: The normalized innovation squared term.

3. Episode-Level Uncertainty Score ($U_i$):
To obtain a single score for an episode, GroundControl fuses the peak dynamical inconsistency with four global trajectory features:

• Peak Dynamical Inconsistency: $\tilde{s}_{skf}^i = \sigma(\max_t H_t)$, capturing brief but significant deviations.
• Normalized Progress ($f_{prog}$): Measures insufficient net reduction in distance-to-goal.
• Monotonicity ($f_{mono}$): Penalizes frequent increases in distance (non-monotonic evolution).
• Path Efficiency ($f_{eff}$): Detects inefficient detours relative to cumulative motion.
• Oscillatory Behavior ($f_{osc}$): Captures repeated action reversals.

The final score is a fixed linear fusion:
$$U_i \triangleq w_{skf}\tilde{s}_{skf}^i + w_{prog}f_{prog}^i + w_{mono}f_{mono}^i + w_{eff}f_{eff}^i + w_{osc}f_{osc}^i$$
Lower values of $U_i$ indicate higher confidence (consistent trajectory).

B. Evaluation Protocol: Selective Risk–Coverage Navigation (SRCN)

To evaluate uncertainty quality independently of raw task success, the authors formalize Selective Risk–Coverage Navigation (SRCN). This protocol measures how effectively an uncertainty score ranks episodes by failure or inefficiency.

• Mechanism: Episodes are sorted by ascending uncertainty ($U_i$). As a threshold $\tau$ increases, high-uncertainty episodes are progressively excluded (abstained), and performance is evaluated on the retained subset.
• Metrics:
  • Selective Risk: The empirical failure rate (or inefficiency rate) among retained episodes.
  • Risk–Coverage Curve: Plots risk against the fraction of retained episodes (coverage).
  • AURC (Area Under the Risk-Coverage Curve): Summarizes the trade-off; lower is better.
  • E-AURC (Excess AURC): Normalized against the oracle lower bound (sorting by true loss). $E\text{-}AURC = 0$ implies perfect ordering.
• Loss Definitions:
  • Success-based: $\ell_i = 1 - \text{Success}_i$.
  • SPL-based: $\ell_i = 1 - \text{SPL}_i$ (penalizes inefficient successful trajectories).

3. Key Contributions

1. Formalization of Trajectory Consistency: The paper establishes trajectory-level consistency of distance-to-goal dynamics as a foundation for uncertainty estimation in VLN, moving beyond instantaneous action entropy.
2. GroundControl Framework: Introduction of a lightweight, interpretable estimator that detects statistically significant deviations from nominal goal-directed motion using a Kalman filter and trajectory descriptors.
3. SRCN Protocol: A new evaluation framework that isolates ranking quality without modifying the underlying navigation policy, enabling fair comparison across heterogeneous uncertainty estimators.
4. Empirical Validation: Demonstration that modeling deviation from goal-directed dynamics provides a robust signal for anticipating failures, outperforming entropy, conformal, and heuristic baselines.

4. Experimental Results

The method was evaluated on five EB-Navigation splits (Base, Common Sense, Complex Instructions, Visual Appearance, Long Horizon) with $N=300$ total episodes across three LLM backbones (GPT-4o, GPT-5-mini, Gemini-1.5-Flash).

• Success-Based Ranking (SRCN):

  • GroundControl achieved near-oracle ordering under success-based selective risk.
  • Weighted-average E-AURC$_{SR}$ = 0.0024 for the GPT-4o model, substantially outperforming entropy, conformal, and heuristic baselines.
  • The advantage was most pronounced in Long Horizon tasks (where success rates dropped to ~16.7%), where entropy and conformal signals approached random ordering, while GroundControl remained close to oracle. This suggests geometric progress dynamics are the dominant signal for detecting failure in deep temporal horizons.

• SPL-Based Ranking:

  • GroundControl consistently achieved the lowest AURC and E-AURC across models and splits when evaluating navigation efficiency (inefficiency).
  • It successfully identified trajectories leading to inefficient paths (e.g., wandering, oscillation) before terminal failure.

• Diagnostic Analysis:

  • Distributional Separation: GroundControl produced a clear upward shift in uncertainty scores for failing trajectories with limited overlap with successful ones, whereas baselines showed substantial overlap.
  • Risk-Coverage Curves: The method closely tracked the oracle boundary, maintaining low failure rates even at high coverage levels.
  • Correlation: Strong negative correlation was observed between uncertainty scores and episode efficiency (SPL), indicating the estimator captures degrading execution quality.

5. Significance and Claims

The paper claims that trajectory-consistent uncertainty is a superior signal for anticipating navigation failures compared to instantaneous decision ambiguity. By modeling the deviation from expected goal-directed dynamics (using a constant-velocity Kalman filter), GroundControl provides an interpretable and robust signal that:

• Detects failure modes (oscillation, stagnation, divergence) as they unfold.
• Enables runtime introspection, potentially allowing for active recovery or halting before catastrophic failure.
• Generalizes across different VLM backbones and navigation splits, including challenging long-horizon scenarios where compounding errors occur.

The authors emphasize that this approach isolates the quality of the uncertainty ranking from the underlying policy, proving that geometric observables (distance-to-goal evolution) provide a model-agnostic foundation for reliable deployment of embodied agents.