Technical Summary: Hide-and-Seek in Trajectories

Problem Statement

Vision-Language-Action (VLA) models have demonstrated significant potential in enabling robots to follow natural language instructions and generalize across diverse tasks. However, their deployment in real-world environments is hindered by susceptibility to execution failures, which can lead to catastrophic outcomes (e.g., dropping objects). Detecting these failures during execution is critical for robustness.

Current failure detection methods face two primary limitations:

1. Supervision Cost: Obtaining step-level failure annotations (identifying the exact timestamp of an error) is expensive and difficult to scale, particularly for long-horizon, stochastic trajectories. Existing approaches that rely on trajectory-level labels often propagate these labels uniformly to every timestep. This "uniform propagation" mislabels normal actions preceding the failure as failures, introducing substantial label noise and obscuring localized failure signals.
2. Computational Practicality: Methods relying on action resampling or external Vision-Language Model (VLM) judges incur prohibitive inference overhead, preventing real-time deployment. Furthermore, many existing detectors are designed for specialist policies on fixed tasks, limiting their applicability to generalist VLAs in open-world settings.

The core challenge addressed is Coarsely Supervised Failure Detection: learning a fine-grained, per-step failure detector using only trajectory-level outcome labels (success/failure) without any step-level annotations.

Methodology: Hide-and-Seek

The authors propose Hide-and-Seek, a lightweight runtime failure detection framework that treats the problem as a coarsely supervised learning task. The method is named for the metaphor that failure signals are "hidden" among largely normal behavior within a trajectory, and the detector must "seek" them out using only the trajectory-level outcome as a clue.

Core Framework

The framework learns a sequential detector $f_\phi$ that maps a trajectory prefix $\tau_{\le t}$ (comprising internal action embeddings) to a scalar failure score $s_t \in [0, 1]$. Instead of uniformly assigning the trajectory label, the method employs a contrastive objective at two complementary granularities to localize failure-indicative actions:

1. Inter-trajectory Contrastive Loss ($\mathcal{L}_{inter}$):

   • Goal: Distinguish the most failure-indicative action in a failure trajectory from the most failure-resembling action (hardest false positive) in a successful trajectory.
   • Mechanism: For a pair of failure ($\tau_f$) and success ($\tau_s$) trajectories, the loss enforces that the maximum failure score in $\tau_f$ exceeds the maximum score in $\tau_s$ by a margin $m_r$.
   • Effect: This guides the model to focus on the most salient failure signals without assuming where they occur, adaptively discovering the critical step.

2. Intra-trajectory Contrastive Loss ($\mathcal{L}_{intra}$):

   • Goal: Shape the temporal dynamics within a failure trajectory to separate pre-failure (normal) and post-failure phases.
   • Mechanism: The method defines a proxy failure onset $t_{onset}$ as the point of sharpest score increase ($\arg\max_t (s_t - s_{t-1})$). The loss encourages the average score after $t_{onset}$ to exceed the average score before $t_{onset}$ by a margin $m_o$.
   • Effect: This converts coarse trajectory-level supervision into a temporally structured signal, ensuring scores remain low during normal execution and rise sharply at the failure onset.

The total objective is $\mathcal{L} = \mathcal{L}_{inter} + \lambda \mathcal{L}_{intra}$.

Runtime Monitoring

During inference, the detector produces a score $s_t$ at each timestep. To determine a failure alarm, the system employs Functional Conformal Prediction (CP). A time-varying threshold $\zeta_t$ is constructed from a held-out calibration set of successful trajectories. This guarantees that for any new successful rollout, the trajectory-level false alarm rate is bounded by a user-specified significance level $\alpha$. A failure is declared at the earliest timestep where $s_t \ge \zeta_t$.

Architecture

The detector $f_\phi$ is a lightweight sequence model (default: single-layer LSTM) that processes action embeddings extracted from the VLA's action token or head. To mitigate temporal redundancy, the authors use non-overlapping sliding window aggregation with average pooling.

Key Contributions

1. Novel Framework: Introduction of Hide-and-Seek, which bridges coarsely supervised learning and embodied failure detection, enabling the discovery of failure-indicative actions from trajectory-level supervision alone.
2. Contrastive Objective Design: A dual-loss mechanism (inter- and intra-trajectory) that distinguishes failure signals across and within trajectories, effectively converting coarse supervision into temporally structured signals without step-level annotations.
3. Comprehensive Evaluation: Extensive evaluation on simulation benchmarks (LIBERO-10, VLABench) and a real-world robotic platform (UFactory xArm 6) using three representative VLA policies: OpenVLA (autoregressive), $\pi_0$, and $\pi_0.5$ (flow-matching).

Experimental Results

The method was evaluated against 12 baselines, including OOD detection methods, multi-sampling approaches, and classifier-based methods (e.g., SAFE).

• Performance: Hide-and-Seek achieves state-of-the-art performance across all metrics (Balanced Accuracy, Weighted Accuracy, Time-weighted Accuracy) on both seen and unseen tasks.
  • On LIBERO-10 with OpenVLA, it outperforms the strongest classifier baseline (SAFE-MLP) by +6.8% in bACC on seen tasks and +7.2% on unseen tasks.
  • On VLABench with $\pi_0.5$, it surpasses SAFE-MLP by +8.1% in TWA on seen tasks and +10.8% on unseen tasks.
  • In real-world experiments, it maintains stable performance across seen and unseen tasks, achieving a +11.7% bACC gain over SAFE-MLP on unseen tasks.
• Efficiency: Compared to a VLM-based runtime monitor (Qwen3-VL-8B), Hide-and-Seek is >2,000× faster (0.001s vs. 2.343s per step) while achieving +13.1% higher accuracy.
• Ablation Studies:
  • Uniform Labeling: The proposed method significantly outperforms a baseline that uniformly assigns trajectory labels to all timesteps (+19.4% bACC on OpenVLA), confirming that uniform propagation introduces label noise.
  • Architecture: Recurrent backbones (LSTM, GRU) outperform MLPs and Transformers, suggesting the benefit of modeling temporal context.
  • Loss Components: Both $\mathcal{L}_{inter}$ and $\mathcal{L}_{intra}$ are necessary; $\mathcal{L}_{inter}$ alone performs well, but combining them yields consistent improvements.

Significance and Claims

The paper claims that Hide-and-Seek provides a practical, lightweight solution for real-time failure monitoring in embodied AI. Its significance lies in:

• Eliminating Annotation Costs: It removes the need for expensive, step-level failure annotations, relying solely on the trajectory-level outcome which is naturally available.
• Generalization: It demonstrates strong generalization to unseen tasks and different VLA architectures (autoregressive and flow-matching), addressing a key limitation of prior specialist-focused methods.
• Real-World Viability: By avoiding heavy inference overhead (unlike resampling or VLM judges) and achieving a favorable accuracy-timeliness trade-off via conformal prediction, it enables robust deployment in real-world robotic systems.

The authors position this work as a novel connection between weakly supervised learning paradigms and the specific challenges of embodied agent safety, offering a scalable path toward reliable VLA deployment.