Failure Identification in Imitation Learning Via Statistical and Semantic Filtering

1. Problem Statement

Imitation Learning (IL) policies excel in controlled environments but are brittle in real-world deployments. They fail when encountering Out-of-Distribution (OOD) states caused by hardware faults, defective parts, unexpected human actions, or environmental shifts.

• The Core Challenge: Existing Vision-based Anomaly Detection (AD) methods can detect deviations from the training distribution but cannot distinguish between benign anomalies (e.g., a fly on a table, a background object moving) and genuine task failures (e.g., dropping a tool, spilling coffee).
• The Gap: Treating all anomalies as failures leads to unnecessary interventions and reduced productivity. Conversely, missing genuine failures compromises safety and task completion. Current methods lack a mechanism to semantically interpret these deviations to filter out benign noise.

2. Methodology: FIDeL Framework

The authors propose FIDeL (Failure Identification in Demonstration Learning), a policy-independent, three-stage monitoring module designed to complement IL policies.

A. Offline Phase: Memory Representation & Calibration

1. One-Class Learning: The system is trained only on nominal (successful) expert demonstrations ($X_N$). No failure data is required for training.
2. Feature Aggregation: A vision encoder (ResNet18 or DinoV2) extracts patch-level features from expert trajectories. These are aggregated into a statistical memory buffer ($M$) containing the mean ($\mu$) and standard deviation ($\sigma$) for each spatial patch at every timestep.
3. Conformal Prediction (CP) Calibration: A subset of data is used to compute adaptive thresholds. The authors extend CP to handle spatio-temporal variations, creating dynamic upper bounds for anomaly scores that account for both time steps and spatial patches.

B. Online Phase: Anomaly Detection

1. Optimal Transport (OT) Alignment: During execution, the current observation is encoded. Instead of simple pixel-wise comparison, FIDeL uses Optimal Transport to align the query features with the stored memory features.
   • It calculates the minimum "cost" to redistribute the feature mass of the current observation to match the nominal distribution.
   • This provides robustness to variable execution speeds and minor spatial shifts.
2. Scoring: An anomaly score is derived from the minimum transport cost across nominal timesteps.
3. Thresholding: The score is compared against the spatio-temporal CP thresholds. If the score exceeds the threshold, an anomaly is flagged, and a heatmap is generated to localize the deviation.

C. Semantic Filtering (The Novelty)

To distinguish between benign anomalies and failures:

• Vision-Language Model (VLM): When an anomaly is flagged, a VLM (Qwen 2.5-7b) is invoked.
• Inputs: The VLM receives:
  1. Task description ($d$).
  2. A reference expert frame ($O_{ref}$).
  3. The current observation ($O_{curr}$).
  4. The anomaly heatmap ($M_c^\dagger$) highlighting the suspicious region.
• Decision: The VLM reasons whether the detected deviation constitutes a failure (task-critical) or a benign anomaly (harmless). It outputs a binary decision to either interrupt the policy or allow it to continue.

3. Key Contributions

1. FIDeL Framework: A novel, policy-independent failure detection module that combines representation-based AD with semantic filtering.
2. Spatio-Temporal Conformal Prediction: An extension of CP that creates dynamic thresholds accounting for both temporal progression and spatial patch variations, ensuring robustness to execution speed changes.
3. Semantic Filtering Mechanism: The integration of a VLM with anomaly heatmaps to explicitly discriminate between harmless deviations and genuine failures, significantly reducing false positives.
4. BotFails Dataset: The introduction of a new multimodal dataset containing 10 real-world tasks (6 domestic, 4 industrial) with labeled anomalies and failures, addressing the scarcity of failure-oriented robotics datasets.
5. Interpretability: The system provides localized heatmaps and human-readable VLM explanations for failure detections.

4. Experimental Results

The method was evaluated on the BotFails dataset and a Real-$\pi$ soldering task, comparing against state-of-the-art baselines (FAIL-Detect, AutoEncoders, STAC, etc.).

• Anomaly Detection Performance:
  • FIDeL achieved +5.30% AUROC on BotFails and +17.38% accuracy on failure detection compared to the best existing baselines.
  • It outperformed reconstruction-based (AutoEncoder) and likelihood-based (Normalizing Flows) methods, demonstrating that compact representation matching via Optimal Transport is superior for capturing deviations.
• Thresholding Efficacy:
  • The proposed CP-time&space thresholding outperformed standard Gaussian fitting and temporal-only CP, particularly in tasks with spatially localized anomalies (like soldering).
• Semantic Filtering Impact:
  • Without semantic filtering, the system suffered a sharp drop in True Negative Rate (TNR) due to false positives from benign anomalies.
  • Adding the VLM filter significantly restored TNR while maintaining high True Positive Rate (TPR) for actual failures, achieving a balanced detection profile (e.g., 85.8% TPR / 74.8% TNR on Real-$\pi$).

5. Significance and Limitations

Significance:
FIDeL bridges the gap between generic computer vision anomaly detection and the specific safety requirements of robotics. By filtering out benign anomalies, it enables the deployment of IL policies in unstructured environments without triggering constant, costly interruptions. The use of a VLM for semantic reasoning represents a shift toward more "understanding" monitoring systems.

Limitations:

• Computational Cost: The reliance on a VLM for every flagged anomaly introduces latency and computational overhead.
• Demonstration Diversity: Performance depends heavily on the diversity of the training demonstrations; rare benign variations not seen in training may still be flagged.
• Temporal Sensitivity: The method's invariance to execution speed and order can reduce sensitivity to non-Markovian anomalies (e.g., performing actions in the wrong sequence), as it focuses heavily on the current state's visual deviation rather than long-term temporal logic.

In conclusion, FIDeL offers a robust, interpretable, and high-performing solution for monitoring robotic policies, effectively separating "noise" from "danger" in real-world deployments.