Technical Summary: FailSafe – Reasoning and Recovery from Failures in Vision-Language-Action Models

Problem Statement

Recent advancements in Vision-Language-Action (VLA) models have enabled robots to execute complex manipulation tasks by integrating low-level control into Vision-Language Models (VLMs). However, these models rely heavily on large-scale datasets composed of clean, ground-truth trajectories, which rarely include failure scenarios. Consequently, while VLAs perform well in distribution, they struggle to detect or recover from unpredictable, abrupt failures encountered during real-world execution.

Existing approaches to failure reasoning face significant limitations:

• Lack of Executable Actions: Many datasets provide only textual explanations of failures (e.g., "the gripper should move left"), which are ambiguous regarding magnitude and endpoints, making them difficult for VLA models to execute directly.
• Reliance on Human Supervision: Methods like OLAF and YAY require external human observers to monitor operations and intervene, which is impractical for fully autonomous systems.
• Inability to Scale: While some pipelines generate failure trajectories, they often fail to produce corresponding corrective actions that are both valid and executable by robots.

The core challenge addressed by this work is the absence of a scalable pipeline that can automatically generate high-level failure reasoning paired with low-level, executable recovery actions to enhance VLA robustness.

Methodology: The FailSafe Pipeline

The authors propose FailSafe, a novel framework designed to automatically generate diverse failure cases paired with executable recovery actions within simulators that support motion planning (demonstrated in ManiSkill). The pipeline consists of five key stages:

1. Failure Generation

FailSafe defines three fundamental failure modes to model common robotic errors:

• Translation Failure: Perturbations along Cartesian axes ($x, y, z$).
• Rotation Failure: Angular deviations in roll, pitch, or yaw.
• No-ops Failure: The robotic arm becoming stuck for a period without movement.

These failures are injected at arbitrary steps of ground-truth rollouts. The system uses a YAML configuration to specify noise ranges (e.g., $\pm0.1$ for translation, $\pm1$ radian for rotation) and the stages where failures occur. This mimics real-world deployment where errors can happen unexpectedly.

2. Action Collection

Unlike previous methods that output text, FailSafe collects 7-DoF end-effector recovery actions ($\Delta A$).

• Mapping: For a deviated pose ($P_d$) in a failure trajectory, the system maps it to a corresponding corrective pose ($P_c$) in the correct trajectory.
• Constraints: To prevent collisions, $P_c$ is selected from a window starting 10 steps after the failure begins and ending 3 steps before the task concludes.
• Diversity: The system generates multiple candidate $(P_d, P_c)$ pairs. The resulting corrective action $\Delta A$ is the 7-DoF difference between these poses. Crucially, even if a specific failure type (e.g., Translation X) is injected, the resulting $\Delta A$ is not restricted to a single dimension; it provides adjustments across all seven dimensions to handle simultaneous or complex failure modes.

3. Systematic Verification

Before inclusion in the dataset, every candidate recovery action undergoes rigorous verification:

• The system replays the trajectory: $A \to P_d \to P_c \to \text{Rest of Task}$.
• If the task, which originally failed, succeeds after applying the corrective pose $P_c$, the pair is validated.
• This ensures that the collected actions are robust and effective when applied at various timesteps.

4. Dataset Construction

The pipeline was applied to three tasks in ManiSkill: Pick Cube, Push Cube, and Stack Cube.

• Scale: The resulting dataset contains approximately 131,000 failure-action pairs and 56,000 ground-truth (successful) trajectories.
• Format: Each entry includes 10 consecutive multi-view images (front, side, wrist), task instructions, sub-task identification, failure classification (yes/no), failure type, and the 7-DoF corrective action vector.

5. Model Training (FailSafe-VLM)

The authors fine-tuned LLaVa-OneVision-7B (LLaVa-OV-7B) on the FailSafe dataset.

• Architecture: The model uses Qwen2-7B-Instruct as the language backbone and SigLIP as the vision tower, with a two-layer GELU MLP projector.
• Training: Full instruction fine-tuning was performed with co-training on a RoboPoint VQA mixture to improve generalization.
• Inference Strategy: FailSafe-VLM operates as an external assistant to a base VLA model. Every 10 steps, it analyzes the current state to determine if a failure is likely. If "Yes," it outputs a corrective action; if "No," control returns to the base VLA.

Key Contributions

1. Scalable Failure-Action Pipeline: The first framework to automatically generate both failure reasoning explanations and accurate, executable 7-DoF recovery actions for robots, adaptable to simulators with motion planning support.
2. Performance Improvement: Demonstrated that the FailSafe dataset enables existing VLMs to reason about failures, significantly boosting the success rates of three state-of-the-art VLA models ($\pi$o-FAST, OpenVLA, OpenVLA-OFT).
3. Generalization: Showed that FailSafe-VLM can generalize across diverse spatial configurations, camera viewpoints, object categories (including unseen objects like spheres and chargers), and robotic embodiments (e.g., transferring from Franka Emika Panda to xArm 6).

Experimental Results

The authors evaluated FailSafe-VLM on the Franka Emika Panda arm in the ManiSkill environment:

• VLA Performance Gains:

  • OpenVLA: Achieved the largest improvement, with an average success rate increase of 22.6% (from 14.7% to 37.3%).
  • OpenVLA-OFT: Improved by 8.0% (from 90.7% to 98.7%).
  • $\pi$o-FAST: Improved by 4.0% (from 78.7% to 82.7%).
  • These gains were achieved even when the camera view used for FailSafe-VLM was novel (different from its training view), while matching the VLA's training view.

• Generalization:

  • Unseen Objects: When tested on Pick Sphere, Place Sphere, and Pick Charger, FailSafe-VLM improved OpenVLA-OFT performance by an average of 17.4%.
  • Unseen Embodiments: When applied to an xArm 6 robot (trained only on Franka data), it improved the Stack Cube success rate from 56% to 76%.

• Comparison with State-of-the-Art VLMs:

  • FailSafe-VLM significantly outperformed GPT-4o, Gemini-2.5-flash, and Qwen2.5-VL in failure detection (Binary Success: 0.9094 vs. ~0.70 for GPT-4o), failure type identification (Accuracy: 0.8368 vs. ~0.20), and recovery action prediction (Cosine Similarity: 0.6522 vs. near 0 for others).
  • Notably, Qwen2.5-VL frequently output "no failure" and zero actions, failing to assist in recovery.

• Efficiency:

  • The integration incurred a marginal inference overhead of +3.8s to +9.1s per task, primarily due to simulator replanning.

Significance and Claims

The paper positions FailSafe as a critical step toward robust and explainable embodied AI. The authors claim that:

• Autonomy: By removing the need for human supervision in failure detection and recovery, FailSafe moves VLA systems closer to full autonomy.
• Actionable Reasoning: The shift from textual failure descriptions to executable delta actions bridges the gap between high-level reasoning and low-level control, a limitation in prior work.
• Scalability: The pipeline's design allows for the scalable creation of failure-recovery data, which is prohibitively difficult to collect in the real world.
• Foundation for Future Work: While the current method focuses on motion-level recovery and does not yet address object-level errors (e.g., dropping an object), it establishes a foundation for future systems capable of self-correction in complex, dynamic environments.

The authors conclude that FailSafe serves as an early attempt to enable stronger failure reasoning and autonomy, paving the way for more reliable robotic manipulation in unstructured environments.