⚡ electrical engineering

VLA-Scope: Shift-Aware Failure Prediction for Vision-Language-Action Models

This paper introduces VLA-Scope, a two-stage framework that enhances failure prediction for Vision-Language-Action models under distribution shifts by combining input-shift characterization with execution history and action features to dynamically update failure risk during robotic rollouts.

Original authors: Kaiwen Zhu, Dongfang Liu, Liangkai Liu

Published 2026-09-21

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Kaiwen Zhu, Dongfang Liu, Liangkai Liu

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots are becoming more capable of understanding the world through sight and speech, but they still struggle when the world changes in unexpected ways. Imagine a robot trained in a clean, well-lit laboratory to pick up a red cup. If you move the cup to a cluttered table, dim the lights, or ask the robot to pick up a blue bowl instead, the robot might freeze or make a mistake. This happens because the robot's "brain," a type of artificial intelligence that connects what it sees with what it is told to do, has never seen that specific combination of sights and instructions before. In the scientific world, this is known as an "out-of-distribution" situation: the robot is facing something outside its training data. For a long time, researchers have tried to build systems that can simply flag these unfamiliar moments as dangerous. However, a robot can often handle a strange situation successfully, or it might fail even when the situation looks normal. Knowing that something is unfamiliar is not enough; we need to know if the robot is about to fail while it is actually trying to do the task.

A team of researchers at Texas Tech University and Purdue University has developed a new method called VLA-Scope to solve this problem. Instead of just checking if a situation looks strange before the robot starts, their system watches the robot as it works, combining what the robot sees with how it moves to predict if a task will go wrong. The system operates in two distinct phases. First, it looks at the initial image and the instruction given to the robot to decide if the situation is familiar or strange. If the situation is strange, the system then classifies exactly how it is different—whether the camera angle has shifted, the lighting has changed, or the objects are in a new arrangement. This classification acts as a context, helping the system understand the type of challenge the robot is facing.

Once the robot begins to move, the second phase of the system kicks in. It does not just watch the robot's eyes; it also watches its hands. The system tracks the specific commands the robot sends to its motors, such as how far it moves its arm, how much it rotates its wrist, and whether it is opening or closing its gripper. Crucially, it also looks at the robot's internal "thoughts" as it generates these commands, creating a running summary of the robot's decision-making process over time. By combining the type of strange situation identified at the start with the real-time history of the robot's movements and decisions, the system calculates a risk score. This score tells us, at any given moment, how likely it is that the robot will fail to complete its task.

The researchers tested this approach using a powerful robot model called OpenVLA on ten different tasks involving moving objects in a simulated environment. They created hundreds of scenarios where the robot faced changes in background, lighting, camera view, and object layout. In these tests, the system proved remarkably good at spotting when the robot was entering an unfamiliar situation, correctly identifying the type of change in about 91 percent of cases. More importantly, when the robot was actually performing the task, the system could predict failure with high accuracy. After the robot had performed sixty actions, the system's ability to distinguish between a task that would succeed and one that would fail reached a level of precision that was significantly better than previous methods.

The study found that simply knowing the robot was in a strange situation was not enough to predict failure. The system needed to see how the robot was reacting to that situation. For example, if a robot was trying to pick up an object in a noisy visual environment, the system could detect if the robot was making small, hesitant adjustments or large, erratic movements that signaled an impending crash. The researchers discovered that the most accurate predictions came from combining the initial context of the strange situation with the accumulated evidence of the robot's behavior over time. This approach allowed the system to catch failures that other methods missed, such as a robot that seemed to be working fine but was actually stuck in a loop of corrections that would eventually run out of time.

One of the key insights from this work is that failure is often a process, not a single moment. A robot might start a task correctly, but as it encounters difficulties, its movements change in subtle ways that signal trouble. By watching these changes and comparing them to the type of challenge the robot is facing, the system can warn of failure early. The researchers showed that this method works even when the robot is in the middle of a task, providing a safety net that could allow a human supervisor to intervene before a mistake becomes permanent. The system achieved these results without changing the robot's underlying brain or requiring it to learn new skills; it simply added a layer of observation that interpreted the robot's actions in real-time.

The findings suggest that for robots to be truly reliable in the real world, we cannot rely on static checks that happen only before a task begins. We need dynamic monitors that understand the relationship between the environment and the robot's behavior. The VLA-Scope framework demonstrates that by looking at both the context of the problem and the history of the robot's actions, we can build a much clearer picture of what is likely to go wrong. This does not mean the robot is perfect, but it means we have a better way to know when it is struggling. The research provides a practical tool for making robotic systems safer and more trustworthy, ensuring that when they face the unexpected, we can see the trouble coming before it happens.

Technical Summary: VLA-Scope

Problem Statement
Vision-Language-Action (VLA) models map visual observations and natural language instructions to robotic actions. While effective, their reliability is compromised by distribution shifts (e.g., changes in camera viewpoint, object layout, or instructions) that move inputs out-of-distribution (OOD). Existing approaches often rely on binary OOD detection, which identifies unfamiliar inputs but fails to distinguish between shift types or predict whether a specific execution will actually fail. A VLA policy may succeed under certain OOD conditions, meaning that detecting an OOD input alone is insufficient for failure prediction. Furthermore, runtime monitors often lack the ability to combine initial shift context with evolving execution evidence to assess risk dynamically.

Methodology: VLA-Scope Framework
The authors propose VLA-Scope, a two-stage diagnostic framework designed to predict execution failure during OOD rollouts without modifying the underlying VLA policy. The framework operates on a frozen OpenVLA model and consists of:

Stage 1: OOD Characterizer
- Input: Pooled image and language representations ( $z_0$ ) derived from the initial observation and instruction before any action is executed.
- Function: A binary logistic regression classifier distinguishes In-Distribution (ID) from OOD inputs. If an input is flagged as OOD, a separate multi-class classifier assigns it to one of seven predefined shift categories: Background, Camera, Language, Light, Noise, Layout, or Robot.
- Gate: A threshold ( $\tau_{OOD} = 0.5$ ) determines whether the system proceeds to risk prediction. Only inputs exceeding this threshold receive a failure risk score.
Stage 2: Failure Risk Predictor
- Input Features:
  - Shift Context: The predicted OOD category (encoded as a 7D one-hot vector), which remains fixed throughout the rollout.
  - Action-Prefix Features: A 14-dimensional vector summarizing executed commands ( $a_{1:k}$ ), including mean absolute magnitudes for translation/rotation axes, mean adjacent-command changes, mean gripper command, and gripper switching rate.
  - Cumulative Execution Representation: A temporally aggregated representation of the execution history. For each step $i$ , the final-layer hidden vector ( $h_i$ ) from the VLA model is extracted. From step 10 onwards, these vectors are cumulatively averaged, standardized, and compressed via Principal Component Analysis (PCA) to 32 dimensions ( $r_k$ ).
- Model: A shared logistic regression model processes the concatenated 53-dimensional input vector (7D category + 14D action features + 32D representation). The model outputs a failure probability score ( $\hat{p}_k$ ) that updates as execution proceeds.
- Training: The model is trained using leave-one-group-out cross-validation on LIBERO-Spatial tasks, using final rollout outcomes (success/failure) as supervision for all prefixes from step 10 to $T-1$ .

Key Contributions

Two-Stage Diagnosis: The framework explicitly separates initial input shift characterization (ID/OOD detection and type classification) from sequential failure-risk prediction, acknowledging that shift type provides context but does not deterministically dictate outcome.
Hybrid Feature Integration: It introduces a method to combine inferred shift context, explicit action-prefix statistics (magnitudes, changes, switching), and cumulatively averaged execution-step representations to assess partial executions.
Shared Logistic Architecture: A single logistic regression model serves all OOD categories, learning to separate the contributions of action features and compressed cumulative representations without requiring an ID action reference or policy modification.

Experimental Results
Evaluated on 1,400 OOD rollouts across ten LIBERO-Spatial tasks using OpenVLA:

OOD Characterization: The OOD detector achieved a ROC-AUC of 0.9454, and the shift classifier achieved 91.00% accuracy (macro-F1: 0.9123).
Failure Prediction: Evaluated independently of the OOD gate on all OOD rollouts, the failure predictor achieved a ROC-AUC of 0.8497 after 60 executed actions.
- Ablation: Removing execution progress features (cumulative representation) reduced the ROC-AUC to 0.7906, demonstrating the value of temporal aggregation.
- Baselines: VLA-Scope outperformed ActProbe (0.8132) and SAFE-MLP (0.8055) at the 60-action checkpoint.
Efficiency: The CPU scoring latency was 0.893 ms at step 60, significantly lower than neural baselines like SAFE-LSTM (6.773 ms).
Temporal Stability: The cumulative averaging strategy reduced mean absolute risk changes between steps compared to current-step fusion, providing smoother risk curves without explicit output smoothing.

Significance and Claims
The paper claims that VLA-Scope demonstrates the value of combining action behavior with execution progress features for accurate failure prediction under input shifts. The authors emphasize that while OOD detection identifies unfamiliar inputs, predicting actual failure requires analyzing how the robot executes tasks over time. By integrating shift context with a temporally aggregated representation of execution steps, the framework improves failure prediction accuracy (an absolute gain of 0.0591 in ROC-AUC over using only shift and action features) while maintaining low computational overhead. The results suggest that separating input characterization from outcome likelihood, and using a shared linear model to weigh action and representation features, offers a robust and efficient approach to monitoring VLA reliability in dynamic environments.

Technical Summary: VLA-Scope

More like this