💻 computer science

Evaluating Uncertainty and Quality of Visual Language Action-enabled Robots

This paper proposes and empirically validates a set of uncertainty and quality metrics for Vision-Language-Action (VLA) robots, demonstrating through a large-scale study that these metrics correlate well with human expert judgments and offer a more nuanced evaluation of task execution than traditional binary success rates.

Original authors: Pablo Valle, Chengjie Lu, Shaukat Ali, Aitor Arrieta

Published 2026-07-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Pablo Valle, Chengjie Lu, Shaukat Ali, Aitor Arrieta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots aren't just mindless machines following a strict list of "if this, then that" commands, but are instead like curious, super-smart assistants that can see the world, understand your spoken words, and figure out how to move their own bodies to help you. This is the exciting frontier of Vision-Language-Action (VLA) robots. Think of them as the ultimate "smart butlers" that can look at a messy room, listen to you say, "Please put the red cup in the basket," and then actually do it. But here's the catch: just because a robot does the task doesn't mean it did a good job. It might have knocked over three other cups, dropped the red one twice, or wobbled so much it looked like it was having a panic attack before finally succeeding.

For a long time, scientists have judged these robots with a simple, binary switch: Did it succeed? Yes or No? It's like grading a student's essay with only a checkmark for "finished" and no comment on whether the handwriting was messy or the spelling was terrible. This paper argues that this "pass/fail" system is broken. It's like saying a chef is great just because they served you a burger, even if the burger was burnt, dropped on the floor, and reassembled with a fork. To fix this, the researchers needed a way to measure not just if the robot finished the job, but how it felt while doing it—was it smooth and confident, or shaky and unsure?

The Robot's "Panic Meter" and "Smoothness Score"

In this study, the researchers decided to stop asking robots "Did you do it?" and start asking, "How sure were you, and how graceful were you?" They treated the robot's brain like a nervous student taking a test. They introduced two new ways to grade the robot: Uncertainty (how confident the robot was in its choices) and Quality (how smooth and safe the movement was).

To measure Uncertainty, they looked at the robot's "inner monologue." Imagine the robot is trying to decide whether to grab a cup. A confident robot thinks, "I'm 99% sure that's a cup, and I'll grab it right here!" A confused robot, however, might be thinking, "Is that a cup? Or a can? Maybe I should grab it here... no, maybe there? Oh no, I'm not sure!" The researchers created eight different "panic meters" to catch this confusion. Some of these meters listened to the robot's digital thoughts (checking how scattered its probability guesses were), while others watched its physical movements. If a robot's arm started jittering, shaking, or making sudden, jerky corrections, the meters flagged it as "high uncertainty," meaning the robot was struggling to decide what to do.

To measure Quality, they looked at the robot's dance moves. They used five different "smoothness scores" to see if the robot moved like a graceful ballerina or a clumsy toddler. They checked for things like sudden jerks, weird accelerations, or if the robot had to backtrack and re-plan its path. A high-quality execution was like a smooth, flowing motion where the robot knew exactly where it was going. A low-quality one was full of stops, starts, and near-misses.

The Great Robot Reality Check

The team didn't just guess; they put this to the test with a massive experiment. They took three of the smartest, most advanced robot brains available (called OpenVLA, SpatialVLA, and $\pi_0$ ) and asked them to perform 908 different tasks, like picking up objects, moving them near other objects, stacking things, or putting items inside containers.

Here is the big surprise they found: The robots that "passed" the test often failed the "quality" test.

When they looked at the results, they saw that a robot could successfully complete a task (like picking up a cup) but do it in a way that was terrible. For example, one robot model ( $\pi_0$ ) was very good at getting the job done, but when human experts watched the videos, they realized the robot was often clumsy, dropping things, or moving in a shaky, uncertain way. In fact, for some tasks, more than half of the "successful" attempts were actually labeled as "low quality" by the experts. This proved that the old "pass/fail" test was lying to us. A robot could pass the test by luck or by brute force, even if it wasn't actually doing a good job.

What the Meters Told Them

The researchers then checked if their new "panic meters" and "smoothness scores" could actually predict what the human experts thought.

The Good News: Several of their new metrics worked surprisingly well. Specifically, metrics that measured Action Velocity Instability (how much the robot's speed changed suddenly) and Action Acceleration Instability (how jerky the movements were) showed a strong link to human opinions. If the robot was jittery, the humans said it was low quality. If the robot was smooth, the humans agreed. This suggests that we can use these math formulas to automatically tell if a robot is doing a good job without needing a human to watch every single video.
The "Did You Move?" Trick: They also found a clever way to spot total failures. They used a metric called Optimal Trajectory Difference (OT), which basically asks, "Is the robot getting closer to the goal?" If the robot is stuck or hasn't moved at all, this metric instantly knows it's a failure. This was the best tool for telling the difference between a robot that succeeded and one that failed completely.
The Bad News: One metric, called Execution Variability (EV), which tried to measure uncertainty by running the robot's brain multiple times to see if it gave different answers, was too slow and expensive to use in real life. It took too much computer power, making it impractical for real-time use. Also, they found that some metrics couldn't tell the difference between a robot that was "okay" (medium quality) and one that failed completely, suggesting that as robots get worse, they start to look more like total failures.

The Takeaway

The main lesson from this paper is that we need to stop judging robots like a simple pass/fail exam. Just because a robot finishes a task doesn't mean it's safe, efficient, or reliable. The researchers showed that by using these new "confidence" and "smoothness" scores, we can get a much clearer picture of how robots are actually performing.

They suggest that in the future, we should use these metrics to build better "test oracles" (the rules that decide if a robot passes). Instead of just checking if the cup is in the basket, we should check if the robot got there smoothly and confidently. This is crucial for the future, because if we want robots to work in our homes or hospitals, we need to know they aren't just lucky—they need to be consistently good, smooth, and sure of themselves. The paper concludes that while we have the tools to measure this now, we need to start using them to make sure our future robot helpers are truly ready for the real world.

Technical Summary: Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robots

Problem Statement

Vision-Language-Action (VLA) models represent a significant advancement in robotic control, integrating visual perception, natural language understanding, and action planning to enable autonomous task execution. However, the evaluation of these systems currently suffers from a critical limitation: reliance on binary success rates (i.e., whether a task was completed) as the primary test oracle. This approach, exemplified by frameworks like VLATest, captures functional correctness but fails to assess the quality of task execution (e.g., smoothness, safety, efficiency) or the robot's confidence in its decisions. Consequently, a robot may be labeled as "successful" despite exhibiting poor-quality behaviors such as collisions, unstable trajectories, or inefficient movements, leading to a false sense of reliability in real-world deployments.

Methodology

To address this gap, the authors propose a comprehensive evaluation framework comprising eight uncertainty metrics and five quality metrics, specifically adapted for VLA-enabled robotic manipulation.

1. Metric Adaptation

Uncertainty Metrics (8 total): These quantify the model's confidence and the consistency of its generated actions.
- Token-based (4): Adapted from Large Language Models (LLMs) to measure uncertainty in the discrete token probabilities of the VLA model's output. These include Token Probability (TB-TP), Prediction Confidence Score (TB-PCS), DeepGini (TB-D), and Entropy (TB-E).
- Action-based (4): These measure uncertainty directly from the continuous action vectors generated by the policy, capturing temporal inconsistencies. They include Action Position Instability (A-PI), Action Velocity Instability (A-VI), Action Acceleration Instability (A-AI), and Execution Variability (EV).
Quality Metrics (5 total): These evaluate the characteristics of the executed trajectory and task progression.
- Trajectory Position Instability (TCP-PI), Trajectory Velocity Instability (TCP-VI), and Trajectory Acceleration Instability (TCP-AI): Adapted from classical robotics smoothness measures (e.g., jerk-based metrics) to assess motion stability.
- Trajectory Instability (TI): A Root Mean Square (RMS) jerk metric to quantify overall motion smoothness.
- Optimal Trajectory Difference (OT): Measures the spatial proximity of the end-effector to the goal, evaluating goal-oriented behavior.

2. Empirical Study Setup

The authors conducted a large-scale empirical study involving:

Models: Three state-of-the-art VLA models: OpenVLA, SpatialVLA, and $\pi_0$ .
Tasks: Four representative manipulation tasks: Pick Up, Move Near, Put On, and Put In.
Embodiments: Two robotic platforms: the Google Robot (Everyday Robot) and the WidowX robot.
Data: 908 successful task executions were manually labeled by three domain experts into High, Medium, and Low quality categories.
Analysis: The study analyzed the correlation between the proposed metrics and expert judgments (Spearman's rank correlation) and evaluated the metrics' ability to discriminate between successful and failing tasks (Mann-Whitney U test and Vargha-Delaney $\hat{A}_{12}$ effect size).

Key Contributions

Evaluation Framework: The design of a unified framework integrating eight uncertainty and five quality metrics tailored for VLA-enabled robots.
Quality Guidelines: The proposal of specific guidelines for human experts to evaluate the quality of VLA task execution, moving beyond binary success.
Large-Scale Empirical Analysis: A rigorous study across three VLA models, four tasks, and two robot embodiments, involving 908 expert-labeled executions.
Replication Package: The release of a complete replication package including code, configuration files, and experimental results to facilitate reproducibility.

Results

Limitations of Success Rates: The study confirmed that success rates alone are insufficient. For instance, while the $\pi_0$ model achieved high success rates, a significant portion of its "successful" executions were labeled as low-quality by human experts (e.g., >50% low-quality in "Pick up" tasks). Conversely, SpatialVLA demonstrated higher overall execution quality despite varying success rates.
Correlation with Expert Judgment:
- Uncertainty: Metrics A-VI (Action Velocity Instability) and A-AI (Action Acceleration Instability) showed moderate to strong correlations with human quality assessments across multiple models and tasks. Notably, orientation-based variants of these metrics often correlated better than positional ones, suggesting orientation uncertainty is critical for perceived quality.
- Quality: TCP-VI (Trajectory Velocity Instability) emerged as the most consistent quality metric, showing strong correlation across models and tasks.
Discrimination Capabilities:
- Optimal Trajectory (OT) was the most effective metric for distinguishing between successful and failing tasks across all models, acting as a model-agnostic indicator of failure.
- Most metrics could effectively distinguish high-quality executions from failures. However, distinguishing medium/low-quality successful tasks from outright failures proved more difficult, as low-quality successes often exhibited behavioral patterns similar to failures.
Computational Overhead:
- Execution Variability (EV) incurred high overhead due to the need for multiple model inferences, making it impractical for real-time use.
- Optimal Trajectory (OT) is largely limited to simulation environments due to the difficulty of defining optimal trajectories in complex real-world scenarios.
- The remaining metrics (Token-based, AI-based, TCP-based, TI) introduced negligible overhead, making them suitable for runtime monitoring.

Significance and Claims

The paper argues that current evaluation practices for VLA-enabled robots are inadequate because they rely on binary success oracles that ignore behavioral quality and model confidence. The authors claim that their proposed metrics offer a more nuanced and reliable assessment of robot performance.

Beyond Binary Testing: The findings challenge the sufficiency of success rates, demonstrating that high success rates can mask poor execution quality and weak behavioral correctness.
Runtime Monitoring: The study suggests that specific metrics (like A-VI, A-AI, and TCP-VI) can serve as effective runtime oracles for monitoring robot behavior, detecting anomalies, and identifying potential failures without relying on symbolic task completion checks.
Future Directions: The work lays the groundwork for next-generation test oracles that incorporate behavioral quality, supports search-based testing and fuzzing by providing objective functions for quality degradation, and enables adaptive enhancement of robotic systems through real-time uncertainty monitoring.

The authors maintain a modest stance, acknowledging that metric effectiveness is dependent on the specific underlying policy and that further validation is needed for real-world deployment, particularly regarding the simulation-to-reality gap and the handling of object orientation in specific tasks.