💻 computer science

LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

The paper introduces LIBERO-PRO, a robust benchmark that exposes the severe reliance of current Vision-Language-Action models on rote memorization by demonstrating their complete performance collapse when evaluated under generalized perturbations of objects, states, instructions, and environments, thereby urging the community to adopt fairer evaluation standards.

Original authors: Xueyang Zhou, Yangming Xu, Guiyao Tie, Yongchao Chen, Guowen Zhang, Duanfeng Chu, Pan Zhou, Lichao Sun

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xueyang Zhou, Yangming Xu, Guiyao Tie, Yongchao Chen, Guowen Zhang, Duanfeng Chu, Pan Zhou, Lichao Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a robot chef to make a salad. You show it a video of someone picking up a bottle of salad dressing and putting it in a basket. The robot watches, learns, and then, when you ask it to do the same thing, it does it perfectly. You give it a high score: 95%! You think, "Wow, this robot is a genius at following instructions."

But here's the twist: The robot isn't actually thinking. It's just acting like a parrot that has memorized a specific script.

This is the core story of the paper LIBERO-PRO. The authors argue that the current way we test these "Vision-Language-Action" (VLA) robots is broken. It's like giving a student a math test where the questions are identical to the ones they practiced, just with the numbers shifted by one millimeter. The student gets 100%, but if you change the numbers slightly, they fail completely.

Here is a breakdown of what the paper found, using simple analogies:

1. The "Rote Memorization" Trap

The current standard test (called LIBERO) is too easy. The robots are trained on a set of tasks and then tested on the exact same tasks, with only tiny, almost invisible changes (like moving a bowl an inch to the left).

The Result: Robots score over 90%.
The Reality: They aren't understanding the task; they are just replaying a memorized movie of what they saw during training. They are "cheating" by memorizing the layout of the room and the sequence of moves, rather than actually understanding what a "basket" or "salad dressing" is.

2. The "LIBERO-PRO" Stress Test

To fix this, the authors created LIBERO-PRO. Think of this as a "surprise pop quiz" designed to catch cheaters. They systematically messed with four things to see if the robot could actually adapt:

The Object Swap (The "Imposter"):
- The Test: You tell the robot, "Pick up the salad dressing." But in the scene, there is no salad dressing. Instead, there is a can of alphabet soup.
- The Robot's Reaction: It ignores the soup, looks at the empty space where the dressing usually sits, and tries to grab thin air. It's so stuck on its memory that it doesn't even notice the object is missing or different.
The "Nonsense" Instruction:
- The Test: You tell the robot, "Pick up the salad dressing," but then you replace the words with gibberish like "fdsgfdsgsd."
- The Robot's Reaction: It doesn't pause or say, "I don't understand." It just grabs the salad dressing anyway. It proves the robot isn't actually listening to your words; it's just guessing based on what it sees.
The "Moving Target" (Position Shift):
- The Test: You move the object just a little bit away from where it was in the training videos.
- The Robot's Reaction: It reaches for the old spot and misses the object completely. It has no concept of "where" the object is right now; it only knows "where it was supposed to be."
The "Combo" Task:
- The Test: You ask the robot to do two simple things it knows how to do separately (pick up a bowl, then turn on the stove) but combine them into one new instruction.
- The Robot's Reaction: It fails. It can't stitch two learned actions together to make a new plan. It's like a musician who can play a scale and a chord, but if you ask them to play a song, they freeze.

3. The Big Reveal

When the authors ran these "stress tests" on the top robots (like OpenVLA and Pi0), the results were shocking.

Standard Test Score: 90%+ (Looks amazing).
LIBERO-PRO Score: 0% (Total failure).

The robots collapsed instantly. They couldn't handle a different object, a moved object, a weird instruction, or a new combination of tasks.

The Takeaway

The paper concludes that the current "gold standard" for testing robots is misleading. It's like judging a driver's skill only by how well they can drive a car on a perfectly straight, empty track they've driven a thousand times.

The authors are calling on the scientific community to stop using the old, easy tests. They want everyone to use LIBERO-PRO instead, which forces robots to prove they can actually see, understand, and adapt to the messy, unpredictable real world, rather than just reciting a memorized script.

In short: The robots aren't smart yet; they are just really good at memorizing. LIBERO-PRO is the test that finally exposes the difference.

Technical Summary: LIBERO-PRO

Problem Statement

Vision-Language-Action (VLA) models have become a central paradigm for embodied AI, yet their evaluation relies heavily on the LIBERO benchmark. The authors identify a critical flaw in the current LIBERO training and evaluation configuration: the test tasks are nearly identical to the training tasks, differing only by marginal, often visually imperceptible perturbations in initial object states.

This setup leads to inflated performance estimates. Models achieving over 90% success on standard LIBERO are shown to rely on rote memorization of action sequences and environment layouts rather than genuine task understanding or environmental perception. Qualitative analysis reveals systematic failure modes, including:

Visual Hallucinations: Executing grasps on unrelated objects or "grasping the air" when target objects are removed.
Instruction Insensitivity: Ignoring meaningless or corrupted instructions and proceeding with memorized trajectories.
Spatial Fragility: Failing immediately when object positions shift slightly from training distributions.
Lack of Compositionality: Inability to combine learned subtasks into coherent multi-step behaviors.

Consequently, the current evaluation protocol fails to rigorously assess the reasoning, compositionality, and generalization capacity of modern VLA architectures.

Methodology: LIBERO-PRO

To address these limitations, the authors introduce LIBERO-PRO, an extended, plug-and-play benchmark designed to systematically evaluate model performance under reasonable perturbations. The framework formalizes a VLA task $T = (l, e, G)$ and introduces controlled variations across four orthogonal dimensions:

Manipulated Objects (Object Attribute Perturbation): Modifying non-essential visual attributes (color, texture, size) while preserving semantic equivalence, or replacing objects with unrelated items.
Initial States (Spatial Configuration Perturbation): Altering the absolute and relative positions of objects within physically plausible bounds.
Task Instructions (Instruction Perturbation): Introducing semantic variations (paraphrasing) and structural variations, including the injection of nonsensical tokens to test language grounding.
Environments (Environment Perturbation): Changing visual context such as backgrounds, lighting, and textures.

Additionally, the framework includes Task Perturbations (modifying target objects or actions within the training distribution) and supports multi-dimensional randomized combinations of these perturbations, excluding only task-level perturbations from cross-type combinations to ensure feasibility.

The evaluation enforces constraints to ensure perturbed tasks remain semantically coherent and executable, varying perturbation magnitude ( $\delta_k$ ) to characterize policy sensitivity.

Key Contributions

The paper makes three primary contributions:

Empirical Demonstration of Overfitting: The authors provide evidence that high LIBERO scores stem from memorizing training configurations rather than transferable reasoning, exposing severe flaws in current evaluation practices.
Introduction of LIBERO-PRO: A new benchmark suite that integrates systematically designed perturbations across object attributes, initial states, task instructions, and environments to support rigorous, single- and multi-dimensional evaluation.
Comprehensive Benchmarking: The evaluation of state-of-the-art models (OpenVLA, Pi0, Pi0.5) on LIBERO-PRO, revealing significant performance degradation that underscores the limitations of the original protocol.

Experimental Results

The authors evaluated leading VLA models (OpenVLA, Pi0, Pi0.5) on the LIBERO-PRO suite. The results indicate a near-complete performance collapse under generalized settings:

Standard vs. Generalized Performance: While models achieve >90% accuracy on standard LIBERO, their performance drops to 0.0% under the generalized LIBERO-PRO setting.
Sensitivity to Perturbations:
- Position Shifts: Success rates for OpenVLA and Pi0 drop to zero once object displacement exceeds 0.2 units. Even the stronger Pi0.5 degrades to zero beyond 0.4 units.
- Object Replacement: Models fail to adapt when target objects are replaced with unrelated items, often executing grasps on the wrong objects.
- Instruction Corruption: Models execute actions despite receiving meaningless instructions (e.g., "fdsgfdsgsd"), indicating a lack of language grounding.
- Task Composition: Models fail to combine simple subtasks into complex multi-step instructions, relying instead on spurious trajectory fragments.
Model Differences: While most models fail severely, Pi0.5 showed slightly better resilience in specific tasks (e.g., 0.38 success rate in libero-goal under position changes vs. 0 for others), highlighting subtle capability differences masked by standard benchmarks.

Significance and Claims

The paper claims that the current "common currency" of VLA evaluation (LIBERO) is fundamentally flawed because it rewards memorization over generalization. The authors argue that reported accuracies exceeding 90% do not reflect genuine perceptual grounding or task-level reasoning.

The significance of LIBERO-PRO lies in its ability to provide a fairer, more faithful, and reproducible evaluation standard. By exposing the reliance of current models on shortcut feature-action correlations (e.g., canonical scene layouts and recurring object positions), the authors call on the community to abandon misleading methodologies. They advocate for evaluation protocols incorporating structured perturbations to ensure that reported progress aligns with the requirements of robust and generalizable embodied intelligence.