💻 computer science

VLAs are Confined yet Capable of Generalizing to Novel Instructions

This paper demonstrates that Vision-Language-Action models (VLAs) can overcome their struggle with task extrapolation and spatial overfitting by manipulating internal text latents through interpolation, a technique that achieves an 83% success rate on novel instruction benchmarks and reveals the potential for hidden, human-unreadable prompt injection.

Original authors: Quanyi Li

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Quanyi Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot chef how to cook. You show it two specific recipes:

Recipe A: "Put the cream cheese in the bowl."
Recipe B: "Put the bowl on top of the cabinet."

The robot learns these two moves perfectly. It can do Recipe A, and it can do Recipe B. But then, you ask it a new question: "Put the cream cheese on top of the cabinet."

Logically, the robot should be able to do this. It knows how to grab the cheese, and it knows how to reach the cabinet. It just needs to combine the two skills it already has. However, according to this paper, most advanced robot brains (called Vision-Language-Action Models or VLAs) fail miserably at this. They get stuck, acting as if they've never seen the cabinet or the cheese before, even though they just used them in the previous steps.

The Problem: The Robot is a "Parrot," Not a "Chef"

The authors discovered that these robots aren't truly understanding the world. Instead, they are memorizing specific scenes.

Think of it like a student who memorizes the answers to a practice test but doesn't understand the math. If you ask, "What is 2 + 2?" they say "4." But if you ask, "What is 2 + 2 if the numbers are written in red ink?" they might panic.

In the robot's case, it has learned that "Cream Cheese" is always associated with the specific spot on the table where it saw the cream cheese during training. It doesn't understand that "Cream Cheese" is an object that can move; it thinks "Cream Cheese" is that specific location. The paper calls this "Spatial Overfitting." The robot is so focused on where things were in the training videos that it ignores where they actually are in the real moment.

The Solution: The "Magic Recipe Card" (Text Latent)

The researchers wanted to know: Is the robot actually smart deep down, or is it just broken?

They found a hidden "ingredient" inside the robot's brain called Text Latent.

The Analogy: Imagine the robot's brain is a giant library. When you give it a command like "Put the cheese in the bowl," the robot pulls out a specific, invisible "recipe card" from its internal memory. This card isn't written in words you can read; it's a complex pattern of electrical signals (a vector) that tells the robot exactly how to move.
The Discovery: The researchers found that if they took this invisible "recipe card" for "putting things in a bowl" and injected it back into the robot's brain, the robot would perform that action perfectly—even if you didn't give it any words at all. The card was the instruction.

The Breakthrough: Mixing the Cards (Text Latent Interpolation)

The real magic happened when they tried to solve the "Cream Cheese on the Cabinet" problem.

They took the invisible "recipe card" for Task A (Cheese to Bowl) and the card for Task B (Bowl to Cabinet). Then, they created a smooth blend of the two cards.

The Analogy: Imagine you have two music tracks playing. One is a jazz song, the other is a rock song. Instead of switching abruptly from one to the other, you use a mixer to slowly fade the jazz out while fading the rock in.
The Result: By "mixing" these two invisible recipe cards inside the robot's brain, the robot successfully combined the skills. It grabbed the cheese, moved it to the cabinet, and dropped it.

The Stats:

Without this trick, the robot (named π0) succeeded only 9% of the time on these new, combined tasks.
With the "mixing" trick, success jumped to 83%.

This proved that the robot did have the skills inside it all along; it just couldn't figure out how to mix them on its own. The "recipe cards" were there, but the robot needed a human to help them blend together.

The Big Test: The "Libero-OOD" Benchmark

To prove this wasn't just a fluke, the authors created a new test called Libero-OOD. This test contains 20 tricky tasks where the robot has to combine skills it already knows in new ways.

The Result: They tested the best robot brains in the world (like UniVLA, OpenVLA, and π0-fast). None of them could do it on their own. They all scored below 21%.
The Conclusion: Even the smartest robots are currently "stuck" in their training data. They can't generalize or "think outside the box" without help.

The Warning: "Private Instructions"

The researchers also found something a bit spooky. Because these "recipe cards" are just patterns of numbers, you can turn them back into text.

When they tried to read the "recipe card" for a task, the resulting text was gibberish (like QSize, black, plate).
However, if you fed this gibberish back into the robot, it still worked!
The Metaphor: It's like having a secret code that only the robot understands. You could hide a secret instruction inside a normal-looking sentence, and the robot would follow it without anyone else knowing. This is called a "backdoor," and it shows that these models are more mysterious than we thought.

Summary

The paper tells us that today's smartest robots are like musicians who can play two songs perfectly but can't improvise a new melody. They memorize the exact notes and positions from their practice sessions but fail when the music changes slightly.

The authors showed that if we manually mix the "musical scores" (the text latents) from two different songs, the robot can play the new melody. This proves the robot has the talent, but it lacks the ability to combine its own skills. The paper introduces a new test (Libero-OOD) to help researchers build robots that can actually learn to mix their own skills, rather than just memorizing scenes.

1. Problem Statement

Vision-Language-Action models (VLAs) have demonstrated strong performance on in-distribution tasks when fine-tuned with specific demonstrations. However, they struggle significantly with Out-of-Distribution (OOD) tasks, particularly those requiring extrapolation (recombining learned skills in novel ways).

The Core Issue: A VLA might successfully learn "put cream cheese in a bowl" and "put a bowl on a cabinet" individually but fail to execute the extrapolated task "put cream cheese on the cabinet," even though the constituent motion primitives (picking, reaching) are already learned.
The Question: Do VLAs merely overfit to demonstrated trajectories, or do they learn composable internal representations that support broader generalization?
The Gap: There is a lack of suitable benchmarks to test this, as existing OOD tasks are often too difficult (e.g., requiring entirely new physics) or too easy. The authors propose that a valid benchmark should consist of tasks the model should be able to solve by combining existing skills but currently fails to do so autonomously.

2. Methodology

The paper introduces a framework based on Mechanistic Interpretability to probe and manipulate the internal states of VLAs, specifically focusing on the model $\pi_0$ .

A. Text Latent Extraction

The authors hypothesize that task-specific skill representations are encoded in the hidden states of text tokens (task descriptions).

Definition: A Text Latent ( $T$ ) is a task-specific vector derived by averaging the hidden states of text tokens across all transformer layers and multiple demonstration episodes for a specific task.
Extraction: $T = \frac{1}{K} \sum_{k=1}^{K} \sum_{i \in B_k} h_T(i)$ , where $h_T(i)$ represents the hidden states of text tokens at timestep $i$ .
Function: Injecting $T$ back into the model's residual stream (specifically the text token hidden states) can reconstruct the associated task behavior even without the original text prompt.

B. Text Latent Interpolation (TLI)

To achieve extrapolation, the authors propose blending the text latents of two base tasks to create a new, composite behavior.

Mechanism: Instead of interpolating discrete text tokens (which is impossible), they interpolate the text latents ( $T_1$ and $T_2$ ) within the residual stream.
Formula: At timestep $i$ , the hidden state is modified as:
$h_T(i) = h_T(i) + [(1-\alpha)T_1 + \alpha T_2] - [(1-\alpha)T_2 + \alpha T_1]$
where $\alpha = i/\lambda$ controls the transition speed from Task 1 to Task 2.
Effect: This guides the model to start with the behavior of Task 1 and smoothly transition to Task 2, effectively "stitching" sub-trajectories together to solve a novel extrapolated task.

C. The Libero-OOD Benchmark

The authors introduce libero-ood, a new benchmark suite containing 20 extrapolated tasks derived from the standard LIBERO suites (libero-goal, libero-spatial, libero-object).

Design: Each task requires combining a grasping location learned from one task and a placement location learned from another. The specific trajectory connecting these two points is never seen in training.
Goal: To test if VLAs can autonomously recombine learned skill representations.

3. Key Contributions

Discovery of Composable Skill Representations: The paper demonstrates that VLAs (specifically $\pi_0$ ) possess internal representations of skills that are individual yet composable. While the model cannot combine them autonomously, it can be forced to do so via text latent manipulation.
Text Latent Interpolation (TLI): A novel inference-time intervention technique that blends task representations to enable extrapolation, boosting success rates on novel tasks without retraining.
The Libero-OOD Benchmark: A rigorous benchmark designed to isolate the ability to recombine skills. It filters out tasks that are impossible for the robot, focusing instead on tasks that are theoretically solvable by the model's existing knowledge base.
Identification of Spatial Overfitting: The paper reveals a critical failure mode where VLAs associate object names with fixed spatial locations from the training data rather than understanding object identity or current state.

4. Experimental Results

Performance on Libero-OOD

Baseline Failure: State-of-the-art VLAs ( $\pi_0$ , $\pi_0$ -fast, UniVLA, openvla-oft) achieved < 21% success rate on the 20 extrapolated tasks of libero-ood, despite achieving ~95% on standard LIBERO tasks.
TLI Success: Applying TLI to $\pi_0$ increased its success rate from 9% to 83% on libero-ood.
Comparison: No other VLA achieved a success rate higher than 21% on the benchmark, highlighting a systemic limitation in current architectures regarding extrapolation.

Task Reconstruction & Security Implications

Reconstruction: Injecting text latents into a blank prompt allowed $\pi_0$ to complete tasks with >80% success, proving the latent captures essential task knowledge.
Unembedding: Unembedding text latents into token IDs produced "obscure prompts" (often unreadable) that could still instruct the model. This suggests potential vulnerabilities for private instructions or backdoor attacks.

Analysis of Failure Modes: Spatial Overfitting

Observation: When instructed to pick up an object (e.g., "cream cheese") that was moved to a new location, the model often ignored the object's current position and moved to the location where that object appeared in the training demonstrations.
Evidence: In experiments where object locations were swapped, $\pi_0$ (and other VLAs) consistently moved to the "training location" of the object name rather than the actual object.
Conclusion: VLAs exhibit spatial overfitting, mapping object names to fixed coordinates rather than learning true object-goal understanding.

5. Significance

Redefining OOD Generalization: The paper argues that current VLAs are not "broken" but "locked." They possess the necessary skills but lack the mechanism to autonomously recombine them. The proposed benchmark (libero-ood) provides a standard to measure this latent capability.
Interpretability as a Tool: By using mechanistic interpretability (text latents), the authors prove that the failure is not a lack of knowledge but a failure of composition. This shifts the research focus from collecting more data to improving the model's internal composition mechanisms.
Safety and Security: The discovery that text latents can be unembedded into obscure prompts highlights new vectors for adversarial attacks and backdoors in robotic systems.
Future Direction: The results suggest that future VLA architectures need to be designed to explicitly support the recombination of learned representations, rather than relying on post-training tricks like TLI to unlock their potential.

In summary, this paper provides strong evidence that VLAs have the potential for generalization but are currently hindered by spatial overfitting and an inability to autonomously compose learned skills. The introduction of libero-ood and TLI offers both a new benchmark for the community and a method to probe the internal "black box" of robotic policies.