🤖 AI

Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time

This paper proposes a retrieval-augmented approach that extends frozen Vision-Language-Action models to new tasks at test time by indexing demonstrations from a cheaper embodiment, thereby eliminating the need for per-task fine-tuning while achieving superior performance on unseen tasks and embodiments.

Original authors: Jeongeun Park, Juhan Park, Taekyung Kim, Sungjoon Choi, Dongyoon Han, Sangdoo Yun

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jeongeun Park, Juhan Park, Taekyung Kim, Sungjoon Choi, Dongyoon Han, Sangdoo Yun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot that needs to learn new jobs. Traditionally, teaching a robot a new task (like opening a specific cabinet or placing a bottle in a box) is like hiring a personal tutor for every single new skill. You have to:

Record the robot doing the task manually (which is slow and expensive).
Re-train the robot's brain specifically for that one task (which takes a lot of computer power and time).

If you want the robot to do 100 different tasks, you have to do this expensive training process 100 times.

The Paper's Big Idea: "Retrieval, Don't Retrain"

The authors of this paper propose a smarter way called RECAP. Instead of retraining the robot's brain every time, they give the robot a library of "cheaper" examples and teach it how to look them up.

Here is how it works, using simple analogies:

1. The Two "Bodies" (Embodiments)

Imagine the robot is a heavy, clumsy construction worker (the "Target"). It's hard to get them to demonstrate new tasks because they are slow and require expensive equipment to control.

Now, imagine a nimble human hand (the "Pool"). It's easy to film a human doing tasks. They are fast, cheap to record, and can do almost anything.

The problem is that the human hand moves differently than the construction worker. If you just tell the worker to "copy the human," they might break things because their arms are different.

2. The Old Way vs. The New Way

The Old Way (Retrain): Every time you want the worker to learn a new task, you hire a trainer to stand next to them, show them the task, and then spend hours tweaking their brain so they can do it. This is slow and expensive.
The New Way (RECAP):
1. Train Once: You teach the worker's brain one time how to translate "Human Hand moves" into "Construction Worker moves." You teach them: "When you see the human hand do X, you do Y."
2. Freeze the Brain: Once that translation skill is learned, you lock the brain. No more training.
3. The Library (Retrieval): You create a library of videos showing the human hand doing many different tasks.
4. The Magic: When the worker needs to do a new task they've never seen before, you don't retrain them. Instead, you ask the library: "Do we have a video of a human doing something similar to this?"
5. The Result: The worker finds the closest human video, looks at it, and uses their "translation skill" to figure out how they should move to achieve the same result.

3. The "Residual" Trick (The Secret Sauce)

The paper uses a special type of AI model (called a "World-Action Model") that acts like a predictive storyteller.

The Human Video (The Plan): The retrieved video gives the robot a "coarse plan." It says, "The goal is to move the object from here to there."
The Robot's Job (The Correction): The robot doesn't try to copy the human exactly. Instead, it calculates the difference (the "residual") between how the human moves and how it needs to move.
- Analogy: Imagine the human is a dancer doing a pirouette. The robot is a forklift. The robot doesn't try to spin like a dancer. It looks at the dancer's plan ("Spin to the left") and calculates, "Okay, I need to turn my wheels left to achieve that same result."

The AI model is trained to predict not just the robot's next move, but also what the scene will look like next. This acts as a "reality check." If the robot's plan would result in a mess (like dropping the bottle), the model catches it and corrects the move.

4. What They Found

The researchers tested this in three ways:

A Simple 2D Game (PushT): They had a robot push a block to different angles. By adding more human videos to the library, the robot got better at pushing the block to new angles it had never seen before, without any extra training.
A Complex Simulation (RoboTwin): They tested a two-armed robot doing complex tasks. The "Retrieval" method beat other methods that tried to retrain the robot for every new job.
A Real Robot: They put this on a real physical robot. They trained it on one task (opening a cabinet) using human videos. Then, they froze the brain. When they asked the robot to do new tasks (closing the cabinet, placing a bottle), they simply added human videos of those tasks to the library. The robot successfully figured out how to do them, whereas a standard robot failed completely.

The Bottom Line

This paper shows that you don't need to retrain a robot's brain for every new job. Instead, you can train it once to understand how to translate "cheap human examples" into "expensive robot actions." Then, to teach it a new job, you just add a new video to the library.

It turns robot learning from "building a new brain for every job" into "looking up a recipe in a cookbook."

Technical Summary: Retrieve, Don't Retrain (RECAP)

Problem Statement

Extending Vision-Language-Action (VLA) policies to new tasks or new robot embodiments (e.g., from a human hand to a specific robot arm) typically requires collecting expensive, task-specific teleoperated demonstrations and performing per-task fine-tuning. This process incurs two significant costs:

Data Collection Cost: Teleoperated robot data is slow to acquire (roughly 18× slower than human-hand demonstrations) and hardware-bound.
Compute Cost: Fine-tuning modern World-Action Models (WAMs) or large VLA policies for each new task is computationally intensive (e.g., ~24 GPU-hours per task for Cosmos Policy) and scales with model size and context length.

The authors argue that this per-task adaptation cost is avoidable. Instead of retraining, behavioral knowledge from a cheap, data-rich source (the "pool" embodiment, e.g., human hands) can be transferred to the target embodiment (the "query") via a retrieval-based paradigm.

Methodology: RECAP

The paper proposes RECAP (Retrieval-Conditioned Action Policy), a framework that shifts adaptation from parameter optimization to data retrieval. The core premise is that while embodiments differ in execution dynamics, they largely agree on the high-level requirements of a task.

1. Training Phase

Backbone: The method utilizes a World-Action Model (WAM), specifically Cosmos Policy, which treats actions and future visual observations as a single denoised video sequence.
Retrieval-Augmented Conditioning: The policy is trained once on paired demonstrations from the query (target robot) and pool (e.g., human hand) embodiments.
Residual Parameterization: Instead of predicting the full action sequence, the policy predicts a residual ( $\Delta a$ $Δ a$ ) on top of the retrieved pool trajectory.
- $\hat{a}^{query} = a^{pool}_{retrieved} + \Delta a$
- This forces the model to learn only the embodiment-specific corrections (dynamics, contact geometry) rather than the entire motion plan.
Joint Objective: The model is trained with a flow-matching loss that jointly supervises the predicted action and the predicted future state (image). The future-image objective acts as a visual consistency signal, ensuring the retrieved trajectory aligns with the physical evolution of the scene on the target robot.

2. Test-Time Adaptation (Inference)

Frozen Policy: Once trained, the policy parameters are frozen. No gradient updates occur for new tasks.
Dynamic Retrieval Pool: New tasks are handled by appending relevant pool-embodiment demonstrations (e.g., human videos) to a retrieval pool.
Step-by-Step Retrieval: At every control step $t$ $t$ , the system:
1. Observes the current query state $s^{query}_t$ .
2. Retrieves the most similar state-action chunk $(s^{pool}_{t':t'+H}, a^{pool}_{t':t'+H})$ from the pool database based on a composite descriptor (language goal, object pose, proprioception, and visual features).
3. Feeds the current observation and the retrieved chunk into the frozen policy.
4. The policy outputs the next action chunk for the target robot, conditioned on the retrieved high-level plan.

Key Contributions

Test-Time Adaptation Paradigm: A method to adapt policies to new tasks entirely at test time by extending a retrieval pool with cheap demonstrations, keeping the policy frozen.
Retrieval-Conditioned Residual Policy on WAMs: A specific architecture where retrieval supplies the coarse high-level motion, and the WAM learns only the embodiment-specific correction. The authors highlight that the WAM's future-image prediction objective is uniquely beneficial here, providing a visual alignment signal that strengthens retrieval-conditioned actions.
Empirical Validation: Demonstrated consistent gains over cross-embodiment baselines across three settings:
- PushT (2D): Generalization to unseen goal angles.
- RoboTwin 2.0 (Simulation): Multi-task, dual-arm manipulation.
- Real Robot: Transfer from human-hand videos to a physical robot arm.

Experimental Results

PushT Benchmark (Cross-Embodiment Generalization)

Setup: A triangle pusher (target) trained on $\pm 45^\circ$ angles, evaluated on 9 angles (7 unseen). The pool consisted of disc-pusher demonstrations.
Results:
- Pool Progression: Expanding the retrieval pool at test time without retraining monotonically improved success on unseen angles, rising from 6.0% (no retrieval) to 34.9% (full pool).
- Backbone Comparison: Retrieval improved both standard VLA ( $\pi0.5$ ) and WAM (Cosmos) backbones, but the gain was significantly larger for the WAM (25.1% vs 34.9%), confirming the synergy between retrieval and the future-image objective.
- Mechanism: Attention analysis showed the policy uses early layers to attend to the retrieved trajectory and later layers to adapt the plan to target dynamics.

RoboTwin 2.0 Simulation

Setup: Target (Aloha-Agilex) vs. Pool (UR5). Trained on 5 paired tasks, evaluated on 5 unseen tasks.
Results:
- Performance: RECAP achieved 31.5% success on unseen tasks, outperforming the strongest baseline (Cosmos Policy without retrieval) at 26.0% and "Retrieval Only" (naive execution) at 26.0%.
- Pool Growth: Success rates increased monotonically as the pool grew from 11 to 35 tasks, reaching 31.5% with the full pool.
- Qualitative: In tasks like "hand-over-mic," naive retrieval failed (colliding with objects), while RECAP successfully adjusted the grip orientation.

Real Robot Experiments

Setup: Target robot trained on "open cabinet" (25 paired demos). Evaluated on held-out tasks: "place bottle" and "close cabinet."
Results:
- The baseline (trained only on open cabinet) failed on held-out tasks (0% and 10% success), replaying the training trajectory.
- RECAP, using only 10 human-hand demonstrations per new task added to the pool, achieved 80% success on placing the bottle and 30% on closing the cabinet.

Significance and Claims

The paper claims that indexing cheaper pool-embodiment data at deployment can stand in for collecting new target-robot demonstrations.

Cost Reduction: The method replaces the ~24 GPU-hours/task fine-tuning cost and the 18× slower teleoperation cost with a retrieval process.
Scalability: By freezing the policy, the system can absorb new tasks simply by adding data to the retrieval pool, avoiding the compounding costs of traditional fine-tuning.
Synergy with WAMs: The work demonstrates that World-Action Models, when paired with retrieval, can effectively leverage the "coarse" planning from one embodiment to learn the "fine" dynamics of another, a capability that standard action-only policies lack.

The authors acknowledge limitations, noting that the method requires shared action spaces (e.g., end-effector pose) and that the current retrieval descriptors (combining language, pose, and vision) may need further refinement for more complex or temporally mismatched scenarios. However, the results suggest a viable path toward general-purpose robot policies that adapt via retrieval rather than retraining.