🤖 AI

EXIMO: VLM Guided Exploration of VLA Policies

This paper introduces EXIMO, a three-stage algorithm that combines a Vision-Language Model (VLM) planner for task decomposition and data collection with subsequent imitation learning and residual reinforcement learning to efficiently finetune large Vision-Language-Action (VLA) policies for new robotic tasks.

Original authors: Bhavya Sukhija, Oliver Groth, Mohit Shridhar, Tim Hertweck, Michael Bloesch, Markus Wulfmeier, Abbas Abdolmaleki, Martin Riedmiller

Published 2026-08-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bhavya Sukhija, Oliver Groth, Mohit Shridhar, Tim Hertweck, Michael Bloesch, Markus Wulfmeier, Abbas Abdolmaleki, Martin Riedmiller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long struggled to learn new tasks without a human standing right beside them, guiding every movement. For years, the most successful approach has been to show a robot thousands of hours of human demonstrations, teaching it to copy those specific motions. This works well for simple, repetitive jobs, but it fails when a robot faces a situation it has never seen before, like figuring out how to put a specific fruit into a bowl based on a riddle about what a monkey likes to eat. The alternative, letting a robot learn by trial and error, is often too slow and dangerous for the real world, as the robot might need millions of attempts to succeed. The challenge for scientists is finding a way to give robots the ability to think through a problem and learn quickly, without needing a human to record every single step.

A team of researchers at Google DeepMind has developed a new method called Eximo to solve this problem. They started with a robot that already knew how to perform basic skills, such as picking up a banana or placing a cup, but could not figure out how to combine these skills for complex, multi-step goals. To bridge this gap, they introduced a second, highly intelligent computer program that acts as a strategic planner. This planner, known as a vision-language model, can look at the robot's surroundings and understand the overall goal, such as "put the plate and bowl on the rack." Instead of trying to move the robot's arms directly, this planner breaks the big goal down into a series of simple, natural language instructions, like "pick up the blue plate with your left hand," and passes them to the robot.

The process happens in three distinct stages. First, the researchers let the robot and the planner work together to explore new tasks. The planner watches the robot's progress through a camera feed and constantly updates its instructions, guiding the robot through the steps it needs to take to succeed. When the robot completes a task successfully, the team saves that sequence of events. In the second stage, they use these saved successes to teach the robot directly. They take the robot's original brain and fine-tune it using the successful examples, effectively showing it how to solve the problem on its own without needing the planner to whisper instructions in its ear. Finally, in the third stage, they let the robot practice the task again, this time using a learning method that allows it to make small adjustments to its movements to become even more precise and reliable.

The results of this approach were tested in a simulated environment with twenty-two different tasks, ranging from placing dishes on a rack to sorting tools into specific compartments. The researchers found that the robot, when guided by the planner during the initial exploration, was able to solve these complex tasks much more often than a robot trying to figure them out alone. More importantly, after the robot was taught using the successful examples from the planner, it became significantly better at the task than even the most advanced robots currently available. The robot learned to solve problems that required reasoning, such as identifying an object by its description rather than its name, and it did so with far fewer attempts than traditional methods would require.

This work demonstrates that combining a robot's physical skills with a powerful, thinking computer program can create a much more capable learner. The key insight is that the robot does not need to keep the thinking program running forever; instead, it can learn from the program's guidance and then perform the task independently. By using this method, the researchers showed that robots can be taught new, complex behaviors quickly and efficiently, moving a step closer to machines that can adapt to the unpredictable nature of the real world without needing constant human supervision.

Technical Summary: Eximo: VLM Guided Exploration of VLA Policies

Problem Statement

The paper addresses the challenge of efficiently finetuning robotic policies to learn new, complex tasks on the fly without relying on expensive human teleoperation. While state-of-the-art robotic manipulation policies are based on large-scale behavior cloning (LBC) of Vision-Language-Action (VLA) models, these models struggle with long-horizon, compositional, or reasoning-heavy tasks that lie outside their training distribution.

Two primary existing approaches have limitations:

Reinforcement Learning (RL): While capable of self-improvement, RL is notoriously sample-inefficient, particularly for long-horizon tasks where exploration is difficult. Applying RL directly to massive VLA models is further complicated by their size and architectural design (e.g., diffusion heads).
Teleoperation-based Behavior Cloning: Collecting new teleoperation datasets for every new task requires hundreds of hours of expensive human labor and fails to capture the full diversity of real-world scenarios.

The authors propose a setting where a pretrained, language-conditioned VLA policy is adapted to new tasks specified only by natural-language goals and a success detector, aiming for sample efficiency without additional teleoperation.

Methodology: Eximo

The authors propose Eximo, a three-stage algorithm designed to bridge the gap between semantic understanding and motor control. The pipeline consists of Explore, Imitate, and Optimize.

1. Explore: VLM-Guided Data Collection

In this phase, the system leverages a state-of-the-art Vision-Language Model (VLM) to act as a high-level planner and orchestrator for the VLA.

Mechanism: The VLM (specifically Gemini) is provided with the environment's visual history and the natural language goal. It decomposes the complex, long-horizon goal into a sequence of tractable, short-horizon sub-goals or instructions.
Interaction: The VLM operates in a closed loop, providing step-by-step natural language instructions to the VLA policy (e.g., "pick up the blue plate with your left hand"). The VLA executes these instructions.
Data Filtering: The system rolls out these orchestrated policies. Only episodes that successfully complete the task (verified by a ground-truth success detector) are stored in a data buffer. Unsuccessful attempts are filtered out.
Goal: To collect high-quality, diverse trajectories for new tasks without human intervention, leveraging the VLM's world knowledge for semantic exploration.

2. Imitate: Supervised Finetuning (SFT)

The collected successful trajectories are used to distill the VLM's orchestration knowledge back into the VLA.

Process: The VLA is finetuned via supervised behavior cloning on the filtered dataset.
Distillation: Crucially, the VLA is trained to predict actions conditioned directly on the original task goal ( $g$ ), rather than the intermediate instructions provided by the VLM. This allows the VLA to internalize the multi-step reasoning and chaining of skills demonstrated by the VLM.
Outcome: The resulting policy becomes a standalone agent capable of executing the new task without requiring online VLM queries, thereby eliminating latency and reducing computational overhead during deployment.

3. Optimize: Residual Online RL

The final stage further refines the policy using online Reinforcement Learning.

Challenge: Directly training large VLAs via RL is difficult due to their size and the sparsity of rewards in long-horizon tasks.
Approach: The authors employ a residual policy architecture. A smaller residual policy ( $\pi_{ref}$ ) learns to output a correction term ( $\Delta a$ ) based on the current state, goal, and the action predicted by the finetuned VLA ( $a_{VLA}$ ). The final executed action is $a = a_{VLA} + \Delta a$ .
Training: The residual policy is trained using off-policy RL (specifically the MPO algorithm) to maximize the probability of success. The pre-finetuned VLA provides a strong initialization, making the exploration for the residual policy more sample-efficient.

Key Contributions

Eximo Framework: A novel three-stage pipeline (Explore-Imitate-Optimize) that combines VLM-guided exploration with VLA finetuning and residual RL.
VLM as Orchestrator: Demonstrating that VLMs can effectively orchestrate VLA policies in natural language to solve complex, long-horizon tasks without requiring explicit grounding to robot APIs or pre-defined skill sets.
Knowledge Distillation: Showing that the strategic distillation of VLM-guided trajectories into the VLA via SFT significantly improves performance and sample efficiency, creating a standalone policy that outperforms both the base VLA and the VLM-orchestrated VLA at inference time.
Residual RL Integration: Successfully integrating online RL with VLA finetuning via a residual policy, achieving further performance gains while maintaining sample efficiency.

Experimental Results

The authors evaluated Eximo on 22 manipulation tasks in simulation using the ALOHA robotic platform. The tasks included dish placement, reasoning-based object selection (e.g., "the fruit the monkey likes"), and spatial reasoning (e.g., left vs. right compartments).

Exploration Efficiency: VLM orchestration significantly increased the success rate of the base VLA (GROD) and reduced episode lengths compared to the base VLA alone, demonstrating more efficient data collection.
Finetuning Gains: Supervised finetuning (SFT) on the VLM-orchestrated data drastically improved the base VLA's performance. The finetuned VLA outperformed the VLM-orchestrated VLA at inference time, proving the success of the distillation process.
RL Performance: When online RL was applied, the "GROD + SFT + RL" configuration achieved the highest success rates and fastest convergence. It significantly outperformed the base GROD model even when the base model was trained for many more environment steps.
Ablation: The paper confirms that all three stages contribute to the final performance. Specifically, distilling the VLM's behavior into the VLA is superior to keeping the VLM as an online orchestrator during evaluation.

Significance and Claims

The paper claims that Eximo establishes a new paradigm for training VLAs. By combining the general world knowledge of VLMs with the sensory-motor skills of VLAs, the approach enables robots to acquire new skills without expensive teleoperation hours.

The authors emphasize that this method allows for:

Sample Efficiency: Drastically reducing the data required to learn new tasks compared to pure RL or pure teleoperation.
Autonomy: Creating a fully autonomous learning loop where the robot can adapt to new, reasoning-heavy tasks specified only by natural language.
Deployment Viability: Producing a standalone policy that does not rely on the high-latency VLM orchestrator during execution, making it suitable for real-time control.

The work suggests that future systems can leverage VLMs not just as planners, but as tools for self-distillation, enabling VLAs to continuously expand their capabilities through self-generated, high-quality data.