💻 computer science

Adaptation of Generalist Robot Policies with Minimal Data

This paper introduces MiDAS, a simple offline-to-online reinforcement learning framework that enables generalist robot policies to rapidly adapt to new tasks and achieve robust autonomous improvement starting from as little as a single human demonstration.

Original authors: Shreyas Kowshik, Sreyas Venkataraman, Leo Wang, Niharika Pant, Max Simchowitz, Aviral Kumar

Published 2026-08-13

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Shreyas Kowshik, Sreyas Venkataraman, Leo Wang, Niharika Pant, Max Simchowitz, Aviral Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots don't just follow a rigid script written by a human engineer, but instead learn and improve on their own, just like we do. This is the holy grail of robotics: building machines that can walk into a new room, figure out what needs to be done, and get better at it through trial and error. To understand how this works, think of a robot's brain as having two main parts. First, it needs a "general knowledge" base, like a library of how the world works, which it learns by watching millions of videos of people doing things. Second, it needs to learn specific skills, like how to open a specific jar or stack a specific set of blocks. The big challenge scientists face is that while robots are getting great at that general knowledge, they are still terrible at learning new specific tasks on their own. If you ask a robot to do something it hasn't seen before, it often just freezes or flails around because it doesn't know where to start. It's like giving a person a map of the entire world but no idea which street they are currently on; they have all the tools to navigate, but they can't take the first step without a little help.

This is the story of a new method called MiDAS, which tries to solve that "first step" problem. The researchers asked a simple question: What if we only gave the robot one single video of someone successfully doing the task, and then let the robot try to figure out the rest on its own? Usually, one video isn't enough. It's like watching someone solve a Rubik's cube once and then being told to solve it yourself; you might know the general idea, but you'll likely fail a lot before you get it right. The paper suggests that by combining that single video with a smart way of learning from mistakes, a robot can actually learn to do the task reliably. They tested this in computer simulations and on a real robot with two arms, finding that even with just one example, the robot could learn to succeed after a few hours of practice, whereas other methods would have given up immediately.

The Problem: The Robot's "Blank Stare"

Imagine you are teaching a robot to make a sandwich. You have a super-smart robot that has watched millions of videos of humans cooking. It knows what bread looks like, what cheese is, and how a knife works. This is its "pre-trained" brain. But if you walk into the kitchen and say, "Make me a sandwich with the blue cheese on the left," the robot might just stare at you. It knows the ingredients, but it doesn't know this specific sandwich.

If you let the robot try to learn by itself (autonomous learning), it will probably fail over and over. It might grab the wrong bread, drop the cheese, or try to put the knife in the toaster. In the world of robotics, these failures are called "sparse rewards." It's like playing a video game where you only get a "You Win!" message at the very end, and you get no hints or points for getting closer. If the robot fails 99 times out of 100, it never gets that "You Win!" signal, so it never learns what to do. It's stuck in a loop of failure.

The Solution: One Video and a "Residual" Helper

The researchers behind MiDAS (Minimal-Data Adaptation Strategy) came up with a clever two-step recipe to break this loop.

Step 1: The "Anchoring" Video
First, they show the robot just one successful video of the task. They don't just let the robot watch; they use a technique called "behavior cloning." Think of this as the robot taking a quick, intense study session. It looks at the one video and tries to copy the movements.

What happens: The robot gets much better. It stops staring blankly. It knows to grab the bread, then the cheese, then the knife.
The catch: It's still clumsy. It might grab the bread correctly but drop the cheese because its grip is too tight, or it might miss the plate because it's slightly off-center. It has the idea of the task, but it lacks the precision. In the paper's tests, this "one-video" robot could get close to the goal but often failed to finish the job.

Step 2: The "Residual" Coach
This is where the magic happens. Instead of trying to re-teach the robot everything from scratch, MiDAS adds a tiny, lightweight "coach" on top of the robot's brain.

How it works: The robot's main brain (the one trained on the one video) says, "I think I should move the arm this way." The new "coach" (called a residual policy) listens to that suggestion and says, "Actually, move it just a tiny bit more to the left."
The Learning: The robot then tries this adjusted move. If it succeeds, the coach gets a point. If it fails, the coach learns to adjust differently next time.
Why it's special: The main brain stays frozen (it doesn't change), so the robot doesn't forget what it learned from the video. The coach is small and fast, learning only the tiny corrections needed to turn a "good try" into a "perfect success."

The Results: From One Try to Mastery

The team tested this idea in two different virtual worlds (called LIBERO-Long and RoboCasa) and on a real robot with two arms (called YAM).

In the Simulations:
The results were surprisingly strong. With just one demonstration:

Standard methods that tried to learn from scratch or just copy the video failed almost completely (0% success).
Other advanced methods that tried to tweak the robot's behavior managed to get about 30% to 40% of the tasks right.
MiDAS succeeded in 91% of the tasks on average. In some specific tasks, like putting a pot on a stove, it reached 98% success.

In the Real World:
They took the method to a real robot with two arms.

Task 1: Placing a green block in one container and a blue block in another. The robot started with a 40% success rate after seeing the one video. After about 6 hours of practicing on its own, it improved to 67%.
Task 2: Placing a knife and a donut on a plate. This was harder. The robot started with only a 27% success rate. But after 5 to 6 hours of autonomous interaction, it jumped to 80% success.

The paper suggests that this works because the robot's pre-trained brain already knows how to move its arms and what objects look like. The single video tells it what to do. The online learning (the coach) just fills in the gaps for how to do it perfectly.

What This Means (and What It Doesn't)

The researchers are careful to point out what this method can and cannot do.

What it does well:

It turns a "maybe" into a "yes" for tasks the robot already has some general knowledge about.
It allows the robot to generalize. If you move the objects slightly or change the lighting, the robot can still figure it out because its "brain" is smart enough to handle those changes.
It works with very little human help. You don't need to record 100 videos; one is enough to get started.

What it doesn't do (yet):

It can't teach the robot to do something completely new that it has no prior knowledge of. If the robot has never seen a knife or a donut before, one video won't be enough.
It struggles if the task requires a completely different way of moving. For example, if the robot needs to swap the positions of two objects in a way that contradicts the video, it might get confused.
The "6 hours" of practice is still a long time for a robot. The paper suggests that while this is a huge step forward, we still need to make the learning faster and more efficient.

The Big Picture

Think of MiDAS as a bridge. Before this, there was a huge gap between "robots that know everything in general" and "robots that can do a specific job." You either needed a human to program every single move, or you needed hundreds of hours of trial and error. MiDAS shows that if you give a robot a tiny bit of human guidance (one video) and let it practice with a smart, lightweight coach, it can cross that gap on its own.

The authors suggest that this is the first time anyone has shown that a robot can reliably adapt to a new task starting from just a single demonstration. It's not a magic wand that solves every problem, but it's a powerful new tool that suggests we are getting closer to robots that can truly learn from us, with just a little bit of help.

Technical Summary: Adaptation of Generalist Robot Policies with Minimal Data

Problem Definition: Minimal-Data Adaptation (MDA)

The paper addresses the challenge of enabling generalist robot policies to adapt to new tasks with minimal human supervision. While recent Vision-Language-Action (VLA) models pre-trained on diverse datasets offer broad priors, they often fail as zero-shot learners in new environments due to sparse rewards and weak exploration. Conversely, fully autonomous learning from scratch is intractable for current policies.

The authors propose the Minimal-Data Adaptation (MDA) regime as a tractable proxy for fully autonomous improvement. In this setting, a pre-trained policy must learn a new task using only $K$ successful demonstrations (where $K=1$ is the sharpest case) followed by autonomous online interaction. The goal is not to solve the task solely from demonstrations, but to use them as a scaffold to make subsequent autonomous learning feasible. The core challenge is that with $K=1$ , the initial policy is often too weak to reach reward-bearing regions consistently, leading to policy collapse under sparse rewards.

Methodology: MiDAS

The authors introduce MiDAS (Minimal-Data Adaptation Strategy), a two-stage offline-to-online reinforcement learning recipe designed to bridge the gap between a weak pre-trained policy and a robust, task-specific controller.

Stage I: Behavior Cloning (BC) for Task Anchoring

Objective: Transform the zero-shot pre-trained policy ( $\pi_{base}$ ) into a task-directed policy ( $\pi^K_{base}$ ) that can roughly attempt the task, even if execution is brittle.
Mechanism: The pre-trained VLA backbone is fine-tuned using Low-Rank Adaptation (LoRA) on the $K$ demonstrations, while the action head is fine-tuned using standard flow matching.
Outcome: This stage recovers coarse, task-relevant behavior (e.g., approaching the correct object) but fails to achieve reliable control (e.g., precise grasping or alignment). It serves to "anchor" the policy to the task, narrowing the search space for online learning.

Stage II: Online Improvement via Residual RL

Objective: Learn fine-grained corrections to overcome the brittleness of $\pi^K_{base}$ using autonomous interaction.
Mechanism:
- Frozen Backbone: The task-anchored policy $\pi^K_{base}$ is frozen.
- Residual Actor: A lightweight residual policy ( $\pi^{res}_{\theta}$ ) is trained on top of the frozen representations. Unlike additive residual formulations that predict small offsets, MiDAS predicts the executed action directly conditioned on the base policy's proposal ( $a_{exec} \sim \pi^{res}_{\theta}(\cdot | s, a_{base})$ ). This allows the policy to shift the action distribution beyond the immediate neighborhood of the base proposal if necessary.
- Value-Based Learning: The system uses Policy-Agnostic RL (PA-RL) with a critic ensemble trained via Temporal Difference (TD) learning. The actor is updated using a "Best-of-N" distillation objective, where candidate actions are refined via gradient ascent on the Q-function before being distilled back into the policy.
Stabilization Techniques:
1. Offline Warmup: The residual actor and critic are initialized using a buffer containing the $K$ demonstrations and a few rollouts from $\pi^K_{base}$ . The actor is initialized to reproduce the base policy's actions ( $\mathcal{L}_{warm}$ ), ensuring early exploration remains task-relevant.
2. Success Balancing: To combat the sparsity of rewards (where success rates may be as low as 2-5%), the replay buffer is split into a general buffer ( $\mathcal{B}$ ) and a success buffer ( $\mathcal{B}_{succ}$ ). Critic updates sample from a mixture of these buffers, oversampling successful trajectories to prevent the value function from being dominated by failures.

Key Contributions

Definition of MDA: The paper formalizes the MDA regime, identifying it as the critical threshold where minimal supervision ( $K=1$ ) makes autonomous online learning tractable for generalist policies.
MiDAS Algorithm: A novel recipe combining few-shot behavior cloning with frozen-backbone residual RL. It demonstrates that a single demonstration can bootstrap autonomous learning when combined with pre-trained representations and value-guided correction.
Empirical Validation:
- Simulation: On LIBERO-Long and RoboCasa, MiDAS achieves high success rates (e.g., ~91% on LIBERO) from a single demonstration, significantly outperforming baselines like Filtered BC, Diffusion-Steering RL (DSRL), and DICE-RL.
- Real-World: The method was successfully transferred to a bimanual YAM platform. Starting from a fragile policy with ~27% success on a complex task (placing a knife and donut), MiDAS improved robustness to 80% after approximately 6 hours of autonomous interaction.
Mechanistic Insights: The authors empirically validate that:
- Pre-trained VLA representations are essential for sample efficiency; generic visual pretraining (e.g., DINO) or training from scratch fails to converge with $K=1$ .
- Behavior cloning provides coarse task grounding but not reliable control.
- Online residual RL is necessary to learn actions that lie outside the effective support of the few-shot policy, a capability that baselines like DSRL lack.

Results and Generalization

Performance: MiDAS consistently recovers strong performance across diverse long-horizon tasks. In the "One Demonstration" setting, it achieves near-perfect success on many LIBERO tasks where zero-shot and standard baselines fail completely.
Generalization:
- Observation Shifts: The method inherits robustness to visual (color, texture) and language shifts from the frozen VLM backbone.
- State Shifts: Online RL improves robustness to moderate state shifts (e.g., object position, shape changes) where the required manipulation strategy remains similar. However, performance degrades on shifts requiring fundamentally new strategies (e.g., object swaps or category changes), indicating the limits of a single demonstration.
- Curriculum Learning: The paper shows that progressively widening the reset distribution during online RL can recover generalization performance comparable to policies trained on 50 demonstrations, without requiring additional human data.

Significance and Claims

The paper claims to provide the first demonstration of reliable robot policy adaptation from a single task demonstration using a generalist pre-trained policy.

The authors argue that MDA represents a pivotal regime where the interplay of three components enables autonomous improvement:

Pretraining: Provides reusable features and broad action priors.
Demonstrations: Anchor the policy to the specific task, making task-relevant behavior reachable.
Autonomous Interaction: Supplies the necessary corrections to achieve robustness and handle distribution shifts.

The work suggests that modern pre-trained robot policies have reached a maturity level where they are "close enough" to task success that a single demonstration can make the remaining gap learnable through online interaction, provided the adaptation strategy (MiDAS) is designed to stabilize the learning process and expand the action distribution beyond the initial few-shot policy's support. The paper remains modest, acknowledging that while MDA works for behaviors well-represented in pretraining distributions, adapting to tasks requiring entirely new behavioral modes or dexterous manipulation may still require more supervision or different mechanisms.