🤖 machine learning

RECALL: Recovery Experience Collection for Active Lifelong Learning in Vision-Language-Action Models

This paper proposes an active, uncertainty-guided continual learning paradigm for Vision-Language-Action models that improves adaptation efficiency through targeted recovery demonstrations, while empirically analyzing the tradeoffs between plasticity and catastrophic forgetting when integrating such data.

Original authors: Ulas Berk Karli, Tesca Fitzgerald

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ulas Berk Karli, Tesca Fitzgerald

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant that has already learned how to do many chores, like folding laundry or setting the table. This robot is a "Vision-Language-Action" (VLA) model—it sees the world, understands your voice commands, and moves its arms to get things done.

However, when you put this robot in a new room or give it a slightly different task, it sometimes trips up. The old way of fixing this was Passive Learning: you'd wait for the robot to fail, then have a human step in, show it the whole task from the very beginning, and hope that helps.

This paper proposes a smarter way called Active Learning, and then solves a tricky problem that comes with it. Here is the breakdown using simple analogies:

1. The Problem with the "Old Way" (Passive Learning)

Imagine you are teaching a student to drive.

The Passive Method: You let the student drive until they almost crash. Then, you stop them, take the wheel, and drive the entire route from the start to the finish, even though they knew how to drive the first half perfectly.
The Waste: This is inefficient. The student wastes time re-learning what they already know, and you only get help after a mistake happens, which is dangerous.

2. The New Idea: "Uncertainty-Guided" Active Learning

The authors suggest a better approach using a "help detector" (a tool called INSIGHT).

The Analogy: Imagine the student has a panic button. They only press it when they feel unsure or about to make a mistake.
The Fix: When the robot feels "uncertain" (like a student who doesn't know the next turn), it stops. A human then comes in only at that specific moment to show the robot how to recover and finish the task.
The Result: The paper found that this is much more efficient. By only collecting data when the robot is confused, the robot learns faster and better than if you just collected random data from the start of every task.

3. The Big Trap: "Catastrophic Forgetting"

Here is the twist. When the robot learns only from these "panic button" moments (the recovery data), it gets really good at fixing mistakes, but it starts forgetting how to do the easy parts it used to know.

The Analogy: It's like a student who studies only the hardest questions on a practice test. They get great at solving the hard problems, but when they take the real test, they forget how to answer the easy questions they used to get right every time.
The Paper's Finding: If you only train the robot on the "recovery" data, it forgets its old skills. This is called Catastrophic Forgetting.

4. The Solutions: How to Keep the Old Skills

The authors tested several ways to fix this forgetting problem so the robot can learn new tricks without losing old ones.

Solution A: The "Low Learning Rate" (Taking Small Steps)
- Analogy: Instead of a heavy workout that changes your muscles drastically, you do light stretching. It helps you adapt without losing your current shape.
- Result: It helps a little, but the robot still doesn't learn the new tricks very well.
Solution B: "Elastic Weight Consolidation" (The Elastic Band)
- Analogy: Imagine the robot's brain has elastic bands holding its old knowledge in place. When it tries to learn something new, the bands pull back if it tries to change the old stuff too much.
- Result: It keeps the old skills safe, but it also makes it hard for the robot to learn the new recovery skills. It's a trade-off: too much safety, and you don't learn; too little safety, and you forget.
Solution C: "Replay" (The Best Solution)
- Analogy: This is like a student studying for a test by mixing the new hard questions with a few old easy questions they already know.
- Result: This worked the best. By showing the robot a mix of the new "recovery" data and some of its old "successful" data, it learned the new tricks without forgetting the old ones.

5. A Surprising Discovery: "Online" vs. "Offline"

The authors also asked: Do we need to record every single moment the robot is unsure, or just the first time it gets stuck?

The Finding: You only need to record the first time the robot gets stuck.
The Analogy: If a student gets lost at the first turn of a maze, showing them how to fix that one turn is enough to get them through the rest of the maze. You don't need to record every single wrong turn they make after that first mistake. This saves a huge amount of human effort.

Summary of the Paper's Main Points

Be Smart About Data: Don't just collect data randomly. Collect it only when the robot is confused (uncertain). This is more efficient.
Don't Forget the Basics: If you only train on the "fixes," the robot forgets how to do the easy stuff.
Mix It Up: The best way to teach the robot is to mix the new "fix" data with some of its old "success" data (Replay).
Stop Early: You only need to collect data from the first moment of confusion, not every moment after that.

The Bottom Line: To make robots that can learn continuously without forgetting, we need to let them ask for help when they are confused, but we must also remind them of what they already know while they learn the new stuff.

Technical Summary: RECALL – Recovery Experience Collection for Active Lifelong Learning in Vision-Language-Action Models

Problem Statement

Vision-Language-Action (VLA) models, such as RT-1, RT-2, and $\pi_0$ -FAST, have demonstrated broad manipulation capabilities but often require fine-tuning when deployed in new environments or task distributions. Current adaptation relies on passive imitation learning, a process where experts collect additional demonstrations starting from task initial states only after the policy fails. This approach suffers from three primary inefficiencies:

Reactive Nature: Data collection is triggered only after failure, which can be time-consuming and unsafe.
Lack of Guidance: It provides no indication of which specific states require supervision, leading to potential under- or over-collection.
Redundancy: Demonstrators often waste effort on states already well-represented in the original training data, failing to address the specific sub-steps or distribution-shifted intermediate states that cause failure.

Furthermore, simply fine-tuning on newly collected data introduces the risk of catastrophic forgetting, where the model loses proficiency in previously learned tasks. The paper posits that active, continual learning paradigms are necessary to address these issues.

Methodology

The authors propose RECALL, an active, continual learning framework for autoregressive VLAs (specifically $\pi_0$ -FAST). The methodology consists of two main components:

1. Active Data Collection via Uncertainty Quantification

The system leverages INSIGHT [6], an inference-time introspection method that uses token-level probability features (entropy, log-probability, and Dirichlet-based uncertainty estimates) to predict when a robot requires assistance.

State Selection: Instead of collecting from initial states, the system identifies high-uncertainty states during policy rollouts.
Collection Modes:
- Online: The robot requests a demonstration immediately upon encountering the first high-uncertainty state in a rollout.
- Offline: The robot completes a full rollout, recording all high-uncertainty states to prompt for demonstrations later.
Baselines: These are compared against Passive collection, where demonstrations are gathered from task initial states, matching the volume and task distribution of the active datasets.

2. Continual Learning Strategies

To prevent catastrophic forgetting while adapting to new recovery data, the paper evaluates several strategies:

Replay-based Data Mixing: Fine-tuning on a mixture of new recovery data ( $D_{new}$ ) and prior data ( $D_{old}$ ). Variants include full replay, LIBERO-10 replay, and targeted replay (only tasks relevant to the new data).
Regularization:
- Elastic Weight Consolidation (EWC): Penalizes changes to parameters important for previous tasks using a Fisher information matrix.
- Learning Rate Reduction: Using a low, constant learning rate to limit the magnitude of updates.

Key Contributions

The paper makes five primary contributions:

Active Framework: An active data collection pipeline for autoregressive VLAs using INSIGHT to select uncertain states for prompting new demonstrations.
Active vs. Passive Evaluation: Empirical evidence that demonstrations from high-uncertainty states yield better policy performance than unguided, passive demonstrations.
Online vs. Offline Efficiency: A finding that collecting data from the first uncertain state (online) is as effective as collecting from all uncertain states (offline), offering higher demonstration efficiency.
Continual Learning Analysis: A comparative study showing that replay-based data mixing is currently the most reliable strategy for preventing catastrophic forgetting, outperforming EWC and low learning rates in balancing plasticity and stability.
Design Lessons: Identification of open challenges in incorporating targeted new data into large robot policies, specifically the trade-offs between adaptation efficiency and retention of prior skills.

Experimental Results

The study was conducted on the LIBERO-10 benchmark using the $\pi_0$ -FAST model. Key findings include:

Superiority of Active Collection: Online recovery collection guided by Strong INSIGHT achieved 72.4% overall success, significantly outperforming matched passive collection (60.2%) and the original baseline (59.8%).
Efficiency of Online Collection: Online collection (one demonstration per rollout) achieved performance comparable to offline collection (all uncertain states), with no statistically significant difference in overall success (72.4% vs. 68.4%). This suggests the first uncertain state marks the divergence from successful behavior, making subsequent dense collection redundant.
Catastrophic Forgetting in New-Only Training: Fine-tuning only on new recovery data caused severe forgetting. For instance, a new-only model achieved only 28.4% overall success compared to 72.4% for the replay-based model, with retained-task performance collapsing.
Limitations of Regularization: While EWC and low learning rates reduced forgetting compared to standard new-only fine-tuning, they failed to match the performance of replay-based mixing. Strong regularization preserved old behaviors but suppressed adaptation to new data; weak regularization allowed adaptation but risked forgetting.
Replay Coverage: Full replay and LIBERO-10 replay (using the full prior dataset) performed similarly. However, targeted replay (using only prior data from the specific tasks being recovered) significantly degraded performance on retained tasks, indicating that replay must cover the full spectrum of previously learned behaviors, not just the specific tasks being updated.

Significance and Claims

The paper claims that active learning for VLAs cannot be reduced merely to selecting high-value states; it must be formulated as active continual learning. The core argument is that while uncertainty predictors (like INSIGHT) effectively identify informative states for recovery, the resulting data is distributionally narrow. Therefore, effective adaptation requires a dual mechanism:

Targeted Data Collection: Gathering demonstrations specifically from high-uncertainty states to address failure modes.
Retention Mechanisms: Integrating this new data via replay-based mixing to preserve previously learned competencies.

The work establishes that uncertainty-guided recovery demonstrations can significantly improve adaptation efficiency but highlights that without robust continual learning strategies (specifically replay), targeted data incorporation leads to catastrophic forgetting. The authors note that their findings are based on simulation (LIBERO-10) and one autoregressive architecture, suggesting that physical deployment and other VLA architectures (e.g., diffusion-based) may present different challenges regarding uncertainty calibration and data collection costs.