🤖 AI

MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation

MemoryVAM introduces an episodic memory mechanism with a Recap-Cue module that injects compressed historical context into video-world-model policies, enabling robots to overcome non-Markovian challenges and significantly improve long-horizon manipulation success rates on both simulated and real-world tasks.

Original authors: Yuxin Jiang, Chang Yu, Yunuo Chen, Xiang Feng, Yin Yang, Nishank Gite, Chenfanfu Jiang

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Yuxin Jiang, Chang Yu, Yunuo Chen, Xiang Feng, Yin Yang, Nishank Gite, Chenfanfu Jiang

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to perform a complex task, like "pick up a red block three times and put it down."

If you give the robot a camera that only shows the last few seconds of video, it gets confused. When it sees the red block on the table for the second time, it looks exactly the same as the first time. Without a memory of what happened before, the robot doesn't know if it has already done this once or twice. It might try to pick it up a fourth time, or stop too early. In the world of robotics, this is called a "non-Markovian" problem: the current picture isn't enough to decide the next move; you need the story of the past.

The paper MemoryVAM introduces a solution: giving the robot a "mental scrapbook" (episodic memory) that it can flip through while it works.

Here is how it works, broken down into simple concepts:

1. The Problem: The Robot Has Short-Term Amnesia

Current robot brains (called "Video World Models") are great at predicting what will happen next based on what they see right now. They act like a movie director who can guess the next scene based on the current frame.

The Flaw: They only look at a short window of time (like the last 5 seconds). If a task takes 50 seconds and involves repeating the same motion, the robot forgets the beginning of the movie. It sees a familiar scene and repeats the action, even if the task is already done.

2. The Solution: The "Recap-Cue" System

The authors built a system called MemoryVAM that acts like a helpful assistant sitting next to the robot's brain. This assistant has two main jobs:

The Recap Compressor (The Summarizer):
Imagine you are watching a long movie. Instead of remembering every single frame, you write a short summary of the key plot points as you go. The robot does this too. It takes all the video history of the episode and compresses it into a few "memory tokens" (tiny, efficient notes).
- Analogy: It's like turning a 2-hour movie into a 3-sentence plot summary that the robot can read instantly.
The Cue Gate (The Finish Line Checker):
This is a small module that constantly asks, "Are we done yet?" It looks at the summary notes and the current instructions to decide if the task is complete.
- Analogy: It's like a race official checking a runner's bib number to see if they have finished the required number of laps, rather than just looking at where they are standing right now.

3. How It Connects: The "Dual Injection"

The clever part of this paper is that the memory isn't just fed to the part of the robot that moves its arm. It is injected into two places at once:

The "Imagination" Engine (Video Backbone): The robot uses its memory to predict the future. If the robot remembers it has already picked up the block twice, its "imagination" of the future will show the block being picked up for the third time, not the first. This ensures the robot's prediction of "what happens next" matches the reality of "where we are in the story."
The "Action" Engine (Action Decoder): The robot uses the same memory notes to decide what to do right now.

The Metaphor: Think of a chef cooking a 3-course meal.

Without Memory: The chef looks at the stove, sees a pot boiling, and adds salt. They don't know if this is the soup (Course 1) or the sauce (Course 3). They might add salt to the dessert.
With MemoryVAM: The chef has a recipe card (the memory) that says, "We are on Course 3." The chef uses this card to predict what the sauce should look like next (Imagination) and to decide to add sugar instead of salt (Action).

4. The Results: From Confused to Competent

The researchers tested this on a benchmark called LIBERO-Mem, which is full of tasks that require remembering history (like counting, finding hidden objects, or repeating actions).

Before MemoryVAM: The robots were essentially guessing. They succeeded only 5% of the time on average.
After MemoryVAM: The robots succeeded 42.5% of the time.
On Real Robots: When they tried this on actual physical robots (not just simulations), the success rate jumped even higher for specific tasks:
- Counting tasks: 78.3% success.
- Finding hidden objects: 80.0% success.
- Tracking sequences: 75.0% success.

5. Why This Matters

The paper claims that by treating memory as a core part of the robot's "world model" (how it understands the world), rather than just an add-on, the robot learns much better. It doesn't need a human to label every single step of the task as "step 1," "step 2," etc. The robot learns to track its own progress by trying to predict the future accurately. If the robot predicts the future correctly, it proves it understands the history.

In short: MemoryVAM gives robots a "storybook" of their own actions, allowing them to know exactly where they are in a long task, preventing them from getting stuck in a loop or forgetting what they've already done.

Technical Summary: MemoryVAM

Problem Statement

Video-world-model policies learn action-relevant representations by predicting future observations, conditioning on a short, fixed-length observation window. While effective for approximately Markovian tasks, this design fails in long-horizon, multi-stage manipulation tasks where the correct action depends on earlier events that are no longer visible in the current window. In such scenarios (e.g., "pick up the red block three times"), visually similar states require different actions based on hidden task progress (e.g., the current count). Without episodic memory, these tasks become non-Markovian, causing policies to fail when they cannot distinguish between different stages of a repetitive sequence or recall spatial history.

Methodology: MemoryVAM

The authors propose MemoryVAM, an episodic memory mechanism designed to integrate history into video-world-model policies. The core innovation is dual memory conditioning, where a compact set of memory tokens influences both the video backbone's future predictions and the action decoder's control decisions.

1. Episode History Encoding

At each timestep $t$ , the system maintains an episode history $\mathcal{H}_t$ consisting of CLIP visual embeddings of past frames. Using CLIP embeddings allows the memory to track task-relevant semantic changes rather than low-level pixel differences.

2. The Recap-Cue (RC) Module

The RC module processes the variable-length history into a fixed-size memory representation:

Recap Compressor: A Perceiver-based Resampler maps the episode history into a compact matrix of memory tokens $\mathbf{G}$ . Using learned queries, it selectively attends to events explaining task progress, capturing both what happened and when it happened via order-aware positional encoding.
Cue Gate: A lightweight module estimates a task completion score $q_\omega$ based on pooled backbone features, memory tokens, and language instructions. This enables autonomous task termination without requiring per-frame progress labels during training; supervision is derived from episode boundaries in demonstrations.

3. Dual Memory Conditioning (Injection)

The memory tokens $\mathbf{G}$ are injected via two complementary pathways:

Pathway 1 (Video Backbone): $\mathbf{G}$ $G$ is injected into the video backbone's cross-attention layers.
- For Cosmos DiT backbones, memory tokens are appended to the text conditioning matrix.
- For SVD UNet backbones (e.g., VPP), a separate memory branch is introduced alongside the text branch.
- Effect: This ensures that the predicted future frames and the intermediate backbone features are conditioned on the full episode history, aligning the model's "imagination" with task progress.
Pathway 2 (Action Decoder): The same memory tokens $\mathbf{G}$ $G$ are concatenated with the observation tokens (derived from the backbone) and fed into the action decoder.
- Effect: This directly constrains the action distribution based on the episodic memory, ensuring the policy selects controls consistent with the history.

4. Training Strategy

The model employs a two-stage training procedure:

Stage 1: Trains the video backbone and memory module using video prediction objectives, a delta-reconstruction auxiliary loss, and the Cue Gate loss. This teaches the memory to encode information useful for predicting future states and determining task completion.
Stage 2: Trains the action policy on the resulting memory-conditioned features while keeping the backbone stable.

The mechanism is architecture-agnostic, demonstrated on both UNet (VPP) and Diffusion Transformer (DiT) (Cosmos) backbones.

Key Contributions

Dual Injection Interface: A two-pathway mechanism that integrates compact Recap tokens into both the video backbone and the action decoder, allowing history to jointly influence imagined futures and control features.
Recap-Cue (RC) Module: A novel module combining a Perceiver-based compressor for history summarization and a Cue Gate for autonomous task completion, trained without explicit per-step progress labels.
Backbone Agnosticism: The method is validated across distinct video backbone paradigms (UNet and DiT), demonstrating that episodic memory is a general requirement for long-horizon video-world-model policies.

Experimental Results

The authors evaluated MemoryVAM on simulation benchmarks and real-robot tasks requiring episodic recall.

LIBERO-Mem Benchmark:
- On tasks requiring episodic recall (counting, sequencing, spatial reasoning), the model improved average success rates from 5% to 42.5% for the UNet backbone and from 3.0% to 40.5% for the DiT backbone.
- Ablation studies showed that using only Pathway 1 (backbone injection) yielded 28.5% success, while Pathway 2 (action decoder injection) alone yielded only 7.5%. Full dual injection was necessary for optimal performance.
- MemoryVAM outperformed memory-augmented VLA baselines (e.g., MemoryVLA, Gated Memory Policy) and memoryless baselines.
Real-Robot Experiments:
- Toy Pickup ( $k\times$ ): 78.3% success on counting tasks (1x, 3x, 5x repetitions).
- Place and Return: 80.0% success on spatial recall tasks.
- Shell Game: 75.0% success on sequential tracking tasks involving occluded objects.
- In contrast, memoryless policies collapsed (0% success) on these tasks when task progress was not directly observable.
Video Prediction Quality:
- Memory injection significantly improved the fidelity of long-horizon autoregressive rollouts. For example, on the UNet backbone, FVD scores improved from 43.36 to 18.32 at a single horizon and from 37.81 to 19.84 at an 8x horizon.
- The Cue Gate successfully reduced the "stop ratio" (post-completion motion), preventing the model from imagining unnecessary actions after a goal is reached.

Significance and Claims

The paper posits that episodic memory should be treated as an integral component of the world-model state, rather than merely an auxiliary input to the action policy. By allowing episode history to shape both what the robot imagines (future prediction) and how the policy acts (control decoding), MemoryVAM addresses the non-Markovian nature of long-horizon tasks.

The authors claim that this approach offers a scalable path for memory systems, as the memory module is supervised by the generative video objective (predicting future frames) rather than relying solely on action-labeled demonstrations. This allows the memory to learn task progress signals that are consistent with the visual dynamics of the environment. The work demonstrates that explicit episodic memory is crucial for diverse video backbone architectures to succeed in complex, multi-stage robotic manipulation.