💻 computer science

NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation

NativeMEM is a novel Vision-Language-Action (VLA) policy that integrates an efficient, native memory compression scheme to enable real-time, long-horizon robotic manipulation with significantly improved success rates and data efficiency, all without relying on external memory modules or incurring substantial latency overhead.

Original authors: Ziye Wang, Modi Shi, Chaojun Ni, Jiazhi Yang, Mengdi Li, Zhizhong Su, Tianwei Lin, Hongyang Li

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziye Wang, Modi Shi, Chaojun Ni, Jiazhi Yang, Mengdi Li, Zhizhong Su, Tianwei Lin, Hongyang Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to do a complex chore, like setting a table or organizing a grocery store. You give the robot a camera and a set of instructions. The problem is that most robots today are like people with very short attention spans: they only look at what is happening right now. If you ask them to "put the red cup back where it started," they might look at the cup, but if they don't remember where "started" was because they only saw the current frame, they will fail.

To fix this, previous methods tried to give the robot a notebook. But these notebooks were either too slow to read, too heavy to carry, or required a second "brain" (an external module) to write in them, which confused the robot's main brain.

NativeMEM is a new way to give robots a long-term memory without adding any extra weight or confusion. Here is how it works, using simple analogies:

1. The Problem: The "Amnesiac" Robot

Current robot brains (called VLA models) are amazing at looking at a picture and saying, "Okay, I see a cup, I should pick it up." But if the task requires remembering what happened 10 seconds ago (like, "I already pressed the blue button, so now I must press the pink one"), the robot gets lost. It forgets the story of what it just did.

2. The Old Solutions: The "Clunky Notebook"

The Text Note Approach: Some researchers tried to have a second AI write text notes about what happened ("I pressed blue") and feed that to the robot. This is like asking a robot to read a diary while trying to walk. It's slow, and the robot might miss tiny details that are hard to describe in words.
The External Hard Drive Approach: Others tried to add a separate memory chip inside the robot. This is like adding a second brain that the main brain has to constantly talk to. It makes the robot slower and more complicated, and the main brain doesn't always understand the new chip's language.

3. The NativeMEM Solution: The "One-Word Summary"

NativeMEM takes a different approach. Instead of adding a new notebook or a second brain, it teaches the robot's existing brain to summarize its own past.

The Compression Trick: Imagine you are watching a 1-hour movie. Usually, to remember it, you'd need to save the whole movie file (huge size). NativeMEM is like a super-efficient editor that watches the movie and writes down one single word for every scene that perfectly captures the most important part of that scene.
Native Language: Because this "one-word summary" is created by the robot's own eyes (its vision encoder), the robot understands it perfectly. It doesn't need to translate from a foreign language (like text notes) or talk to a new chip. The summary is just another word in the sentence the robot is already reading.

4. How They Taught It: The "Two-Step Training"

The researchers didn't just turn this feature on; they trained it in two stages:

Stage 1: Learning to Summarize. They froze the robot's main brain (so it wouldn't forget what it already knew) and trained a small helper to look at past videos and turn them into those "one-word summaries." They taught this helper by saying, "If you summarize the past this way, the robot will make the right move."
Stage 2: Putting it to Work. Once the robot learned how to read these summaries, they unlocked the main brain and let it practice specific tasks using these summaries. It's like giving the robot a cheat sheet of the past that fits perfectly into its existing thought process.

5. The Results: Super Memory, No Slowdown

The paper shows that this method is a game-changer:

Success Rate: In simulations, robots using NativeMEM went from being successful only 32% of the time to 84%. On real robots, they hit 98.7% success.
Speed: Even though the robot is remembering minutes of history (hundreds of frames), it doesn't slow down. It can look back at 160 frames of history in the time it usually takes to look at just one.
Data Efficiency: The robot learned these skills using only 20% of the training data that other methods needed. It's like learning to drive a car after just a few hours of practice instead of a whole semester.

In a Nutshell

NativeMEM is like giving a robot a "superpower" to remember its entire history without making it slow or confusing. Instead of carrying a heavy backpack of video files or reading a slow text diary, the robot compresses its entire past into tiny, efficient "memory tokens" that it can read instantly, allowing it to solve complex, long-term puzzles that it previously couldn't handle.

Technical Summary: NativeMEM – Native Memory Compression for Long-Horizon Robotic Manipulation

1. Problem Statement

Pretrained Vision-Language-Action (VLA) models have shown promise in embodied decision-making but are fundamentally limited by their reactive nature. Most existing VLAs condition actions solely on the current observation and language instruction, lacking the capacity to retain long-horizon visual histories. This limitation is critical for non-Markovian manipulation tasks where the correct action depends on past interactions, such as tracking object counts, remembering occluded states, recalling failed attempts, or monitoring task progress.

Current approaches to memory augmentation face significant trade-offs:

VLM-Driven Memory: Methods that use external high-level Vision-Language Models (VLMs) to generate textual notes or subtasks often require costly annotations and struggle to faithfully encode subtle visual details in language.
External Memory Modules: Approaches that introduce freshly initialized memory modules (e.g., recurrent states, retrieval banks) inside the policy increase architectural complexity. Since these modules are unseen during the VLA's pretraining, they risk degrading the pretrained policy's performance and introduce a "compression dilemma": insufficient compression leads to high computational costs, while excessive compression discards action-critical details.

The core challenge is to design a mechanism that allows a VLA to retain long-horizon, fine-grained visual histories with minimal memory cost while remaining fully compatible with the pretrained policy's token space and attention mechanisms.

2. Methodology: NativeMEM

The authors propose NativeMEM, a VLA policy that integrates long-term, real-time updated memory directly into the pretrained model's architecture without external planners or new memory modules. The core innovation is Native Memory Compression.

2.1 Native Memory Compression

Instead of introducing a separate memory architecture, NativeMEM repurposes the VLA's own vision encoder to compress each historical frame from every camera view into a single native memory token.

One-Token Compression: Each frame-view pair is summarized into a single token ( $M=1$ ), drastically reducing the sequence length compared to retaining full visual tokens.
Native Integration: These tokens are produced by the same visual pathway as the current observations and are appended to the original token sequence. This allows the pretrained VLA to attend to long-term history using its existing attention mechanisms, interpreting the tokens as "native visual evidence" rather than external states.

2.2 Two-Stage Training Pipeline

To ensure these compact tokens are both information-rich and aligned with the pretrained policy, the authors introduce a two-stage training process:

Stage 1: Learning a Generic Memory Tokenizer (Action-Supervised)
- The pretrained VLA is frozen.
- A memory tokenizer, initialized from the VLA's native vision encoder, is trained to compress past observations.
- Objective: The tokenizer is optimized using the VLA's original action prediction loss. This ensures the compressed tokens are "action-supervised" (encoding information relevant to predicting actions) rather than "reconstruction-supervised" (which might retain irrelevant visual details).
- Data: The tokenizer is trained on a mixture of standard manipulation data and memory-demanding tasks to learn a general representation aligned with the VLA's token space.
Stage 2: Task-Specific Finetuning
- The learned memory tokenizer is frozen.
- The VLA backbone and action head are unfrozen and finetuned on a specific target task using limited demonstrations.
- Input: The input sequence consists of the current observation, language instruction, proprioceptive state, and the appended sequence of cached memory tokens derived from the history.
- This stage converts a single-frame VLA into a memory-augmented policy using minimal task-specific data.

2.3 Real-Time Inference

During deployment, the memory tokenizer operates independently from the VLA policy. As new observations arrive, the tokenizer converts them into memory tokens at a high frequency (e.g., 10Hz+). The VLA policy simply concatenates the latest compact memory queue to its input tokens. This decoupling allows for minute-level history retention with negligible latency overhead.

3. Key Contributions

Native Memory Compression: A novel scheme that compresses historical frames into single tokens using the VLA's own vision encoder, eliminating the need for external memory modules or planners.
Action-Supervised Tokenization: A training strategy that learns memory representations directly through the VLA's action prediction objective, ensuring the compressed history is relevant for decision-making.
Efficient Long-Horizon Retention: The method enables attention over minute-level histories (over 160 frames) with a 9× to 40× longer horizon than prior methods, while maintaining low inference latency and GPU memory usage.
High Data Efficiency: The approach achieves competitive performance with prior state-of-the-art methods using only 20% of the training data.

4. Experimental Results

The authors evaluated NativeMEM on memory-dependent manipulation tasks in both simulation and the real world.

Simulation Performance: On a benchmark including RMBench tasks and button-pressing tasks, NativeMEM achieved an average success rate of 84.0%, significantly outperforming the best baseline (Mem-0 at 32.4%) and other memory-augmented methods.
Real-World Performance: On real robots, NativeMEM achieved a success rate of 98.7% across three tasks (Click Buttons, Put Back Block, and Grocery Checkout Scanning). Notably, it performed well on the "Grocery Checkout Scanning" task, which was unseen during the first-stage tokenizer training, demonstrating strong generalization.
Efficiency:
- Inference: NativeMEM supports up to 5,000 historical frames within a 32 GB memory budget and maintains real-time reactiveness (attending to ~200 frames under 100ms latency).
- Data: With only 10 demonstrations, NativeMEM reached 60% average success, outperforming Mem-0 by 3.5×.
Ablation Studies:
- Removing the two-stage training (unfreezing the VLA during tokenizer learning) caused performance to drop to 23.2%, highlighting the necessity of keeping the VLA frozen to preserve pretrained priors.
- Using generic visual features (mean-pooled) instead of learned native tokens resulted in failure on tasks requiring fine-grained details (e.g., Put Back Block).
- Sparse updates or short horizons significantly degraded performance, confirming the need for dense, long-horizon coverage.

5. Significance and Claims

The paper claims that NativeMEM offers a unified solution for long-horizon manipulation by repurposing the VLA's existing visual encoder. Its significance lies in:

Compatibility: It allows pretrained single-frame VLAs to access long-term history without architectural changes or external modules, preserving the "native" nature of the policy.
Scalability: By compressing history to one token per frame, it overcomes the context length and computational bottlenecks that typically limit memory retention in transformers.
Practicality: It demonstrates that complex, memory-dependent tasks can be solved on real robots with minimal fine-tuning data and low computational overhead.

The authors acknowledge limitations, noting that the system is not designed for persistent memory over hours or days and that the reliance on action supervision might create a semantic gap for highly abstract memory requirements in future, more complex tasks. However, they assert that NativeMEM effectively bridges the gap between reactive VLA policies and the needs of long-horizon robotic manipulation.