RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies

This paper introduces RoboMME, a large-scale standardized benchmark and a suite of memory-augmented VLA models designed to systematically evaluate and advance robotic generalist policies in long-horizon, history-dependent manipulation tasks.

The Gist

Imagine you are teaching a robot to do chores. If you just tell it, "Pick up that red cup," and it sees the cup right in front of it, that's easy. But what if the robot needs to remember that it already picked up two cups, or that the blue cup is hiding under a box it can't see right now, or that it needs to repeat a specific dance move it saw you do five minutes ago?

This is where memory comes in. Without memory, a robot is like a goldfish with a 3-second attention span—it can only react to what it sees right now.

The paper introduces RoboMME, a giant "test drive" designed to see how good robots are at remembering things. Think of it as a robotic driver's license exam, but instead of parallel parking, the robot has to pass four specific memory challenges.

Here is a breakdown of the four types of memory the paper tests, using simple analogies:

1. Temporal Memory: The "Counting Sheep" Test

• The Challenge: The robot is told, "Put three green blocks in the bin, then press the button."
• The Analogy: Imagine you are at a concert, and the bouncer says, "Let in exactly 50 people, then close the door." If you don't remember how many people you've let in, you'll either let in too few or let in a crowd of 100.
• The Robot's Job: It has to count its own actions. If it forgets it already moved two blocks, it might move a third one when it should have stopped. This tests if the robot can keep a running tally of events over time.

2. Spatial Memory: The "Shell Game" Test

• The Challenge: The robot watches a video where a green cube is hidden under a cup. Then, the cups are shuffled around while the robot isn't looking (or while its view is blocked). It has to find the green cube.
• The Analogy: Think of the classic street magic trick where a magician hides a ball under one of three cups and shuffles them. If you have good spatial memory, you know exactly where the ball is, even if you can't see it. If you have bad memory, you guess randomly.
• The Robot's Job: It has to track objects even when they are invisible or moving. It's like playing "Where's Waldo" but the Waldo is a cube that keeps getting covered up.

3. Object Memory: The "Who's Who" Test

• The Challenge: The robot sees a video where a specific cube is highlighted for a split second. Later, it has to pick up that exact same cube from a pile of identical-looking cubes.
• The Analogy: Imagine you meet a friend at a party, they briefly wear a red hat, and then take it off. Later, you have to find that same friend in a crowd of people wearing the same clothes. You have to remember, "That's the guy who had the red hat," not just "a guy in a blue shirt."
• The Robot's Job: It has to link a fleeting visual clue (the highlight) to a specific object and remember that identity later, even if the object looks exactly like its neighbors.

4. Procedural Memory: The "Dance Recital" Test

• The Challenge: The robot watches a video of a human moving a stick in a specific, complex pattern (like drawing a circle or a zigzag). It then has to copy that exact movement.
• The Analogy: This is like watching a dance tutorial on YouTube and then trying to do the moves yourself. You aren't just looking at the room; you are trying to remember the sequence of your own body's movements.
• The Robot's Job: It has to recall a "muscle memory" routine. It's not just about where the object is, but how to move to get there.

The Big Discovery: One Size Does Not Fit All

The researchers didn't just build the test; they built 14 different "student robots" with different types of memory brains to take the test. They found something surprising:

• The "Note-Taker" Robot (Symbolic Memory): This robot writes down notes like "Step 1: Pick up green block." It's great at counting and simple tasks but gets confused when things move fast or require precise hand-eye coordination.
• The "Photographer" Robot (Perceptual Memory): This robot keeps a mental photo album of what it saw. It's amazing at copying dance moves and tracking moving objects but sometimes gets overwhelmed by too many photos.
• The "Compressor" Robot (Recurrent Memory): This robot tries to squish all its memories into a tiny summary. It turned out to be the least effective, like trying to remember a whole movie by remembering just one sentence about it.

The Verdict: There is no single "super-brain" for robots.

• If you need a robot to count or follow a checklist, give it a "Note-Taker" brain.
• If you need a robot to imitate a dance or track a moving ball, give it a "Photographer" brain.

Why This Matters

Right now, most robots are like toddlers: they can do simple things but forget everything the moment you look away. RoboMME is a standardized way to measure how much a robot has grown up. By understanding which type of memory works best for which job, we can build robots that don't just react to the world, but actually remember it, making them reliable helpers for long, complex tasks like cleaning a whole house or organizing a warehouse.

The paper concludes that while robots are getting smarter, they still struggle with these memory tasks—sometimes even more than humans do! But now, we have a clear map of where they need to improve.

Technical Summary

Here is a detailed technical summary of the paper "RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies."

1. Problem Statement

Current Vision-Language-Action (VLA) models for robotics have made significant strides in short-horizon tasks but struggle with long-horizon, history-dependent manipulation. Real-world tasks often require the robot to:

• Count repeated actions (e.g., "pick up 3 cubes").
• Track objects that become temporarily occluded or move while hidden.
• Identify objects based on transient visual cues or language references.
• Replicate complex motion trajectories from demonstrations.

Existing benchmarks (e.g., RLBench, CALVIN, MemoryBench) fail to systematically evaluate these capabilities because they either:

1. Rely on immediate perception (Markovian), making memory unnecessary.
2. Contain only a few tasks that are already "solved" by current policies.
3. Lack standardized evaluation protocols and high-quality demonstrations for imitation learning.

There is a critical need for a unified benchmark to evaluate memory-augmented policies and a systematic understanding of which memory representations (symbolic, perceptual, recurrent) work best for specific types of tasks.

2. Methodology

A. The RoboMME Benchmark

The authors introduce RoboMME, a large-scale simulation benchmark built on the ManiSkill environment with a 7-DOF Franka Panda arm. It is grounded in cognitive theories of human memory, categorizing tasks into four distinct suites:

1. Counting (Temporal Memory): Tasks requiring the accumulation and ordering of events (e.g., BinFill, PickXTimes, StopCube). The robot must count actions or track time-sensitive events.
2. Permanence (Spatial Memory): Tasks requiring tracking object locations under occlusion or scene changes (e.g., VideoUnmask, ButtonUnmaskSwap). The robot must remember where objects are when they are hidden or swapped.
3. Reference (Object Memory): Tasks requiring the identification of specific objects based on transient visual or linguistic cues (e.g., PickHighlight, VideoPlaceOrder). The robot must recall which object was highlighted or referenced in the past.
4. Imitation (Procedural Memory): Tasks requiring the reproduction of demonstrated motion patterns (e.g., PatternLock, RouteStick). The robot must replicate specific trajectories or manipulation strategies.

Dataset Scale: 16 diverse tasks, 1,600 demonstrations, and 770k high-quality timesteps.

B. MME-VLA Suite (Memory-Augmented Models)

To evaluate the benchmark, the authors developed a family of 14 VLA variants based on the $\pi_{0.5}$ backbone. They systematically tested three Memory Representations combined with three Integration Mechanisms:

Memory Representations:

1. Symbolic Memory: Summarizes history as natural language subgoals (generated by VLMs like QwenVL or Gemini).
2. Perceptual Memory: Retains raw visual tokens from past frames using strategies like Token Dropping (removing redundant patches) or Frame Sampling.
3. Recurrent Memory: Compresses history into fixed-size latent states using Test-Time Training (TTT) or Recurrent Memory Transformers (RMT).

Integration Mechanisms:

1. Memory-as-Context: Concatenating memory tokens with current observations and instructions.
2. Memory-as-Modulator: Using memory tokens to modulate the action expert via adaptive LayerNorm (AdaLN) and cross-attention.
3. Memory-as-Expert: Adding a dedicated transformer expert to process memory tokens, interacting with the action expert via block-wise causal attention.

3. Key Contributions

1. First Comprehensive Memory Benchmark: RoboMME is the first benchmark to explicitly cover four cognitive memory dimensions (Temporal, Spatial, Object, Procedural) with non-Markovian, long-horizon tasks.
2. Systematic Taxonomy of Memory: The paper provides a rigorous framework for categorizing memory needs in robotics, moving beyond ad-hoc evaluations.
3. Extensive Empirical Analysis: The authors evaluated 14 novel VLA variants against 4 prior methods (including $\pi_{0.5}$, SAM2Act+, and MemER) and human performance baselines.
4. Real-World Validation: The study includes real-world experiments on a Franka Panda robot, confirming that simulation trends transfer to physical hardware.

4. Key Results & Findings

• No "One-Size-Fits-All" Solution: No single memory representation or integration strategy dominates across all tasks. Effectiveness is highly task-dependent.
  • Symbolic Memory excels at Counting and short-horizon reasoning (e.g., BinFill, PickXTimes) where high-level subgoals are sufficient.
  • Perceptual Memory is critical for Motion-Centric and Time-Sensitive tasks (e.g., PatternLock, StopCube) where precise visual history and low-level control are required.
  • Recurrent Memory performed the worst, likely due to unstable training when fine-tuning shallow recurrent layers on top of a frozen backbone.
• Best Integration Strategy: Memory-as-Modulator (using AdaLN) combined with Perceptual Memory (specifically Frame Sampling) achieved the best balance of performance and computational efficiency.
• Human Performance: Humans achieved ~90.5% success but still struggled with long-horizon tasks (e.g., PatternLock) and time-sensitive tasks (StopCube), indicating RoboMME remains a challenging testbed even for humans.
• Efficiency vs. Performance: Perceptual memory offered the best trade-off. External VLM-based symbolic memory (e.g., using Gemini) introduced significant computational overhead (3x–5x cost) with diminishing returns on complex manipulation tasks.
• Real-World Transfer: The trends observed in simulation (Perceptual for motion, Symbolic for counting) held true in real-world experiments.

5. Significance

• Advancing Robotic Generalists: The paper argues that for robots to operate reliably in open worlds, they must move beyond immediate perception to robust memory mechanisms. RoboMME provides the necessary testbed to drive this progress.
• Guiding Future Architecture: The findings suggest that future VLA designs should not rely on a single memory type. Instead, hybrid approaches that combine symbolic reasoning for high-level planning with perceptual memory for low-level control are likely the path forward.
• Standardization: By providing a standardized benchmark and a suite of baseline models, RoboMME enables fair comparison and reproducible research in memory-augmented robotics, addressing a major gap in the current literature.

Conclusion: RoboMME establishes that memory is not a monolithic capability but a set of distinct cognitive functions. The success of robotic generalists will depend on developing policies that can dynamically select and integrate the appropriate memory representation based on the specific demands of the task.