EchoVLA: Synergistic Declarative Memory for VLA-Driven Mobile Manipulation

EchoVLA is a memory-enhanced Vision-Language-Action model for mobile manipulation that synergizes scene and episodic declarative memories to improve navigation and task performance, validated by the new MoMani benchmark and demonstrating significant gains over existing baselines in both simulation and real-world settings.

The Gist

Imagine you are teaching a robot to clean your house. It's not just about picking up a toy; it's about walking from the living room to the kitchen, opening the fridge, grabbing a juice box, and putting it in the sink.

Current robots are like amnesiacs. They can see what's in front of them right now, but they don't remember where they were five seconds ago, or that they already opened the cabinet door. If the robot turns a corner and sees a wall that looks exactly like the wall it saw before, it gets confused: "Did I open this door already? Or am I about to open it?"

This paper introduces EchoVLA, a new kind of robot brain designed to solve this "short-term memory" problem.

Here is the breakdown using simple analogies:

1. The Problem: The Robot with No Memory

Most robots today operate on a "Markovian" basis. This is a fancy way of saying: "I only know what I see right now."

• The Analogy: Imagine playing a video game where every time you turn your head, the game resets to the last checkpoint. You can't plan a route across the map because you forget where you started.
• The Result: These robots are great at simple tasks on a table (like stacking blocks), but they fail miserably at "mobile manipulation" (walking around a house to do chores) because they lose track of the big picture.

2. The Solution: EchoVLA's "Dual-Brain" System

The authors looked at how human brains work. We have two main types of memory that work together:

1. Spatial Memory: "Where is the kitchen? Where is the fridge?"
2. Episodic Memory: "I just opened the fridge. I haven't opened the microwave yet."

EchoVLA builds a robot brain with these two specific "memory banks":

A. The "Scene Memory" (The Mental Map)

• What it is: A persistent, 3D map of the room.
• The Analogy: Think of this as a Google Maps or a mental blueprint of your house. It remembers that the fridge is always in the kitchen, the table is in the middle, and the floor is clear. It doesn't care what the robot is doing right now; it just cares about the layout of the world.
• Why it helps: Even if the robot walks away and comes back, it knows, "Ah, I'm back in the kitchen. The fridge is to my left."

B. The "Episodic Memory" (The To-Do List)

• What it is: A short-term log of recent actions and events.
• The Analogy: Think of this as a sticky note or a short-term diary. It remembers, "Three seconds ago, I grabbed the cup. Two seconds ago, I turned left."
• Why it helps: This solves the "amnesia" problem. If the robot sees a cabinet, the Episodic Memory tells it, "Wait, I already opened that cabinet in the last step. Don't open it again!"

3. How They Work Together: The "Conductor"

The robot doesn't just look at these memories; it uses a special attention system to blend them.

• The Analogy: Imagine a conductor in an orchestra.
  • The Scene Memory is the sheet music (the structure of the song).
  • The Episodic Memory is the soloist's recent improvisation (what just happened).
  • The Conductor (EchoVLA) listens to both to decide the next note. It asks: "Based on the map (Scene) and what we just did (Episodic), should the robot's arm move left or should the robot's wheels turn right?"

This allows the robot to coordinate its legs (base) and its hands (arm) perfectly, even over long tasks.

4. The Training Ground: MoMani

To teach this robot, the authors couldn't just use old data. They built a new training ground called MoMani.

• The Analogy: Instead of just showing the robot a few videos of people cleaning, they built a virtual video game simulator where an AI "Director" (a Large Language Model) generates thousands of unique cleaning scenarios.
• The Twist: They also filmed real robots doing these tasks. This is like having a student watch a master chef cook (simulation) and then actually tasting the dish (real-world data) to make sure the lesson sticks.

5. The Results: From Clumsy to Capable

When they tested EchoVLA:

• The Baseline (Old Robots): Got confused easily. In complex tasks, they succeeded only about 20-30% of the time. They would walk into a wall or forget to pick up the item.
• EchoVLA (The New Robot): Succeeded about 44-52% of the time.
• The Real-World Win: In a real room, EchoVLA could successfully navigate to a different room, open a microwave, rotate a knob, and place items in a sink. The old robots would often get stuck or give up.

Summary

EchoVLA is a robot that finally has a "memory."

• It has a Map (to know where things are).
• It has a Diary (to know what it just did).
• It uses both to plan its next move.

This turns a robot that stumbles around blindly into a helpful assistant that can actually clean your house, cook your dinner, and remember where it left the keys.

Technical Summary

Here is a detailed technical summary of the paper "EchoVLA: Synergistic Declarative Memory for VLA-Driven Mobile Manipulation."

1. Problem Statement

Current Vision-Language-Action (VLA) models have achieved success in short-horizon, table-top manipulation tasks. However, they face significant limitations in mobile manipulation (coordinating a mobile base with an arm) due to:

• Markovian Control: Most existing VLAs rely solely on the current observation, lacking the memory to reason over extended task sequences.
• Spatial Context Loss: They struggle to maintain consistent spatial understanding when navigating changing environments or performing multi-step tasks where visual states are ambiguous (e.g., distinguishing between "about to open a cabinet" and "cabinet already opened").
• Data Scarcity: There is a lack of large-scale, high-quality benchmarks that cover diverse mobile manipulation tasks with both simulation and real-world data.

2. Methodology: EchoVLA

The authors propose EchoVLA, a neuro-inspired, memory-aware VLA framework designed for mobile manipulation. It mimics the human brain's declarative memory system by integrating two complementary memory banks: Scene Memory and Episodic Memory.

A. Neuro-Inspired Dual-Memory Architecture

1. Scene Memory (Spatial-Semantic Map):
   • Inspiration: Analogous to the Parahippocampal Cortex (PHC).
   • Function: Maintains a persistent, voxelized 3D feature map of the environment. It accumulates spatial information across episodes, encoding geometric structures (surfaces, free space, object boundaries).
   • Update Mechanism: Uses a discrepancy-driven rule. New observations are compared to the existing map; only regions with high reconstruction errors are updated, ensuring the map evolves consistently without redundant overwrites.
2. Episodic Memory (Task-Level Experience):
   • Inspiration: Analogous to the Hippocampus.
   • Function: Stores a time-indexed buffer of recent multimodal token sequences (observations, proprioception, instructions).
   • Role: Captures fine-grained temporal progress (e.g., "drawer opened," "object grasped") to resolve non-Markovian ambiguities where visual inputs alone are insufficient.

B. Synergistic Retrieval & Fusion

• Hierarchical Attention: The model retrieves information using a coarse-to-fine strategy:
  • Coarse-grained Attention: Queries the Scene Memory using the current 3D voxel features to retrieve global spatial context.
  • Fine-grained Attention: Queries the Episodic Memory using current multimodal tokens to retrieve specific task history.
• Fusion: The retrieved representations from both memories are fused and concatenated to form a memory-augmented state ($H_t$).

C. Per-Part Diffusion Policy

• The fused memory state conditions a Diffusion Transformer (DiT) policy.
• Decoupled Action Generation: The policy generates actions for the mobile base and the arm in separate but coordinated denoising processes. This allows for structured learning of locomotion and manipulation behaviors simultaneously.

3. Key Contributions

A. The EchoVLA Model

• Introduces the first VLA architecture specifically designed for mobile manipulation that explicitly separates and synergizes spatial (scene) and temporal (episodic) memory.
• Demonstrates that this dual-memory system enables coherent reasoning over long-horizon tasks, outperforming single-memory or Markovian baselines.

B. MoMani Benchmark

• Automated Data Generation: Introduces MoMani, a pipeline that uses Multimodal Large Language Models (MLLMs) to generate expert-level trajectories via planning and feedback-driven refinement.
• Scale and Diversity: Covers a wide range of mobile manipulation tasks (navigation + manipulation) in simulation.
• Real-World Integration: Uniquely includes a real-robot dataset collected on a TidyBot++ platform (holonomic base + Kinova Gen3 arm), bridging the sim-to-real gap. The dataset contains ~7,889 simulation episodes and 1,200 real-world episodes.

4. Experimental Results

Simulation (RoboCasa)

• Metrics: Success Rate (SR) on Manipulation, Navigation, and Mobile Manipulation tasks.
• Performance:
  • Mobile Manipulation: EchoVLA achieved an SR of 0.31, significantly outperforming the strong baseline $\pi0.5$ (0.20) and Diffusion Policy (0.06).
  • Manipulation/Navigation: Achieved an SR of 0.52, exceeding $\pi0.5$ by +0.20.
• Ablation Studies:
  • Removing Point Cloud (3D) input caused a drastic drop in performance, highlighting the need for geometric cues.
  • Removing either Episodic or Scene memory led to substantial degradation, proving both are essential.
  • Optimal hyperparameters were found at a window size ($L$) of 8 and an update threshold ($\tau$) of 0.5.

Real-World Experiments

• Setup: Deployed on a 7m $\times$ 7m arena with tasks like opening drawers, closing microwaves, and cross-room navigation (EnP).
• Performance: EchoVLA achieved a mean SR of 0.44, outperforming $\pi0.5$ (0.33) and Diffusion Policy (0.32).
• Robustness: EchoVLA showed superior stability in long-horizon tasks (e.g., "Enter and Place") and complex manipulation (e.g., rotating a microwave knob), where baselines failed due to memory drift.
• Failure Analysis: Performance dropped in scenarios with dynamic occlusion (e.g., opening a fridge door), where the explicit 3D scene map struggled with rapid geometric changes, suggesting a need for future integration with visual-SLAM.

5. Significance

• Advancing Embodied AI: EchoVLA addresses a critical gap in mobile manipulation by moving beyond Markovian control to memory-augmented reasoning, enabling agents to handle complex, multi-stage household tasks.
• Neuro-Inspired Design: The successful application of a dual-memory system (spatial vs. episodic) provides a new blueprint for designing robust VLA agents that can "remember" where objects are and how tasks were previously executed.
• Data Infrastructure: The MoMani benchmark provides a scalable, automated solution for generating high-quality mobile manipulation data, which is a bottleneck for training general-purpose robots.
• Practical Impact: The results demonstrate that combining explicit 3D spatial maps with temporal experience allows robots to operate more reliably in unstructured, real-world environments.