Technical Summary: Benchmarking Robot Memory Under Interference

Problem Statement

As robots are deployed in realistic environments, they must operate across numerous sessions and unrelated tasks. A critical challenge for real-world deployment is the ability to recall information from sessions that occurred significantly in the past, buried behind a history of irrelevant interactions. While current robot-memory benchmarks (e.g., RoboMME) evaluate memory within single episodes or short contexts, they fail to measure system robustness when a relevant prior session is embedded within a longer history of unrelated sessions. The core question addressed is whether current robot-memory systems can successfully recall useful prior information when it is obscured by interference from unrelated sessions.

Methodology

The authors introduce RoboMME-Interference, a cross-session benchmark built upon the existing RoboMME framework [Dai et al., 2026]. The methodology involves constructing a "history buffer" for each query episode that serves as the memory input for the Vision-Language-Action (VLA) policy.

Benchmark Construction

• Structure: For each query, the history buffer contains the relevant prior demonstration (the "lesson") followed by $k$ unrelated distractor sessions.
• Interference Levels: The number of distractor sessions ($k$) is swept across $\{0, 1, 3, 7\}$, representing no, light, moderate, and strong interference, respectively. A "no-history" baseline with an empty buffer is also included.
• Distractor Design: Distractors are drawn from different task families than the query to avoid introducing contradictory facts (e.g., conflicting targets) that would make performance degradation uninterpretable. Each distractor contributes a fixed-length section of 32 stored frames, ensuring comparability across sessions.
• Task Selection: The benchmark utilizes nine task families from RoboMME where the necessary prior information is contained in a self-contained demonstration video distinct from the execution phase. These include:
  • Imitation: MoveCube, InsertPeg, PatternLock, RouteStick.
  • Video-conditioned Reference: VideoRepick, VideoPlaceButton, VideoPlaceOrder.
  • Video-conditioned Permanence: VideoUnmask, VideoUnmaskSwap.
  • Excluded: Seven other RoboMME tasks were excluded because their cues are interleaved within a single execution episode, preventing the isolation of a separable prior session.

Evaluated Systems

The study evaluates nine memory-augmented variants of the $\pi_0.5$ policy [Physical Intelligence et al., 2025], running them unmodified through the benchmark. The systems vary by memory representation and integration mechanism:

• Representations: Perceptual (FrameSamp, TokenDrop), Recurrent (TTT, RMT), and Symbolic (excluded from this specific evaluation due to reliance on pre-defined annotations).
• Integration Mechanisms: Context concatenation, Modulator (adaptive LayerNorm), and Expert (block-wise causal attention).
• Baseline: The standard $\pi_0.5$ policy with no memory.

Key Contributions

1. Cross-Session Interference Benchmark: An extension of RoboMME that explicitly measures the impact of session length and unrelated memory on accuracy by inserting controlled numbers of unrelated sessions between relevant memories and queries.
2. Comprehensive Result Grid: A full evaluation of nine memory systems across nine task families, providing a detailed breakdown of performance degradation as interference increases. This includes code, data, figures, and analysis released alongside the paper.

Results

The evaluation reveals a stark contrast between performance with clean memory and performance under interference:

• Performance with Clean Memory ($k=0$): When the history buffer contains only the relevant lesson, perceptual memory variants significantly outperform the no-history baseline. The best-performing system, FrameSamp-Modulator, achieves a success rate of 45.3% (a +27.1 percentage point lift from its 18.2% no-history baseline).
• Degradation Under Interference: As unrelated sessions accumulate ($k=1, 3, 7$), the performance of all systems decays steadily.
  • FrameSamp-Modulator drops by approximately 26 percentage points from $k=0$ to $k=7$, returning to roughly 19.3%, which is near its original no-history floor.
  • TokenDrop-Modulator similarly drops by ~16 points, ending at 19.8%.
• Integration Mechanism Sensitivity: Systems with the same representation behave differently based on integration. For instance, FrameSamp with a Modulator integration vastly outperforms FrameSamp with Context or Expert integration at $k=0$, but all converge toward the baseline as interference increases.
• Recurrent Variants: Recurrent variants (TTT) show no meaningful gain over the no-history baseline at any session length, remaining at or below 18% success.
• Task and Difficulty Variance: Memory benefits are uneven across task families and difficulty levels. Gains are most pronounced on easy and medium-difficulty episodes, while hard episodes remain near the performance floor for all systems.

Significance and Claims

The paper claims that while perceptual memory variants improve success when provided with a clean history, they fail to maintain robustness as unrelated sessions accumulate. Current systems largely fail at long-context memory and interference robustness, with performance collapsing to the level of a system with no memory once enough distractors are introduced.

The authors position this work as a necessary step to make long-horizon cross-session recall measurable for robot policies, similar to how long-term recall is benchmarked for Large Language Models (LLMs). They conclude that reliable cross-session recall is currently out of reach for robot policies but note that the LLM memory literature offers a mature body of work to build upon as robot memory systems evolve.

Limitations noted by the authors:

• Evaluation is conducted in simulation using released checkpoints; real-robot validation is future work.
• Results reflect sampling over episodes but not run-to-run or training variation (single seed per system).
• The benchmark is limited to task families with separable prior demonstrations.