Technical Summary: SMMBench

Problem Definition

Current benchmarks for multimodal agent memory primarily evaluate systems within pre-assembled, coherent contexts (e.g., a single long conversation or a unified retrieval corpus). However, real-world agent tasks often require reasoning over evidence that is source-distributed: fragmented across multiple independently originated artifacts such as separate group chats, private messages, profiles, screenshots, tables, and documents.

The core challenge identified is not merely reading long inputs, but source-distributed memory composition. In this setting:

1. Source-Level Incompleteness: No single source contains sufficient information to answer a query; evidence must be aggregated from multiple distinct origins.
2. Cross-Source Bridging: Agents must identify relevant sources and bridge their independent local contexts, which are often created for different purposes and at different times.
3. Conflict Resolution: Information across sources may conflict or be outdated, requiring agents to reason over authority levels and temporal states to update their memory.

Existing benchmarks largely fail to evaluate this specific bottleneck, focusing instead on long-context retrieval within a single unified stream.

Methodology: SMMBench Construction

The authors introduce SMMBench (Source-distributed Multimodal Memory Benchmark), a dataset designed to operationalize the challenge of composing evidence across independent sources.

Dataset Construction Pipeline

The benchmark is constructed through a three-stage pipeline to ensure naturalness and evidence dependency:

1. QA Preparation: Raw samples from diverse public multimodal benchmarks (e.g., ChartQA, SlideVQA, MLLMKC, MMPB, M3Bench) are converted into question-answer pairs with unified evidence units.
2. Conversational Source Synthesis: Agents with predefined profiles are instantiated to generate multi-turn conversations across multiple channels (group and private chats). These conversations serve as the "host" environment, simulating realistic long-horizon interactions without initially containing the target evidence.
3. Source-Aware Evidence Insertion: A key design choice involves routing complementary evidence units into different independent sources (e.g., placing a chart in one chat, a table in another, and a profile update in a third). An LLM-based inserter ensures these insertions maintain conversational flow and temporal consistency, creating a scenario where the answer-critical evidence is scattered and incomplete in any single source.

Benchmark Statistics and Scope

• Scale: 1,877 evaluation samples grounded in 264 distinct sources.
• Distribution: On average, each question requires evidence from 2.82 sources and 4.54 evidence items.
• Modality: Rich multimodal composition, including text, images, tables, and documents. The average sample contains 2.31 text items, 0.95 image items, and 1.24 document items.
• Task Types: The benchmark evaluates four core capabilities across five task categories:
  1. Single-Hop & Multi-Hop QA: Retrieving and reasoning over evidence from multiple sources.
  2. Conflict Resolution: Identifying and prioritizing newer/authoritative evidence over outdated or conflicting records.
  3. Preference Reasoning: Inferring user preferences from implicit cues distributed across interactions.
  4. Function Call (Action Prediction): Executing precise tool invocations based on parameters recovered from distributed multimodal content.

Evaluation Protocol

The benchmark supports both open-book (full context access) and retrieval-based settings. It evaluates systems on:

• Cross-source reasoning: Can the agent locate and combine fragments?
• Conflict handling: Can the agent resolve contradictions?
• Action grounding: Can the agent convert remembered facts into executable actions?

Key Results

Experiments were conducted on 1877 samples using representative memory-style baselines (e.g., MemGPT, MIRIX, Mem0) and RAG-style baselines (e.g., Native RAG, HMRAG, VRAG).

1. Performance Gap: Current systems struggle significantly. Even the strongest baseline (HMRAG) achieved an overall score of 0.4933, leaving a substantial gap (0.2540) compared to the Golden Evidence Baseline (0.7473), which provides the model with the correct evidence directly.
2. Source-Distributed Difficulty: When the same evidence was concentrated into a single source (Source-Concentrated setting), performance improved across all baselines. This confirms that the difficulty stems specifically from the distribution of evidence across independent sources, not just the volume of content.
3. Task-Specific Challenges:
   • Function Calling proved the most difficult task (best score: 0.1111), indicating that recovering precise parameters from scattered multimodal evidence for action execution is a major bottleneck.
   • Preference Reasoning and Conflict Resolution also showed significant drops compared to factual QA tasks.
4. Retrieval vs. Composition: While increasing the retrieval budget ($K$) improved recall, it often introduced noise that degraded precision. The results suggest that retrieval alone is insufficient; the ability to compose and reconcile retrieved evidence across source boundaries is the primary failure point.
5. Modality Impact: Native multimodal access did not uniformly solve the problem. Some systems (e.g., MIRIX) improved with native multimodal inputs, while others (e.g., MemGPT) performed worse, suggesting that the bottleneck lies in cross-source memory architecture rather than just modality perception.

Error Analysis

Diagnosis of 600 failure cases revealed a layered failure mode:

1. Access Failure (61.8%): The dominant error type is the inability to surface the necessary distributed evidence from memory.
2. Utilization Failure (16.5%): Even when evidence is retrieved, agents fail to compose clues across sources or prioritize updated evidence over stale records.
3. Action Failure (8.2%): Agents fail to convert remembered information into correct executable outputs (Function Call errors).

Significance and Claims

The paper claims that source-distributed memory composition is an under-evaluated but critical bottleneck for multimodal agents.

• Problem Identification: It highlights that the difficulty in real-world agent memory is not just "long-context" but "cross-source" reasoning.
• Benchmark Contribution: SMMBench provides the first systematic evaluation of agents' ability to retrieve, align, and compose evidence from heterogeneous, independently originated sources.
• Empirical Insight: The results demonstrate that current state-of-the-art memory and RAG systems are far from effective in this setting. The gap between retrieval and successful composition suggests that future progress requires architectures capable of source-aware memory management and cross-source evidence reconciliation, rather than simply scaling context windows or retrieval pools.

The authors conclude that progress in multimodal agent memory must be evaluated not only by performance within coherent contexts but by the ability to handle the fragmented, source-distributed nature of real-world information.