Technical Summary: MEMDREAMER

Problem Statement

Current Vision-Language Models (VLMs) face significant bottlenecks when processing hours-long videos. The primary challenges are token explosion and attention dilution.

• Token Explosion: Processing a 2-hour video at standard frame rates (e.g., 1 FPS) generates over 1.6 million tokens, exceeding the context limits of even the most advanced models.
• Attention Dilution & "Lost in the Middle": Existing end-to-end paradigms rely on brute-force frame sampling and ultra-long context windows. This flattens the video into massive token streams, causing critical reasoning signals to be submerged by noise. Models struggle with long-range dependencies and the "lost in the middle" phenomenon, where relevant information is missed amidst redundant tokens.
• Structural Limitations: Current memory-based approaches often use flat or chunk-based storage, which fails to capture the hierarchical nature of video content (plot $\to$ scenes $\to$ events $\to$ actions) and severs essential spatiotemporal and causal links.

Methodology: MEMDREAMER

MEMDREAMER introduces a paradigm shift by decoupling perception and reasoning. It transforms long-video understanding from a passive token ingestion task into an agentic exploration process. The framework consists of two distinct stages:

1. Hierarchical Graph Memory Construction (Perception)

A perception model ($P$) processes the video stream in a streaming fashion to construct a persistent, purely textual Hierarchical Graph Memory ($G$). This memory is organized into a three-tier coarse-to-fine topology:

• Video Root: A global summary capturing the title, themes, and key entities.
• Super Events: Narrative phases spanning broader temporal scales.
• Macro Events: Episode-level summaries.
• Local Subgraphs (Leaf Tier): Each Macro Event is instantiated as a local subgraph containing:
  • Entity Nodes: Persistent referents (people, objects, locations).
  • Micro-Event Nodes: Atomic actions with explicit temporal extents.
  • Topological Edges: Three distinct edge types connect these nodes:
    1. Spatial-attribute: Static layouts and coreference.
    2. Subject-object: Binding entities to their actions.
    3. Temporal-causal: Chronological progression and causal dependencies between micro-events.

Construction Process:

• Streaming Adaptive Segmentation: Instead of fixed-length chunks, the system uses a semantic-driven sliding window to ensure Macro Events are self-contained and semantically intact.
• Downward Subgraph Extraction: Fine-grained visual details are extracted into local subgraphs for each Macro Event.
• Upward Hierarchical Aggregation: Local descriptions are distilled and clustered to form Super Events and the Video Root, creating a global navigation backbone.

2. Agentic Tool-Augmented Retrieval (Reasoning)

A separate reasoning model ($R$) interacts with the text-based memory $G$ via an Observation-Reason-Action loop. It does not access raw video frames during inference. Instead, it utilizes a dedicated Agentic Tool Bank comprising three categories:

• Navigation Tools: Enable vertical traversal (e.g., GetSummary, GetSuperEvent, GetMacroEvent, GetSubgraph) to move from global context to specific details.
• Search Tools: Enable rapid node localization via semantic similarity (SearchNodes) or temporal filtering (SearchByTime).
• Graph Traversal Tools: Enable multi-hop reasoning along logical chains (GetRelationGraph) to trace causal or spatial relationships.

At each step, the model selects a tool, executes it, and distills the raw observation into only the task-relevant evidence cues ($c_t$) before updating its trajectory. This prevents the accumulation of irrelevant textual noise.

Key Contributions

1. Decoupled Paradigm: The first framework to explicitly separate perception (memory construction) from reasoning (agentic retrieval) for long-video understanding, bypassing context window limits.
2. Hierarchical Graph Memory: A novel three-tier architecture that preserves both global narrative structure and local spatiotemporal/causal relations, addressing the limitations of flat or chunk-based storage.
3. Agentic Retrieval Mechanism: A tool-augmented loop that shifts understanding from passive consumption to active, multi-step exploration, allowing the model to navigate complex logical chains.
4. Discovery of Agentic Correlation: The paper empirically establishes a strong positive linear correlation between a VLM's intrinsic agentic reasoning capability (measured on logic benchmarks like AIME 2025) and its long-video understanding performance. This suggests that scaling agentic capacity is a new paradigm for multimodal comprehension.

Experimental Results

MEMDREAMER was evaluated on four mainstream benchmarks: LVBench, LongVideoBench, Video-MME, and EgoSchema.

• State-of-the-Art Performance: The framework achieved SOTA results across all benchmarks. On LVBench, using the Gemini-3.1-Pro backbone, it reached 90.7% accuracy, narrowing the gap to human experts (94.4%) to just 3.7 points.
• Efficiency: MEMDREAMER constrains the reasoning context window to merely 2% of the full-context ingestion required by end-to-end models (approx. 6K tokens vs. 240K–784K tokens).
• Performance Gains: Compared to end-to-end baselines using the same backbone, MEMDREAMER delivered absolute accuracy gains of 12.5 points on LVBench. For example, Qwen3-VL improved from 63.6% to 84.8%.
• Correlation Analysis: Statistical analysis revealed a Pearson correlation of 0.90 ($p < 0.01$) between agentic reasoning scores and long-video performance under the MEMDREAMER paradigm, compared to only 0.70 ($p = 0.052$) for end-to-end models. This confirms that decoupling allows models to fully leverage their reasoning capabilities.
• Ablation Studies:
  • Removing the hierarchical structure or graph edges significantly degraded performance.
  • The full tool-augmented retrieval outperformed static embedding search by 20.2 points.
  • The system demonstrated robustness; swapping perception models had minimal impact on final performance, indicating high error tolerance in the perception phase.

Significance

The paper posits that MEMDREAMER establishes a new design principle for multimodal memory systems. By decoupling perception and reasoning, it overcomes the structural bottlenecks of long-context processing. The discovery that agentic capability scaling directly correlates with long-video understanding performance suggests a new optimization direction for future research: rather than simply expanding context windows, future VLMs should focus on enhancing their agentic retrieval and reasoning loops within structured memory environments. This approach offers a scalable path toward embodied intelligence and open-world interaction without the prohibitive costs of token explosion.