Technical Summary: MERIT (Multi-key Episodic Retrieval with Inference-time Temporal expansion)

Problem Statement

The rapid evolution of Multi-modal Large Language Models (MLLMs) has shifted video understanding from short clips to ultra-long streams spanning hours or even days. However, directly processing such ultra-long videos end-to-end is computationally infeasible due to context window limitations. While Retrieval-Augmented Generation (RAG) frameworks offer a solution, prior approaches suffer from two critical inefficiencies in the ultra-long setting:

1. Query-Agnostic Overhead: Existing methods (e.g., hierarchical summarization or graph-based memory) invest significant computational resources to pre-model high-level semantic relations and structures before knowing the downstream query. This forces a "query-agnostic abstraction" that may not align with the specific information needed.
2. Intelligence Misallocation: Using powerful MLLMs for complex, static memory construction (e.g., merging segments into day-level summaries or building knowledge graphs) expends high-capability reasoning on tasks that could be deferred.

The authors argue that the bottleneck should shift from building static, complex semantics in advance to enabling high-recall retrieval that consistently delivers the right fine-grained evidence, deferring semantic composition to inference time when the query is known.

Methodology: MERIT

MERIT is a lightweight, agentic framework designed for ultra-long video understanding. It adopts a minimalist two-stage paradigm: Episodic Memory Construction (query-agnostic) followed by Agentic Retrieval and Answering (query-aware).

1. Episodic Memory Construction (Stage 1)

Instead of constructing complex hierarchical graphs or multi-scale summaries, MERIT partitions the video $V$ into non-overlapping 30-second clips. For each clip, it generates a dense caption and extracts a set of Multi-keys ($K_i$).

• Multi-Key Representation: Each clip is indexed by four distinct, complementary keys to capture diverse perspectives:
  • Event/Action Key: Observable physical actions and interactions.
  • Dialogue/Mention Key: Spoken content, commands, or specific words.
  • Object Key: Specific items handled, requested, or moved, including state transitions.
  • Summary Key: A compact, keyword-style abstraction of the core narrative.
• Memory Store: The memory $M$ is a simple key-value store where keys ($K_i$) enable retrieval and values ($v_i, d_i$) provide the raw evidence.

2. Agentic Retrieval and Answering (Stage 2)

At inference time, the system operates in an iterative multi-turn loop:

• Multi-Key Retrieval: The solver generates a retrieval query $q^{(j)}$ and matches it against the stored keys using a sentence embedding model. Retrieval relevance is determined by the Maximum Similarity across all keys for a given clip:
  $$S_i^{(j)} = \max_{k \in K_i} \text{sim}(E(q^{(j)}), E(k))$$
  This allows a clip to be retrieved if any of its keys align with the query, significantly improving recall.
• Neighbor Filtering (Temporal Expansion): Recognizing that evidence often spans multiple adjacent moments, MERIT does not rely solely on the retrieved 30-second anchor. Upon retrieving a set of clips, the system dynamically expands the temporal context by including neighboring clips (e.g., $\pm \Delta$ clips).
• Query-Aware Filtering: The solver acts as a filter on this expanded local context, distilling only the information relevant to the specific query. This reconstructs coherent context without pre-computed global structures.
• Iterative Reasoning: The agent assesses if the accumulated evidence is sufficient. If not, it refines the query and repeats the retrieval; otherwise, it integrates the evidence to generate the final answer.

Key Contributions

1. Paradigm Shift in Intelligence Allocation: The paper proposes deferring semantic composition to inference time rather than investing in query-agnostic preprocessing. This leverages the full reasoning capacity of the MLLM only when the task is specified.
2. Simple yet Effective Multi-Key Indexing: By attaching four heterogeneous keys to minimal episodic segments, MERIT achieves high-recall retrieval without the computational overhead of hierarchical or graph-based memory construction.
3. On-Demand Temporal Expansion: The "Neighbor Filtering" mechanism captures broader semantic context by expanding the temporal scope exclusively around retrieved segments at inference time, avoiding the massive cost of global memory construction.
4. Minimalist Architecture: MERIT eliminates the need for expensive multi-stage memory consolidation, relying instead on a simple key-matching mechanism and dynamic context expansion.

Experimental Results

MERIT was evaluated on three long-video benchmarks: EgoLifeQA (ultra-long, egocentric), LVBench (hour-long), and Video-MME (Long).

• State-of-the-Art Performance: MERIT achieved new SOTA results across all benchmarks.
  • On EgoLifeQA, using GPT-5 as the solver, MERIT achieved 71.2% average accuracy, outperforming the previous SOTA (WorldMM with GPT-5) by +5.6%.
  • On LVBench, MERIT (GPT-5) reached 71.8%, surpassing the prior SOTA by +9.9%.
  • On Video-MME (Long), MERIT achieved 77.7%, outperforming WorldMM (76.6%).
• Efficiency: MERIT demonstrated significant efficiency gains in memory construction. Compared to WorldMM, it reduced input tokens by 8.4× and output tokens by 8.6×, while reducing LLM calls from ~23,700 to ~6,200.
• Retrieval Quality: MERIT achieved a higher Hit Rate (0.56) compared to hierarchical (EgoRAG: 0.15) and graph-based (WorldMM: 0.53) baselines, confirming that precise, fine-grained retrieval is critical for QA performance.
• Ablation Studies:
  • Neighbor Filtering: Removing this component significantly dropped performance, particularly in categories requiring broader context (e.g., RelationMap, HabitInsight).
  • Multi-Key Combinations: Using all four keys simultaneously yielded the highest accuracy, validating the complementary nature of the keys.
  • Solver Dependency: Performance scaled significantly with stronger solvers (e.g., GPT-5 vs. Qwen3-VL-8B), reinforcing the hypothesis that the memory should be lightweight while the solver handles complex reasoning.

Significance and Claims

The paper claims that deferring semantic composition to query time is a more principled and effective approach than investing in query-agnostic preprocessing. The results suggest that:

1. Simplicity is Effective: A simple episodic multi-key memory, combined with on-demand temporal expansion, can outperform complex hierarchical and graph-based architectures.
2. Retrieval is the Bottleneck: High-recall retrieval of fine-grained evidence is more critical than pre-computed semantic structures.
3. Scalability: The framework is highly scalable for arbitrarily long videos because it avoids the cascading computational costs of updating complex global memory structures as video length increases.

The authors position MERIT as a practical and scalable foundation for future ultra-long video understanding, emphasizing that the "intelligence" required for complex reasoning should be allocated to the inference stage where the query defines the necessary abstractions.