Technical Summary: EGOMEMREASON

Problem Statement

Next-generation visual assistants, including smart glasses and embodied agents, must reason over continuous visual streams spanning entire days or weeks. In ultra-long videos, relevant information is sparsely distributed across hours or days, creating a fundamental memory challenge: models must accumulate information over time, recall prior states, track temporal order, and abstract recurring patterns.

Existing week-long video benchmarks (e.g., EgoLife, MM-Lifelong) primarily target perception and recognition tasks, such as moment localization or global summarization. These benchmarks typically require reasoning within short temporal windows (under ten minutes) or rely on single-moment retrieval. They fail to address the specific demands of memory-driven reasoning, where answering a question requires integrating evidence distributed across multiple days. Current models struggle to selectively accumulate information, maintain temporal coherence over long horizons, and distinguish between sparse, repeated observations.

Methodology

Benchmark Design: EGOMEMREASON

The authors introduce EGOMEMREASON, a comprehensive benchmark for week-long egocentric video understanding. The benchmark decomposes long-horizon memory into three complementary types, each targeting distinct reasoning operations:

1. Entity Memory: Focuses on tracking how object states evolve and change across days.
   • Cumulative State Tracking: Identifying how an object's location or condition changes across observations separated by hours or days (e.g., a bowl moving from a counter to a sink to a table).
   • Temporal Counting: Counting distinct instances of a category observed up to a specific query timestamp, requiring a global inventory of entity instances.
2. Event Memory: Focuses on recalling and ordering activities separated by hours or days.
   • Event Ordering: Arranging events drawn from different days into a correct temporal sequence.
   • Event Linking: Identifying a specific event matching contextual constraints (location, activity type) from previous inputs, requiring reasoning over long intervals.
3. Behavior Memory: Focuses on abstracting recurring patterns from sparse, repeated observations over the whole week.
   • Spatial Preference Inference: Inferring habitual associations, such as where a person typically performs an activity.
   • Activity Pattern Inference: Predicting likely next states based on learned behavioral patterns (e.g., what happens after lunch).

Construction Pipeline

The benchmark is constructed using a four-stage pipeline based on the EgoLife dataset (week-long, continuous egocentric recordings):

1. Evidence Preparation: Raw videos are segmented into 30-second clips and processed by a strong MLLM (GPT-5) to generate dense, object-centric captions and hierarchical event summaries (spanning 3-minute, 2-hour, and day-level granularities).
2. Query Generation: Task-specific generators (using GPT-5.2) create multiple-choice questions constrained to a specific query timestamp, ensuring only past observations are accessible. Distractors are generated from similar contexts to ensure non-trivial difficulty.
3. Automatic Filtering: Candidates undergo "blind tests" (answering without visual input) to remove text-leakage cases and are verified for temporal grounding (ensuring evidence exists before the query time).
4. Human Verification: Annotators review questions against query-time context and evidence clips, assessing clarity, answer correctness, and option quality. Only ~15% of initial candidates survive this rigorous process.

The final dataset comprises 500 questions across the three memory types and six core challenges. On average, each question requires aggregating 5.1 distinct video segments and backtracking through 25.9 hours of video duration.

Evaluation Setup

The authors evaluated 17 systems across three paradigms:

• General-purpose MLLMs: (e.g., Qwen-3-VL, InternVL3.5, GPT-5, Gemini-3-Flash/Pro).
• Video-specific MLLMs: (e.g., LongVA, InternVideo-2.5, VideoLLaMA3, Molmo2).
• Agentic Video Frameworks: (e.g., SiLVR, Ego-R1, WorldMM, AVP).

Evaluation metrics include accuracy across the six capability dimensions and overall performance.

Key Results

Overall Performance

Long-horizon memory reasoning remains a significant open challenge. Even the best-performing model, Gemini-3-Flash, achieved only 39.6% overall accuracy. No model reached 51% accuracy on any single capability dimension.

Memory-Type Specific Bottlenecks

The analysis reveals that the three memory types fail for fundamentally different reasons, indicating that scaling model size alone is insufficient:

• Entity Memory: Bottlenecked by fine-grained visual grounding combined with long-context modeling. Models relying heavily on text-centric reasoning (e.g., LongVA) performed poorly (<25% on Temporal Counting), while models with pixel-level grounding pretraining (e.g., Molmo2-8B) showed superior performance.
• Event Memory: Bottlenecked by long-range temporal coherence. Even top models struggled to order or link events separated by days, with accuracy dropping sharply as the temporal span increased. This contrasts with high performance on single-event retrieval in prior benchmarks.
• Behavior Memory: Bottlenecked by abstraction over sparse repeated evidence. Models struggled to distinguish recurring patterns from noise, often defaulting to stereotypes rather than aggregating observed frequencies.

Ablation Studies

• Temporal Certification: Accuracy decreases monotonically as the evidence span (temporal certification) increases. Event memory showed the sharpest decline.
• Auxiliary Inputs: Adding transcripts or captions provided marginal or negative benefits. Captions often fragmented temporal continuity, while transcripts offered only slight gains for behavior memory.
• Frame Sampling: Increasing the number of sampled frames improved Entity memory up to a saturation point (~256 frames) but did not resolve bottlenecks in Event or Behavior memory, suggesting the issue lies in memory storage and retrieval mechanisms rather than visual coverage.
• Prompting Strategies: Chain-of-Thought (CoT) prompting degraded performance, suggesting that explicit step-by-step reasoning amplifies errors in memory-intensive tasks.

Significance and Claims

The paper claims that EGOMEMREASON establishes a strong foundation for evaluating and advancing long-context, memory-aware multimodal systems. Its primary contributions are:

1. Redefining the Benchmark Scope: It moves beyond retrieval-centric evaluation (locating a few relevant moments) to require aggregation, relational, and inductive reasoning over evidence distributed across days.
2. Structured Memory Diagnosis: By decomposing memory into entity, event, and behavior types, the benchmark provides a diagnostic framework that reveals distinct failure modes, guiding future research toward specific capabilities (e.g., perceptual precision for entities, structured temporal modeling for events).
3. Highlighting the Gap: The results demonstrate that current state-of-the-art MLLMs and agentic frameworks are far from solving long-horizon memory reasoning, with performance degrading significantly as temporal horizons extend.

The authors conclude that progress requires advances on three orthogonal axes: perceptual precision with long-context retention, structured temporal modeling, and aggregation-based reasoning, rather than simply scaling model parameters or input lengths.