Technical Summary: Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding

Problem Statement

Video reasoning is a core capability for egocentric and embodied agents. While recent advancements in multimodal Large Language Models (LLMs) have improved performance on long-form video benchmarks, a critical evaluation gap remains for complex, long-form egocentric video reasoning. Existing benchmarks (e.g., EgoSchema, EgoTempo) typically evaluate only the final answer, overlooking the intermediate reasoning process. Furthermore, standard benchmarks often fail to assess reasoning over extended temporal contexts or the specific challenges of the egocentric perspective, which involves close-up, frequent object interactions and continuous activity flows. Current state-of-the-art (SOTA) models struggle with perceptual identification and temporal grounding of objects in these settings, leading to significant performance gaps compared to human capabilities.

Methodology

The authors introduce Minerva-Ego, a new benchmark designed to evaluate complex egocentric visual reasoning through dense, human-annotated reasoning traces.

1. Dataset Construction

• Source: Built upon the HD-EPIC dataset, a high-quality, in-the-wild egocentric video collection focused on kitchen activities.
• Scale: Contains 1,160 complex, hand-crafted multiple-choice questions covering 156 unique videos (ranging from 10 seconds to 75 minutes).
• Annotation: Unlike prior works that rely on automated pipelines or simple QA pairs, Minerva-Ego features spatiotemporally-dense human-annotated reasoning traces.
  • Each question includes five answer choices.
  • Each correct answer is accompanied by a detailed reasoning trace that links specific timestamps and segmentation masks to the relevant objects of interest required to solve the question.
  • Annotations are performed by highly skilled raters with a rigorous multi-step verification process to ensure quality and avoid model bias.
• Statistics: Reasoning traces average 122 words, referencing a mean of 6.3 timestamps and 2.8 objects per question.

2. Evaluation Framework

The paper employs two primary experimental approaches:

• SOTA Benchmarking: Evaluating leading video understanding models (Gemini 2.5 Pro/Flash, GPT-4.1/5, Claude Sonnet 4, Qwen3) on the dataset. The evaluation goes beyond final accuracy (MCQ) to analyze reasoning traces using the MINERVA Rubric (MiRA), which scores models on four dimensions: Perceptual Correctness, Temporal Grounding, Logical Reasoning, and Completeness.
• Spatiotemporal Highlighting Experiments: To investigate the root causes of model failures, the authors test whether providing explicit "hints" regarding where (spatial) and when (temporal) to look improves performance.
  • Oracle Setting: Using ground-truth segmentation masks (from HD-EPIC and interpolated via SAM2) to highlight relevant objects (via masks, bounding boxes, circles, or class names) and select relevant frames.
  • Practical Setting: Using an open-vocabulary segmentation model (OWLv2) to detect and highlight objects based on categories suggested by the LLM itself.

Key Results

1. Performance Gap

SOTA models perform poorly on Minerva-Ego compared to humans:

• Human Performance: 91.8% accuracy.
• Best Model (Gemini 2.5 Pro): 40.1% accuracy (with "thinking" enabled).
• Gap: A substantial gap remains, indicating current models are not yet capable of robust long-form egocentric reasoning.

2. Error Analysis

Analysis of reasoning traces reveals that models fail primarily due to perceptual grounding errors rather than logical reasoning flaws:

• Perceptual Correctness: This is the weakest dimension for all evaluated models.
• Object Recall: Models reference only 20–50% of the relevant objects mentioned in ground-truth reasoning traces.
• Temporal Grounding: Models often miss the correct temporal span or fail to identify actions due to the egocentric viewpoint or fast motion.

3. Impact of Spatiotemporal Hints

Providing explicit hints significantly bridges the performance gap:

• Oracle Experiments: Combining spatial highlighting (encircling objects with circles and displaying class names) with temporal selection (feeding only frames where relevant objects appear) yielded a peak improvement of 5.6% (from 44.5% to 50.3% for Gemini 2.5 Pro).
• Practical Experiments: Using predicted segmentation (OWLv2) with temporal selection and class name visualization provided modest gains (up to 2.5% improvement).
• Generalization: These improvements were also observed on the EgoTempo dataset, suggesting the technique generalizes to other egocentric benchmarks.

Significance and Claims

The paper claims the following contributions and significance:

1. First Egocentric Benchmark with Dense Traces: Minerva-Ego is the first benchmark to pair complex, multi-step egocentric questions with dense spatiotemporally-grounded reasoning traces. This allows for "diagnosable evaluation" rather than simple accuracy metrics.
2. Diagnosis of Failure Modes: The authors demonstrate that current frontier models fail primarily due to perceptual grounding errors (identifying what and when to look) rather than a lack of logical reasoning capability.
3. Value of Explicit Hints: The study proves that prompting models with explicit spatiotemporal hints ("where" and "when") yields substantial performance improvements. This suggests that the bottleneck in current video reasoning is often the model's ability to attend to the correct visual evidence, not necessarily its reasoning engine.
4. Methodological Insight: The work highlights that simple visual prompting techniques (like highlighting objects and selecting frames) can circumvent reasoning errors in fixed, frontier models without requiring architectural changes or retraining.

The authors conclude that while SOTA models show promise, there is a significant need to improve their ability to identify and focus on key objects of interest over time in egocentric settings to achieve human-level performance.