EgoIntent: An Egocentric Step-level Benchmark for Understanding What, Why, and Next

1. Problem Definition

Current Multimodal Large Language Models (MLLMs) have shown promise in video reasoning, but their ability to understand human intent at a fine-grained, step-level in egocentric (first-person) videos remains largely unexplored.

• Limitations of Existing Benchmarks: Most existing benchmarks focus on episode-level or clip-level reasoning (e.g., "What is happening in this 3-minute clip?" or "What is the overall goal?"). They often overlook the dynamic evolution of intent within individual procedural steps.
• The Gap: Real-world applications (e.g., robotic imitation, intelligent assistants, AR guidance) require understanding not just what is happening, but why a specific step is being taken and what will happen next, all based on limited visual context before the outcome is visible.
• Core Challenge: Models must infer Local Intent (What), Global Intent (Why), and Next-Step Plan (Next) without access to future frames or the completed outcome of the current action, preventing "future-frame leakage."

2. Methodology: The EgoIntent Benchmark

The authors introduce EgoIntent, a comprehensive benchmark designed to evaluate step-level intent understanding in egocentric procedural activities.

A. Dataset Construction

• Source: Built upon the GoalStep annotations from the Ego4D dataset.
• Scale: Comprises 3,014 steps extracted from 32 event-level videos.
• Diversity: Covers 15 distinct scenarios (both indoor and outdoor), including cooking, repairing, cleaning, gardening, and assembly.
• Annotation Dimensions: Each step is annotated with three complementary labels:
  1. Local Intent: The immediate goal of the current step (e.g., "prepare the drill for installation").
  2. Global Intent: The higher-level functional role of the step within the entire procedure (e.g., "aim to repair the wheel").
  3. Next-Step Plan: The most likely action to follow immediately after the current step (e.g., "replace the socket").

B. Temporal Truncation Strategy (Key Innovation)

To ensure a rigorous evaluation of anticipatory reasoning, the authors implemented a strict pre-outcome temporal truncation mechanism:

• Mechanism: Each video clip is truncated immediately before the key outcome of the queried step becomes visually explicit (e.g., before a tool makes contact or an object is grasped).
• Constraint: The clip contains no frames from subsequent steps.
• Purpose: This prevents models from "cheating" by observing the result of an action to infer the intent. The model must rely solely on pre-outcome cues (hand-object interactions, scene context, procedural flow) to predict the intent and future.

C. Evaluation Protocol

• Task Type: Open-ended generation (not multiple-choice) to avoid reliance on semantic cues from predefined options.
• Metric: Since exact string matching is unsuitable for open-ended intent descriptions, the authors use an LLM-based semantic consistency evaluation (using Qwen3) to score the generated answers against ground truth.
• Models Evaluated: 15 representative MLLMs, including state-of-the-art closed-source (e.g., Qwen3.5-plus, Gemini-3-flash) and open-source models (e.g., InternVL, LLaVA-Video, Molmo).

3. Key Contributions

1. EgoIntent Benchmark: The first step-level benchmark for egocentric videos that jointly evaluates Local Intent (What), Global Intent (Why), and Next-Step Plan (Next).
2. Truncation Mechanism: A novel data processing strategy that removes future information leakage, forcing models to perform true predictive reasoning based on incomplete visual evidence.
3. Comprehensive Evaluation: A systematic analysis of 15 MLLMs, revealing significant limitations in current models' ability to handle fine-grained, proactive intent inference.

4. Experimental Results

The evaluation of 15 MLLMs yielded the following key findings:

• Overall Performance is Low: Even the best-performing model (Qwen3.5-plus) achieved an average score of only 33.31 across the three dimensions. This indicates that step-level intent understanding is a highly unsolved problem for current AI.
• Dimension Imbalance: Models performed relatively better on Global Intent (understanding the "Why" or high-level goal) compared to Local Intent (immediate "What") and Next-Step Plan (predicting "Next").
  • Example: Qwen3.5-plus scored 36.80 on Global Intent vs. 31.06 on Next-Step Plan.
• Scenario Variability: Performance varied significantly across different environments. Models struggled in scenarios requiring complex context integration or where the link between action and goal was implicit.
• No Dominant Model: No single model consistently dominated all metrics or scenarios, suggesting that current architectures lack robust, transferable procedural reasoning capabilities.

5. Significance and Impact

• Diagnostic Tool: EgoIntent serves as a critical diagnostic testbed, exposing the specific weaknesses of MLLMs in fine-grained temporal reasoning and proactive planning.
• Advancing Proactive AI: The benchmark addresses the gap between passive video understanding and proactive AI assistance. Real-world agents need to anticipate user needs before explicit instructions are given; EgoIntent provides the metrics to measure this capability.
• Future Research Direction: The results highlight the need for models that can better integrate temporal context, understand procedural hierarchies, and reason about "pre-outcome" states without relying on hindsight.

In summary, EgoIntent establishes a new standard for evaluating how well AI can "read the room" in first-person videos, proving that while MLLs can recognize actions, they still struggle to understand the intent behind them and predict the future in real-time.