Technical Summary: EgoCoT-Bench

Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced video understanding, existing benchmarks for egocentric (first-person) video understanding suffer from significant limitations. Current datasets often lack grounded rationale evaluation, failing to support fine-grained operation-centric reasoning. Specifically, they rarely examine whether a model's generated reasoning is actually grounded in explicit spatio-temporal evidence. In egocentric scenarios, objects are frequently partially visible, occluded by hands, or intermittently leave the field of view, requiring models to reason over temporally evolving evidence rather than relying on instantaneous frames. Existing benchmarks either focus on generic third-person video QA or provide limited support for explicit rationale supervision and fine-grained spatial grounding in first-person manipulation tasks.

Methodology: EgoCoT-Bench Construction

To address these gaps, the authors introduce EgoCoT-Bench, a fine-grained benchmark designed for grounded and verifiable operation-centric reasoning. The construction follows a rigorous Spatio-Temporal Scene Graph (STSG)-guided generation framework combined with human refinement:

1. Data Collection: The benchmark aggregates 351 egocentric video clips from diverse sources, including public datasets (Ego4D, EPIC-KITCHENS, MECCANO, Charades-Ego, HD-EPIC) and self-recorded videos, covering daily activities, kitchen tasks, and structured assembly.
2. STSG-Guided Generation:
   • Video clips are converted into ego-adapted Spatio-Temporal Scene Graphs (STSG), serving as an intermediate, structured representation.
   • STSGs are manually refined to correct unreliable object identities, inconsistent interaction links, and ambiguous state transitions.
   • Candidate QA pairs are derived by traversing task-specific evidence paths within the STSG, connecting answers to concrete first-person interaction cues (timestamps, bounding boxes, action history).
   • An LLM renders these verified structural facts into natural language questions, options, and rationales, ensuring the underlying evidence is not invented.
3. Human Refinement: A multi-round manual screening process ensures semantic correctness, egocentric relevance, and evidence consistency. Annotators verify that questions, answers, rationales, and supporting evidence are mutually consistent.
4. Task Taxonomy: The benchmark comprises 3,172 QA pairs organized into four task groups and 12 fine-grained subtasks:
   • Egocentric Grounding & Perception: Active Object Grounding (AOG), Hand-Object Association (HOA), Manipulation State Perception (MSP).
   • Spatio-Temporal Retrospection: State Retrospection (SR), Location/Visibility Retrospection (LVR), Hand-Object Temporal Retrospection (HOTR).
   • Predictive & Causal Inference: Next State Anticipation (NSA), Next Action Anticipation (NAA), Local Cause-Outcome Inference (LCOI).
   • High-level Grounded Reasoning: Progress Reasoning (PR), Hand-Object Grounded Chain of Thought (HGC), Goal-Oriented Object Tracking (GOT).
5. Evaluation Protocol: The benchmark employs a three-metric protocol to assess models beyond simple answer accuracy:
   • Answer Accuracy (Acc): Strict exact-match on multiple-choice questions.
   • Reasoning Score (R): A 0–5 scale score (judged by a strong LLM) assessing logical coherence, factual consistency, and alignment with visual evidence.
   • Spurious Correct Rate (SCR): The proportion of cases where the model answers correctly but provides weak or ungrounded reasoning (R ≤ 2).

Key Results

The authors evaluated a broad set of 4 proprietary and 15 open-source MLLMs (including GPT-5, Qwen, LLaVA, and InternVL series) on EgoCoT-Bench:

• Overall Performance: The benchmark remains highly challenging. The best-performing model, Qwen3.5-27B, achieved an overall accuracy of 71.28%, significantly trailing human performance (95.93%).
• Task Difficulty:
  • Predictive & Causal Inference was the most tractable group (best accuracy: 78.21%).
  • Spatio-Temporal Retrospection and High-level Grounded Reasoning proved most difficult, with best accuracies of 69.10% and 68.08%, respectively.
  • Goal-Oriented Object Tracking (GOT) was the hardest subtask (best accuracy: 45.28%), while Manipulation State Perception (MSP) was the easiest (87.79%).
• Reasoning vs. Accuracy: A critical finding is the misalignment between answer correctness and reasoning quality. For instance, while Qwen3.5-27B had the highest accuracy, Qwen3-VL-Plus achieved the highest mean Reasoning Score (3.08/5). Conversely, GPT-5.2 demonstrated the lowest Spurious Correct Rate (4.27%), indicating the most faithful reasoning among correct answers.
• Spurious Correctness: Many models exhibited high Spurious Correct Rates, particularly in High-level Grounded Reasoning tasks, suggesting they often guess the correct option without providing reasoning grounded in the video's spatio-temporal evidence.

Significance and Claims

The paper claims that EgoCoT-Bench serves as a necessary testbed for advancing grounded, verifiable, and temporally coherent reasoning in egocentric video understanding. Its primary contributions are:

1. A Fine-Grained Benchmark: It provides the first large-scale dataset specifically targeting operation-centric reasoning with explicit step-by-step rationale annotations and spatio-temporal grounding.
2. A Structured Construction Pipeline: It demonstrates an STSG-guided generation and human refinement pipeline that ensures every sample is traceable to explicit visual evidence, moving beyond free-form video descriptions.
3. A Faithful Evaluation Protocol: By introducing the Reasoning Score and Spurious Correct Rate, the authors advocate for evaluation protocols that explicitly assess whether model reasoning is consistent with underlying evidence, rather than relying solely on final answer accuracy.

The authors conclude that current MLLMs struggle with evidence grounding and rationale consistency in first-person scenarios. They posit that future progress in embodied AI and assistive systems requires models that can justify decisions using verifiable spatio-temporal evidence, a capability EgoCoT-Bench is designed to measure and improve.