EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes

Technical Summary: EgoGapBench

Problem Statement
Existing benchmarks for egocentric vision primarily rely on first-person-view (FPV) data (e.g., Ego4D, EPIC-KITCHENS). The authors argue that these benchmarks conflate two distinct abilities: understanding FPV input (interpreting visual cues like hands, feet, or camera distortion) and understanding an egocentric perspective (adopting an observer's standpoint to determine appropriate actions). In current datasets, models often succeed by detecting visible body cues of the camera wearer rather than reasoning about the viewpoint itself. Consequently, it is difficult to isolate whether a model genuinely understands the egocentric perspective or merely recognizes first-person artifacts. Furthermore, in multi-agent scenes, models struggle to distinguish between actions available to the observer and actions performed by other visible agents, often incorrectly selecting the latter.

Methodology: EgoGapBench
To address this, the authors introduce EgoGapBench, a diagnostic benchmark designed to measure Egocentric Action Selection (EAS). EAS is defined as the ability to select an appropriate action from an agent's perspective in the presence of other agents, without relying on visible first-person body cues.

The benchmark construction adheres to two core design principles:

1. No First-Person-Specific Cues: Images are drawn from general datasets (COCO, Wikimedia) and filtered to exclude visible camera-wearer body parts (hands, feet) or FPV-specific distortions.
2. Singleton Constraint: Distractor options are constructed using "singleton actions"—actions that cannot be duplicated by multiple agents in the same situation (e.g., a batter swinging a bat). If a visible agent is already performing a singleton action, the viewpoint agent cannot perform it. This makes copying another agent's action a structurally incorrect error rather than a plausible alternative.

The dataset consists of 1,000 items derived from 440 images, featuring multi-agent scenes. Each item presents a question asking for the next appropriate action, with options including:

• Correct: An action available from the assigned viewpoint.
• Motor Intrusion: An action observed being performed by another agent in the scene (structurally incorrect due to the singleton constraint).
• Off-Vantage: An action that fits the activity but requires a position the viewpoint agent does not occupy.

Items are formatted as 3-option (n3) or 4-option (n4) multiple-choice questions.

Key Contributions

1. Formalization of EAS: The paper formally separates egocentric perspective understanding from first-person-view understanding, defining EAS as a specific capability required for action selection in multi-agent scenarios.
2. Diagnostic Benchmark: The introduction of EgoGapBench, which utilizes singleton-action distractors to create a clear diagnostic signal for self-other confusion.
3. Empirical Analysis: A comprehensive evaluation of open-source and proprietary Multimodal Large Language Models (MLLMs) revealing a significant gap between human and model performance, characterized by a systematic tendency to copy observed actions.

Results

• Human-Model Gap: Humans achieve 94.5% accuracy on EgoGapBench. In contrast, the strongest proprietary model (GPT-5.4) achieves only 66.1%, while most open-source models perform near or below chance (random guessing).
• Systematic Error Patterns: Models do not fail randomly; they systematically select "motor intrusion" errors (actions performed by other visible agents). This tendency is particularly strong for "direct" actions (active interventions) compared to "indirect" actions (passive responses like observing).
• Ineffectiveness of Existing FPV Tuning: Fine-tuning models on existing egocentric instruction datasets (EgoIT-99K) fails to improve EAS performance. In fact, it degrades performance on EgoGapBench by an average of 33.9%, while leaving performance on standard FPV scene understanding benchmarks (EgoThink) largely unchanged. This suggests that existing FPV data teaches models to rely on body cues, which are absent in EgoGapBench, leading to negative transfer.
• EAS-Specific Tuning: Fine-tuning on EgoGapBench training data improves accuracy (e.g., Qwen3.5 rises from 29.0% to 75.2%) but does not close the gap to human performance. Furthermore, while "motor intrusion" errors decrease, "off-vantage" errors increase, indicating that the models learn to avoid copying others but still struggle with precise viewpoint localization.
• Robustness: The performance gap persists across different prompt formulations (including explicit instructions to ignore other agents' actions) and out-of-domain datasets (Wikimedia), suggesting the issue is fundamental to the models' current architecture or training paradigms.

Significance
The paper claims that EAS is a distinct capability that cannot be acquired solely from first-person-view data. The results indicate that current MLLMs, even those fine-tuned on egocentric data, lack the ability to ground action selection in a specific viewpoint when body cues are absent and other agents are present. The authors posit that improving EAS requires structural advancements beyond standard scene understanding, moving models toward truly adopting an observer's standpoint rather than merely interpreting visual artifacts. EgoGapBench is proposed as a necessary diagnostic tool to measure progress toward this goal.