EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

1. Problem Statement

Egocentric video understanding involves analyzing first-person videos captured from the actor's viewpoint. This setting presents unique challenges compared to standard third-person (exocentric) video analysis:

• Partial Observability & Occlusion: The camera often misses objects or actions due to the actor's body or hands blocking the view.
• Continuous Motion: Rapid camera movement and shifting viewpoints make temporal alignment difficult.
• Procedural Reasoning Deficits: Current Multimodal Large Language Models (MLLMs) struggle to predict how actions unfold over time. They often generate plausible-sounding but visually inconsistent responses, fail to maintain coherent object representations, and suffer from "catastrophic forgetting" of general (exocentric) knowledge when fine-tuned on egocentric data.

Existing methods either rely on sparse, rule-based rewards (limiting complex reasoning) or require synchronized ego-exo video pairs (which are scarce and expensive to collect).

2. Methodology: The EgoVITA Framework

EgoVITA (Egocentric Video Intelligence and Thinking Agent) introduces a Plan-then-Verify framework that decomposes reasoning into two distinct, complementary stages, trained via Reinforcement Learning (RL).

A. Reasoning Decomposition

The model generates a structured reasoning trajectory $Y$ consisting of three segments:

1. Egocentric Planning (<ego_plan>): The model predicts a causal sequence of anticipated actions from a first-person perspective (e.g., "1. Pick up knife, 2. Place on board"). This stage specializes in procedural reasoning and action ordering.
2. Exocentric Verification (<exo_verify>): The model audits the generated plan from a third-person perspective. It assesses whether the predicted actions are physically plausible given the scene layout, object affordances, and observable context (e.g., "The knife is visible on the counter; the board is within reach"). This stage acts as a consistency check without requiring paired ego-exo video data.
3. Final Answer (<answer>): The final response is conditioned on both the plan and the verification.

B. Training Strategy

The framework is trained in two phases using Group Relative Policy Optimization (GRPO):

• Stage I: Supervised Fine-Tuning (SFT):

  • Initializes the model to produce the three-part structure.
  • Uses human-annotated datasets (EgoProceL for planning, HD-EPIC for verification) to generate high-quality chain-of-thought traces via a teacher model (Qwen2.5-VL-72B).
  • Filters and verifies traces using Seed1.5-VL to ensure quality.

• Stage II: GRPO Optimization:

  • Samples groups of candidate trajectories (rollouts) for each input.
  • Updates the policy based on the relative quality of these trajectories using a Composite Reward Function ($R$) consisting of four components:
    1. Format Reward ($R_{format}$): Ensures correct output structure.
    2. Answer Reward ($R_{answer}$): Measures alignment with ground-truth labels.
    3. Anticipatory Cross-Modal Grounding (ACMG) Reward ($R_{ACMG}$): A dense reward that aligns predicted plan clauses with future visual frames. It projects text embeddings of plan steps into visual space and calculates cosine similarity with the next $N=16$ frames. This encourages the model to "look ahead" and predict future states.
    4. Confidence Reward ($R_{confidence}$): A dense reward that reinforces consistent third-person validation. It uses a teacher-guided warm-up phase followed by self-ranking to prefer verifications that logically reconcile the plan with visual evidence.

• Exocentric Regularization: To prevent catastrophic forgetting of general video understanding, the training interleaves GRPO updates with a lightweight cross-entropy loss on an exocentric dataset (MSR-VTT). This ensures the model retains third-person reasoning capabilities while specializing in egocentric tasks.

3. Key Contributions

1. Dual-Perspective Framework: EgoVITA is the first framework to explicitly separate egocentric planning from exocentric verification, enabling cross-perspective feedback without requiring synchronized ego-exo video pairs.
2. Dense Reward Mechanisms:
   • ACMG: Introduces a mechanism to ground procedural plans in future visual observations, forcing the model to learn temporal dynamics rather than static scene features.
   • Confidence Reward: Refines the verification stage through teacher-guided warm-up and self-ranking.
3. Generalization: The approach achieves state-of-the-art performance on egocentric benchmarks while maintaining (and often improving) performance on standard exocentric benchmarks, solving the trade-off between specialization and generalization.

4. Experimental Results

The authors evaluated EgoVITA on multiple foundation models (Qwen2.5-VL-7B, InternVL3.5-8B, Qwen3-VL-8B) across various benchmarks.

• Egocentric Performance:

  • Qwen2.5-VL-7B: Improved by +7.7 on EgoBlind (visual assistance for the blind) and +4.4 on EgoOrient.
  • InternVL3.5-8B: Improved by +3.6 on EgoBlind and +3.8 on EgoOrient.
  • Outperformed the previous state-of-the-art (EgoThinker) significantly, despite using only 47k training samples compared to EgoThinker's 5M samples.

• Exocentric Performance:

  • Unlike prior methods that degrade on general video tasks, EgoVITA maintained or improved scores on benchmarks like Video-MME, MVBench, and DocVQA.
  • For example, InternVL3.5 gained +2.7 on Video-MME, demonstrating that the model did not suffer from catastrophic forgetting.

• Ablation Studies:

  • ACMG vs. Present Grounding: Grounding in future frames (ACMG) yielded significantly higher gains (+7.7 on EgoBlind) compared to grounding only in the current frame (+4.1).
  • Teacher Warm-up: Removing the teacher-guided phase reduced performance, confirming the utility of the initial alignment.
  • Regularization: Without exocentric regularization, performance on third-person tasks dropped; with it, performance recovered and surpassed the base model.

5. Significance

• Solving the Ego-Exo Gap: EgoVITA provides a scalable solution to transfer knowledge between first-person and third-person views without expensive paired data, addressing a major bottleneck in egocentric AI.
• Procedural Reasoning: By forcing models to anticipate future visual states and verify them from an external perspective, the framework significantly improves the model's ability to understand complex, multi-step human activities.
• Efficiency: The method achieves superior results with a fraction of the training data required by previous large-scale fine-tuning approaches, making it more accessible and efficient.
• Safety and Reliability: In assistive applications (e.g., for the blind), the verification step acts as a safety mechanism, reducing hallucinations and ensuring that suggested actions are visually grounded and physically possible.

In conclusion, EgoVITA represents a paradigm shift in egocentric video understanding, moving from simple pattern matching to structured, self-correcting procedural reasoning through a novel plan-and-verify architecture.