Technical Summary: From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision

1. Problem Statement

Vision Language Models (VLMs) have demonstrated strong performance in general video understanding tasks but struggle to infer egocentric properties (e.g., fine-grained human-object interactions, specific hand manipulations) when observing exocentric videos (third-person, fixed-camera views). This limitation is critical for applications like Activities of Daily Living (ADL) monitoring, where wearable cameras are impractical, yet understanding specific interactions (e.g., "slicing a tomato" vs. "slicing an onion") is essential.

Existing VLMs are typically trained on mixed datasets containing both viewpoints or solely on exocentric data. Consequently, they learn to prioritize scene context and full-body motion, often missing subtle interaction cues that are visually prominent in egocentric views but small or partially occluded in exocentric views. The core challenge is: How can a VLM be trained to infer egocentric properties from exocentric inputs when egocentric data is unavailable at inference time?

2. Methodology: Ego2ExoVLM

The authors propose Ego2ExoVLM, a framework that leverages privileged egocentric supervision available only during training to teach a model to reason about exocentric inputs. The framework operates on time-synchronized ego-exo video pairs $(V^{ego}, V^{exo})$ and employs a teacher-student paradigm:

• Teacher ($T$): A frozen VLM that processes the egocentric video. It generates language sequences (answers) that inherently capture fine-grained egocentric details.
• Student ($S$): A trainable VLM that processes the exocentric video. It is supervised to reproduce the teacher's outputs, effectively learning to "see" egocentric details through the exocentric lens.

The method introduces two complementary components to facilitate this cross-view knowledge transfer:

A. Ego2Exo Sequence Distillation

Instead of standard feature-level distillation, the authors utilize language-level sequence distillation.

• The teacher generates a response $A^{ego}$ based on the egocentric view.
• The student is trained to auto-regressively maximize the likelihood of generating $A^{ego}$ given the exocentric video $V^{exo}$ and the query $Q$.
• Insight: The authors found that feature-level knowledge transfer (common in prior cross-view works) is ineffective for VLMs. Language serves as a more robust medium to bridge the viewpoint gap, forcing the student to align its reasoning with the teacher's egocentric perspective.

B. Ego Adaptive Visual Tokens

To address the issue that exocentric videos contain visual distractors (background, full-body motion) that dilute interaction cues, the authors introduce a set of learnable visual tokens.

• These tokens act as visual probes, interacting with intermediate representations of the vision encoder via cross-attention.
• They are supervised by the egocentric signal to selectively extract spatio-temporal regions corresponding to human-object interactions (HOIs) that are less prominent in the exocentric view.
• These tokens are projected into the language model's embedding space and concatenated with standard visual embeddings before decoding.

3. Key Contributions

1. Ego2ExoVLM Framework: The first VLM framework explicitly trained to infer egocentric properties from exocentric inputs using time-synchronized ego-exo pairs. It addresses the underexplored ego $\to$ exo transfer direction, distinct from the more common exo $\to$ ego transfer.
2. Novel Transfer Mechanisms:
   • Ego2Exo Sequence Distillation: Demonstrates that language-level distillation is superior to feature-level distillation for transferring viewpoint-specific reasoning in VLMs.
   • Ego Adaptive Visual Tokens: A mechanism to surface relevant interaction cues in exocentric representations, guided by privileged egocentric supervision.
3. Ego-in-Exo Perception Benchmark: A new benchmark consisting of 3,881 multiple-choice questions (MCQs) derived from the EgoExo4D dataset.
   • Design: Questions are curated exclusively from the egocentric viewpoint (ensuring they target egocentric properties), but evaluation is performed using only the paired exocentric videos.
   • Tasks: Action Understanding, Task-relevant Region identification, Human-Object Interactions (HOI), and Hand Identification.
4. Empirical Validation: Comprehensive evaluation showing state-of-the-art performance on the ADL-X benchmark suite and significant improvements over strong baselines on the proposed Ego-in-Exo Perception benchmark.

4. Experimental Results

Performance on Ego-in-Exo Perception

• Ego2ExoVLM achieved an average accuracy of 74.2%, outperforming the strong baseline VideoLLaMA3 (71.2%) and the exo-only fine-tuned variant EE4D-VideoLLaMA3 (65.7%).
• It significantly outperformed LLAVIDAL (33.3%), which struggles with ADL tasks despite being tailored for them.
• The model approached the "upper bound" performance of the teacher model (77.9%), which has direct access to the egocentric view.

Performance on ADL-X

• Ego2ExoVLM achieved 78.8% on ADL-MCQ tasks, surpassing VideoLLaMA3 (76.9%) and EE4D-VideoLLaMA3 (75.0%).
• It also led in video description tasks (Charades and Toyota Smarthome), demonstrating robustness in generating detailed, interaction-aware descriptions from fixed-camera inputs.

Ablation Studies

• Component Necessity: Simply training on both ego and exo videos without distillation yielded negligible gains (+0.3%). Both Sequence Distillation and Adaptive Tokens were required for optimal performance, with the tokens providing significant gains on HOI tasks.
• Transfer Strategy: Language-level distillation outperformed visual feature distillation (67.8% vs 74.2%).
• Robustness: The method remains effective even when trained on non-time-synchronized (pseudo-paired) ego-exo data, suggesting strict temporal alignment is not strictly required for the language-mediated transfer to work.

5. Significance and Claims

The paper claims that Ego2ExoVLM represents a fundamental shift in how VLMs can be adapted for ADL monitoring. By treating the egocentric view as a privileged supervisor, the framework enables models to recover interaction cues that are otherwise latent in exocentric observations.

Key claims include:

• First of its Kind: It is the first VLM to explicitly learn exocentric representations from privileged egocentric supervision.
• Superiority of Language Mediation: The work challenges the prevailing assumption that feature-level transfer is sufficient for cross-view tasks in VLMs, demonstrating that language-level supervision is more effective for bridging the ego-exo gap.
• Practical Applicability: The framework addresses a real-world constraint where egocentric cameras are impractical for deployment (e.g., elderly care), yet the detailed understanding they provide is necessary for safety and cognitive assessment.
• Generalizability: The approach is shown to be effective across different VLM backbones and even with non-synchronized training data, making it a viable solution for large-scale ADL monitoring systems.