EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answering

This paper introduces EgoCross, a comprehensive benchmark comprising 1,000 QA pairs across four challenging domains (surgery, industry, extreme sports, and animal perspective) to evaluate and expose the poor cross-domain generalization capabilities of current Multimodal Large Language Models in egocentric video question answering.

The Gist

Imagine you've trained a brilliant student to be the ultimate "House Helper." This student has watched thousands of hours of videos about cooking, cleaning, and gardening. They are a wizard at answering questions like, "What ingredient is the chef adding?" or "When does the soup start boiling?"

Now, imagine you hire this same student to work in a surgical theater, a factory floor, a ski jump, or even strapped to the back of a cat.

Suddenly, the student freezes. They can't tell a scalpel from a screwdriver. They get dizzy trying to predict where a skier will land next. They have no idea what a cat sees when it chases a laser pointer.

This is exactly the problem the paper EgoCross is solving.

The Problem: The "Daily Life" Bubble

For a long time, AI researchers have been testing video-understanding AI (called Multimodal Large Language Models, or MLLMs) only on "daily life" videos. It's like only testing a car on a smooth, flat parking lot. The AI gets an A+ because it's never seen a pothole, a mountain road, or a muddy trail.

But in the real world, AI needs to work everywhere. If a robot is going to help a surgeon, it can't just know how to chop vegetables; it needs to know how to hold a surgical tool. If a drone is going to help a firefighter, it needs to understand smoke and fire, not just a sunny picnic.

The current AI models are like that "House Helper" student: they are great at home, but they panic when they leave the house.

The Solution: EgoCross (The "Field Trip" Exam)

The authors created a new test called EgoCross. Think of it as a "Field Trip Exam" designed to see if these AI students can handle totally different environments.

Instead of just cooking videos, EgoCross tests the AI in four very different, high-stakes worlds:

1. Surgery: Watching a surgeon's hands from their own eyes. The tools are tiny, the movements are precise, and the vocabulary is medical.
2. Industry: Watching someone repair complex circuit boards. It's about logic, tools, and assembly lines.
3. Extreme Sports: First-person views of skiing or skydiving. The camera is shaking, moving fast, and the scenery is wild.
4. Animal Perspective: Videos from the point of view of a dog, cat, or turtle. The world looks different (lower to the ground, different motion), and the "human" logic doesn't apply.

The Test: "What Happens Next?"

The exam isn't just "What do you see?" It's much harder. The AI has to answer questions like:

• Prediction: "The surgeon just cut this artery; what tool will they use next?"
• Localization: "Where exactly was the screwdriver when the hand touched it?"
• Counting: "How many different tools did the mechanic use in this 10-second clip?"
• Recognition: "Is that a 'grasper' or a 'scissors'?" (Even though they look similar).

The test includes about 1,000 questions, split into two types:

• Multiple Choice (CloseQA): Like a standard quiz.
• Free Response (OpenQA): Like a short essay where the AI has to explain its reasoning.

The Results: The AI Got Stuck

The authors ran the test on the smartest AI models available today (including the big ones from Google and OpenAI).

The verdict? Most of them failed miserably.

• On the "daily life" tests, they scored high.
• On the EgoCross test, their scores dropped dramatically. They were barely better than guessing randomly.

It turns out, these AIs are "specialists" in daily life, not "generalists." They haven't learned the deep rules of how the world works; they just memorized patterns of cooking and cleaning. When the visual style and the logic change (like switching from a kitchen to an operating room), they get lost.

The Silver Lining: Trying to Fix It

The paper didn't just point out the problem; they tried to fix it. They tested three methods to help the AI "study" for the new subjects:

1. Prompting: Giving the AI a "cheat sheet" of context (e.g., "Remember, you are in a hospital"). Result: A little help, but not enough.
2. Fine-Tuning: Letting the AI study a small amount of the new videos. Result: Better, but it only learned the specific examples it saw.
3. Reinforcement Learning: Letting the AI try, fail, get feedback, and try again (like training a dog with treats). Result: This was the winner! It helped the AI adapt much better to the new worlds.

The Big Takeaway

EgoCross is a wake-up call. It tells us that while AI is getting very good at understanding our daily lives, it is still very fragile when it steps out of its comfort zone.

If we want AI to truly help us in hospitals, factories, or rescue missions, we can't just train it on YouTube cooking videos. We need to build models that are robust enough to handle the messy, weird, and specialized parts of the real world. This paper provides the map and the exam to help us get there.

Technical Summary

Here is a detailed technical summary of the paper "EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answering."

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced significantly in Egocentric Video Question Answering (EgocentricQA), existing benchmarks and models are heavily biased toward common daily-life activities (e.g., cooking, cleaning, gardening).

• The Gap: Real-world deployment of egocentric AI (e.g., in surgery, industrial repair, or wildlife monitoring) requires handling domain shifts where visual styles, semantic contexts, and object interactions differ drastically from daily life.
• The Question: Can current MLLMs generalize effectively to these unfamiliar, high-stakes, domain-specific scenarios?
• Current Limitation: Existing datasets (like EgoSchema or EgoTaskQA) lack the diversity to test cross-domain robustness, leading to an overestimation of model capabilities in practical applications.

2. Methodology: The EgoCross Benchmark

The authors introduce EgoCross, a comprehensive benchmark designed to evaluate cross-domain generalization in EgocentricQA.

A. Dataset Construction

• Domains: EgoCross covers four distinct, challenging domains that deviate significantly from daily life:
  1. Surgery: High-precision tool interactions and procedural phases (sources: EgoSurgery, CholecTrack20).
  2. Industry: Complex workflows, circuit board repair, and tool logic (source: ENIGMA-51).
  3. Extreme Sports: High-speed motion, blur, and rare environments (source: ExtremeSportFPV).
  4. Animal Perspective: Non-human viewpoints with unique motion patterns and scales (source: EgoPet).
• Scale: The dataset comprises 798 video clips and approximately 1,000 QA pairs.
• Task Taxonomy: The benchmark evaluates four core capabilities, decomposed into 15 specific sub-tasks:
  • Identification: Recognizing objects, actions, and events (e.g., specific surgical instruments, animal species).
  • Localization: Identifying spatial (region) and temporal (timestamp) locations of interactions.
  • Prediction: Forecasting future actions, next phases, or movement directions based on causal reasoning.
  • Counting: Tracking and counting distinct instances in dynamic environments.
• Format: Each QA pair is provided in two formats for rigorous evaluation:
  • CloseQA: Multiple-choice (4 options) to test discriminative ability.
  • OpenQA: Free-form generation to test generative and reasoning capabilities.

B. Data Curation Pipeline

The authors employed a multi-stage pipeline to ensure high quality:

1. Meta Annotation Refinement: Unified disparate annotation formats from source datasets and performed manual reviews to ensure ground-truth accuracy.
2. Template Design: Created initial QA templates for the 15 sub-tasks, then used an LLM (Gemini 2.5 Pro) to expand them with domain-specific variations.
3. Quality Control: Human annotators verified all generated questions for clarity, logic, and answerability. A final 10% random sampling check was performed.

3. Experimental Setup & Evaluation

• Models Evaluated: 8 State-of-the-Art (SOTA) MLLMs were tested in a zero-shot setting:
  • Proprietary: GPT-4.1, Gemini 2.5 Pro.
  • Open-Source General: Qwen2.5-VL (3B/7B), VideoLLaMA3, InternVL3.
  • Egocentric-Specialized: EgoVLPv2, EgoGPT.
• Metrics:
  • CloseQA: Standard accuracy.
  • OpenQA: Exact match + LLM-as-a-Judge (using Qwen-MAX) to evaluate semantic correctness.
• Baseline Comparison: Performance was compared against EgoSchema (daily-life domain) to quantify the domain shift penalty.

4. Key Results

• General Struggle: Most models struggle significantly on EgoCross.
  • CloseQA Accuracy: Average < 55% (random chance is 25%).
  • OpenQA Accuracy: Average < 35%.
  • Specialized Models Fail: Surprisingly, models specifically trained on egocentric data (EgoVLPv2, EgoGPT) performed worse than general-purpose models, highlighting that daily-life egocentric training does not transfer to specialized domains.
• Domain Variance:
  • Animal Perspective was the easiest (closest to daily life).
  • Industry and Extreme Sports were the most difficult due to complex workflows and rapid motion.
  • Surgery showed a unique pattern: models performed relatively well visually but struggled with deep semantic reasoning (e.g., predicting the next surgical phase).
• Domain Shift Penalty: When comparing the same task types between EgoSchema (in-domain) and EgoCross (cross-domain), accuracy dropped drastically (e.g., Action Temporal Localization dropped from 92.3% to 34.1%).
• Pilot Studies (Improvement Strategies):
  • Prompt Learning: Provided modest gains.
  • Supervised Fine-Tuning (SFT): Improved performance in some domains (e.g., Industry) but failed in others (Animal Perspective) due to limited training data.
  • Reinforcement Learning (RL): Showed the most significant improvement (+22% average), suggesting that learning from feedback helps models adapt to complex decision-making in new domains.

5. Key Contributions

1. First Cross-Domain Benchmark: Introduced EgoCross, the first benchmark specifically designed to evaluate MLLM generalization across diverse, non-daily-life egocentric domains.
2. Comprehensive Taxonomy: Defined a structured framework covering 15 sub-tasks across Identification, Localization, Prediction, and Counting.
3. Empirical Evidence of Limitations: Demonstrated that current SOTA MLLMs, even those specialized for egocentric vision, lack robustness in cross-domain settings, with a massive performance gap compared to daily-life benchmarks.
4. Actionable Insights: Provided pilot studies showing that Reinforcement Learning is a promising direction for improving cross-domain generalization, while SFT alone is insufficient without sufficient domain-specific data.

6. Significance

EgoCross addresses a critical bottleneck in the deployment of Embodied AI and wearable assistants. By proving that current models fail in high-stakes, specialized environments (like surgery or industrial repair), the paper calls for a shift in research focus from "daily life" benchmarks to domain-robust models. It establishes a new standard for evaluating whether AI can truly understand and interact with the physical world beyond the kitchen and living room.

Code & Resources:

• GitHub: https://github.com/MyUniverse0726/EgoCross
• Challenge Website: https://egocross-benchmark.github.io/