💻 computer science

EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos

The paper introduces EgoExoMem, the first benchmark for cross-view memory reasoning using synchronized egocentric and exocentric videos, along with the training-free E $^2$ -Select method that leverages complementary dual-view cues to achieve state-of-the-art performance on this challenging task where current models struggle.

Original authors: Ruiping Liu, Junwei Zheng, Yufan Chen, Di Wen, Shaofang Quan, Chengzhi Wu, Jiaming Zhang, Kailun Yang, Kunyu Peng, Rainer Stiefelhagen

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruiping Liu, Junwei Zheng, Yufan Chen, Di Wen, Shaofang Quan, Chengzhi Wu, Jiaming Zhang, Kailun Yang, Kunyu Peng, Rainer Stiefelhagen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery in a busy kitchen. You have two ways to remember what happened:

The "Chef's View" (Egocentric): You wear a GoPro on your forehead. You see exactly what your hands are doing, the spices you grab, and the knife you use. But you can't see the person standing behind you, or the whole layout of the room.
The "Security Camera View" (Exocentric): You have a camera mounted on the ceiling. You see the whole room, everyone moving around, and where things end up. But you can't see the fine details of what the chef is holding in their hand or the specific expression on their face.

The Problem:
For a long time, AI researchers have only built "memories" for robots based on the Chef's View. The paper argues this isn't enough. If you only have the Chef's View, you might miss that someone else moved a chair. If you only have the Security Camera, you might miss that the chef cut the onion into tiny pieces.

The Solution: EgoExoMem
The authors created a new "exam" called EgoExoMem. It's a giant test bank of 2,600 questions designed to trick AI. These questions can only be answered if the AI combines both the Chef's View and the Security Camera View at the same time.

Example Question: "Where did the second person put the bowl after the chef handed it to them?"
- The Chef's View sees the handoff but loses the bowl once it leaves the frame.
- The Security Camera sees the bowl moving across the room but might miss the exact moment of the handoff.
- The AI needs both to get the answer right.

The Results: AI is Still Struggling
The authors tested the smartest AI models available (like Gemini and others) on this exam.

The Score: The best AI got about 55% correct. That's barely passing a high school test.
The Lesson: Even the most advanced AI struggles to "remember" and "reason" when it has to juggle two different camera angles at once. They often get confused about which view to trust.

The New Tool: E2-Select
Since AI is bad at looking at every single frame of two long videos (it's too much data), the authors invented a smart "highlight reel" maker called E2-Select.

Think of it like a film editor who has to cut a 10-minute video down to 32 seconds for a movie trailer.

Old Way: Just pick frames randomly or pick frames that look "important" in one camera.
E2-Select Way: It acts like a detective. It asks, "What is the question asking?"
- If the question is about where something is in the room, it picks more frames from the Security Camera.
- If the question is about what the chef is holding, it picks more frames from the Chef's Camera.
- It then uses a special mathematical trick (called k-DPP) to make sure the selected frames don't all look the same (avoiding redundancy) and cover the whole story.

The Outcome:
When the AI used this new "highlight reel" tool, its score jumped to 58.2%. It's still not perfect, but it proved that the tool works better than previous methods.

The Big Surprise (The "Third Person" Glitch)
The researchers found a weird quirk. When asking about what a third person (someone other than the camera wearer) was doing, the AI actually performed better when it only looked at the Chef's View, even though the Security Camera sees the whole room better.

Why? The authors discovered that the questions were written in a way that made the AI focus on the Security Camera, but the answers were actually hidden in the Chef's View. It's like a riddle where the clue is in one room, but the answer is in another, and the AI got confused by the wording. This shows that simply having two cameras isn't enough; the AI needs to learn how to match the question to the right camera view.

In Summary:
This paper says: "Robots need to see the world from two angles to truly understand it. We built a test to prove current robots are bad at this, and we built a new tool to help them pick the best moments to remember. We are getting closer, but there is still a lot of work to do."

Technical Summary: EgoExoMem

Problem Statement

While multimodal large language models (MLLMs) are increasingly capable of perceiving and reasoning about complex environments, existing memory frameworks for embodied agents remain fundamentally limited by their reliance on purely egocentric (first-person) observations. This singular viewpoint creates blind spots regarding out-of-view regions, spatial relationships, and the interactions of other agents within a shared space. Although humans naturally encode experiences from both "field" (first-person) and "observer" (third-person) perspectives to support situation awareness, current benchmarks and methods largely treat these views independently or focus on unidirectional knowledge transfer. There is a lack of a standardized evaluation for cross-view memory reasoning, where an agent must synthesize synchronized egocentric and exocentric video streams to answer questions that are unresolvable from either viewpoint alone.

Methodology

1. EgoExoMem Benchmark

The authors introduce EgoExoMem, the first benchmark designed to evaluate memory-based reasoning over synchronized egocentric and exocentric videos.

Data Source: The benchmark utilizes 390 synchronized videos from the LEMMA and EgoExo4D datasets, focusing on scenarios with substantial environmental interaction (cooking and bike repair) and multi-agent activities.
Task Definition: The benchmark comprises 2,600 human-verified Multiple Choice Questions (MCQs) categorized into eight distinct reasoning types:
1. Habitual Location (HL): Persistent spatial map retention.
2. Instantaneous Position (IP): Bridging egocentric action with allocentric position.
3. Resulting Location (RL): Tracking object location after manipulation across event boundaries.
4. Egocentric Direction (ED): Fine-grained spatial awareness relative to the camera wearer.
5. Object State (OS): Identifying physical states of manipulated objects.
6. Allocentric Relation (AR): Spatial relations between objects from a third-person perspective.
7. Third Person Activity (TPA): Simultaneous spatial tracking of multiple agents.
8. Temporal Ordering (TO): Recalling the sequence of procedural sub-actions.
Quality Control: A rigorous pipeline ensures data quality, including human editing to correct ambiguities and a "text-only check" using GPT-4o to ensure questions are strictly vision-dependent (removing common-sense answers).

2. E2-Select: Frame Selection Method

To address the challenge of retrieving relevant information from dual, synchronized, and asymmetric video streams, the authors propose E2-Select, a training-free frame selection method. Unlike prior methods that treat views independently or use hard selection, E2-Select is designed to handle view asymmetry and cross-view temporal consistency through three stages:

Independent View Scoring: Query-frame relevance is computed separately for ego and exo views using CLIP cosine similarity. This prevents a dominant view from overwhelming the other during joint retrieval.
Relevance-Based Budget Allocation: The total frame budget ( $K$ ) is allocated proportionally to the aggregated relevance scores of each view. This allows the system to dynamically favor the view more relevant to the specific query while maintaining a soft balance.
Per-View k-DPP Sampling: To reduce temporal redundancy within each view, the method employs k-Determinantal Point Processes (k-DPP). This ensures the selected subset of frames is both relevant to the query and visually diverse, maximizing the volume spanned by frame embeddings in feature space.
Timestamp-Ordered Merge: Selected frames from both views are merged chronologically to preserve the temporal structure of the synchronized memory stream.

Key Results

Performance of MLLMs

Extensive evaluations of proprietary (Gemini 2.5 Flash) and open-source (InternVL3.5, LLaVA-OV, Qwen2.5-VL) MLLMs reveal significant challenges:

Complementarity: Combining ego and exo views consistently outperforms single-view inputs across all models. For instance, Gemini 2.5 Flash improves from 52.2% (Ego) and 49.5% (Exo) to 55.3% with Ego+Exo.
Model Limitations: Even the best-performing model achieves only 55.3% accuracy, indicating that current MLLMs struggle with comprehensive cross-view memory reasoning.
View Preferences: Specific QA types show distinct view dependencies. Instantaneous Position favors exocentric views, while Egocentric Direction favors egocentric views. Notably, Third Person Activity showed a counterintuitive preference for the egocentric view in baseline models, which the authors attribute to a mismatch between question framing and answer grounding.

Performance of E2-Select

E2-Select demonstrates state-of-the-art performance among memory retrieval strategies:

Benchmarking: E2-Select achieves an average accuracy of 58.2%, outperforming existing frame selection (AKS, BOLT), text retrieval (BM25, DPR), and visually-grounded retrieval (VideoRAG) baselines.
Ablation Studies: The study confirms that k-DPP sampling is more effective for cross-view integration than temporal structure estimation methods (like AKS/BOLT) which degrade under cross-view budget allocation. Furthermore, soft allocation (relevance-based) significantly outperforms hard view selection, proving the necessity of utilizing complementary evidence from both streams.

Significance and Claims

The paper claims that EgoExoMem fills a critical gap in embodied AI research by establishing the first benchmark for cross-view memory reasoning. Its significance lies in:

Demonstrating Insufficiency of Single Views: It provides empirical evidence that egocentric memory alone is insufficient for holistic spatial-temporal understanding, necessitating the integration of exocentric observations.
Highlighting Current Limitations: The low performance of state-of-the-art MLLMs (max 55.3%) underscores the complexity of the task and the need for specialized memory mechanisms.
Introducing E2-Select: The proposed method offers a simple yet effective, training-free solution for dual-view frame selection that explicitly addresses view asymmetry and temporal consistency, achieving a new state-of-the-art of 58.2%.
Revealing Systemic Conflicts: The analysis of failure cases, particularly the "view-preference conflict" in Third Person Activity questions, highlights a nuanced challenge where the view most relevant to the question may differ from the view most relevant to the answer, motivating future work on joint query-answer view routing.

The authors position EgoExoMem and E2-Select as foundational resources for advancing cross-view memory reasoning in embodied intelligence, particularly for applications in healthcare, smart homes, and multi-agent environments where comprehensive situational awareness is required.