EgoSelf: From Memory to Personalized Egocentric Assistant

1. Problem Statement

Egocentric assistants aim to provide personalized services by analyzing first-person (ego-centric) video data. However, current approaches face two critical challenges:

1. Unstructured Data Organization: Egocentric video is a continuous, long, and unstructured multimodal stream. Existing methods (like Long-context VLMs or simple RAG) treat content uniformly or rely on semantic similarity, failing to model the structured relationships between interactions, entities, and time. As user history grows, simple similarity-based retrieval suffers from precision degradation due to irrelevant but semantically similar entries.
2. Lack of Genuine Personalization: Most systems focus on factual recall (e.g., "Where did I put my keys?") rather than internalizing user-specific behavioral patterns. True personalization requires understanding latent habits, preferences, and routines to anticipate future actions, which current frameworks do not address simultaneously with long-term data organization.

2. Methodology: The EgoSelf Framework

EgoSelf addresses these challenges by drawing inspiration from Conway's Self-Memory System (SMS), which distinguishes between episodic memory (specific events) and semantic self-memory (generalized knowledge). The framework consists of three core components:

A. Graph-Based Interaction Memory

Instead of a flat list of events, EgoSelf constructs a heterogeneous interaction graph ($G = (V, E)$) to structure long-term observations:

• Nodes:
  • Event Nodes ($V_{event}$): Discrete interaction events extracted from video segments. Each node contains a language caption, involved objects, speech content, location, and timestamp.
  • Entity Nodes ($V_{entity}$): Persistent representations of objects and people. These act as stable anchors connecting disjoint events over time.
• Edges:
  • Event-Event Edges: Captures Causal (prerequisite $\to$ consequence), Co-activity (same high-level activity), and Temporal relationships.
  • Event-Entity Edges: Links events to the specific objects or people involved.
• Benefit: This structure transforms fragmented observations into a queryable memory that supports efficient retrieval and pattern-level reasoning.

B. User Profile Construction

To complement the fine-grained graph, EgoSelf generates a compact User Profile ($P$):

• Process: Discrete action records are grouped by semantic meaning and clustered. High-frequency patterns are retained while sporadic behaviors are filtered out.
• Output: An LLM summarizes these clusters into high-level preference descriptions (e.g., "Prefers fried food," "Routine: Go to Dining Hall $\to$ Order Fast Food").
• Role: This profile provides stable contextual priors for long-term reasoning, ensuring consistency even when specific event details are distant in time.

C. Self-Supervised Habit Learning

To internalize user behavior, EgoSelf introduces a dedicated training task:

• Objective: Predict future interactions based on historical interaction chains within the graph.
• Mechanism: The system splits the interaction graph into "past" (history) and "future" segments. The model is trained to predict a summary of the future events given the past graph structure and video frames.
• Data Generation: A reasoning LLM identifies suitable partition points based on causal/temporal logic, and a verifier refines the captions to ensure high-quality training pairs.
• Goal: This forces the model to learn individual behavioral regularities and latent preferences rather than just memorizing facts.

D. Retrieval Strategy

When a user query arrives, EgoSelf employs a Filter-then-Expand retrieval mechanism:

1. Embedding Similarity: Select top-$k$ candidate event and entity nodes.
2. Graph-Constrained Filtering: Retain only event nodes connected to the selected entity nodes. This ensures retrieved events are grounded in relevant entities, reducing noise.
3. Context Expansion: Expand the filtered set by including multi-hop neighboring events to provide comprehensive temporal and causal context.

3. Key Contributions

1. EgoSelf Framework: A novel architecture that organizes long-term first-person observations into a heterogeneous interaction graph with persistent entity nodes and compact user profiles.
2. Habit Learning Task: A self-supervised learning objective that predicts future interactions from historical graph data, enabling the model to internalize user-specific behavioral patterns.
3. State-of-the-Art Performance: Demonstrated superior performance on the EgoLifeQA benchmark across diverse personalization tasks, outperforming strong baselines including EgoGPT, EgoRAG, and various LLMs.

4. Experimental Results

The system was evaluated on the EgoLife dataset (6 participants, 7 days of continuous recording).

• Overall Performance: EgoSelf achieved the highest average score (40.6%) on the EgoLifeQA benchmark, surpassing the second-best baseline (EgoGPT at 36.0%). It showed significant gains in HabitInsight (44.3% vs. 31.1%) and EventRecall (42.9% vs. 36.5%).
• Long-Context Robustness:
  • In tasks requiring reasoning over histories longer than 24 hours, EgoSelf maintained high accuracy (38.8%), whereas baselines like EgoGPT+EgoRAG dropped significantly.
  • EgoSelf demonstrated stable retrieval performance (Hit@7) over a 7-day period, while EgoRAG showed continuous performance degradation as the temporal window expanded.
• Qualitative Analysis: Case studies showed that EgoSelf successfully retrieves causally and temporally connected events, filtering out noise. In contrast, baselines often retrieved isolated, irrelevant events, leading to fragmented and incorrect reasoning.

5. Significance

EgoSelf represents a paradigm shift in egocentric AI from passive retrieval to active behavioral modeling.

• Bridging the Gap: It successfully bridges the gap between organizing massive multimodal data and internalizing the "self" (habits and preferences) required for true personalization.
• Scalability: The graph-based approach solves the "retrieval precision degradation" problem common in long-term RAG systems by enforcing structural constraints (entity grounding) during retrieval.
• Future Impact: By enabling assistants to anticipate user needs based on learned behavioral patterns rather than just past facts, EgoSelf paves the way for truly adaptive, context-aware, and natural human-AI collaboration in AR/VR and wearable computing.