EgoGraph: Temporal Knowledge Graph for Egocentric Video Understanding

EgoGraph is a training-free, dynamic knowledge graph framework that overcomes the limitations of existing methods in ultra-long egocentric video understanding by constructing a unified schema and temporal relational modeling to capture long-term cross-entity dependencies, thereby achieving state-of-the-art performance on long-term video question answering benchmarks.

The Gist

Imagine you are wearing a camera on your glasses for an entire week, recording every single moment of your life: what you eat, who you talk to, where you go, and what you do. Now, imagine trying to answer a question about that week, like, "Who did I have coffee with on Tuesday, and did they bring a blue mug?"

If you tried to remember this by just scrolling through hours of video, you'd get lost. If you tried to summarize it by writing a long diary entry for every hour, you'd end up with a messy, disorganized stack of papers that's hard to search.

EgoGraph is a new computer system designed to solve this exact problem. It turns your ultra-long, chaotic video life into a smart, living memory map.

Here is how it works, using some simple analogies:

1. The Problem: The "Messy Diary" vs. The "Smart Map"

Current AI systems try to understand long videos by breaking them into small chunks (like 1-minute clips) and summarizing each one separately.

• The Analogy: Imagine trying to understand a whole movie by reading a one-sentence summary of every scene, but the summaries are just piled in a random box. If you ask, "How did the hero's relationship with the villain change from the beginning to the end?" the AI has to dig through thousands of disconnected notes to find the connection. It often misses the big picture because it doesn't see how Scene A connects to Scene Z.

2. The Solution: EgoGraph (The "Living Memory Map")

EgoGraph doesn't just summarize; it builds a Knowledge Graph. Think of this as a giant, interactive mind-map or a subway system for your memories.

• The Nodes (Stations): Instead of just text, the system identifies key "stations" in your life: People (John), Objects (the yellow mug), Places (the kitchen), and Events (the meeting).
• The Edges (Tracks): It draws lines between these stations to show how they connect. "John" is connected to "the yellow mug" because he held it. "The yellow mug" is connected to "the kitchen" because it was there.
• The Time Travel (The Secret Sauce): This is the most important part. Every connection has a timestamp. It's not just a map; it's a time-traveling map. It knows that John held the mug on Day 1 at 9:00 AM, but on Day 3, he held a red cup.

3. How It Learns: The "Human Brain" Approach

The paper mentions that human memory works by organizing who, where, and what happened. EgoGraph copies this:

• The Schema (The Filing System): It has a strict rulebook (a schema) for what matters. It knows that "John" is a person, "coffee" is an object, and "kitchen" is a location. It doesn't get confused by random words.
• The Update (The Growing Tree): As you record more days, the graph doesn't just get bigger and messier. It merges things. If you see "John" on Day 1 and Day 5, the system realizes, "Oh, that's the same John!" and updates his profile with new habits or preferences, rather than creating a duplicate "John #2." It keeps the memory compact but rich.

4. Answering Questions: The "Detective"

When you ask a question like, "Who did I see in the kitchen yesterday?", EgoGraph acts like a detective:

1. Time Filter: It immediately ignores everything that happened after yesterday. It only looks at the "yesterday" part of the map.
2. Search: It zooms in on the "Kitchen" station and follows the tracks to see who was connected to it at that specific time.
3. Reasoning: If the question is tricky, like "Did I start drinking tea after I moved to the new house?", the system can trace the timeline of "Tea" and "New House" to see if the tea-drinking habit started after the move.

Why This Matters

Previous AI models were like students who studied for a test by memorizing isolated facts. If the question required connecting two facts from different days, they failed.

EgoGraph is like a student who keeps a structured, time-stamped journal. It can look back at a week of data, find the specific connections between people and objects, and answer complex questions about your daily life with high accuracy.

In short: EgoGraph turns a chaotic, endless video stream into a clean, organized, time-traveling map of your life, allowing computers to finally understand the "story" of your day, not just the individual frames.

Technical Summary

1. Problem Statement

The paper addresses the significant challenge of ultra-long egocentric video understanding, specifically videos spanning multiple days captured by wearable cameras (e.g., AR glasses, robots). Existing approaches face two primary limitations:

• Fragmented Processing: Current methods typically process videos as short, independent clips (often <1 hour) or rely on hierarchical summarization (e.g., hourly/daily captions). This breaks the continuity of long-term dependencies, making it difficult to reason about events separated by large time gaps.
• Limited Temporal Modeling: Most models lack explicit mechanisms to track when events occur and how they relate temporally. They often rely on static frames or commonsense reasoning rather than dynamic temporal dynamics, leading to "temporal leakage" (confusing past and future events) and inefficient retrieval in massive data spaces.

The core difficulty lies not just in recognizing what happens, but in capturing when it happens and establishing causal or associative links across days.

2. Methodology: EgoGraph

The authors propose EgoGraph, a training-free, dynamic knowledge graph framework designed to encode long-term, cross-entity dependencies in egocentric video streams.

A. Core Definition

EgoGraph is defined as a temporal knowledge graph $G = (V, E)$ where:

• Nodes ($V$): Represent entities (Person, Location, Object, Event) with attributes, names, and a list of timestamps ($T_v$) and descriptions ($D_v$).
• Edges ($E$): Represent relationships between entities, annotated with timestamps ($T_e$) and relation descriptions ($D_e$).
• Temporal Filtering: For any query at time $t_q$, the system retrieves a subgraph $G_{\le t_q}$ containing only nodes and edges observed up to that timestamp, ensuring causal consistency.

B. Construction Pipeline

1. Temporal Grounding: Long video transcripts and captions are partitioned into chunks, each anchored to a specific timestamp (the earliest time in the chunk).
2. Egocentric Schema: To ensure semantic consistency and avoid redundancy, the authors define a specific schema with four core entity types:
   • Person: Attributes include name, gender, appearance, habits, hometown.
   • Location: Name and description.
   • Object: Name, type, color, owner, condition.
   • Event: Name, description, start time, subject, object, location.
3. Extraction & Merging: Large Language Models (LLMs) extract entities and relations based on the schema. Crucially, the system employs a merging strategy:
   • Redundant nodes are merged based on textual embedding similarity.
   • New observations are appended as new timestamps and descriptions to existing nodes rather than creating duplicates.
   • This allows the graph to capture the evolution of an entity (e.g., a person's changing habits or an object's location changes) without exploding in size.

C. Question Answering Framework

The QA process involves:

1. Retrieval: Semantic keywords are extracted from the query. The system performs vector similarity search to retrieve top-$k$ relevant nodes and edges.
2. Temporal Filtering: A subgraph is constructed containing only information up to the query's timestamp ($t_q$).
3. LLM-Based Temporal Reasoning: The retrieved subgraph is fed to an LLM with explicit instructions to interpret time relative to $t_q$. The system handles natural language temporal expressions (e.g., "yesterday," "last time," "2 hours ago") by mapping them to the structured timestamp format [DAY HH:MM:SS].

3. Key Contributions

1. EgoGraph Framework: A novel, training-free temporal knowledge graph that explicitly models long-term dependencies in ultra-long egocentric videos, overcoming the fragmentation of clip-based methods.
2. Egocentric Schema & Temporal Modeling: The introduction of a structured schema (Person, Location, Object, Event) and a temporal relational strategy that accumulates stable long-term memory, enabling complex reasoning (e.g., inferring habits from multi-day observations).
3. State-of-the-Art Performance: Demonstrated superior performance on ultra-long benchmarks, validating that structured, time-aware graphs outperform both hierarchical summarization and static graph approaches.

4. Experimental Results

The authors evaluated EgoGraph on two benchmarks: EgoLifeQA (500 QAs over 7 days) and EgoR1-Bench (300 QAs across 6 perspectives).

• Performance:

  • EgoLifeQA: EgoGraph achieved 45.8% average accuracy, outperforming the best graph-based baseline (LightRAG, 39.2%) by +6.6% and the strongest MLLM (Gemini-1.5-Pro, 36.9%) by +8.9%.
  • EgoR1-Bench: EgoGraph scored 41.3%, surpassing Gemini-1.5-Pro (38.3%) by +3.0%.
  • It achieved the highest scores in complex reasoning tasks like TaskMaster (60.3%) and EventRecall (46.8%).

• Ablation Studies:

  • Graph vs. Hierarchy: EgoGraph outperformed hierarchical methods (EgoGPT) by 29.3% on temporal reasoning tasks, proving the superiority of instance-centric representations for long-term dependencies.
  • Component Analysis: Removing the temporal filter or the egocentric schema caused significant performance drops, confirming that temporal awareness is a fundamental requirement, not just an enhancement.
  • Scalability & Robustness: As video context grew from 1 to 7 days, "Plain-text" methods failed (dropping to 8.8% accuracy due to token limits), while EgoGraph maintained high stability (dropping only slightly from 51% to 45.8%), demonstrating effective indexing and retrieval without being overwhelmed by data volume.

5. Significance

EgoGraph represents a paradigm shift in video understanding for long-duration, first-person scenarios.

• Memory Efficiency: By structuring information into a compact, evolving graph rather than storing unstructured text or video clips, it enables scalable reasoning over weeks or months of data.
• Temporal Precision: It solves the "when" problem in video QA, allowing systems to distinguish between past, present, and future events relative to a query, which is critical for episodic memory retrieval.
• Generalizability: The framework is training-free and relies on off-the-shelf LLMs and encoders, making it a practical solution for embodied agents and personal AI assistants that need to understand and recall long-term daily activities.