Technical Summary: Keep It in Mind: User-Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams

Problem Definition

The paper addresses a critical gap in modern computer vision and AI research: the lack of User-Centric Continual Spatial Intelligence. While humans continuously track their location, orientation, and the relative positions of objects in dynamic environments (even when those objects are out of sight), current AI systems struggle with this capability.

Existing benchmarks and methods primarily focus on:

1. Static or Offline Settings: Evaluating spatial reasoning on single images or short, pre-recorded video clips.
2. Third-Person Perspectives: Questions are often framed from a "God's eye" view rather than the user's first-person perspective.
3. Instantaneous Perception: They rely on cues visible at the moment of the query, failing to account for the need to recall past observations to answer questions about the current state of the world relative to the user's movement.

The core challenge is egocentric video question answering (QA) where the model must maintain a coherent, evolving spatial memory that updates with the user's real-time movement (ego-motion), handling viewpoint changes, occlusions, and long temporal gaps between observation and query.

Methodology

1. UCS-Bench: A New Benchmark

To diagnose and advance this capability, the authors introduce UCS-Bench, a dataset specifically designed for user-centric continual spatial reasoning.

• Scale & Scope: It comprises 532 long videos (over 170 hours) from six diverse sources (e.g., EgoLife, HourVideo, ScanNet, EPIC-KITCHENS), covering both indoor and outdoor daily activities.
• Annotations: The dataset includes 8,114 timestamped QA pairs. Crucially, questions are not static; the same query is often posed at different timestamps to test the model's ability to update its answer based on the user's new location.
• Task Taxonomy: Questions are categorized into four dimensions:
  1. Position & Orientation: Tracking relative directions (e.g., "behind you," "to your left") as the user moves.
  2. Trajectory & Movement: Recalling paths taken by the user or objects.
  3. Proximity & Reachability: Assessing distance and feasibility of interaction from the current viewpoint.
  4. Category & Quantity: Tracking object existence and counts over time.
• Construction: The dataset was built via a rigorous pipeline involving evidence-grounded annotation, manual verification, and automated bias diagnostics (using NLI models and blind tests) to ensure questions cannot be answered via language priors alone.

2. DirectMe: A Training-Free Framework

The authors propose DirectMe, a framework that enhances existing Multimodal Large Language Models (MLLMs) without retraining. It operates by constructing and maintaining a structured spatial scene graph as a memory module.

• Background Processing (Offline/Streaming):
  • Multi-modal Extraction: From the incoming video stream (sampled at 1 fps), the system extracts metric depth maps and ego-pose trajectories using DepthAnything3. It also performs open-vocabulary object detection (Grounding-DINO/YOLO-World) and tracking (SAM2) to generate persistent object masks and identities.
  • Global Metric-Semantic Map: These signals are fused into a world-coordinate frame, creating a consistent 3D representation of the environment and object locations relative to the global origin.
  • Evolving Scene Graph: A scene graph $G=(V, E)$ is incrementally built. Nodes represent persistent objects and semantic places, while edges encode spatial relations (distance, direction) computed in the world frame.
• Online Reasoning (Query Time):
  • When a question is asked at time $t$, the system retrieves a minimal, query-conditioned subgraph from the memory.
  • Pose-Aware Transformation: The world-frame coordinates of retrieved objects are transformed into the current egocentric frame using the inverse of the current camera pose ($T_{e \leftarrow w}(t)$).
  • Prompting: The MLLM receives the relevant video frames, a spatial summary of the subgraph (rendered in the user's current perspective), and the question. This allows the model to reason about "out-of-sight" objects by leveraging the structured memory rather than relying solely on visual context windows.

Key Contributions

1. UCS-Bench: The first benchmark explicitly targeting user-centric continual spatial intelligence in streaming egocentric videos, featuring long-horizon, dynamic, and perspective-dependent queries.
2. DirectMe: A novel, training-free framework that decouples memory construction from reasoning. It introduces a pose-anchored scene graph to maintain robust spatial memory, enabling models to track object locations relative to the user's movement over time.
3. Empirical Analysis: A comprehensive evaluation demonstrating that current state-of-the-art MLLMs (both general-purpose and spatially specialized) significantly underperform on this task, revealing specific failure modes like "Visual Memory Amnesia" and "Temporal State Tracking Failure."

Experimental Results

The authors evaluated 18 representative MLLMs (ranging from 4B to 38B parameters) on UCS-Bench.

• Performance Gap: There is a substantial gap between human performance (91.0%) and current models (best SOTA around 50%).
• DirectMe's Superiority: DirectMe significantly outperforms all baselines. When paired with strong backbones like Qwen3-VL-32B, it achieves 50.5% overall accuracy, surpassing specialized spatial models and long-streaming video models.
• Robustness to Time Gaps: DirectMe shows a much slower degradation in accuracy as the time interval between the evidence and the query increases (e.g., >3 minutes), whereas baseline models suffer severe performance drops.
• Error Analysis:
  • Baseline Failures: Current models struggle most with ego-motion reasoning (tracking the user's own path) and temporal state tracking (counting or tracking object states over time).
  • DirectMe Failures: Even with DirectMe, the primary failure modes are tracking failures (40%) and ego-pose relation errors (23%), indicating that maintaining a temporally consistent, user-centric spatial state remains the core bottleneck.

Significance and Claims

The paper posits that User-Centric Continual Spatial Intelligence is a fundamental capability required for next-generation AI assistants, particularly for wearable devices and assistive technologies (e.g., for the visually impaired).

• Beyond Static Perception: The work argues that true spatial intelligence requires more than just recognizing objects in a frame; it demands the ability to integrate streaming observations into a dynamic, user-referenced world model.
• Limitations of Current Architectures: The results suggest that simply scaling model size or context windows is insufficient. The gap highlights a need for explicit mechanisms to track ego-motion and maintain structured spatial memories.
• Foundation for Future Research: By providing a rigorous benchmark and a modular framework (DirectMe), the authors aim to catalyze research into robust, long-horizon spatial reasoning, paving the way for AI systems that can "keep it in mind" as they navigate the real world.

The authors conclude that while DirectMe sets a new state-of-the-art, the task remains far from solved, offering a challenging new avenue for advancing multimodal reasoning in embodied AI.