Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory

Problem Formulation: Online Visual Query 2D (OVQ2D)

The paper addresses the limitations of existing Episodic Memory (EM) retrieval systems, specifically the Visual Query 2D (VQ2D) task introduced in the Ego4D benchmark. Current VQ2D formulations operate in an offline setting, assuming full access to the video history at query time. This approach requires storing the entire video sequence (which can reach terabytes for wearable devices) and entails significant computational overhead, as the system must re-scan the video for every new query.

To enable practical deployment on power- and storage-constrained wearable devices, the authors introduce Online Visual Query 2D (OVQ2D). In this formulation:

• Streaming Processing: The system processes the egocentric video ($V_{ego}$) online, observing each frame only once.
• Compact Memory: Instead of storing raw video, the system must maintain a compact, dynamic memory of relevant objects and events.
• Retrieval: Given a visual query patch $Q$ (an image of an object), the system must retrieve the spatio-temporal localization (a sequence of bounding boxes and frames) of the object's last appearance using only the stored memory, without accessing the original video stream.

This shift introduces three primary challenges not present in offline settings:

1. Unsupervised Discovery: The system must identify potentially queryable objects in real-time without prior knowledge of what the user will ask about.
2. Pre-emptive Tracking: The system must track object instances and store their response tracks in advance, rather than initiating a search only after a query is received.
3. Representation Association: The system must associate potential response tracks with visual representations suitable for future retrieval, as it cannot directly compare the query patch to raw video candidates during the search phase.

Methodology: ESOM (Egocentric Streaming Object Memory)

The authors propose ESOM, a framework designed to tackle OVQ2D through three integrated modules:

1. Object Memory Representation ($M_{ego}$)
The core data structure is a dynamic list of object instances $O_i$. Each instance is a temporally indexed list of records $o_{i,t}$, containing:

• $t$: Timestamp.
• $b_{i,t}$: Bounding box coordinates.
• $F_t$: The associated RGB frame (or a patch extracted from it).
• $c_{i,t}$: A binary relevance label indicating if the visual content is distinctive enough for retrieval.
  This structure allows for indexed, lightweight, and human-readable storage of observed objects.

2. Object Memory Population (OMP)
This algorithm continuously updates $M_{ego}$ as the video streams in. It coordinates two sub-modules:

• Object Tracking (OT): Updates the positions of previously memorized objects in the current frame $F_t$ using a visual object tracker. It filters results based on confidence scores and writes updated tracks to memory.
• Object Discovery (OD): Detects new objects of interest in $F_t$ using an object detector. It filters detections based on confidence and checks for Intersection over Union (IoU) with existing memory entries. New, non-overlapping detections are added to memory as new object instances.
• Relevance Filtering: Before writing to memory, a binary classifier evaluates the visual patch to determine if it is distinctive ($c_{i,t}=1$). This step filters out low-quality or non-distinctive patches to improve retrieval efficiency.

3. Query Retrieval and Localization (QRL)
When a user provides a query $Q$, the QRL algorithm:

• Extracts feature representations $\Phi(Q)$.
• Compares these against the features $\Phi(\phi_{i,t})$ of all relevant object patches ($c_{i,t}=1$) stored in $M_{ego}$ using a similarity function $\Psi$.
• Aggregates scores per object instance. If the maximum similarity score exceeds a threshold, the system returns the most recent contiguous sequence of bounding boxes and frames for that object as the visual response track $r$.
• Crucially, this process relies exclusively on $M_{ego}$ and does not access the original video $V_{ego}$.

Key Contributions

1. OVQ2D Task Formulation: The paper introduces a new, application-ready benchmark for episodic memory retrieval that enforces online, single-pass processing and compact memory usage, reflecting the constraints of wearable devices.
2. ESOM Architecture: A novel framework integrating object discovery, long-term tracking, and dynamic memory management. It prioritizes practical deployment and benchmarking over algorithmic novelty, leveraging state-of-the-art detection and tracking components.
3. Real-World Benchmarking: The authors provide a benchmark on the Ego4D dataset. They argue that this benchmark serves as a valuable resource not just for EM retrieval, but for evaluating the real-world applicability of object detection and tracking algorithms in downstream tasks, moving beyond isolated component evaluation.

Results and Performance

Experiments were conducted on the Ego4D Episodic Memory VQ2D benchmark (433 hours of video, 22K queries).

• Baseline Comparison: ESOM outperforms other online approaches (adapted from AMEGO and VideoAgent) in terms of success rate and efficiency. The best ESOM configuration (using a YOLO-based detector and EgoSTARK tracker) achieves a 4.02% success rate with a memory footprint of 1.7 GB and a retrieval time of 3.1 seconds.
• Comparison with Offline Methods: While offline VQ2D methods achieve higher success rates (e.g., 55.89% for VQLoC), they require storing the full video (12.1 GB) and take significantly longer to retrieve (e.g., 41 seconds). ESOM reduces memory usage by up to 7$\times$ and retrieval time by up to 13$\times$ compared to offline baselines.
• Oracle Analysis: The paper highlights that the low success rates of current methods are primarily due to limitations in object detection and tracking in real-world egocentric videos, not the memory architecture itself.
  • With perfect tracking (Oracle OT) and standard detection, success rises to 31.91%.
  • With perfect detection (Oracle OD) and standard tracking, success rises to 40.55%.
  • With both perfect detection and tracking, ESOM achieves an 81.92% success rate, surpassing the best offline methods.
• Component Analysis:
  • The QRL algorithm using Siamese R-CNN features consistently outperforms DINOv2-based approaches.
  • Relevance filtering significantly improves retrieval speed and accuracy by discarding suboptimal representations.
  • Tracking performance (specifically association accuracy, AssA) is more critical for OVQ2D success than raw detection accuracy (DetA), as coherent tracks are essential for constructing valid response sequences.

Significance and Claims

The paper claims that OVQ2D represents a necessary step toward deployable episodic memory systems for wearable technology. While the current success rate of ~4% on real-world data is modest, the authors emphasize that this is a reflection of the current state of object detection and tracking in egocentric vision, rather than a failure of the memory retrieval paradigm.

The significance of the work lies in:

• Feasibility: Demonstrating that episodic memory retrieval can be performed online with manageable memory and computational budgets.
• Diagnostic Value: The benchmark and its "oracle" configurations provide a principled testbed to identify bottlenecks in object detection and tracking algorithms when applied to real-world, continuous video streams, suggesting that future improvements in these foundational components will directly translate to better episodic memory systems.
• Efficiency: Proving that compact, object-centric memory representations can offer substantial gains in storage and speed over raw video storage without a catastrophic loss in retrieval capability (assuming detection/tracking are robust).