Technical Summary: EGOSTREAM

Problem Statement

Autonomous agents operating in dynamic, real-world environments require continuous episodic memory to store and recall events across varying temporal horizons. While egocentric vision provides a natural testbed for studying this capability—capturing first-person experience as it unfolds—current evaluation methods are insufficient for diagnosing specific memory failures. Existing offline Video Question Answering (VideoQA) benchmarks often fail to isolate memory as a measurable variable, rarely controlling the temporal distance between evidence and query. Furthermore, they do not distinguish between genuine model forgetting and the natural invalidation of answers due to scene evolution. Recent streaming benchmarks focus on aggregate responsiveness or accuracy, masking the qualitative trade-offs of different memory-management architectures. Consequently, models with vastly different memory profiles (e.g., one preserving coarse event gist while forgetting object locations, another retaining short-term spatial details but failing long-horizon causal recall) may achieve similar scores, hindering the diagnosis of which forms of episodic information degrade first.

Methodology

The authors introduce EGOSTREAM, a diagnostic benchmark and a unified evaluation framework designed to assess streaming episodic memory in egocentric vision.

1. The EGOSTREAM Benchmark

• Data Curation: The benchmark aggregates 2,250 curated questions from five existing egocentric VideoQA datasets (Ego4D, EgoLife, EgoTempo, Multi-Hop EgoQA, and HD-EPIC). These were standardized into a 4-way multiple-choice format using LLM-generated hard negatives to prevent superficial solving.
• Cognitive Taxonomy: Questions are organized along seven cognitive dimensions grounded in memory theory: Detail (object identity/attributes), Spatial (location), Temporal (order/sequence), Event (action gist), Social (people/interactions), Causal (reasons), and Prospective (future plans).
• Answer Validity Window (AVW): A core innovation is the AVW, which defines the precise temporal span during which a specific answer remains factually correct before being overturned by subsequent events. This allows the benchmark to separate "forgetting" from "scene evolution."
• Recall Regimes: Questions are evaluated across seven temporal regimes ranging from Instant (0s) to Ultra-Long-term (>8h). By sampling query times within the AVW, the benchmark expands the 2,250 questions into 8,528 recall-conditioned evaluations, testing memory retention at specific delays.

2. Unified Streaming MLLM Framework

To enable controlled diagnostic comparisons, the authors implement a unified Multimodal Large Language Model (MLLM) framework based on a frozen Qwen3-VL backbone. This framework isolates memory management as an independent variable by keeping the encoder, projector, and decoder fixed.

• Cache Architecture: The KV cache is partitioned into three logical regions:
  • Sink Tokens ($SINK$): Protected system prompt tokens.
  • Hat Tokens ($HAT$): Protected recent visual observations.
  • Mutable Region: Historical visual tokens subject to memory management.
• Memory Management Strategies: The framework evaluates various state-of-the-art strategies operating strictly on the mutable region:
  • Reduction: Intra-frame and inter-frame token pruning (based on cosine redundancy, pseudo-attention salience, value-norm, sink-guided attention) and token merging.
  • Overflow Policies: First-In-First-Out (FIFO) discarding vs. Offloading (storing evicted tokens on disk for retrieval).
  • Quantization: Int8 quantized offloading to reduce storage costs.

Key Contributions

1. EGOSTREAM Benchmark: A new diagnostic testbed comprising 2,250 validated questions expanded into 8,528 evaluations across seven cognitive dimensions and seven recall regimes.
2. Answer Validity Window (AVW): A novel metric that constrains recall evaluations to the timeframe where an answer is factually valid, ensuring that performance drops are attributed to memory failure rather than environmental changes.
3. Unified Diagnostic Framework: A standardized streaming MLLM pipeline that allows for the plug-and-play comparison of memory-management mechanisms (pruning, merging, offloading) under identical architectural constraints, revealing that aggregate accuracy often masks distinct memory profiles.

Results

Experiments within the unified framework reveal that while aggregate performance gaps among strategies are narrow, the specific types of information retained and the temporal horizons they support vary significantly.

• Aggregate vs. Profile: Top-performing methods achieve only about 45% accuracy on average, and all mechanisms operate well below real-time speeds (>1s per frame). Comparable aggregate scores often hide starkly different memory profiles.
• Pruning vs. Merging: Token pruning consistently outperforms token merging. Pruning preserves fine-grained details and temporal structure significantly better, whereas merging tends to blur fine-grained evidence. For instance, a pruning-based strategy achieved 45.19% semantic accuracy compared to 41.87% for a merging-based approach.
• Temporal Decay: Performance degrades as the recall horizon increases. While spatial and detail memory are relatively robust in the short term, ultra-long-term recall (>8h) is particularly challenging.
• Offloading: External retrieval strategies, particularly quantized offloading (Q-OFF), are critical for ultra-long-term memory. While offloading alone is weaker than in-memory pruning, combining pruning with quantized offloading (e.g., StreamingTOM) rescues ultra-long-term recall (reaching 34.21% for Ultra-Long regime) and significantly reduces processing time (down to 1.43s) compared to pure in-memory pruning.
• Protected Tokens: The inclusion of Sink and Hat tokens is essential. Adding Sink tokens to a naive sliding window baseline boosted semantic accuracy from ~29% to ~44%, confirming that preserving structural context and recent observations is vital for stable inference.

Significance and Claims

The paper claims that EGOSTREAM provides the necessary diagnostic testbed to close the gap in current streaming episodic memory research. By moving beyond aggregate accuracy, the benchmark exposes that current state-of-the-art architectures have critical limitations:

• They fail to maintain high-fidelity memory over long horizons without significant latency penalties.
• Different architectural choices (pruning vs. merging) induce specific, predictable vulnerabilities in what information is retained.
• The field requires evaluation methods that explicitly measure what is remembered and for how long, rather than just whether a model answers correctly.

The authors conclude that while current methods can specialize in different memory profiles, achieving true real-time streaming ingestion with reliable long-term episodic memory remains an open challenge. The benchmark and framework are intended to guide the design of future streaming architectures by providing a precise tool to diagnose memory vulnerabilities in embodied agents.