Technical Summary: S-EMBER

Problem Statement

Current AI assistants lack the capability for episodic memory—the ability to recall past events and reason about their temporal relationships over long time horizons. Existing benchmarks for video understanding often rely on offline evaluation, where models have access to entire video files, failing to simulate the streaming reality of wearable intelligence (e.g., smart glasses). Furthermore, traditional benchmarks often prioritize global search over causal, active recall, where a query is triggered by a specific visual event in a continuous stream. There is a critical gap in evaluating how effectively models can locate and utilize information within long-form, unscripted visual histories while adhering to strict temporal constraints.

Methodology

The S-EMBER Benchmark

The authors introduce S-EMBER (Streaming Egocentric Memory Benchmark for Episodic Retrieval), a large-scale diagnostic foundation designed to evaluate Grounded Streaming Episodic Retrieval (GSER).

• Data Collection: The dataset comprises 3,141 unique videos totaling 388 hours of organic human activity. Unlike prior datasets using research rigs, S-EMBER uses Ray-Ban Meta smart glasses worn by 613 unique users, providing a hardware-authentic, head-centered perspective that mirrors natural human attention.
• Task Definition (GSER): In this task, an assistant receives a video stream truncated at a specific "trigger" timestamp $T$. The model must answer a question based only on the historical context $V[0, T]$. Crucially, the model must also identify the temporal interval $[t_{start}, t_{end}]$ that serves as visual proof for its answer.
• Annotation Pipeline:
  1. Event-Driven Generation: Annotators observe streams in real-time to generate questions triggered by specific events, ensuring a "look-back" requirement. Questions are categorized into eight cognitive types: Visual Detail Recall, Sequential Action, Time Duration, Counting, Temporal Ordering, Object Comparison, Location Trace, and Spatial Reasoning.
  2. Multi-Granular Grounding: For each question, three independent groups generate responses of varying verbosity (Concise, Balanced, Comprehensive). Simultaneously, annotators mark the precise evidence interval. A strict temporal padding constraint (max 15% of duration or 15s) prevents lazy grounding.
  3. Quality Assurance: Triplets are audited by human experts and a Blind-LLM to remove questions solvable by common sense or linguistic priors without visual context.
• Evaluation Metrics:
  • Semantic Accuracy: Measured via Free-form evaluation (LLM-as-a-Judge against tri-tiered references) and 5-way Multiple Choice Questions (MCQ). Metrics include Clean Accuracy (penalizing hallucinations) and Overall Accuracy.
  • Temporal Grounding: Measured via Mean Intersection over Union (mIoU), Recall@1 (IoU $\ge \tau$), and Grounded QnA (GQ@$\tau$), which jointly requires correct semantics and precise localization.

Key Results

The Localization Paradox

Extensive benchmarking of frontier models (e.g., Gemini 3.1 Pro, GPT-5.4, InternVL3.5, Qwen3VL) reveals a fundamental disconnect:

• Semantic Reasoning Scales: As model parameter counts increase (e.g., InternVL3.5 from 4B to 38B), semantic reasoning accuracy improves significantly (from 19.7% to 26.1%).
• Temporal Grounding Stagnates: Despite scaling, temporal grounding precision remains a stagnant architectural bottleneck. Increasing model size, visual resolution (up to 720p), or frame density (up to 768 frames) yields diminishing returns for localization. For instance, while accuracy for Qwen3VL nearly triples with increased frame density, grounding performance remains trapped below 3%.

Performance Benchmarks

• Top Performers: Gemini 3.1 Pro leads with 42.88% Clean Accuracy and 0.479 mIoU, significantly outperforming other models but still far below the Human Ceiling (91.43% accuracy, 0.777 mIoU).
• Recall Decay: All models exhibit a universal recall decay as the temporal gap between evidence and query increases. For example, Gemini 3.1 Pro drops from 50% accuracy for immediate recall (0–30s) to 22% for events occurring 10 minutes prior.
• Hallucination: High-performing models show significant hallucination rates (e.g., 40.17% for Gemini 3.1 Pro), often generating plausible but unverified details. The "Blind LLM" baseline (text-only) achieves only 2.68% Clean Accuracy, confirming the dataset requires genuine multimodal reasoning.

Significance and Claims

The paper claims that S-EMBER establishes a hardware-authentic foundation for developing grounded, reliable episodic memory in the next generation of wearable AI agents. Its primary contributions are:

1. A New Paradigm: Shifting evaluation from static, offline search to causal, streaming recall triggered by visual events.
2. Rigorous Verification: Providing manual visual proof (intervals) for every query to distinguish between authentic visual grounding and temporal hallucinations.
3. Identification of Architectural Gaps: The authors identify that current scaling laws (parameter count, resolution, frame density) do not solve the problem of precise temporal localization. They argue that future progress requires specialized architectural improvements, such as adaptive temporal indexing and hierarchical memory buffers, rather than brute-force scaling.

The work underscores that while models are becoming better at understanding what happened in a video, they remain fundamentally challenged in determining when it happened, a critical capability for proactive, memory-aware personal assistants.