Technical Summary: StreamMemBench

Problem Statement

Personal agents built on Large Language Models (LLMs) require memory mechanisms to transform stored information and prior interactions into future-oriented assistance. While existing benchmarks evaluate long-term memory, they largely fail to assess the critical trajectory from streaming observations to future assistance. Current evaluations typically rely on scripted dialogues or synthesized chats, testing only dialogue recall or isolated task improvement. This approach evades the core challenge of identifying relevant user-specific information from continuous, unstructured egocentric streams and applying it to future tasks without explicit prompting. Furthermore, existing benchmarks often lack verifiable grounding for user feedback, making it difficult to distinguish whether an agent's success stems from genuine evidence use or merely plausible-sounding generation. Consequently, there is a gap in understanding whether stored memory actually translates into reliable, future-oriented behavior.

Methodology: StreamMemBench

To address these limitations, the authors introduce StreamMemBench, a streaming benchmark designed to evaluate whether personal-agent memory supports future-oriented assistance across egocentric observations and interactions. The benchmark is constructed upon EgoLife, a dataset of continuous daily life recordings from wearable devices.

Benchmark Construction

The evaluation process is anchored around evidence anchors extracted from the EgoLife stream. For each anchor, a two-step task sequence is generated:

1. Initial Task Episode: The agent receives an assistance request dependent on a specific piece of evidence extracted from the stream. The evidence is never explicitly stated in the query. The agent responds, and a simulated user provides feedback (either confirming the response or correcting it by supplying the missing evidence).
2. Follow-up Task Episode: A new request, grounded in the same evidence anchor but presenting a different scenario, is issued. This tests whether the agent has consolidated the initial interaction (including feedback) to handle the follow-up task.

Evaluation Metrics

The benchmark diagnoses agent performance at four distinct points along the trajectory:

• Fidelity: Measures whether the target evidence is retained in the agent's memory delta before the task sequence begins.
• Initial Evidence Use (IEU): Evaluates whether the agent's initial response correctly utilizes the anchored evidence.
• Feedback Incorporation (FI): Assesses whether the agent incorporates corrective feedback into its immediate post-feedback response.
• Follow-up Reuse (FUR): Determines whether the agent reuses the evidence or the corrected interaction experience in the follow-up task.

The evaluation is conducted in a black-box manner against the agent's text responses, with a separate check for memory fidelity based on the exposed memory delta.

Key Contributions

1. A Streaming Benchmark: StreamMemBench is the first benchmark to evaluate personal-agent memory across a continuous stream of egocentric observations, moving beyond static dialogue histories.
2. Evidence-Anchored Trajectory: By using evidence anchors to link initial and follow-up tasks, the benchmark isolates specific failure modes, distinguishing between failures in evidence retention, initial use, feedback incorporation, and long-term reuse.
3. Comprehensive Evaluation: The framework evaluates eight diverse memory systems (including retrieval baselines and agentic memory systems) across two different LLM backbones, providing a granular diagnosis of where the "memory-to-action" pipeline breaks down.

Experimental Results

Experiments involving systems such as RAG (raw and extracted), Mem0, EverMemOS, A-Mem, MemOS, MemoryOS, and MemSkill reveal significant limitations in current architectures:

• Storage $\neq$ Usage: Systems with high Fidelity (successfully storing evidence) often exhibit low IEU and FUR. For instance, RAGraw achieves 100% Fidelity but only ~28-42% IEU, indicating that stored evidence does not automatically translate into task behavior.
• Feedback vs. Reuse: A notable gap exists between Feedback Incorporation (FI) and Follow-up Reuse (FUR). Many systems can correct their behavior immediately upon receiving feedback (high FI) but fail to consolidate this correction for future tasks (low FUR). For example, MemOS shows high FI (77%) but extremely low FUR (4%).
• Temporal Drift: Performance varies across the stream. RAGraw maintains high fidelity but shows a steady decline in IEU as the stream grows, suggesting retrieval mechanisms become less reliable for initial use over time. Conversely, MemSkill shows improved reuse but declining fidelity.
• Failure Modes: Analysis categorizes failures into lifecycle stages: Formation failure (evidence not stored), Initial evidence use failure, Feedback incorporation failure, Correction consolidation failure, and Evidence-use persistence failure. The results indicate that different memory mechanisms fail at different stages; for example, extraction-based systems often fail at formation, while consolidation-based systems often fail at reuse.

Significance and Claims

The paper argues that the current premise—that storing or managing past experience inherently improves future assistance—is not guaranteed. StreamMemBench demonstrates that even advanced systems struggle to turn observed evidence into initial task support or to convert interaction feedback into reliable, reusable behavior.

The authors claim that memory should be judged not merely by what is stored, but by whether it supports useful future behavior. The benchmark provides a necessary diagnostic tool to trace the path from observed evidence to initial use, feedback incorporation, and follow-up reuse, revealing that a single aggregate score is insufficient to capture the temporal drift and specific failure points in personal-agent memory. The work highlights the need for memory systems that can effectively bridge the gap between passive storage and active, future-oriented assistance.