LifeBench: A Benchmark for Long-Horizon Multi-Source Memory

This paper introduces LifeBench, a novel benchmark designed to evaluate AI agents' long-horizon, multi-source memory capabilities by simulating complex, real-world scenarios that require integrating both declarative and non-declarative memory, revealing that current state-of-the-art systems struggle with a 55.2% accuracy rate on these tasks.

The Gist

Imagine you are trying to teach a robot how to be the perfect personal assistant for a human being.

Currently, most AI assistants are like amnesiacs with a photographic memory for facts. If you tell them, "I like pizza," they remember that forever. But if you ask, "What did I have for dinner last Tuesday?" or "Why do I always run when I'm stressed?", they often get lost. They can recall explicit facts, but they can't figure out your habits, your routines, or the subtle patterns of your life.

The paper "LifeBench" introduces a new way to test if AI is ready to be a true, long-term companion. Here is the breakdown in simple terms:

1. The Problem: The "Goldfish" vs. The "Biographer"

Most current AI memory tests are like a quiz show. They ask simple questions like, "What color was the car I mentioned yesterday?"

• The Issue: Real life isn't a quiz show. Real life is a messy, 24/7 movie.
• The Missing Piece: Humans have two types of memory:
  1. Declarative: "I went to Paris in 2020." (Facts)
  2. Non-Declarative: "I always feel anxious before a big meeting, so I drink extra coffee." (Habits, skills, feelings)
• The Gap: Current AI is great at the first type but terrible at the second. It doesn't know why you do things, only what you did.

2. The Solution: Building a "Digital Twin"

To fix this, the researchers built LifeBench. Think of this as a massive, hyper-realistic video game where they created 10 fake people (avatars) and lived their entire lives for a whole year inside a computer.

• The Simulation: They didn't just write a diary. They simulated every text message, every calendar appointment, every photo taken, every health record (steps, sleep, heart rate), and every chat with an AI assistant.
• The "Noise": Just like your real phone, these fake people have thousands of notifications, ads, and random texts that have nothing to do with the main story. The AI has to find the signal in the noise.
• The Scale: For just one person, they generated over 5,000 events and 8,000 pieces of digital data (texts, calls, photos, etc.). That's a lot of data to remember!

3. The Challenge: The "Needle in a Haystack" Test

Once they built these digital lives, they asked the AI some tricky questions to see if it could understand the person.

• Simple Question: "What time did I wake up on Christmas?" (Easy, just look at the calendar).
• Hard Question: "How many times did I go running this year?" (Hard, because the data is scattered across health apps, photos, and text messages).
• Super Hard Question: "I seem to run more when I'm stressed. Can you prove it?" (This requires connecting the dots between a work email, a high heart rate, and a running log).

4. The Results: The AI Got Stuck

They tested the smartest AI memory systems available today on this new benchmark.

• The Score: The best AI only got about 55% of the answers right.
• The Verdict: Even the smartest AIs are struggling. They are like a librarian who can find a specific book if you give them the exact title, but if you ask, "What books did I read when I was sad last summer?", they get confused.

5. Why This Matters

This isn't just about passing a test. If we want AI to be a true partner in our lives, it needs to understand us better than we understand ourselves.

• Health: It could notice, "You've been sleeping poorly and eating more sugar since your promotion; let's adjust your schedule."
• Personalization: It could say, "You usually hate Mondays, but you loved the hiking trip last month. Let's plan a hike for next Monday."

The Big Metaphor

Imagine you are trying to hire a butler.

• Old AI: A butler who only remembers the exact words you said to them. If you say, "I'm hungry," they bring food. If you don't say it, they don't know you're hungry.
• LifeBench AI: A butler who has watched you for a year. They know you get hungry at 4 PM, that you prefer tea over coffee when you're stressed, and that you usually skip breakfast on rainy days. They anticipate your needs before you speak.

LifeBench is the new "interview" to see if our AI butlers are ready to stop being just robots and start being real helpers. Right now, they are still in training, but this paper gives us the map to teach them how to truly remember us.

Technical Summary

Technical Summary: LifeBench – A Benchmark for Long-Horizon Multi-Source Memory

1. Problem Statement

Current benchmarks for AI agent memory (e.g., LOCOMO, LongMemEval) primarily focus on declarative memory (semantic and episodic) derived from explicit dialogue. They fail to capture the complexity of real-world human cognition, which relies heavily on non-declarative memory (habits, procedural skills, emotional conditioning) and requires inferring patterns from fragmented, multi-source digital traces (e.g., health records, SMS, calendar entries, photos) rather than coherent conversations.

Existing datasets often suffer from:

• Sparse Data: Low frequency of events per day.
• Single-Source Limitations: Relying almost exclusively on chat logs.
• Lack of Non-Declarative Reasoning: Inability to test an agent's ability to infer habits or skills.
• Scalability Issues: Generating long-horizon, causally linked synthetic data is computationally expensive and difficult to maintain with high fidelity.

2. Methodology: The LifeBench Framework

LifeBench introduces a synthetic data generation framework designed to simulate a full year of densely connected human life for multiple personas. The system is built on two core principles: Human Cognition-Inspired Synthesis and Partonomic Hierarchy.

2.1 Synthesis Pipeline Architecture

The framework utilizes a five-stage pipeline powered by Large Language Models (LLMs), specifically DeepSeek-R1:

1. Persona Synthesis: Generates detailed user profiles including demographics, personality (MBTI), social networks, and long-term goals. These are grounded in anonymized survey data and real-world priors to ensure behavioral rationality.
2. Hierarchical Outline Planning: Constructs a "partonomic tree" of events.
   • Root: Yearly outline.
   • Intermediate: Thematic events (monthly/weekly) based on plot templates (e.g., career promotion, health journey).
   • Leaves: Atomic events (<1 day duration).
   • Optimization: Adjusts atomic events based on contextual factors (weather, mood, habits) to ensure consistency.
3. Dual-Agent Daily Activity Simulation:
   • Subjective Agent: Simulates human reasoning to propose daily activities based on habits and emotional states.
   • Objective Agent: Validates plans against real-world constraints (map APIs for travel time, venue availability) to ensure physical and temporal feasibility.
4. Phone Data Generation: Converts daily activities into fragmented digital artifacts. This includes:
   • Structured Data: Contacts, Calendar, Health Records, Push Notifications.
   • Unstructured Data: SMS, Call logs, Photos (with OCR/captioning), Notes, Agent Chats.
   • Noise: Irrelevant data (ads, random browsing) to simulate real-world signal-to-noise ratios.
5. Question Answering (QA) Generation: Dynamically generates 2,003 questions during simulation to ensure exact grounding. Questions are categorized into five types:
   • Information Extraction (IE): Direct fact retrieval.
   • Multi-hop Reasoning (MR): Aggregating multiple memory points.
   • Temporal & Knowledge Updating (TKU): Handling evolving information over time.
   • Non-Declarative Memory Reasoning (ND): Inferring habits, skills, and preferences.
   • Unanswerable (UA): Questions that cannot be answered from available data.

2.2 Scalability Optimization

To address the high computational cost of sequential generation (originally 58 hours/user/year), the authors implemented a parallelization strategy:

• Parallel Event Decomposition: Recursively decomposes thematic events into sub-events across multiple threads.
• Time-Sliced Daily Simulation: Assumes activities in two-week windows are conditionally independent, allowing parallel generation of daily logs.
• Parallel Data Generation: Phone and health data are generated independently for each day.
• Result: Reduced generation time from 58 hours to 8 hours per user-year.

3. Key Contributions

1. Human Cognition-Inspired Synthesis: A novel framework that models both declarative and non-declarative memory systems, organizing events via a partonomic hierarchy to maintain consistency across temporal scales.
2. Comprehensive Multi-Source Dataset: A year-long dataset for 10 diverse personas containing:
   • 51,491 events (avg. 14 events/day).
   • 80,460 phone artifacts (avg. 23 artifacts/day) across 9 types (SMS, calls, health, photos, etc.).
   • Context Depth: 3.66M tokens per user, exceeding existing benchmarks by >2x.
3. Specialized Benchmarking: The first benchmark to explicitly evaluate Non-Declarative Memory Reasoning (habits, skills) and Multi-Source Integration (inferring facts from fragmented, noisy data).
4. Reproducibility: Release of the full dataset, synthesis code, and documentation under Apache License 2.0.

4. Experimental Results

The authors evaluated state-of-the-art memory systems (MemU, Hindsight, MemOS) using a unified base LLM (GPT-5.1-Mini) and an LLM-as-a-judge evaluation protocol.

• Overall Performance: Even the top-performing system, MemOS, achieved only 55.22% accuracy.
• Comparison: Hindsight, which scores ~90% on existing benchmarks (LOCOMO, LongMemEval), dropped to 40.99% on LifeBench, highlighting the inadequacy of current benchmarks for real-world complexity.
• Category Breakdown:
  • Systems performed relatively well on Information Extraction (IE) and Temporal Updating (TKU) due to better handling of factual and versioned data.
  • Systems struggled significantly with Non-Declarative Reasoning (ND) and Unanswerable (UA) questions, often hallucinating habits or failing to recognize when data was insufficient.
• Error Analysis: Common failure modes included:
  • Failure to retrieve correct evidence from multi-source data.
  • Incomplete evidence retrieval (ignoring subtle cues).
  • Hallucination during memory unit summarization.
  • Lack of temporal constraints in retrieval (similarity-based retrieval failing to respect time windows).

5. Significance and Future Impact

• Advancing Personalized AI: LifeBench provides a rigorous testbed for developing AI agents that can truly understand user lifestyles, not just chat history. This is crucial for applications in digital health coaching, proactive personal assistants, and behavioral research.
• Bridging the Gap: It shifts the paradigm from "chat-based memory" to "lifelog-based memory," forcing systems to reason over fragmented, real-world data traces.
• Research Directions: The benchmark identifies critical bottlenecks for future research, specifically the need for:
  • Specialized temporal indexing and time-aware query processing.
  • Mechanisms for multi-source integration and cross-modal reasoning.
  • Better modeling of non-declarative memory (habits and skills).
• Ethical Considerations: By using high-fidelity synthetic data, LifeBench allows for large-scale experimentation without the privacy risks associated with real user data, while still maintaining behavioral realism.

In conclusion, LifeBench demonstrates that current memory systems are ill-equipped for the complexities of long-horizon, multi-source human life simulation, setting a new standard for the development of truly personalized and adaptive AI agents.