Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues

This paper introduces EverMemBench, the first benchmark designed to evaluate long-horizon memory in multi-party collaborative dialogues, revealing that current LLM systems struggle with multi-hop reasoning, temporal versioning, and implicit relevance retrieval in realistic, complex interaction scenarios.

The Gist

Imagine you are the new manager of a massive, chaotic, 100-person company. You have thousands of emails, Slack messages, and meeting notes spanning an entire year. People are constantly changing their minds, updating plans, arguing in different group chats, and working on projects that overlap in confusing ways.

Now, imagine you ask your AI assistant: "Who was responsible for the final version of the marketing budget, and what was the exact date we approved it?"

If your AI is like current systems, it might get lost. It might confuse the draft with the final version, mix up who said what in a group of 20 people, or forget that a rule changed three months ago.

This paper introduces EverMemBench, a new "stress test" for AI memory, designed specifically to see if AI can handle this kind of real-world, multi-person chaos.

Here is the breakdown using simple analogies:

1. The Problem: The "Single-Thread" vs. The "Web"

Most current AI memory tests are like a one-on-one diary. They ask: "You and the AI talked for a long time. Do you remember what I said on Tuesday?"

• The Reality: Real life isn't a diary; it's a busy subway station. Hundreds of people are talking at once. Information is scattered across different groups (the "Marketing Group," the "Engineering Group"). People change their minds, and a decision made in one group affects another group.
• The Gap: Current AI benchmarks don't test if the AI can navigate this subway station. They only test if it can read a long book.

2. The Solution: EverMemBench (The "Chaos Simulator")

The authors built a fake company with 170 employees and 5 massive projects (like a tech startup, a bank, and a marketing firm). They generated over 4 million words of simulated chat history.

• The Analogy: Think of this as a giant, living simulation game. They didn't just write a story; they programmed the characters to have personalities, skills, and roles. They made them argue, revise plans, and update rules over time.
• The Goal: To see if an AI can act like a competent human employee who remembers who said what, when, and why, even when the information is buried in a sea of noise.

3. The Three Big Challenges (The "Boss Levels")

The paper tests the AI on three specific skills, which they call "Dimensions":

A. Fine-Grained Recall (The "Detective")

• The Test: "Find the specific link to the final budget document."
• The Trap: There are 10 links to budget documents. One is a draft, one is a Figma design, and one is the final Confluence link. They were all sent by the same person, just two days apart.
• The Failure: Current AIs often grab the wrong link because it looks similar or was mentioned more recently. They can't tell the difference between a "draft" and a "final version."
• The Analogy: It's like asking a detective to find the original murder weapon in a room full of identical-looking knives. Current AIs grab the first shiny knife they see.

B. Memory Awareness (The "Smart Assistant")

• The Test: "The boss wants to use a new tool, but does that break our old rules?"
• The Trap: The AI needs to remember a rule established months ago ("We only use Tool X") and realize that the new request violates it, even if the boss sounds very convincing.
• The Failure: AIs often forget the rule or get tricked by the boss's urgency. They act like a sycophant who agrees with everything, rather than a smart assistant who says, "Wait, we agreed on this last year."
• The Analogy: It's like a GPS that ignores the "Do Not Enter" sign you set last week because the traffic looks good right now.

C. Profile Understanding (The "Chameleon")

• The Test: "Write an email for 'Sarah'."
• The Trap: Sarah is a casual, emoji-loving engineer. The AI needs to write like her, not like a generic robot.
• The Failure: Even if the AI knows the facts, it writes in a stiff, formal tone because it forgot Sarah's personality.
• The Analogy: It's like hiring an actor who knows the script perfectly but forgets to wear the costume and speak with the character's accent. They sound like a robot reading a script, not the character.

4. The Results: The "Reality Check"

When they ran these tests on the smartest AI models available (like GPT-4 and Gemini):

• The Good: They are great at remembering simple facts if the answer is right in front of them.
• The Bad: They fall apart when things get complex.
  • Multi-Party Confusion: If three people are talking, the AI gets confused about who owns which idea.
  • Time Travel: They struggle to understand that a plan from January is different from a plan in June. They treat time like a flat list, not a timeline with updates.
  • The "Needle in a Haystack" isn't the problem: The problem isn't that the memory is too long; it's that the AI doesn't know how to search through the mess.

5. Why This Matters

This paper is a wake-up call. It says: "We can't just make AI read longer books. We need to teach it how to be a human teammate."

To build AI that can actually work in offices, hospitals, or schools, we need systems that understand:

• Context: Who is talking to whom?
• Evolution: How did this idea change over time?
• Identity: Who is this person, and how do they speak?

In short: EverMemBench is the "driver's license test" for AI memory. It proves that while AI can read a whole library, it still struggles to navigate the messy, changing, multi-person reality of a real workplace.

Technical Summary

Here is a detailed technical summary of the paper "Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues" (EverMemBench).

1. Problem Statement

Current Large Language Model (LLM) memory benchmarks fail to reflect the complexity of real-world collaborative environments. Existing benchmarks (e.g., LoCoMo, LongMemEval) primarily focus on dyadic (two-person) or single-user interactions, often treating memory as a simple retrieval task over long contexts.

However, practical applications (e.g., workplace collaboration, personal assistance) involve multi-party, multi-group dialogues where:

• Information is distributed: Facts are scattered across different speakers, groups, and channels.
• Context is dynamic: Decisions are revised, roles evolve, and constraints change over time.
• Attribution is complex: Systems must track "who said what" and how information propagates across a network.
• Personas are implicit: User preferences and expertise are inferred from long-term interaction patterns, not explicit profiles.

The paper argues that current systems fail not just due to context length limits, but due to structural limitations in multi-hop reasoning, temporal versioning, and profile consistency under collaborative conditions.

2. Methodology: EverMemBench

The authors introduce EverMemBench, the first benchmark designed specifically for long-horizon, multi-party collaborative memory.

A. Data Construction Pipeline

The benchmark is built on a rigorous, three-stage curation pipeline to ensure coherence and controllability:

1. Blueprint & Profile Generation:
   • Simulates an enterprise with 170 employees across 7 departments and 5 projects (Technology, Operations, Marketing, Financial Services, Governance).
   • Each employee has a unique persona defined by demographics, skills (40–60 skills), and an 8-dimensional communication style (formality, verbosity, humor, etc.).
   • Projects are decomposed into sub-tasks with explicit dependencies, timelines, and role assignments.
2. Conversation Generation:
   • Simulates 365 days of daily group chats (51,023 turns, ~4.2M tokens total).
   • Uses hierarchical summarization (weekly/monthly summaries + leader instructions) to maintain long-range coherence without exposing the full history to the generator during synthesis.
   • Ensures cross-group dependencies (e.g., one team's output is another's input) to force multi-hop reasoning.
3. QA Construction & Quality Control:
   • Generates 2,400 QA pairs with evidence grounding.
   • Applies a three-phase filter:
     • Blind Test: Ensures questions cannot be answered by parametric knowledge alone.
     • Evidence Grounding: Verifies sufficiency (answerable with specific evidence) and uniqueness (not answerable with other evidence).
     • Human Audit: Checks for logical and pragmatic consistency.

B. Evaluation Dimensions

EverMemBench evaluates memory systems across three core dimensions, broken down into nine sub-tasks:

1. Fine-Grained Recall: Retrieving precise facts from dense discussions.
   • Single-hop: Entity grounding (e.g., finding a specific link).
   • Multi-hop: Tracing a person's work timeline across fragmented threads.
   • Temporal: Calculating durations or identifying event boundaries (start/end) amidst noisy timestamps.
2. Memory Awareness: Reasoning over stored knowledge to handle novel scenarios.
   • Constraint: Applying implicit organizational norms to new requests.
   • Proactivity: Detecting conflicts between user requests and established rules.
   • Update: Resolving rule versioning (e.g., identifying the current policy after a revision).
3. Profile Understanding: Inferring stable user models from distributed signals.
   • Style: Matching communication tone (formality, emoji usage).
   • Skill: Recommending solutions within a user's expertise boundary.
   • Role: Framing responses from the user's professional perspective.

3. Key Contributions

1. EverMemBench Benchmark: A high-fidelity dataset featuring multi-party group chats with interdependent tasks, role-conditioned personas, and dynamic knowledge updates, totaling ~1M tokens per project.
2. New Evaluation Framework: Proposes a tripartite evaluation framework (Recall, Awareness, Profile) that moves beyond simple token retrieval to assess relevance recognition and persona consistency.
3. Diagnostic Insights: Systematically isolates failure modes in current architectures, distinguishing between retrieval failures and reasoning limitations.

4. Empirical Results

The authors evaluated state-of-the-art Long-Context LLMs (Gemini-3-Flash, GPT-4.1-mini, LLaMA-4) and Memory-Augmented Systems (Zep, Mem0, MemOS, MemoBase).

Key Findings:

• Multi-Hop Reasoning Collapse:
  • While Single-hop recall is high (~97% for top models), Multi-hop accuracy drops drastically (e.g., Gemini-3-Flash drops to 26.5%).
  • Performance degrades significantly as the number of involved groups increases (from 54.5% for 1 group to 19.7% for 3 groups).
  • Oracle Analysis: When ground-truth evidence is provided directly, accuracy jumps to ~88%, proving the bottleneck is attribution and retrieval, not reasoning capability.
• Temporal Reasoning Failure:
  • Systems struggle with version semantics (distinguishing drafts from final decisions) and working-day calculations.
  • Even with oracle evidence, temporal accuracy remains low (max 60%), indicating a gap in understanding event lifecycles (initiation vs. completion vs. archival).
• Retrieval vs. Reasoning Trade-off:
  • Memory-augmented systems often degrade the performance of strong long-context models (e.g., Gemini-3-Flash drops from 97% to ~76% with retrieval) because retrieval pipelines discard crucial contextual cues.
  • Retrieval fails to surface inferentially relevant information (e.g., implicit constraints) that requires reasoning to connect.
• Profile Understanding Limitations:
  • Systems perform poorly on Style (67% max) because communication style is an emergent pattern across many interactions, not a discrete fact.
  • Systems fail to maintain Role consistency, often defaulting to generic "best practice" advice rather than role-specific recommendations.

5. Significance and Future Directions

• Paradigm Shift: The paper argues that scaling context length or improving retrieval similarity is insufficient. Future memory architectures must explicitly encode versioned states, episodic boundaries, and cross-group dependencies.
• Architectural Implications: Effective memory systems need to integrate retrieval and reasoning more tightly, moving beyond surface similarity to inferential relevance.
• Benchmark Utility: EverMemBench serves as a diagnostic tool to identify why systems fail (e.g., is it retrieval granularity or reasoning capability?), guiding the development of next-generation LLMs capable of realistic, time-aware, and socially grounded collaboration.

The benchmark and code are publicly available, providing a standardized foundation for advancing long-horizon memory research.