💬 NLP

MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning

This paper introduces MemoryDocDataSet, a synthetic benchmark designed to evaluate AI systems on the joint challenge of navigating multi-session conversational memory and performing deep reasoning within long documents, revealing a significant performance gap in current models when handling questions that require retrieving relevant documents based on conversation history.

Original authors: Qiyang Xie, Jialun Wu, Xinjie He, Su Liu, Shuai Xiao, Zhiyuan Lin, Weikai Zhou

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qiyang Xie, Jialun Wu, Xinjie He, Su Liu, Shuai Xiao, Zhiyuan Lin, Weikai Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant to help you manage a complex legal case. This assistant needs to do two very difficult things at the same time:

Remember a long, messy history: They need to recall who said what during dozens of phone calls and meetings over the last six months.
Read a massive library: They need to find specific details inside thousands of pages of dense legal contracts and court rulings.

Until now, researchers have tested assistants on just one of these skills at a time. Some tests see if they can remember a conversation, but they don't give them any books to read. Other tests see if they can read a huge book, but they don't ask them to remember what was said in a meeting.

The Problem: Real life doesn't work that way. In the real world, you might ask your assistant, "What did we decide about the penalty clause in the contract we discussed last Tuesday?" To answer this, the assistant first has to remember the conversation to figure out which contract you are talking about, and then read that specific contract to find the answer.

The Solution: MemoryDocDataSet
The authors of this paper created a new "test" called MemoryDocDataSet to see if AI can handle this two-step challenge.

How the Test Works (The "Micro-World")

Instead of just giving the AI a question, they built 50 tiny, self-contained "worlds" for the AI to explore. Think of each world like a mini-mystery novel with these ingredients:

The Characters: 3 to 5 people with specific jobs and relationships.
The Timeline: A graph of events happening over six months.
The Books: 3 to 5 real, massive legal documents (each as long as a short novel, with 20,000 to 50,000 words).
The Chats: 5 different conversation sessions where the characters talk about these documents.
The Questions: 20 questions per world.

The "Hybrid" Twist

The most important part of this test is a special category of questions called "Hybrid."

Chat-only questions: "What time was the meeting?" (Answer is in the chat).
Doc-only questions: "What is the penalty amount in Contract A?" (Answer is in the book).
Hybrid questions: "What was the penalty amount in the contract we discussed during the March 15th meeting?"

To answer a Hybrid question, the AI must act like a detective:

Step 1: Scan the conversation history to realize, "Ah, on March 15th, they were talking about Contract B."
Step 2: Go open Contract B and read it to find the penalty amount.

If the AI skips Step 1 and just guesses, or if it opens the wrong book, it fails.

What They Found (The Results)

The researchers tested six different types of AI "strategies" to see how well they could solve these puzzles. Here is what happened:

The "Big Brain" (Long-Context LLM): They gave the AI the entire conversation and all the books at once (about 60,000 words). You might think this would be easy, but the AI got confused. It was like trying to find a specific needle in a haystack while staring at the whole haystack. It performed poorly on the tricky "Hybrid" questions.
The "Document Hunter" (RAG-Doc): This AI was great at finding answers inside the books if you told it exactly which book to look at. But when the question required remembering a conversation first, it collapsed. It couldn't figure out which book was relevant.
The "Hybrid Hunter" (RAG-Both): This was the best performer. It looked at both the chats and the books. It did better than the others, but it still struggled. It was like having a librarian who could find books and chats, but didn't have a clear strategy for connecting the two steps.

The Big Takeaway:
The paper shows that current AI systems are bad at connecting the dots. They are good at remembering chats OR reading books, but they are terrible at using a chat to tell them which book to read.

The authors conclude that future AI assistants need a new kind of "brain architecture." They need a system that doesn't just dump all the information into a giant pile, but one that can actively say, "Wait, the conversation points to this specific document. Let me go there and read it."

Summary

This paper introduces a new, difficult test that forces AI to combine memory (remembering a conversation) with reading (finding facts in a huge document). The results show that today's AI is still struggling to do both at once, revealing a gap that future technology needs to fill.

Technical Summary: MemoryDocDataSet

Problem Statement

Current AI benchmarks evaluate conversational memory and long-document reasoning in isolation, failing to capture the demands of real-world AI assistants that must operate simultaneously across both domains. Existing memory benchmarks (e.g., LoCoMo, LongMemEval) utilize multi-session dialogues but lack long documents, allowing systems to succeed via perfect conversation memorization without reading capability. Conversely, long-document benchmarks (e.g., L-Eval, ZeroSCROLLS) provide book-length texts but lack conversational structure, allowing systems to succeed via large context windows without memory navigation.

There is a critical gap in evaluating the joint capability required to:

Navigate multi-session conversation history to identify context.
Determine which specific long document is relevant based on that context.
Perform deep reading comprehension within that identified document to extract an answer.

This "joint-retrieval gap" is exemplified by scenarios such as a legal assistant recalling which contract a client discussed in a prior session and then navigating 40,000 tokens of that contract to find a specific clause.

Methodology

The MemoryDocDataSet Benchmark

The authors introduce MemoryDocDataSet, a synthetic benchmark comprising 50 micro-worlds and 1,000 question-answer (QA) pairs. Each micro-world is a self-contained scenario defined by the tuple $M = (P, G, D, C, Q)$ :

Personas ( $P$ ): 3–5 entities with defined roles, expertise, and relationships.
Event Graph ( $G$ ): A temporal Directed Acyclic Graph (DAG) of 5–10 time-stamped events spanning at least six months, encoding causal dependencies.
Documents ( $D$ ): 3–5 real long documents (20,000–50,000 tokens each) sourced from the Caselaw Access Project (US court opinions).
Conversations ( $C$ ): 5 multi-session dialogue sessions anchored to specific events, with at least 40% of sessions explicitly referencing documents.
QA Pairs ( $Q$ ): 20 pairs per micro-world across five reasoning categories (Single-hop, Multi-hop, Temporal, Knowledge Update, Adversarial).

The Source Dimension:
The defining innovation is the source tag annotation for every QA pair, categorizing the required information source:

Chat-only: Answer derivable from conversation history alone.
Doc-only: Answer derivable from long documents alone.
Hybrid: Requires a two-step process: (1) using conversation context to identify the relevant document, and (2) reading that document to extract the answer.
Distribution: Hybrid questions constitute 75.1% of the dataset, with a design floor of 30% per micro-world.

Generation Pipeline

The dataset is generated via a fully automated, config-driven seven-stage pipeline using LLMs (Claude Sonnet 4.6):

Document Collection: Sourcing and bundling real legal texts.
Persona & Event Graph Generation: Creating synthetic roles and temporal events.
Conversation Generation: Generating dialogues grounded on the event graph.
QA Generation: Creating questions with evidence references.
Quality Verification: An LLM-as-judge protocol using strict and lenient prompts to measure prompt-sensitivity self-consistency (median Cohen's $\kappa = 0.634$ ).
Dataset Packaging: Splitting into train/val/test (70/14/16) with document isolation.
Baseline Evaluation.

Baseline Experiments

Six baseline configurations were evaluated on the test split (160 QA pairs, 8 micro-worlds) using Token-level F1 and Exact Match (EM):

Base LLM: Truncated context (4,096 tokens).
Long-Context LLM: Full context (60,000 tokens).
RAG-Conv: Retrieval over conversation chunks only.
RAG-Doc: Retrieval over document chunks only.
RAG-Both: Joint retrieval over both modalities.
Memory System: Graph-based fact extraction from conversations.

Key Results

Performance on Hybrid Questions

The results reveal a significant "joint-retrieval gap":

RAG-Doc achieved a strong 0.453 F1 on Doc-only questions but collapsed to 0.267 F1 on Hybrid questions. This demonstrates that document retrieval without conversational navigation is insufficient for joint reasoning.
RAG-Both achieved the highest overall performance (0.358 F1) and improved Hybrid performance to 0.342 F1, confirming that incorporating conversation retrieval recovers much of the gap.
Long-Context LLM (0.323 overall) performed only marginally better than the Base LLM (0.317) despite a 15x context increase. It exhibited the lowest abstention accuracy (0.844), suggesting that expanding context windows without structured retrieval leads to increased hallucination on unanswerable questions ("lost in the middle" effect).

Specialization Asymmetry

A clear specialization asymmetry was observed: systems strong on one source tag (e.g., RAG-Doc on Doc-only) were weak on the complementary tag (Hybrid). Even the best baseline (RAG-Both) underperformed significantly compared to human expectations, indicating that current architectures do not natively support the two-step navigational strategy required for Hybrid tasks.

Significance and Claims

The paper claims the following contributions and significance:

First Joint Benchmark: MemoryDocDataSet is the first benchmark to simultaneously evaluate multi-session conversation history and long-document reasoning, formalized through the Hybrid source tag.
Measurable Gap: The benchmark quantifies a specific architectural gap: the inability of current systems to first navigate conversation to identify a document, and then retrieve from that specific document.
System Design Implications: The results suggest that simply increasing context window size or using standard RAG is insufficient. The next generation of AI assistants requires architectures that unify conversational memory with long-document navigation, specifically implementing a two-stage retrieval strategy (conversation $\to$ document identification $\to$ document retrieval).
Resource Release: The authors release the dataset, the generation pipeline, and all baseline implementations to facilitate research into this joint capability.

The paper maintains a modest tone regarding human performance, noting that a formal human upper bound has not been measured and that the current LLM-as-judge verification serves as a proxy for consistency rather than absolute correctness. The limitations acknowledged include the domain specificity (US legal text), the synthetic nature of the conversations, and the English-only scope.