Verbatim Chunks Beat Extracted Artifacts: A Controlled Ablation of Memory Representations for Long LLM Conversations

Technical Summary: Verbatim Chunks Beat Extracted Artifacts

Problem Statement
A prevalent class of conversational memory systems operates on the premise that raw dialogue history should be compressed into structured, distilled artifacts (e.g., extracted facts, decisions, events) to improve retrieval efficiency and accuracy. The underlying assumption is that a distilled structure is a superior retrieval target compared to the raw text it replaces. However, published comparisons often conflate representation changes with other system variables (retrieval stacks, answerers, judges), failing to isolate the specific impact of the stored representation. Furthermore, it remains an empirical question whether the "lossy distillation" inherent in extraction discards critical verbatim details that are necessary for answering specific constraints or questions, potentially causing structure and fidelity to pull in opposite directions.

Methodology
The authors conduct a controlled ablation study to isolate the variable of stored representation within a single, fixed retrieval–rerank–reasoning pipeline.

• Fixed Pipeline: The system utilizes a shared backbone including the bge-m3 embedder, bge-reranker-v2-m3, a hybrid retriever (dense + lexical), a gpt-4o answerer, and a gpt-4o-mini judge. Hyperparameters and token budgets are held constant.
• Manipulated Variable (Axis 1): The stored representation is swapped between two conditions:
  1. Verbatim Chunks: 512-character sliding windows of the raw conversation transcript (lossless, no LLM processing at write time).
  2. LLM-Extracted Artifacts: Typed artifacts (e.g., DECISION, KEYFACT, EVENT) generated by an LLM (gpt-4o-mini) from each turn, anchored by a verbatim quote but otherwise distilled.
• Secondary Variable (Axis 2): A 1-hop semantic similarity graph is toggled on/off for both representations to test if graph expansion can recover lost information.
• Benchmarks: Evaluation is performed on LoCoMo (natural long conversations, ~300 turns) and LongMemEval-S (500 questions across five memory abilities), alongside synthetic probes designed to test specific retention capabilities (single-hop, multi-hop, temporal reasoning).
• Controls: The study includes five confound controls: disabling the graph, enforcing zero lexical overlap between chunks, matching evidence budgets (increasing artifact retrieval count to match token counts), using dense-only retrieval, and varying extraction granularity (session-level vs. turn-level).

Key Results
The study finds that verbatim chunks decisively outperform LLM-extracted artifacts across both benchmarks and all answerable question types.

• Performance Gap:
  • LoCoMo: Verbatim chunks achieve 43.9% accuracy versus 28.0% for extracted artifacts (a 15.9-point gap).
  • LongMemEval-S: Verbatim chunks achieve 67.4% accuracy versus 45.4% for extracted artifacts (a 22.0-point gap).
• Baseline Comparison: The extracted-artifact pipeline never outperforms a naive RAG baseline (using raw chunks without a reranker) in overall accuracy. In fact, artifacts often underperform naive RAG.
• Graph Expansion: Enabling a 1-hop semantic graph improves retrieval recall for artifacts but fails to close the accuracy gap with verbatim chunks. The graph cannot regenerate content absent from the nodes.
• Cost Analysis: While extraction reduces generation token costs, it increases total cost per correct answer ($14.9 vs. $12.5 for chunks) due to lower accuracy.
• Synthetic Probes: In a synthetic multi-hop reasoning task where facts are pre-shaped to match the artifact schema, artifacts outperform chunks. However, this advantage disappears in natural conversation data, suggesting the premise only holds when the world is artificially arranged to satisfy the extraction schema.
• Union Store: Storing both chunks and artifacts together yields accuracy statistically indistinguishable from chunks alone, confirming that artifacts do not add value when verbatim text is present but cause a significant drop when they replace it.

Key Contributions

1. Controlled Ablation Finding: The paper provides the first clean isolation of the representation variable, demonstrating that under an identical pipeline, verbatim chunks consistently beat LLM-extracted artifacts on real-world conversational benchmarks.
2. Mechanistic Account (Lossy Distillation): The authors propose that the failure of extraction is due to "lossy distillation." Extraction commits to relevance at write time, discarding verbatim details (e.g., quantifiers like "everywhere") that are crucial at query time. Retrieval accuracy is shown to track the distance of the stored representation from the source text.
3. Reconciliation of Literature: The findings reconcile negative results with concurrent positive results (e.g., EMem) by noting that successful extraction systems use "near-verbatim" units with source provenance, whereas the tested "typed artifact" systems are too abstract.
4. Evidence Against Objections: The study systematically rules out alternative explanations such as retrieval stack bias, budget asymmetry, lexical overlap artifacts, and graph deficiencies through rigorous controls.

Significance and Claims
The paper claims a cautionary, negative result regarding the replacement of raw text with distilled artifacts in conversational memory. It argues that for the extraction designs tested, structure should augment verbatim text rather than replace it. The core lesson aligns with distributed cognition: external memory systems function best when they preserve the actual trace of the interaction. The authors emphasize that their findings are specific to the tested extraction schema and pipeline; they do not claim that all structured memory is ineffective, but rather that replacing source text with abstracted artifacts incurs a fidelity cost that current retrieval mechanisms cannot recover. The work highlights that the "lossy" nature of extraction is the primary bottleneck, not the retrieval stack or the graph structure.