🤖 machine learning

Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration

This paper introduces two full-text scientific memory benchmarks, PAIM and PTr, to demonstrate that memory leaderboards are often misleading without controlling for retrieval budgets and protocols, ultimately arguing that scientific memory should be evaluated as budgeted, modality-aware context restoration rather than unconstrained architecture performance.

Original authors: Maksim Sheverev, David Finkelstein, Sergey Nikolenko

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maksim Sheverev, David Finkelstein, Sergey Nikolenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, complex mystery. You have a super-smart detective (an AI) who can read a library's worth of books in a split second. But here's the catch: even the smartest detective gets confused if you shove a million pages of text in front of them all at once. They might miss the crucial clue hidden in the middle, or get so overwhelmed they start guessing. This is the problem of "long-context" AI. To fix it, scientists are building "memories" for these detectives—digital filing cabinets where the AI can store facts, stories, and evidence to pull out only when needed.

For a long time, people thought the best memory was simply the one that could grab the most pages of text. It was like a game where the winner was the detective who could read the biggest stack of papers. But this paper asks a better question: Is it better to read a mountain of paper, or to read just the right few pages that actually solve the case? The researchers argue that for scientific research, memory isn't just about storing facts; it's about "context restoration." It's the ability to reconstruct the exact, interconnected web of evidence needed to answer a specific question, without getting lost in the noise. They want to know: if we give every detective the exact same amount of paper to read, who actually solves the mystery best?

The authors of this paper, Maksim Sheverev, David Finkelstein, and Sergey Nikolenko, decided to stop guessing and start measuring. They built two massive, real-world test labs using thousands of actual scientific papers about Artificial Intelligence and computer science. They created a new way to play the game called "budgeted context restoration." Imagine giving every detective a backpack that can only hold 30,000 characters of text. No matter how good their memory system is, they can't carry more than that. They then tested eight different types of "memory systems"—some that just grab chunks of text, some that build complex maps of facts, and some that try to summarize everything into tiny notes.

Here is what they found, and it completely flips the usual leaderboard.

First, they discovered that the "winners" in previous contests were often just cheating by carrying huge backpacks. One system, called Graphiti, was winning by a landslide, but it was because it was allowed to carry 2.6 million characters of text per question—roughly the size of a small novel! When the researchers forced Graphiti to use the same tiny 30,000-character backpack as everyone else, it didn't just lose; it fell to the bottom of the list. The "win" wasn't because the system was smarter; it was just because it had more fuel.

Second, they found that the type of memory matters less than the tools used to find the information. On one of their test sets (called PTr), which was full of specific names like "Kimi Linear" or "Ring-flash," the best strategy wasn't a fancy new architecture. It was simply mixing two search methods: one that looks for meaning (dense) and one that looks for exact words (sparse/BM25). When they added this "word-search" tool to the other systems, the scores jumped up, and the top three systems ended up in a dead heat, separated by less than a single point on a ten-point scale. It suggests that for scientific questions, having the right search engine is more important than having a fancy filing cabinet.

Third, they proved that their way of grading the answers was fair and reliable. They used a super-smart AI to judge the answers, but they were worried the AI might be biased. So, they had three different AIs grade the same answers and even brought in 112 human volunteers to do a side-by-side comparison. They found that the AI judges agreed with the humans about 77% of the time, but only when the difference in scores was clear. If two answers were very close (within one point), even the AI couldn't tell them apart. This means that in the future, we shouldn't get too excited about tiny score differences; they are likely just noise.

The paper introduces a new system called "Theoria," which tries to organize scientific papers into layers of evidence, communities of related ideas, and high-level theories. While Theoria performed very well and was competitive with the best systems, it didn't magically win the whole thing. The researchers suggest that Theoria's real power might come from how it handles long-term research projects over time, not just answering a single question in a test.

In the end, the authors argue that we need to stop treating AI memory like a video game leaderboard where the highest score wins. Instead, we should treat it like a scientific experiment where we control the budget. They showed that if you don't control how much text the AI is allowed to read, you aren't measuring intelligence; you're just measuring who has the biggest backpack. By standardizing the rules, they revealed that simple methods often work just as well as complex ones, provided you give them the right tools to find the needle in the haystack. They have released all their data, code, and test questions to the public, inviting everyone to try and beat their results, but with the new, fair rules in place.

Technical Summary: Beyond Memory Leaderboards

Problem Statement

Current evaluations of Long-Term Memory (LTM) for Large Language Model (LLM) agents are largely inherited from conversational settings, focusing on multi-session dialogues, user preferences, and fact updates. However, research agents operate in a different regime: they must restore evidence from full scientific papers to answer complex research questions. The authors argue that standard benchmarks fail to capture the distinction between recall (retrieving a fact) and context restoration (reconstructing an interconnected body of literature with specific granularity, conditions, and evidence).

A critical flaw in existing leaderboards is the lack of control over the retrieval budget (the volume of text returned) and retrieval modality (dense vs. sparse/hybrid). Without these controls, architectures that return massive amounts of raw text (e.g., full papers) appear superior not because of better memory management, but simply because they provide the synthesizer with more context. This paper posits that scientific memory should be evaluated as budgeted, modality-aware context restoration rather than an unconstrained architecture leaderboard.

Methodology

Benchmarks: Paim and PTr

The authors introduce two full-text scientific QA datasets designed to test context restoration under controlled conditions:

Public AI Memory (Paim): 81 papers on AI agent memory, RAG, and scientific agents (1.1M words). It includes 66 audited questions across three difficulty tiers (L1: single-span recall; L2: mechanism/comparison; L3: cross-document synthesis).
Public Transformers (PTr): 252 papers on Transformer architectures (attention, KV management, etc.). It includes 98 audited questions. This corpus features denser, lexically distinctive named entities (model names, kernel names) compared to Paim.

Both datasets undergo a rigorous rubric audit to ensure facts are verifiable within the cited papers, dependencies on volatile data are removed, and difficulty labels are accurate.

Evaluation Protocol

The authors evaluate eight memory/retrieval systems (including their own, Theoria) plus a no-retrieval baseline under a unified context-restoration protocol:

Ingestion: Systems ingest full papers (or chunks) into their respective stores (vectors, graphs, facts, etc.).
Retrieval: Systems retrieve context under strict budget constraints ( $B \in \{10K, 30K, 50K\}$ characters per query) and modality constraints (dense embeddings vs. sparse-dense hybrids using BM25).
Synthesis: An external synthesizer (GPT-4.1-mini) generates answers from the retrieved context.
Judging: Answers are scored against gold rubrics by an LLM judge (Gemini 3.1 Pro), with cross-validation against other frontier models (Sonnet 4.6, GPT-5.4, DeepSeek V4 Pro) and a blinded human side-by-side (SxS) study.

Systems Evaluated

The study spans diverse architectural families:

Simple RAG: Chunk-based baseline (null hypothesis).
Mem0: Extract-and-update atomic facts.
Hindsight: Narrative/temporal memory with four epistemic networks.
Cognee: Knowledge Graph (KG) completion via ECL pipeline.
Graphiti/Zep: Bi-temporal knowledge graph with episode retrieval.
Theoria: A three-layer system (Evidence, Community, Theory) using RAPTOR trees, Leiden clustering, and theory statements.
Prism: An ablation of Theoria isolating extraction and RAPTOR without community routing.
Direct Read: An oracle anchor using full corpus context.

Key Results

1. Retrieval Volume Dominates Native Leaderboards

In "native" (uncapped) runs, Graphiti achieves the highest scores on Paim (8.04 overall). However, this is entirely due to its retrieval of ~2.6 million characters per query (full paper episodes). When the retrieval budget is constrained to ~30K characters, Graphiti's performance collapses to the bottom of the leaderboard (5.27), while Simple RAG (7.25) and Theoria (7.19) emerge as leaders. This demonstrates that native leaderboards conflate architecture with context volume.

2. Budget Control Reveals Architectural Nuance

Under fixed budgets (~30K chars) on Paim, the top cluster (Simple RAG, Theoria, Mem0) is statistically indistinguishable (within 0.3 points). However, on PTr, where named entities are lexically distinctive, structured memories (Theoria, Mem0) pull slightly ahead of Simple RAG under dense retrieval.

3. The BM25 Hybrid Effect

The single most significant intervention measured was adding lexical BM25 retrieval to dense embeddings (sparse-dense hybrid).

On PTr, this hybrid approach significantly boosted scores for all systems.
The three hybrid variants (Simple RAG+BM25, Mem0+BM25, Theoria+BM25) tied for the lead within 0.03 points at a 50K budget, despite having ingestion costs spanning four orders of magnitude.
On Paim, the hybrid lift was less consistent, suggesting that corpus structure (lexical distinctiveness) dictates the efficacy of hybrid retrieval.

4. Judge Calibration and Resolution

The study validates the reliability of LLM-as-a-judge:

Rankings are highly consistent across frontier judges (Spearman $\rho \in [0.90, 0.97]$ ).
Human SxS evaluation shows that LLM judges agree with human raters on 77% of pairs when a tie-tolerance of $\tau=1$ is applied.
The effective resolution of the protocol is roughly one point on a 0–10 scale; differences smaller than this are indistinguishable from noise.

Significance and Contributions

The paper argues that the field must move beyond unconstrained leaderboards toward budgeted, modality-aware context restoration. Its primary contributions are:

Two New Benchmarks: Paim and PTr, featuring full-text scientific corpora, audited rubrics, and difficulty-tiered questions that stress context restoration rather than simple fact recall.
Unified Evaluation Harness: A framework that isolates the effects of retrieval budget, modality, and architecture, revealing that "winning" architectures often rely on uncontrolled context volume.
Theoria System: A three-layer scientific memory system (Evidence + Community + Theory) that demonstrates competitive performance under matched budgets, though it does not strictly dominate the hybrid leaderboard.
Methodological Rigor: A protocol for calibrating LLM judges against human evaluation and cross-judge consensus, establishing that small score differences (<1 point) are not statistically meaningful.

The authors conclude that while structured memory (Theoria, Mem0) offers advantages in specific domains (like PTr), retrieval modality (specifically hybrid sparse-dense retrieval) is currently the most powerful lever for improving scientific QA. They call for future benchmarks to explicitly report retrieval budgets, modalities, and granularity contracts to ensure fair and interpretable comparisons.