🤖 AI

Rethinking Literature Search Evaluation: Deep Research Helps, and Human Citation Lists Are Not a Ground Truth

This paper demonstrates that a Deep Research pipeline significantly improves literature search recall while revealing that human citation lists are biased toward collaborators and thus insufficient as a sole ground truth, advocating for a multi-dimensional evaluation framework that combines recall, relevance, diversity, and co-authorship distance.

Original authors: Gaurav Sahu, Laurent Charlin, Christopher Pal

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gaurav Sahu, Laurent Charlin, Christopher Pal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. Your goal is to find every single clue (a research paper) that is relevant to your case (a new scientific idea).

For a long time, the standard way to check if a detective did a good job was to look at the notebook of the previous detective who worked on a similar case. If your list of clues matched their notebook, you got a high score. If you missed something they had, you got a low score.

This paper argues that this old way of grading is broken, and it proposes a better way to both find clues and grade the detectives.

1. The Problem with the "Old Notebook" (Human References)

The authors say that treating a human researcher's reference list as the "perfect truth" is like trusting a detective's notebook without question. They found two big issues:

The "Buddy System" Bias: Humans often cite their friends and colleagues. The paper found that human researchers are 2.5 times more likely to cite someone they have worked with directly, even if that person's work isn't the most relevant to the topic. It's like a detective only listing clues found by their own police precinct, ignoring brilliant clues from other cities.
The "Toolbox" Clutter: Humans also cite papers that are just "tools" or "foundations" (like citing the inventor of the hammer when you are writing about building a house). These papers are important for context, but they aren't actually about the specific topic of the new research. The study found that nearly half (48%) of the citations in human notebooks are these "tool" or "friend" citations, not direct topic matches.

When the authors used a neutral AI judge to grade these human notebooks, they found that only 51% of the cited papers were actually topically relevant. In contrast, the best AI systems found 86–88% relevant papers.

2. The Solution: "Deep Research" (The Super-Detective)

The authors built a new system called Deep Research to find clues much better than the old methods.

How it works: Instead of just typing a few keywords into a search engine (like asking a librarian for "books about cats"), the system reads the entire new paper first.
The Breadth-First Search: It then acts like a detective following a trail of breadcrumbs. It finds the papers the new paper mentions, then finds the papers those papers mention, and keeps going deeper. It explores the whole "family tree" of ideas.
The Result: This method is a game-changer. While old search methods only found about 20% of the relevant clues, the Deep Research system found over 80%. It didn't just find a few more; it found a whole new world of relevant information that was previously hidden.

3. The New Way to Grade

The paper suggests we stop using just one score to grade a literature search. Instead, we need a "dashboard" with four different gauges:

Recall: Did you find everything? (The Deep Research system wins here).
Topical Relevance: Are the papers you found actually about the topic? (The AI systems win here, beating the human notebooks).
Diversity: Did you find a mix of different ideas, or just the same thing over and over?
The "Friendship" Check: Did you cite too many of your own friends? (This is a new diagnostic tool to spot bias).

The Big Takeaway

The paper concludes that we need to stop thinking that a human's list of references is the "gold standard" or the final answer. Humans are great at finding foundational tools and acknowledging their team, but they are bad at finding the most relevant, obscure, or distant papers on a specific topic.

The best approach is to use a Deep Research system to cast a wide net and find everything, and then use a mix of metrics (relevance, diversity, and bias checks) to evaluate the results, rather than just comparing them to a human's imperfect notebook.

Technical Summary: Rethinking Literature Search Evaluation

Problem Statement

Literature search is a foundational step in scientific work, yet its evaluation remains problematic. The community traditionally evaluates retrieval systems by measuring precision and recall against the reference list of a query paper, treating that human-curated list as "ground truth." The authors identify two critical flaws in this convention:

Retrieval Limitations: Vanilla API-based search (relying on single or multiple keyword queries) is brittle and fails to capture the long-tail structure of citation graphs, missing relevant work hidden behind chains of references.
Ground Truth Flaws: The human reference list itself is incomplete and idiosyncratic. It includes non-topical citations such as foundational paper attributions, software/library credits, and citations to direct collaborators, which may not reflect topical relevance in a strict sense.

Methodology

1. The Deep Research Pipeline

The authors propose a modular retrieval pipeline designed to overcome the limitations of keyword-based search. The system operates in two phases:

Phase 1 (High-Recall Retrieval): An LLM generates diverse keyword queries based on the full text of the query paper. These are translated into provider-specific syntax for scholarly APIs (arXiv, OpenAlex, Semantic Scholar) to retrieve seed papers.
Phase 2 (Breadth-First Expansion): The system expands the candidate set breadth-first along bibliographic references up to a configurable depth ( $D_{max}=3$ ) and paper budget ( $N_{max}$ ). This leverages the citation graph to uncover relevant literature that keyword queries miss.
Re-ranking: The expanded pool is re-ranked using one of three interchangeable modules:
- QWEN3 Embeddings: Dense retrieval using instruction-conditioned query embeddings.
- LLM Debate: An agent generates arguments for and against citing a candidate, assigning a 0–100 relevance score.
- Ensemble: A combination of Debate and QWEN3 scores.

2. Benchmark and Evaluation Metrics

To evaluate these systems without data contamination, the authors introduce ROLLINGEVAL-JUN25, a benchmark of 250 computer science arXiv submissions from June 2025 (post-dating the training cutoffs of evaluated models). The evaluation employs four complementary metrics:

Precision and Recall: Measured against the human reference list.
Semantic Relevance (SR): A 0–100 score assigned by a neutral LLM judge (GPT-OSS-120B) based on the query paper's full text and the candidate's title/abstract.
$\alpha$ -nDCG: A novelty-aware metric measuring topic diversity and redundancy in the ranked list.
Co-authorship Distance: A diagnostic metric using the OpenAlex co-authorship graph to measure the hop distance between the query authors and candidate authors.

3. Ground Truth Analysis

The authors treat the human reference list not as an absolute truth but as a target to be analyzed. They use a neutral LLM judge to rate the topical relevance of every human citation and compare this distribution against AI-generated lists of matched length. They further analyze the "citation gap" by mapping citations onto the OpenAlex co-authorship graph to quantify network bias.

Key Results

Retrieval Performance

The Deep Research pipeline demonstrates an order-of-magnitude improvement in recall over traditional API-only search:

Recall: Vanilla search peaks at roughly 15% recall even at $K \approx 10^5$ . In contrast, all Deep Research variants exceed 80% recall by $K=10^4$ . The best variant (QWEN3 embeddings) achieves 51.9% recall at $K=1000$ , nearly tripling the performance of an arXiv-API baseline.
Precision: QWEN3 embeddings achieve the strongest top- $K$ precision (0.37 at $K=1$ ).
Diversity: The Debate-based re-ranker achieves the highest $\alpha$ -nDCG, suggesting it distributes relevant items more evenly across the list compared to embedding similarity alone.

The "Ground Truth" Gap

The study reveals a significant discrepancy between human citations and AI-generated lists regarding topical relevance:

Semantic Relevance: When restricted to the same list length, only 51% of human citations are judged as moderately relevant or higher (score $\ge 60$ ). In contrast, the strongest AI re-rankers (QWEN3 and Debate+Qwen) achieve 86–88% high-relevance scores.
Network Bias: Analysis of the OpenAlex co-authorship graph shows that humans are 2.5 times more likely than the best AI re-rankers to cite a direct collaborator ( $d=0$ ). The bias extends to one-hop collaborators, where humans cite at a rate of 12.2% compared to 6.9–7.4% for AI.
Non-Topical Citations: Approximately 48.6% of human citations fall into low-relevance categories (foundational architectures, software libraries, classical algorithms) that lack topical signals in their abstracts, whereas AI re-rankers produce such citations at a rate of only 12–14%.

Significance and Claims

The paper argues against the use of single-axis evaluation for literature search. The authors claim that:

Human Reference Lists are Imperfect Ground Truth: Treating human citations as a clean target for evaluation is problematic because they systematically include non-topical citations (foundational work, tools) and exhibit network bias (collaborators).
Multi-Dimensional Evaluation is Required: No single metric captures the full quality of a literature search. The authors advocate for reporting four complementary properties jointly:
- Recall: To measure coverage against known targets.
- Topical Relevance: To measure semantic alignment with the query.
- Ranked-List Diversity: To ensure the list covers varied topics without redundancy.
- Co-authorship Distance: To diagnose network bias in the citation set.
Deep Research is a Necessary Baseline: The order-of-magnitude recall gain provided by Deep Research expands the candidate pool, raising the achievable ceiling for any retrieval system, regardless of the final ground truth definition.

The authors conclude that future benchmarks should not treat human reference lists as the sole ground truth but should instead use them as one diagnostic axis alongside semantic relevance and network analysis. They release the ROLLINGEVAL-JUN25 benchmark, the Deep Research pipeline, and the OpenAlex distance analysis scripts to facilitate this re-examination.