Why Large Language Models can Secretly Outperform Embedding Similarity in Information Retrieval

Although the study finds that Large Language Model-based relevance judgment systems do not outperform embedding-based retrieval on standard TREC-DL 2019 benchmarks due to the short-sightedness inherent in human annotations, it argues that these models possess the theoretical capability to surpass embedding methods by better understanding relevance through reasoning.

The Gist

Here is an explanation of the paper using simple language and creative analogies.

The Big Idea: The "Smart Librarian" vs. The "Keyword Matcher"

Imagine you are looking for a specific book in a massive library. You have a question, and you need the librarian to find the right page for you.

For a long time, librarians (Information Retrieval systems) have used a method called NERS (Neural Embedding Retrieval Systems). Think of this librarian as a super-fast keyword matcher.

• How it works: If you ask, "What is the difference between a McDouble and a Double Cheeseburger?", this librarian looks for documents that contain the words "McDouble," "Double," "Cheeseburger," and "Difference."
• The Flaw: This librarian is short-sighted. If the answer is hidden in a sentence that doesn't repeat your exact words, the librarian misses it.
  • Example: The answer might be: "One slice of cheese instead of two. It's twenty cents more."
  • The keyword matcher sees: "McDouble" (missing), "Cheeseburger" (missing), "Difference" (missing). It thinks, "This doesn't look like your question," and throws the book away. But actually, it's the perfect answer!

The authors of this paper wanted to test a new kind of librarian: the LLM-RJS (Large Language Model Relevance Judgment System). Think of this as a Smart Reasoner.

• How it works: Instead of just matching words, this librarian reads the question and the answer, then uses logic to figure out if they fit together.
• The Goal: The authors hoped this Smart Reasoner would be much better than the Keyword Matcher because it understands meaning, not just words.

---

The Experiment: The Test Drive

The researchers took a standard test set (TREC-DL 2019) which is like a "driving test" for these librarians. They had a list of questions and a list of potential answers, and they asked: "Who can rank the best answers at the top?"

1. The First Race (Smart Reasoner vs. Keyword Matcher)

They let the Smart Reasoner (LLM) and the Keyword Matcher (NERS) compete.

• The Result: Surprisingly, they tied. The Smart Reasoner did not beat the Keyword Matcher.
• Why? The researchers were confused. They thought the Smart Reasoner should win easily.

2. The Twist: The "Ground Truth" is Flawed

To understand why the Smart Reasoner didn't win, they looked closer at the "answer key" (the human annotations).

• The Discovery: The humans who graded the test were also short-sighted.
• The Analogy: Imagine a teacher grading a student's essay. The student wrote a brilliant, logical answer to a math problem, but they didn't use the exact keywords the teacher was looking for. The teacher marks it wrong because "it doesn't look like the formula."
• In the paper, they found 94 cases where the Smart Reasoner said, "This is a perfect answer!" but the human grader said, "This is irrelevant."
  • Real Example: The question was "Difference between McDouble and Double Cheeseburger." The Smart Reasoner found a text saying "One slice of cheese instead of two." It gave it a perfect score. The human grader gave it a zero because the words didn't match.

The "Reasoning" Superpower

The researchers then tested the Smart Reasoner with a special mode called Chain of Thought (like asking the librarian to "think out loud" before answering).

• What happened: The Smart Reasoner with "thinking" capabilities found even more relevant answers that the humans missed.
• The Conclusion: The Smart Reasoner is actually better at finding the right information, but the test we use to grade it is broken. The test relies on human grades, and humans are just as short-sighted as the old Keyword Matcher.

The Final Verdict

The paper concludes with a paradox:

1. The Smart Reasoner (LLM) is capable of understanding deep relevance and finding answers that don't share keywords with the question.
2. The Keyword Matcher (NERS) is limited to finding only things that look similar.
3. The Problem: We are judging the Smart Reasoner using a ruler made by the Keyword Matcher (human annotations). Because the ruler is flawed, the Smart Reasoner looks like it's not doing better, even though it is.

The Catch (Why we don't use the Smart Reasoner yet)

If the Smart Reasoner is so great, why aren't we using it for everything?

• Cost and Speed: The Keyword Matcher is like a sprinter. It's cheap and incredibly fast. You can run it millions of times for pennies.
• The Smart Reasoner is like a marathon runner who stops to think. It is expensive and slow. To use it for every search query would cost a fortune and take too long.

The Future Solution:
The authors suggest a hybrid approach. Use the fast Keyword Matcher to narrow down the list to the top 100 candidates, and then use the expensive Smart Reasoner to pick the best one from those 100. This way, you get the speed of the sprinter and the brainpower of the thinker.

Summary in One Sentence

Large Language Models are smarter at finding relevant answers than current systems, but they appear to perform the same because the human tests we use to grade them are too focused on matching keywords rather than understanding meaning.

Technical Summary

Here is a detailed technical summary of the paper "Why Large Language Models can Secretly Outperform Embedding Similarity in Information Retrieval" by Matei Benescu and Ivo Pascal de Jong.

1. Problem Statement

The paper addresses a fundamental limitation in modern Information Retrieval (IR): the reliance on Neural Embedding Retrieval Systems (NERS).

• The Limitation of Similarity: NERS rank documents based on semantic similarity (e.g., Cosine Similarity) between query and document embeddings. The authors argue that similarity is a "short-sighted" proxy for relevance. A document can be highly relevant to a query without being semantically similar (e.g., a document explaining why two items differ, rather than listing the items themselves).
• The Hypothesis: Large Language Models (LLMs) equipped with reasoning capabilities (Chain of Thought) can overcome this limitation by understanding the logical relationship between a query and a document, rather than just surface-level similarity.
• The Paradox: Despite the theoretical advantage of LLMs, previous evaluations using standard human-annotated datasets (like TREC) have not shown LLMs outperforming NERS. The authors hypothesize that this is because the human annotations themselves suffer from the same "short-sightedness" as NERS, causing LLMs to be penalized for correctly identifying relevant but non-similar documents.

2. Methodology

The authors conducted two experiments using the TREC-DL 2019 passage retrieval dataset (derived from MS MARCO), which contains query-passage-relevance triples rated by humans on a 0–3 scale.

Experiment 1: Ranking Performance Comparison

• Goal: Compare the ranking performance of NERS vs. LLM-Based Relevance Judgment Systems (LLM-RJS).
• Models Evaluated:
  • NERS: Google's gemini-embedding001 (closed) and nomic-embed-text:v1.5 (open).
  • LLM-RJS: Various models including GPT-5.1, Sonnet 4.5, Gemini 2.5 Pro, and open-weight models (Llama 3.1/3.2, GPT-oss).
  • Baseline: BM25.
• Evaluation Metric: Normalized Discounted Cumulative Gain (NDCG@5 and NDCG@10) against human ground truth rankings.
• Setup: LLMs were prompted to output a relevance score (0–3) for each query-passage pair. Temperature was set to 0 for reproducibility.

Experiment 2: Impact of Reasoning and Annotation Analysis

• Goal: Analyze the effect of "reasoning effort" (Chain of Thought) and investigate discrepancies between LLM and human ratings.
• Setup: GPT-5.1 was tested in two conditions: No Reasoning (direct score) and High Reasoning (generating a Chain of Thought before scoring).
• Evaluation Metric: Instead of ranking metrics, the authors compared the Mean Absolute Error (MAE) and confusion matrices between LLM scores and human scores directly.
• Analysis: The authors manually inspected cases where LLMs gave high relevance scores (3) while humans gave low scores (0), specifically looking for "short-sightedness" in the human annotations.

3. Key Contributions

1. Empirical Validation of "Short-Sightedness": The paper provides evidence that human annotators often fail to recognize relevance when the document does not share lexical or semantic overlap with the query, mirroring the failure mode of embedding models.
2. Re-evaluation of LLM Performance: The authors demonstrate that LLM-RJS with reasoning capabilities can identify relevant documents that NERS and human annotators miss.
3. Critique of Standard Benchmarks: The paper argues that standard IR benchmarks (like TREC-DL) are flawed for evaluating reasoning-based IR systems because the ground truth (human labels) is biased toward similarity-based relevance, effectively penalizing superior reasoning models.
4. Quantification of False Positives: In the analysis of "false positives" (where LLMs rated a document as relevant but humans rated it irrelevant), the authors found that 89 out of 94 cases were actually human annotation errors, not LLM hallucinations.

4. Results

Experiment 1 Results (Ranking)

• Performance Parity: Top-tier LLM-RJS models performed on par with state-of-the-art NERS models (e.g., Gemini Embeddings) but did not demonstrate statistically significant improvements in NDCG scores.
• Cost Efficiency: NERS models are significantly cheaper and faster. For example, Gemini Embeddings cost ~$0.15 per 1M tokens, whereas GPT-5.1 costs significantly more for inference.
• Conclusion: Under standard evaluation metrics, LLMs do not appear to outperform NERS.

Experiment 2 Results (Reasoning & Discrepancies)

• Reasoning Increases Relevance Scores: LLMs with reasoning capabilities assigned higher average relevance scores (1.23) compared to non-reasoning LLMs (1.03) and humans (0.79).
• The "False Positive" Phenomenon: When LLMs rated a document as "Perfectly Relevant" (3) and humans rated it "Irrelevant" (0), the LLM was often correct.
  • Example: Query: "difference between a mcdouble and a double cheeseburger."
  • Document: "One slice of cheese instead of two. It's twenty cents more."
  • Human Rating: 0 (Irrelevant - likely due to lack of keyword overlap).
  • LLM Rating: 3 (Perfectly Relevant - correctly inferred the difference).
• NERS Similarity: In these specific "disagreement" cases, the NERS similarity scores were low (avg 0.44), confirming that embedding models also failed to retrieve these documents.

5. Significance and Implications

• The "Short-Sightedness" Bottleneck: The paper concludes that the reason LLMs haven't surpassed NERS in benchmarks is not a lack of capability, but a bias in the evaluation data. Human annotators, like NERS, struggle to identify relevance without semantic similarity.
• Redefining Ground Truth: Standard annotated datasets may be insufficient for evaluating the next generation of reasoning-based IR systems. To truly measure LLM superiority, new evaluation protocols are needed where trained experts or reasoning chains are used to validate relevance beyond simple similarity.
• Future Architecture: The authors suggest a hybrid approach (CoT re-ranking) where NERS performs a fast, cheap initial retrieval, and a reasoning-LLM re-ranks the top candidates to catch "non-similar but relevant" documents.
• Cost-Benefit Trade-off: While LLMs have the potential to outperform NERS, the current cost and latency make them impractical for large-scale retrieval without a re-ranking architecture.

In summary, the paper posits that LLM-RJS have the inherent ability to solve the short-sightedness problem of NERS, but current evaluation metrics (based on human similarity-biased annotations) mask this advantage, making reasoning models appear no better than simple embedding similarity.