💬 NLP

Topic Is Not Agenda: A Citation-Community Audit of Text Embeddings

This paper demonstrates that current state-of-the-art text embeddings fail to capture fine-grained research agendas despite performing adequately at broader sub-field levels, revealing a critical limitation for scientific retrieval-augmented generation that can be partially addressed by leveraging citation-based signals.

Original authors: Junseon Yoo

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junseon Yoo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific needle in a massive haystack, but instead of looking for the needle, you are asking a very smart librarian to find you "things that feel like this needle."

This paper is a report card on how well modern "smart librarians" (called text embeddings) actually do their job when the library is the entire world of scientific research.

The Big Assumption

The entire system of modern AI search (used in tools like RAG) runs on one big guess: "If two documents have similar words, they must be about the same specific idea."

The authors decided to test this guess. They built a giant map of 3.58 million scientific papers. On this map, they drew two types of circles around groups of papers:

Level 1 (The Neighborhood): A broad circle around a whole field, like "Physics" or "Biology."
Level 2 (The Research Agenda): A tiny, tight circle around a very specific project, like "studying a specific type of star using UV light" or "testing a specific drug on a specific receptor."

The Test: The "Top 10" Game

The researchers asked four different AI models (the "librarians") to look at a paper and list its top 10 closest neighbors. They then checked: Do these 10 neighbors belong to the same tiny circle (Level 2) as the original paper?

Here is what they found:

1. The Librarians are Good at Neighborhoods, Bad at Specifics

Level 1 (Broad): The AI models were decent. About 50% of the time, the top 10 neighbors were in the same broad field. If you asked for a paper on "Biology," the AI gave you other biology papers. That's a pass.
Level 2 (Specific): The AI models failed miserably. When they looked for the specific research agenda, only 15% to 21% of the top 10 neighbors were actually on the same topic.
The Reality: This means that for every 10 papers the AI recommends to a scientist, 8 of them are actually about the wrong specific problem. They are in the same building, but they are working on completely different projects.

2. Bigger Brains Didn't Help
The researchers tested different sizes of AI models, from tiny ones to massive ones. They also tested a model (SPECTER2) that was specifically trained to understand citations (references).

The Result: Bigger models didn't fix the problem. The "citation-trained" model was actually the worst at finding the specific agenda. It turns out that just knowing "Paper A cites Paper B" isn't enough to understand the deep, specific connection between them.

3. The "Old School" Solution Worked Better
The authors tried a simple trick: instead of relying on the AI's "feeling" (cosine similarity), they used a simple rule: "How many other people have cited this paper?"

They took a basic list of papers and simply re-ordered them based on citation counts.
The Result: This simple, old-school method jumped from a 39% success rate to nearly 60%. It found the right specific agenda much better than the fancy AI models.

The Analogy: The "Coffee Shop" vs. The "Project Team"

Imagine a massive coffee shop (the scientific library).

Level 1 (Sub-field): Everyone in the shop is drinking coffee. If you ask the AI for "coffee drinkers," it gives you a list of people holding mugs. This works.
Level 2 (Research Agenda): You are actually looking for people discussing how to brew coffee using a specific French Press method.
The AI Failure: The AI looks at the words "coffee" and "brew" and says, "Oh, you want people talking about coffee!" It hands you a list of people discussing espresso machines, tea blends, and coffee bean farming. They are all "coffee people," but they aren't talking about your specific French Press method.
The Citation Fix: The citation method is like asking, "Who has been sitting at this specific table talking about French Presses for the last hour?" It ignores the general "coffee" buzz and finds the actual group working on your specific problem.

The Bottom Line

The paper concludes that while AI embeddings are great at finding the broad topic, they are currently terrible at finding the specific research question.

In the world of science, where the difference between "treating a disease" and "treating a specific strain of a disease" is everything, relying solely on these AI embeddings means you are likely to get 8 wrong answers for every 1 right one. The "missing signal" that the AI ignores is the structure of how scientists actually cite and build upon each other's work, which a simple count of citations captures much better.

Technical Summary: Topic Is Not Agenda: A Citation-Community Audit of Text Embeddings

Problem Statement

Deployed vector search systems and Retrieval-Augmented Generation (RAG) pipelines operate on the assumption that high cosine similarity between text embeddings serves as a reliable proxy for conceptual relatedness. While recent theoretical work (Steck et al. [2024]; Weller et al. [2026]) questions the intrinsic semantic validity of learned embeddings and the representational limits of single-vector retrievers, there is a lack of empirical measurement regarding where this assumption fails in high-stakes domains and at what level of granularity.

This paper investigates whether text embeddings can distinguish between a broad sub-field and a specific research agenda. The authors posit that while embeddings may successfully group papers within a general domain, they likely fail to capture the finer-grained "agenda" (specific questions, instruments, or methodologies) that defines a research thread, leading to retrieval of papers that are topically related but methodologically or contextually irrelevant.

Methodology

Corpus and Graph Construction

The study utilizes a composite corpus of 3.58 million scientific papers derived from OpenAlex, PubMed, Semantic Scholar, and publisher websites, filtered to SCIE-indexed journals and CORE A*/A conferences (published 2010 or later).

Augmented Citation Graph: To overcome the sparsity of direct citations (where many papers are orphans in the induced subgraph), the authors constructed an augmented graph combining three edge types:
1. Direct Citations: Undirected edges between target papers.
2. Bibliographic Coupling (BC): Edges weighted by Salton cosine similarity between papers sharing at least three external references.
3. Co-citation (CC): Edges weighted by Salton cosine similarity between papers co-cited by at least three common external citers.
  This resulted in a dense graph with ~153 million edges and a mean degree of 85.5.

Two-Level Community Partitioning

Using the Leiden algorithm optimizing the Constant Potts Model (CPM), the graph was partitioned at two hierarchical granularities:

Level 1 (L1 - Sub-field): A global partition with $\gamma = 10^{-4}$ , yielding 73,477 communities. Manual inspection confirmed these map to broad research sub-fields.
Level 2 (L2 - Research Agenda): A hierarchical partition where Leiden was re-run inside each L1 community (size $\ge$ 200) with $\gamma = 10^{-2}$ . This yielded 328,738 communities, representing specific research threads (e.g., separating UV-spectroscopy from X-ray studies within the same sub-field).

Evaluation Setup

Models: Four state-of-the-art embedding models were evaluated: Gemini Embedding, Qwen3-8B, Qwen3-0.6B, and SPECTER2 (a citation-trained scientific model).
Metric: The primary metric is the top-k same-community rate, measuring the percentage of a query paper's top- $k$ nearest neighbors (by cosine similarity) that share the same L1 or L2 community label.
Diagnostic Probe: To isolate the signal missing from embeddings, the authors tested a retrieval pipeline using the citation graph as a reranking signal. This involved a simple citation-count rerank applied to candidates generated by either BM25 or LLM-expanded Boolean queries.

Key Results

1. Failure at Agenda Granularity

The core finding is a sharp performance drop when moving from sub-field (L1) to research agenda (L2) granularity:

L1 Performance: Embeddings perform reasonably well, with top-10 same-community rates ranging from 45% to 52% across models and domains. This is significantly above the random baseline (0.16%–2.63%).
L2 Performance: Performance collapses to 15%–21% for top-10 neighbors. This implies that 8 out of 10 retrieved papers belong to a different specific research agenda than the query, despite being in the same sub-field.
Consistency: This failure is universal across all eight scientific domains (Biology, Biomedical, Chemistry, CS, Engineering, Env./Earth, Materials, Physics) and all four models.

2. Model Comparisons

General vs. Specialized: Contrary to expectations, SPECTER2 (trained with citation-based contrastive learning) was the weakest performer at both L1 and L2 levels.
Scaling: Increasing model size (Qwen3-0.6B to 8B) provided only a marginal gain of 3–5 percentage points at L2, failing to close the gap with the best performer (Gemini).
Ordering: The stable ranking across domains was: Gemini $\approx$ Qwen3-8B > Qwen3-0.6B > SPECTER2.

3. Diagnostic Probe Results

The citation-count rerank probe demonstrated that the augmented graph carries a strong agenda-matching signal that embeddings miss:

Baseline Retrievers: Without citation reranking, BM25 and the four embedding models achieved a top-1 L2 same-rate between 39% and 51%.
Citation Rerank: Applying a simple internal citation count rerank to candidates (from either BM25 or LLM-expanded queries) boosted performance to 57.7%–59.6%.
Significance: This represents a ~9 point improvement over the best cosine retriever (Gemini) and ~20 points over BM25 alone. The gain was consistent regardless of the candidate generator, suggesting the citation graph itself holds the missing signal.

Significance and Claims

The paper positions itself as a measurement study rather than a proposal for a new retrieval architecture. Its primary contributions are:

Empirical Validation of Theoretical Limits: The results provide concrete evidence for the "expressivity bound" of single-vector retrievers (Weller et al. [2026]), showing that even state-of-the-art models converge to a similar failure mode at fine-grained levels.
Granularity of Failure: The study identifies that the "topic" (sub-field) and "agenda" (specific research thread) are distinct retrieval challenges. Embeddings capture the former but systematically miss the latter.
Limitations of Citation-Based Training: The underperformance of SPECTER2 suggests that training on direct citation pairs is insufficient to learn the complex, multi-reference signals (bibliographic coupling and co-citation) required to map to specific research agendas.
Signal Isolation: The diagnostic probe isolates a specific "agenda-matching signal" present in citation graphs but absent in current text embeddings. This suggests that future RAG systems for scientific literature may require hybrid approaches that explicitly incorporate reference graph structures rather than relying solely on dense vector similarity.

The authors conclude that for scientific RAG, relying solely on cosine similarity leads to a "methodologically distant" retrieval pool. While the paper does not propose a new learned reranker, it demonstrates that the reference graph contains the necessary signal to recover agenda-level relevance, a signal currently ignored by standard vector search pipelines.