💬 NLP

Entity Labels Are Not Entity Signals: A Framework for Observable Relevance in Document Re-Ranking

This paper argues that entity-aware retrieval systems should shift from relying on Conceptual Entity Relevance (topical association) to Observable Entity Relevance (discriminative power), demonstrating that the latter significantly improves document re-ranking performance by better distinguishing relevant from non-relevant documents.

Original authors: Utshab Kumar Ghosh, Shubham Chatterjee

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Utshab Kumar Ghosh, Shubham Chatterjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian trying to help a patron find the perfect book on a specific topic, like "The impact of the 1990s tech boom." You have a massive library (the document collection) and a list of keywords or famous names (entities) associated with that topic, like "Steve Jobs," "Microsoft," or "Dot-com bubble."

For years, the standard advice for librarians (search engines) has been: "If a book mentions a famous name related to the topic, it's probably a good book." This is what the paper calls Conceptual Entity Relevance (CER). It's like asking, "Does this book talk about Steve Jobs?" If the answer is yes, the librarian assumes the book is relevant.

However, the authors of this paper argue that this logic is flawed. They propose a new way of thinking called Observable Entity Relevance (OER). Instead of just asking, "Is this name related to the topic?", they ask, "Does seeing this name in a book actually help me tell the difference between a great book and a boring one?"

Here is the breakdown of their findings using simple analogies:

1. The "Famous Name" Trap (CER vs. OER)

Imagine you are looking for books about the 1990s tech boom.

The CER Approach (The Old Way): You look for the name "United States." Since the tech boom happened in the US, a human or an AI would say, "Yes, 'United States' is definitely relevant to this topic!" So, you mark every book mentioning the US as a potential winner.
The Problem: Almost every book in the library mentions "United States," whether it's a brilliant history of the tech boom or a random recipe book. Because the name appears everywhere, it doesn't help you filter out the bad books. It's a "noisy" signal.
The OER Approach (The New Way): The authors look at the actual books in the library. They notice that while "United States" appears in both good and bad books, a specific phrase like "Netscape IPO" appears only in the good books about the tech boom. Even if "Netscape IPO" seems like a minor detail (peripheral), it is a strong signal because its presence reliably predicts a good book.

The Analogy:

CER is like hiring a security guard who stops everyone wearing a red shirt because "red is the color of the team." But since half the crowd is wearing red, the guard stops everyone, including the people you want to let in and the people you want to keep out.
OER is like a guard who stops people based on a specific, rare badge that only the VIPs (relevant documents) have. Even if the badge isn't the "main topic" of the event, spotting it is the best way to find the VIPs.

2. The "Label" vs. The "Signal"

The paper's title, "Entity Labels Are Not Entity Signals," is the core message.

Labels are what we think an entity means (e.g., "Steve Jobs is relevant to Apple").
Signals are what an entity actually does in the real world of search results (e.g., "Steve Jobs appears in 90% of irrelevant documents, so he's useless for filtering").

The authors found that for a long time, search engines have been trained using Labels (topical relevance) but needed Signals (discriminative power). It's like training a dog to sit by showing it a picture of a chair (the concept) but then expecting it to sit only when it sees a specific type of wooden leg (the observable signal). The dog gets confused because the picture doesn't match the reality.

3. The "Closed World" vs. "Open World" Illusion

The paper explains why this mistake was hard to see until now.

Closed-World Evaluation: Imagine a test where you are only allowed to look at a small pile of books that you already know are good. In this small pile, the name "Steve Jobs" might seem helpful because it's there.
Open-World Evaluation: Now, imagine you have to search the entire library. Suddenly, "Steve Jobs" is everywhere, and your search engine fails to find the good books because it's distracted by the noise.

The authors show that many search systems look great in the "Closed World" (the small test pile) but fail miserably in the "Open World" (the real library) because they rely on the wrong kind of clues.

4. The Solution: Listening to the Data, Not the Theory

The researchers tested this by building a system that ignores "topical relevance" and instead focuses on observable patterns. They asked: "Which entities appear frequently in the good books but rarely in the bad books?"

The Results:

When they used the old method (CER), the system could only remove about 4% of the bad books. It was like trying to filter sand with a colander that has holes too big.
When they used the new method (OER), the system could remove up to 10 times more bad books (pruning 10x better).
This led to a significant improvement in finding the right documents, beating the standard baseline methods.

Summary

The paper argues that in the world of search engines, just because a word or name is "about" the topic doesn't mean it's useful for finding the right answer.

Old Way: "Is this entity related to the query?" (Topical Relevance)
New Way: "Does finding this entity help me separate the good documents from the bad ones?" (Observable Relevance)

By shifting their focus from "what the entity means" to "what the entity actually does in the search results," the authors created a much more effective way to rank documents. They proved that the "signal" (the observable evidence) is what matters, not just the "label" (the theoretical connection).

Technical Summary: Entity Labels Are Not Entity Signals

Problem Statement
Entity-aware document retrieval traditionally relies on the assumption that entities topically related to a query (Conceptual Entity Relevance, or CER) serve as effective ranking signals. This assumption underpins most existing supervision strategies, which derive entity labels from human annotations, LLM judgments, or document relevance judgments (qrels) based on topical plausibility.

The authors argue this assumption is insufficient because it ignores the structural reality of entity linking. Unlike terms, which are ground-truth observations, entity links are hypotheses produced by an imperfect linker. An entity can be topically central yet non-discriminative if the linker fires indiscriminately across relevant and non-relevant documents. Conversely, a peripheral entity can be a strong signal if its observed mentions concentrate in relevant documents. The core problem is a mismatch between Conceptual Entity Relevance (CER)—whether an entity is about the query—and Observable Entity Relevance (OER)—whether the observed presence of that entity in a specific collection discriminates relevant from non-relevant documents.

Methodology and Framework
The paper introduces a diagnostic framework to formalize and measure the divergence between CER and OER.

Definitions:
- CER: A label ( $y_{cer} \in \{0, 1, 2\}$ ) assigned based on the query text and entity description alone, representing topical relatedness. This mimics standard human or LLM annotation.
- OER: A score calculated based on the distribution of linked entities within a candidate set. It measures the log-odds of an entity appearing in relevant versus non-relevant documents, weighted by a support function to downweight entities with sparse evidence.
- Signal Taxonomy: Entities are categorized into six modes (e.g., core signal, generic bait, anti-signal) based on their OER scores and frequency.
Experimental Setup:
- Datasets: Four collections (TREC Robust 2004, CODEC, TREC Core 2018, TREC Deep Learning 2019) using Lucene BM25 with RM3 as the first-stage retriever.
- Entity Linker: WAT (Wikipedia Entity Linking).
- Selectors: The study evaluates various entity selectors, including unsupervised baselines (PPR), CER-trained models (BERT trained on LLM-generated or human CER labels), and OER-aligned models (LightGBM LambdaRank trained on OER log-odds).
- Evaluation: The framework distinguishes between Closed-World evaluation (ranking restricted to documents containing selected entities) and Open-World evaluation (ranking over the full candidate set). Key metrics include Bait Rate, Signal Rate, Document Filtering (NonRelPrune, RelRetain), and downstream MAP/nDCG.

Key Contributions

Formal Distinction: The paper formally distinguishes CER from OER, highlighting that entity links are hypotheses subject to linking noise, making topical plausibility a poor proxy for discriminative utility.
Empirical Characterization: Through extensive experiments across four collections and multiple annotation sources (including human judgments), the authors demonstrate that CER and OER exhibit near-chance agreement ( $\kappa \approx 0$ ). In contrast, different operationalizations of OER agree substantially ( $\kappa \approx 0.5$ ), confirming that the disagreement is structured, not random.
Diagnosis of Downstream Failure: The study shows that CER-based supervision leads to the selection of "generic bait" (high-frequency, non-discriminative entities) and misses discriminative entities. This results in poor document filtering capabilities (pruning fewer than 4% of non-relevant documents in some cases) and a failure to improve open-world retrieval performance.
Supervision Alignment: The paper demonstrates that aligning supervision with OER (using observable distributional statistics) significantly improves retrieval effectiveness, whereas CER-based supervision fails to transfer to open-world settings regardless of the underlying architecture.

Results

Label Agreement: Agreement between CER and OER is consistently low ( $\kappa$ ranges from 0.06 to 0.23 across datasets), while OER operationalizations agree strongly ( $\kappa \approx 0.48-0.57$ ).
Selection Quality: CER-trained selectors (e.g., BERT trained on LLM CER labels) produce high "bait rates" (up to 29% of top-20 selections are non-signals) and low discriminative scores. OER-aligned selectors significantly reduce bait rates and increase signal rates.
Document Filtering: CER-based selectors fail to prune non-relevant documents effectively (e.g., pruning only 11.3% on Robust04). OER-aligned selectors improve non-relevant pruning by up to 10 $\times$ (e.g., 21.2% on Robust04).
Retrieval Performance: In open-world evaluation, CER-based selectors collapse to near-BM25 performance. OER-aligned selection (using a fixed QDER ranker) improves open-world MAP by 0.051 over BM25+RM3 and by 0.033 over the best CER-based selector. This gain holds across different architectures and is not an artifact of a specific model.
Coverage-Discrimination Tradeoff: The paper finds a structural tradeoff where increasing relevant document coverage inevitably increases non-relevant coverage in the current linking environment. Post-hoc filtering of CER-selected entities to improve discrimination severely degrades relevant coverage and overall ranking performance.

Significance and Claims
The paper claims that the bottleneck in entity-aware retrieval is not the ranking architecture or the quality of entity representations, but the supervision target. The field has relied on CER as a proxy for retrieval utility, but this proxy is structurally misaligned with the goal of discriminative ranking.

The authors argue that:

Entity labels are not entity signals: Topical relevance does not imply discriminative power.
The gap is structural: The mismatch arises because standard test collections provide document-level relevance judgments but lack entity-level discriminativeness labels, forcing models to learn from noisy proxies.
A shift is required: Future progress requires shifting from conceptual notions of relevance to observable ones. This involves using candidate-set statistics (frequency, distribution) as first-class signals and designing annotation protocols that expose annotators to corpus evidence rather than asking for isolated topical judgments.

The paper concludes that aligning supervision with OER breaks the performance ceiling observed in current entity-aware systems, offering a path to significant improvements in open-world retrieval without requiring new architectures.