🤖 AI

Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies

This paper introduces "Latent Terms," a method demonstrating that dense retrieval models inherently learn representations that can be trivially decomposed via sparse autoencoders into Zipfian-distributed vocabularies suitable for BM25 scoring, thereby enabling high-performance sparse retrieval without additional supervision or expansion objectives.

Original authors: Benjamin Clavié, Sean Lee, Aamir Shakir, Makoto P. Kato

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Benjamin Clavié, Sean Lee, Aamir Shakir, Makoto P. Kato

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart librarian (a Dense Retriever) who has read millions of books. When you ask a question, this librarian doesn't just look up keywords; they understand the vibe and meaning of your question and find books that "feel" right. They do this by turning your question and the books into a single, complex number code (a vector) and checking how close those codes are to each other.

However, the paper argues that this librarian is actually hiding a secret superpower. Inside their brain, they have organized all that knowledge into a massive, hidden list of "concept tags" (a Latent Vocabulary). The problem is, the librarian is only trained to show you the "vibe score" (the dot product), not the list of tags.

Here is how the paper's method, called Latent Terms, unlocks that hidden list:

1. The "Translator" (The Sparse Autoencoder)

The authors built a special tool called a Sparse Autoencoder (SAE). Think of this as a translator or a decoder ring.

They take the librarian's internal "thoughts" (the complex number codes) and feed them into this decoder.
The decoder doesn't try to guess the answer; it just tries to reconstruct the librarian's thoughts.
In doing so, it forces the librarian's brain to break down those complex thoughts into simple, distinct "tags" or "features."

2. The "Zipfian" Surprise

When the decoder pulls out these tags, something magical happens. The frequency of these tags follows a natural pattern found in human language (called Zipf's Law).

The Analogy: Imagine a dictionary where a few words (like "the" or "and") appear constantly, many words appear moderately, and a huge number of words appear very rarely. This is how human language works.
The paper found that the "tags" the decoder pulled out of the AI's brain naturally formed this exact same pattern. They weren't random noise; they were structured like a real dictionary.

3. The "Old School" Scorecard (BM25)

Because these hidden tags look so much like real words, the authors realized they could use a very old, simple, and reliable scoring system called BM25 (which usually just counts how many times a word appears) to rank the results.

The Twist: They didn't teach the AI to use BM25. They didn't even show the AI any search questions during the training of the decoder. They just let the decoder do its job on random text, and then plugged the resulting tags into the old-school BM25 system.
The Result: This "plug-and-play" approach worked incredibly well. In many cases, it found better answers than the librarian's original "vibe score" method.

Why This Matters (The "LIMIT" Test)

The paper tested this on a specific challenge called LIMIT.

The Scenario: Imagine a game where the librarian is asked to find a book based on a very specific, tricky detail that doesn't rely on "vibes" but on exact, rare facts.
The Failure: The librarian's original "vibe score" method completely failed here (scoring near zero). It was like trying to find a needle in a haystack by guessing the color of the needle.
The Success: The Latent Terms method, using the hidden tags and the old-school scorecard, found the needle almost perfectly. It showed that the librarian knew the answer all along; the "vibe score" just wasn't the right way to access that specific piece of knowledge.

The "Hybrid" Nature

When the authors looked closely at the tags the decoder found, they realized they were a mix of two things:

Lexical: Tags that act like specific words (e.g., "bridge," "normal").
Semantic: Tags that act like concepts (e.g., "buy/purchase," "ancient/archaeology").

It's as if the decoder took the librarian's complex understanding and turned it into a list of keywords that cover both the exact words and the deeper meanings.

Summary

The paper claims that Dense Retrievers (the smart, modern librarians) contain a hidden, structured vocabulary of "tags" that are naturally suited for Sparse Retrieval (the old-school, keyword-based method). By using a simple decoder tool (SAE) to extract these tags, you can turn a modern AI into a powerful keyword-search engine without needing to retrain it or teach it how to search. The AI already learned the structure; we just needed the right key to unlock it.

Technical Summary: Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies

Problem Statement

Neural information retrieval models are typically categorized by the specific representations they expose at inference time and the scoring operators used to evaluate them. Dense single-vector retrievers encode queries and documents into fixed-size vectors scored via dot product or cosine similarity, while learned sparse retrievers (e.g., SPLADE) explicitly learn vocabulary weights for efficient indexing. A prevailing assumption is that a model's default scoring interface captures the entirety of its retrieval-relevant structure. However, recent work suggests that single-vector retrievers may encode useful structure beyond their default interfaces.

The central question addressed by this work is whether dense retrievers, even those not trained to produce sparse representations, contain latent sparse, indexable structures that can be recovered without retrieval-specific supervision. Specifically, the authors investigate if the internal representations of frozen dense retrievers can be decomposed into a "quasi-lexical" vocabulary suitable for classical sparse retrieval scoring (BM25) without the need for learned expansion objectives, hard negatives, or sparsity regularization.

Methodology: Latent Terms

The authors propose Latent Terms, a method that converts the token-level activations of a frozen dense retriever into sparse, BM25-searchable representations using a Sparse Autoencoder (SAE). The process involves three primary stages:

SAE Training on Frozen Retrievers:
- A dense retriever $R$ (single- or multi-vector) is frozen.
- An SAE is trained on the token-level activations ( $h_i$ ) produced by $R$ when processing unlabeled web text (specifically from FineWeb-Edu).
- Crucially, the SAE is trained only on a standard reconstruction objective ( $L_{SAE} = \|h - \hat{h}\|^2_2 + \lambda\|z\|_1$ ) with no retrieval supervision, no in-domain data, and no sparse retrieval training signals.
- The SAE maps dense activations to a higher-dimensional sparse code ( $z \in \mathbb{R}^{m}_{\geq 0}$ ) with a fixed vocabulary size (32,768) and a top- $k$ sparsity constraint (16 active features per token).
Constructing Latent Sparse Representations:
- At indexing and query time, the trained SAE encoder projects token-level activations from the frozen retriever into the latent vocabulary.
- Per-token sparse codes are aggregated via sum-pooling to create a single sparse vector per input.
- An element-wise activation transform $\phi$ (typically a square-root transform, $\phi(u) = \sqrt{u}$ ) is applied to the pooled vector. This transformation is necessary because SAE activations represent signal magnitudes rather than simple term counts, differing from natural language term frequencies.
BM25 Scoring:
- The resulting sparse vectors are treated as vocabulary weights. The feature indices serve as vocabulary terms, and the transformed magnitudes serve as term frequencies.
- The Okapi BM25 formula is applied directly over these latent features. The method adapts BM25 by treating the real-valued activation magnitudes as the frequency term $f(t, D)$ , while retaining standard Inverse Document Frequency (IDF) and document length normalization mechanisms.

Key Contributions

The paper makes three primary contributions:

Introduction of Latent Terms: A simple, plug-and-play method to convert frozen retriever activations into BM25-searchable sparse representations using reconstruction-trained SAEs, requiring no retrieval-specific training data or objectives.
Demonstration of Supervision-Free Sparse Retrieval: Evidence that these latent vocabularies support strong sparse retrieval performance without sparse retrieval supervision, outperforming the base model's native scoring in specific contexts.
Analysis of Latent Structure: An analysis showing that the generated vocabulary exhibits term-like collection statistics (approximating a Zipfian distribution) and captures a mix of lexical, narrow semantic, and broad topical units, effectively bridging the gap between dense and sparse retrieval.

Experimental Results

The method was evaluated on three dense retriever backbones: Contriever (single-vector), Nomic-embed-text-v1.5 (single-vector), and GTE-ModernColBERT (multi-vector).

BEIR Benchmark: Latent Terms consistently extracted strong sparse retrieval performance from all frozen backbones.
- It matched or outperformed comparable SPLADE variants (e.g., Latent Terms + Nomic outperformed SPLADE-v3).
- It significantly outperformed the base model's native single-vector cosine similarity scoring, particularly for Contriever.
- On domain-specific tasks (FiQA, TREC-Covid), Latent Terms strongly outperformed SPLADE variants.
LIMIT Benchmark: Designed to expose the failures of single-vector retrieval, LIMIT showed that single-vector models collapse (Recall@20 < 10%). Latent Terms recovered substantial performance, achieving a Recall@20 over 25 times higher than Contriever's native setting, suggesting the latent structure captures signals inaccessible to the single-vector scoring mechanism.
Comparison to MaxSim: While Latent Terms outperformed single-vector baselines, it generally lagged behind the native MaxSim scoring of the multi-vector GTE-ModernColBERT, though it showed better long-tail recall (Recall@1000) than the dense model.

Significance and Claims

The authors claim that their results offer a different perspective on neural retrieval models:

Hidden Expressivity: Dense retrievers contain more expressive and indexable structure than their default scoring functions expose. The "black box" of a dense retriever is not fully utilized by its native inference interface.
Decoupling Scoring from Representation: The study suggests that the scoring operator (e.g., dot product) is not the only way to access a model's retrieval knowledge. The internal representations learned during retrieval training are rich enough to be decomposed into sparse features suitable for classical IR methods like BM25.
Natural Statistics: The SAE successfully recovers a vocabulary with Zipfian collection statistics, a property essential for BM25's effectiveness, despite the SAE being trained solely on reconstruction without retrieval supervision.
Hybrid Nature: Qualitative analysis reveals that the latent terms capture a "hybrid" mix of lexical matches and semantic concepts, potentially explaining why the method performs well across diverse datasets.

The paper concludes that these findings encourage future research into decoupling the study of scoring operators from retrieval representation learning to better understand the limits of retriever expressivity. The authors note limitations regarding the focus on English, the specific SAE variants used, and the potential for further optimization of sparsity and scoring mechanisms.