💬 NLP

Enhancing Lexicon-Based Text Embeddings with Large Language Models

This paper introduces LENS, a novel lexicon-based embedding method that leverages Large Language Models and token clustering to achieve competitive, compact text representations that outperform dense embeddings on the MTEB benchmark and reach state-of-the-art results when combined with them.

Original authors: Yibin Lei, Tao Shen, Yu Cao, Andrew Yates

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yibin Lei, Tao Shen, Yu Cao, Andrew Yates

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a library catalog system for the entire internet. You need a way to turn every book, article, and tweet into a unique "fingerprint" (an embedding) so a computer can find the right information when you ask a question.

For a long time, the best way to do this was to use Dense Embeddings. Think of these like a smoothie. You throw all the words of a sentence into a blender, and out comes a single, uniform liquid. It's efficient and works well, but if you look at the smoothie, you can't tell which specific fruit (word) contributed which flavor. It's a "black box."

Then, there were Lexicon-Based Embeddings. These are more like a spice rack. Instead of blending everything, you keep every single spice (word) separate. If you ask about "cinnamon," the system lights up the "cinnamon" jar. This is great because it's transparent—you know exactly why the system made a match. But, there was a big problem: the spice rack was messy.

The Problem: The Messy Spice Rack (LLM Tokenizers)

Modern AI models (Large Language Models or LLMs) use a tokenizer to break text into pieces. Think of this like a clumsy chef chopping ingredients.

Redundancy: The chef might chop "What," "what," and " what" into three different, confusing piles, even though they are the same word.
Fragmentation: The chef might chop "education" into "edu" and "cation," making it hard to recognize the whole concept.
One-Way Vision: These chefs usually only look at the ingredients they've already chopped, ignoring what's coming next. This makes it hard to understand the full context of a sentence.

Because of this mess, previous attempts to use these "spice rack" embeddings with modern AI were clunky and didn't perform as well as the "smoothie" (dense) models.

The Solution: LENS (The Smart Organizer)

The authors of this paper introduce LENS (Lexicon-based Embeddings with LLMs). They didn't just use the messy chef; they hired a Smart Organizer to fix the spice rack before the AI even starts cooking.

Here is how LENS works, using simple analogies:

1. Grouping the Chaos (Token Clustering)
Instead of having a separate jar for "What," "what," and "What," the Smart Organizer groups them together. It takes all the similar-looking or similar-meaning pieces and puts them in one Cluster.

Analogy: Imagine sorting a messy drawer of socks. Instead of having 50 individual socks scattered everywhere, you fold them and put matching pairs into labeled bins. Now, when you need "socks," you just grab the "socks" bin.
Result: This reduces the size of the vocabulary from a huge, noisy mess to a clean, compact list of about 4,000 to 8,000 "bins."

2. Giving the Chef 360° Vision (Bidirectional Attention)
The original AI chefs could only look at the past (what they had already chopped). LENS tweaks the AI so it can look at the whole sentence at once, both forward and backward.

Analogy: Imagine reading a sentence where you can only see the words before the current one. You might miss the punchline! LENS lets the AI read the whole sentence like a human does, understanding that "bank" means something different in "river bank" vs. "savings bank" because it sees the whole picture.

3. The Best of Both Worlds
LENS creates a "spice rack" that is just as efficient as a "smoothie" but keeps the transparency of knowing exactly which words mattered.

Compact: It's small enough to fit in standard computer memory.
Prunable: If you need a smaller version, you can just throw away the least active "bins" without breaking the system.
Transparent: You can look at the result and say, "Ah, the system matched this because it found the 'fast' and 'rapid' bins."

The Results: Why It Matters

The researchers tested LENS on a massive benchmark called MTEB (think of it as the "Olympics" for text search and understanding).

Solo Performance: LENS performed better than almost all other models that were trained only on public data. It beat the previous "smoothie" champions in many categories.
The Super-Combo: The most exciting part? When they mixed the LENS "spice rack" with a standard "smoothie" (dense embedding), the result was State-of-the-Art.
- Analogy: It's like having a team where one person has perfect memory of every specific detail (LENS), and the other has perfect intuition about the general vibe (Dense). Together, they are unbeatable.

The Catch (Limitations)

Language: Right now, this only works well for English. The "chef" needs to learn how to organize other languages' spices.
Cost: Using a giant AI model to do this organization is more expensive (computationally) than using smaller, older models.

In a Nutshell

LENS takes the messy, fragmented way modern AI sees words, cleans it up by grouping similar words together, and lets the AI see the whole sentence at once. The result is a text embedding system that is fast, small, transparent (you know why it made a decision), and extremely powerful, especially when paired with traditional methods. It proves that you don't have to choose between a "black box" smoothie and a messy spice rack; you can have a perfectly organized, high-performance spice rack.

1. Problem Statement

While Large Language Models (LLMs) have revolutionized text embedding, the field is dominated by dense embeddings (low-dimensional, real-valued vectors). Lexicon-based embeddings (high-dimensional vectors where each dimension corresponds to a vocabulary token) offer advantages in transparency and interpretability but have historically underperformed dense models, particularly when applied to modern LLMs.

The authors identify two primary bottlenecks preventing LLMs from generating effective lexicon-based embeddings:

Token Redundancy and Noise: Modern tokenizers (e.g., subword tokenization) fragment words (e.g., "education" $\to$ "edu", "cation") and create redundant variations based on casing or whitespace (e.g., "what", "What", " what"). This leads to inconsistent lexical matching and inflated vocabulary sizes.
Unidirectional Attention: Causal LLMs (decoder-only) typically use unidirectional attention during pre-training, meaning a token only attends to preceding context. This limits the model's ability to capture full-context dependencies, which is critical for generating accurate token-level scores used in lexicon-based embeddings.

2. Methodology: The LENS Framework

The authors propose LENS (Lexicon-based Embeddings with LLMs), a framework designed to generate low-dimensional, high-quality lexicon-based embeddings.

A. Token Embedding Clustering

To address token redundancy, LENS does not use the raw vocabulary of the LLM. Instead:

Clustering: It applies K-means clustering to the token embeddings from the language modeling head.
Centroid Replacement: The original token embeddings are replaced by the centroids of these clusters.
Dimensionality Reduction: If the target embedding size is $k$ (e.g., 4,000 or 8,000), the vocabulary is effectively reduced to $k$ clusters. Each dimension in the output vector corresponds to a specific cluster of semantically similar tokens (e.g., grouping "rapid," "quickly," and "swift").
Benefit: This merges redundant tokens, reduces dimensionality to be comparable to dense embeddings, and naturally supports sparsity.

B. Architecture Modifications

Bidirectional Attention: Unlike previous approaches that rely on prompt engineering to simulate context, LENS modifies the model architecture during fine-tuning to enable bidirectional attention. This allows every token to attend to the full context, significantly improving the quality of token-level representations.
Pooling Strategy: The framework employs max-pooling across the sequence dimension of the logits. The authors found this superior to sum-pooling or last-token pooling for lexicon-based tasks.

C. Training Procedure

Objective: The model is trained using InfoNCE loss (contrastive learning) on a single-stage pipeline.
Data: It relies exclusively on publicly available data (mixture of retrieval, reranking, clustering, and classification tasks), adhering to the training recipe of the SOTA dense model BGE-en-ICL for fair comparison.
No Specialized Sparsity Training: Unlike methods requiring Matryoshka Representation Learning (MRL) to support variable dimensions, LENS inherently supports efficient dimension pruning without retraining.

3. Key Contributions

First LLM-based Lexicon Embeddings: Introduces the first lexicon-based embedding framework leveraging LLMs that achieves competitive performance across diverse tasks, not just retrieval.
Vocabulary Consolidation: Proposes a novel clustering mechanism to handle LLM tokenizer redundancy, creating coherent, compact representations.
Architectural Insight: Demonstrates that bidirectional attention is critical for lexicon-based embeddings, contrasting with findings that unidirectional attention is optimal for dense embeddings.
Natural Sparsity: Shows that LENS embeddings can be pruned (Top-K) at inference time without performance degradation, eliminating the need for complex training objectives like MRL.
Hybrid State-of-the-Art: Proves that combining LENS with dense embeddings yields superior performance to either method alone.

4. Experimental Results

The authors evaluated LENS on MTEB (Massive Text Embedding Benchmark) and AIR-Bench.

MTEB Performance:
- LENS-8000 (8,000 dimensions) achieved the highest average performance among all models trained exclusively on public data, outperforming the dense counterpart BGE-en-ICL (trained on the same data).
- It achieved SOTA zero-shot performance on the retrieval subset (BEIR) when combined with dense embeddings.
Efficiency & Pruning:
- LENS maintains high performance even when pruned to 256 dimensions (Top-K pruning), outperforming Gecko (a model explicitly trained with MRL for variable dimensions) in similar low-dimensional settings.
Qualitative Analysis:
- Clustering successfully groups semantically equivalent tokens (e.g., "cannot" and "impossible") and morphological variants (e.g., "shows" and "showed").
- The embeddings demonstrate deep semantic understanding (e.g., linking "hypoxia" to "oxygen") rather than just shallow token matching.
Hybrid Approach:
- Concatenating LENS-8000 with BGE-en-ICL achieved a 63.00 average score on BEIR, surpassing NV-Embed-v2 (62.65), which uses a much more complex training pipeline involving synthetic data and hard-negative mining.

5. Significance and Limitations

Significance:

Transparency: LENS offers interpretable embeddings where dimensions correspond to human-understandable token clusters, unlike opaque dense vectors.
Efficiency: It bridges the gap between the interpretability of sparse models and the performance of dense models, offering a path to compact, prunable representations without complex training.
Paradigm Shift: Challenges the notion that unidirectional attention is always optimal for LLM-based embeddings, showing that bidirectional attention is essential for token-level tasks.

Limitations:

Language: Currently limited to English; multilingual capabilities are unexplored.
Model Specificity: Experiments are conducted primarily on Mistral-7B; generalizability to other LLM architectures is not fully verified.
Computational Cost: Using LLMs as backbones increases computational costs compared to smaller, specialized lexicon models like SPLADE.
Failure Cases: The model can still suffer from subword tokenization issues where lexically similar but semantically unrelated clusters are activated (e.g., "hangar" triggering "anger").

In conclusion, LENS demonstrates that with proper architectural adjustments (clustering and bidirectional attention), lexicon-based embeddings can rival and even surpass dense embeddings, offering a powerful, transparent, and efficient alternative for text representation.