💬 NLP

LMK > CLS: Landmark Pooling for Dense Embeddings

This paper introduces Landmark (LMK) pooling, a novel representation learning strategy that partitions sequences into chunks with inserted landmark tokens to effectively balance local salient features and long-context extrapolation, thereby outperforming traditional [CLS] and mean pooling methods on long-context tasks while maintaining short-context performance.

Original authors: Meet Doshi, Aashka Trivedi, Vishwajeet Kumar, Parul Awasthy, Yulong Li, Jaydeep Sen, Radu Florian, Sachindra Joshi

Published 2026-01-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Meet Doshi, Aashka Trivedi, Vishwajeet Kumar, Parul Awasthy, Yulong Li, Jaydeep Sen, Radu Florian, Sachindra Joshi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to summarize a very long novel for a friend. You need to capture the most important parts of the story so your friend understands the whole book without reading every single page.

In the world of Artificial Intelligence (AI), this is exactly what "dense embeddings" do. They turn long texts (like documents, articles, or code) into a single, compact list of numbers (a vector) that represents the meaning of the whole thing. AI uses these summaries to search for information, group similar documents, or classify text.

The problem is: How do you create that summary?

The Old Ways: The "Hero" and The "Average"

For a long time, AI researchers used two main methods to summarize a text, and the paper argues both have flaws when the text gets very long.

The "Hero" Token ([CLS]):
Think of this like appointing one specific character in a play to be the "Hero" who holds all the memory of the story. In AI, this is a special token (a placeholder) placed at the very beginning of the text. The model is trained to force this one token to remember everything.
- The Flaw: The paper found that this "Hero" gets overwhelmed. It tends to remember the beginning of the story very well but starts to "tune out" the middle and end. If the story is 100 pages long, the Hero only really remembers the first few pages. It's like trying to carry a heavy backpack; the further you walk, the more you drop things.
The "Average" (Mean Pooling):
This method is like taking every single word in the text, adding them up, and dividing by the total number of words to get a "middle ground" summary.
- The Flaw: This dilutes the important stuff. If a document has one crucial sentence that solves a mystery, but 999 boring sentences, the "Average" method washes that crucial sentence out. It's like making a soup where you add a tiny drop of salt to a giant pot; the flavor becomes weak and bland.

The New Solution: Landmark Pooling (LMK)

The authors propose a new method called Landmark (LMK) Pooling.

Imagine you are hiking a very long trail. Instead of relying on one person at the start to remember the whole path (The Hero), or trying to remember every single step equally (The Average), you place signposts (Landmarks) every few miles along the trail.

How it works: The AI breaks the long text into chunks. At the end of each chunk, it inserts a special "Landmark" token.
The Summary: To create the final summary, the AI doesn't look at every word, and it doesn't just look at the start. It only looks at the Landmarks. It takes the "memory" of each signpost and averages those together.

Why is this better?

No Overload: Because there are many Landmarks, no single one has to carry the weight of the entire book.
No Dilution: Because the Landmarks are placed frequently, they capture the "flavor" of each section. The crucial sentence in the middle of the book gets its own Landmark, so it doesn't get washed out by the boring parts.
Long Distance: This works great even for massive documents (like 32,000 words long), where the old "Hero" method fails completely.

The Results: A Race Against Length

The researchers tested this new method against the old ones on various tasks:

Short Stories: On short texts, the new method works just as well as the old "Hero" method. It doesn't break anything.
Long Novels: On very long texts, the new method crushes the competition. It finds the right information much better than the "Hero" or the "Average" methods.

They also found that even if they trained the AI on short stories, the "Landmark" method could still handle long stories later on (a concept called "extrapolation"). The old "Hero" method, however, got confused and performed poorly when the text got longer than it was trained on.

The Bottom Line

The paper introduces a simple, practical fix for a big problem in AI: How to summarize long texts without losing the important details.

By replacing the single "Hero" token with a series of "Landmark" signposts, the AI can read a whole library of documents without forgetting the most important parts, making it much better at finding answers in a sea of information.

Technical Summary: Landmark Pooling for Dense Embeddings

Problem Statement

Representation learning is fundamental to downstream tasks such as search, clustering, and classification. State-of-the-art sequence encoders typically collapse variable-length token sequences into a single fixed-dimensional vector using a pooling operator. The two dominant strategies are:

[CLS] Pooling: Utilizing a special [CLS] token at the beginning of the sequence to aggregate information.
Mean Pooling: Averaging the embeddings of all tokens in the sequence.

The authors identify systematic weaknesses in these approaches, particularly regarding long-context extrapolation:

[CLS] Limitations: The [CLS] token tends to concentrate information toward the initial positions of the sequence. Due to the mechanics of Rotary Positional Embeddings (RoPE), attention weights for distant tokens decay significantly, causing the [CLS] representation to under-represent distributed evidence found later in long documents.
Mean Pooling Limitations: While position-independent, mean pooling dilutes salient local signals by uniformly averaging all tokens. This often leads to worse performance on short-context tasks where critical evidence is concentrated in a small subset of tokens, as the averaging process assumes information is evenly distributed.

Existing methods often fail to balance the preservation of local salient features with the ability to generalize to long contexts without sacrificing performance on shorter sequences.

Methodology: Landmark (LMK) Pooling

To address these limitations, the authors propose Landmark (LMK) pooling, a mechanism that partitions a sequence into chunks and inserts special "landmark" tokens between them.

Key Mechanisms:

Tokenization Strategy: Instead of relying on a single special token (like [CLS]), the input sequence is split into fixed-size chunks (e.g., every $k$ tokens). A special landmark token (typically the [SEP] token, or [EOS] if [SEP] is unavailable) is inserted after each chunk.
Pooling Strategy: The final sequence representation is generated by mean-pooling only the embeddings of these inserted landmark tokens, rather than pooling all token embeddings or relying on a single initial token.
Granularity Variants:
- Fixed Chunking: Inserts landmarks at regular intervals.
- Variable Chunking: Randomly samples the insertion interval from a predefined set (e.g., $\{32, 64, 128, 256\}$ ) during training to encourage robustness to granularity choices.
- Sentence-based: Inserts landmarks after sentence boundaries (though this is noted as less generalizable to domains like code).

Theoretical Basis:
The authors argue that LMK pooling alleviates the representational bottleneck of [CLS] in long contexts by distributing the aggregation burden across multiple tokens. Simultaneously, it preserves local salient features better than global mean pooling because each landmark token aggregates a specific, manageable chunk of the sequence. The method introduces a modest computational overhead by adding a small number of special tokens (e.g., 256 extra tokens for a 32k sequence with 128-token granularity).

Key Contributions

Introduction of LMK Pooling: A simple, effective alternative to common pooling methods that delivers substantial gains on long-context embedding benchmarks across various domains and languages.
Systematic Analysis of Pooling Limitations: The paper provides empirical evidence and theoretical analysis (via attention weight distributions) demonstrating the inherent positional bias of [CLS] pooling and the feature dilution of mean pooling, characterizing their respective strengths and weaknesses for short vs. long contexts.
Robustness and Compatibility: The authors demonstrate that LMK pooling is robust across different training regimes (including short-context training) and is compatible with existing pretraining (e.g., RetroMAE) and retrieval optimization strategies.

Experimental Results

The authors evaluated LMK pooling against baselines (CLS, Mean, Latent Attention, and MultiCLS) using ModernBERT-base and gte-en-mlm-base architectures on English and multilingual datasets.

Short-Context Performance: LMK pooling matches or slightly exceeds the performance of [CLS] pooling on standard short-context retrieval benchmarks (e.g., MS MARCO Dev, BEIR, MTEB-v2), outperforming mean pooling.
Long-Context Performance: LMK pooling yields substantial improvements over all baselines on long-context tasks (e.g., MLDR, COIR, LongEmbed).
- On the MLDR dataset, LMK significantly outperformed [CLS] and mean pooling.
- In multilingual settings (mmBERT-base), LMK maintained substantial improvements on long-context evaluations across 13 languages.
Extrapolation: Even when trained on short sequences (512 tokens), LMK pooling demonstrated strong extrapolation capabilities to sequences up to 32k tokens. In contrast, [CLS] pooling performance degraded significantly when extrapolating beyond training lengths.
Training Length Analysis: Increasing the training sequence length for [CLS] models reduced but did not eliminate their positional bias or close the performance gap with LMK on out-of-domain long-context tasks.
Pretraining Integration: When used in RetroMAE pretraining, LMK pooling provided consistent performance gains on downstream long-context retrieval tasks compared to CLS-based pretraining.

Significance and Claims

The paper claims that LMK pooling offers a practical and scalable alternative to existing pooling methods. Its significance lies in:

Balancing Local and Global Information: It successfully mitigates the trade-off between preserving local salient features (often lost in mean pooling) and capturing long-range dependencies (often missed by [CLS]).
Simplicity: The method requires only a modification to the tokenization and pooling logic, without complex architectural changes or additional learnable parameters (unlike Latent Attention).
Generalization: It enables robust long-context extrapolation even when models are trained predominantly on short-context data, addressing a critical gap in current dense embedding systems where long-context supervision is scarce.

The authors conclude that LMK pooling integrates seamlessly with existing training and evaluation pipelines, making it a viable solution for long-context embedding tasks, multi-vector embeddings, and chunking-based Retrieval-Augmented Generation (RAG) systems.