💬 NLP

Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connector

The paper introduces MILCO, a novel multilingual Learned Sparse Retrieval architecture that leverages a specialized connector and a LexEcho head to map cross-lingual queries and documents into a shared English lexical space, achieving state-of-the-art performance and superior efficiency compared to leading dense and sparse baselines.

Original authors: Thong Nguyen, Yibin Lei, Jia-Huei Ju, Eugene Yang, Andrew Yates

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thong Nguyen, Yibin Lei, Jia-Huei Ju, Eugene Yang, Andrew Yates

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the librarian of a massive, global library containing books in 39 different languages. A patron walks up and asks a question in Chinese: "How do I import music from Momo Live to my phone?"

In the old days, the librarian would have to guess which language the answer was in, or rely on a translator who might miss the specific name "Momo" because it's a brand name that doesn't translate well.

MILCO is a new, super-smart librarian system that solves this problem. It's a type of search engine technology called Learned Sparse Retrieval (LSR). Here is how it works, broken down into simple concepts and analogies.

1. The Problem: The "Lost in Translation" Gap

Most modern search engines are like dense fog. They turn words into invisible, mathematical clouds (vectors). They are great at understanding the vibe of a sentence, but they are opaque. You can't see why they picked a result. Also, when they try to translate a query from Chinese to English to find an answer, they often drop the "weird" or specific words (like the brand "Momo") because they don't fit neatly into standard English dictionaries.

2. The Solution: The "Universal Translator" Connector

MILCO introduces a special bridge called a Multilingual Connector.

The Analogy: Imagine every language in the world is a different dialect of a giant game of "Telephone." Usually, if you pass a message from Chinese to English, the details get fuzzy.
MILCO's Trick: It takes the Chinese query, runs it through a special translator (the connector), and forces it to speak English using real English words. But here's the magic: it doesn't just translate; it understands the meaning and maps it to the English words that best describe it.
The Result: Whether you ask in French, Hindi, or Japanese, MILCO converts your question into a list of English keywords (like music, import, phone, step). This allows it to search a massive English-based index using any language.

3. The "LexEcho" Head: The Safety Net

There's a catch. Sometimes, a word is so unique (like "Momo" or a specific local slang) that translating it to English loses the meaning. If the system only relies on the English translation, it might miss the answer.

The Analogy: Think of the LexEcho as a bilingual security guard.
- View 1 (The English View): The guard translates your request to English to find the main answer.
- View 2 (The Source View): If the guard realizes, "Wait, 'Momo' is a specific name that doesn't translate well," they shout out the original Chinese word (陌) to the search engine as a backup.
Why it matters: This ensures that even if the translation isn't perfect, the system still remembers the original unique words. It's like having a safety net that catches the specific details that usually fall through the cracks.

4. The Training: "Learning to Match"

MILCO didn't just wake up knowing this. It went through a two-step training camp:

Sparse Alignment Pretraining (The "Dictionary" Phase): The system was fed millions of pairs of sentences (e.g., a Chinese sentence and its English translation). It learned to map the Chinese words directly to the correct English keywords. This prevented the "semantic collapse" where the system forgets what words actually mean.
Contrastive Training (The "Quiz" Phase): It was then tested on real search questions. If it picked the wrong document, it was corrected. This made it sharp and effective at finding the best answers, not just any answers.

5. The Superpower: Speed and Efficiency

Most powerful search engines are like heavy trucks. They are strong but slow and take up a lot of space (memory).

MILCO is like a high-speed electric scooter.

Pruning: Because MILCO uses "sparse" representations (mostly zeros, with only a few important words active), it can throw away the unnecessary parts of the data after it's learned.
The Analogy: Imagine a dense vector is a 100-page report. MILCO is a 30-word summary.
The Result: MILCO is 3 times faster and creates an index 10 times smaller than its biggest competitors (like Qwen3-Embed), yet it still finds better answers. It can run on standard computer chips without needing massive supercomputers.

Summary

MILCO is a search engine that:

Unifies Languages: It translates everything into a shared English "vocabulary space" so it can search across 39 languages with one single model.
Remembers the Details: Its "LexEcho" feature keeps the original unique words (like brand names) so they don't get lost in translation.
Is Fast and Small: It uses a "sparse" method that cuts out the fluff, making it incredibly fast and cheap to run, while still beating the giant, heavy models in accuracy.

It's the difference between trying to find a needle in a haystack by looking at a blurry photo of the whole haystack, versus having a magnet that instantly pulls out the needle, no matter what language the haystack is written in.

1. Problem Statement

Learned Sparse Retrieval (LSR) offers a unique advantage over dense retrieval by combining the efficiency of bi-encoders with the transparency of lexical matching (interpretable token weights). However, existing LSR approaches face two critical limitations when scaling beyond English:

Semantic Collapse: Directly training multilingual LSR models often leads to "semantic collapse," where the model loses interpretable term semantics, resulting in ungrounded, random outputs and degraded retrieval effectiveness.
Fragmented Support: Current solutions are either English-only (e.g., SPLADE), rely on separate models for specific language pairs (limiting scalability), or combine sparse heads with dense/multi-vector heads (e.g., BGE-M3), where the sparse component often underperforms and lacks robust cross-lingual support.
Entity Loss: When projecting non-English text (especially non-Latin scripts) into a shared English vocabulary, uncommon entities (e.g., specific brand names like "Momo" or "Douyin") are often lost or mistranslated, hurting retrieval accuracy.

2. Methodology: The MILCO Architecture

The authors propose MILCO, a novel architecture designed to map queries and documents from 39+ languages into a shared English lexical space while preserving source-language robustness.

Core Components

Multilingual Encoder: A transformer-based encoder (initialized from bge-m3-unsupervised) processes input text in any supported language, producing contextualized hidden states.
Multilingual Connector: A lightweight Multi-Layer Perceptron (MLP) projects the multilingual hidden states into a dimension aligned with the English embedding space. This acts as a "pivot," unifying representations across languages before they are decoded.
LexEcho Head (Dual-View Representation): This is the core innovation. Instead of a single output, it generates two complementary sparse views:
- Pivot (English) View: Generated by an English MLM head (similar to SPLADE). It maps concepts to English tokens (e.g., "music", "import"), enabling cross-lingual matching.
- Source View: Generated via a special [ECHO] token mechanism. It selectively retains and weights original input tokens (e.g., "陌", "直播") that might be lost in translation or are uncommon entities.
- Final Output: The model combines both views, allowing it to match documents via English semantics or direct source-token matching.

Training Regime: Two-Stage Strategy

To prevent semantic collapse and ensure effectiveness, MILCO uses a specialized two-stage training process:

Sparse Alignment Pretraining (SAP):
- Goal: Ground multilingual inputs into the English lexicon.
- Method: Uses widely available parallel bitext corpora (e.g., OPUS). An oracle English LSR teacher (SPLADEv3) generates target sparse representations for English sentences. The model minimizes a Sparse-Aware MSE (SMSE) loss between the student's English view and the teacher's target, focusing only on active lexical coordinates.
- Significance: This step is a prerequisite; without it, contrastive training leads to semantic collapse.
Sparse Contrastive Training (SCT):
- Goal: Optimize retrieval effectiveness.
- Method: Fine-tunes the model on retrieval datasets using KL Divergence distillation from a cross-encoder teacher. It also applies $\ell_1$ -norm regularization to enforce sparsity.

3. Key Contributions

MILCO Architecture: The first LSR model to unify multilingual and cross-lingual retrieval within a single model by mapping diverse languages to a shared English vocabulary via a multilingual connector.
LexEcho Head: A novel mechanism that enhances robustness against "tail entities" (uncommon names, non-Latin scripts) by maintaining a source-language view alongside the English pivot view.
Sparse Alignment Pretraining (SAP): A new pretraining strategy that leverages parallel corpora to align sparse representations, solving the semantic collapse problem inherent in multilingual LSR.
Efficiency via Post-Hoc Pruning: Demonstrates that MILCO supports dynamic efficiency. By pruning low-weight tokens, it can reduce document representations to ~30 active dimensions without significant performance loss, drastically reducing index size and latency.

4. Experimental Results

The model was evaluated on 39 languages across four major benchmarks: MIRACL (multilingual), MTEB v2 (multilingual/cross-lingual), MLDR (long-document), and MKQA (cross-lingual).

State-of-the-Art Performance:
- On MIRACL, MILCO (560M params) achieves an average nDCG@10 of 72.3, outperforming the dense Qwen3-Embed 8B (+3.6%) and BGE-M3 variants.
- It surpasses BGE-Sparse by 34.1% and T-Splade by 32.7%.
- On MLDR (long documents), it achieves 74.4 nDCG@10, significantly beating dense+sparse ensembles.
Zero-Shot Cross-Lingual Retrieval:
- On MKQA (English docs, 25 non-English queries), MILCO achieves a Recall@100 of 76.6, outperforming dense baselines like Qwen3-Embed 8B (67.9) and E5-Mistral 7B (70.1). This is remarkable as MILCO was not explicitly trained for cross-lingual retrieval tasks.
Robustness of LexEcho:
- Removing the LexEcho head (Source View) drops performance by ~4.2% on MIRACL. The gains are most pronounced in non-Latin languages (e.g., Chinese +8.09%, Telugu +6.8%), proving its effectiveness in handling translation gaps.
Efficiency vs. Effectiveness:
- Index Size: With aggressive pruning (keeping only 30 tokens/document), MILCO reduces index size by 80% (from 61GB to 12GB) while maintaining competitive performance.
- Latency: Pruned MILCO achieves 3x lower latency (0.61ms vs 1.29ms) and 10x smaller index size compared to the similarly sized Qwen3-Embed 0.6B (dense).

5. Significance

MILCO represents a significant leap forward in multilingual information retrieval.

Transparency: Unlike dense models, MILCO provides interpretable, sparse lexical weights, allowing for error tracing and bias inspection.
Scalability: It eliminates the need for language-specific models or complex translation pipelines, supporting 39+ languages in a single 560M parameter model.
Practical Deployment: The ability to dynamically prune representations allows for a "Matryoshka-like" trade-off between latency and accuracy, making it highly suitable for production systems with strict resource constraints.
Solving the "Uncommon Entity" Problem: The LexEcho mechanism addresses a critical failure mode in cross-lingual retrieval where specific entities are lost in translation, ensuring robustness for real-world search queries.

In summary, MILCO successfully bridges the gap between the efficiency of sparse retrieval and the multilingual capabilities of modern dense models, setting a new standard for transparent, efficient, and effective cross-lingual search.