🤖 machine learning

Do Static Embeddings Add Value to Hybrid Dutch Retrieval?

This paper demonstrates that for Dutch retrieval tasks, adding static embedding models to a hybrid system combining BM25 and transformer-based embeddings (Qwen) provides no marginal value and often reduces effectiveness, suggesting that a robust two-retriever lexical-transformer architecture is sufficient and that standalone embedding benchmark scores do not guarantee utility in hybrid fusion.

Original authors: António Pereira Barata

Published 2026-08-04

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: António Pereira Barata

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific needle in a massive, chaotic haystack. In the world of computer science, this is called information retrieval. For decades, the best way to find that needle was to look for the exact words written on it. If you were looking for "red apple," the computer would only show you documents containing the words "red" and "apple." This is like using a strict librarian who only follows a checklist. But what if the document says "crimson fruit" instead? The strict librarian misses it.

To fix this, modern computers started using "smart" helpers called embeddings. These are like magical translators that understand the idea behind the words. They know that "crimson fruit" is the same as "red apple," even if the words don't match. However, these smart helpers can be slow and expensive to run. So, engineers started building hybrid systems, combining the strict word-matching librarian with the smart idea-understanding translator to get the best of both worlds. But here is the big question: Do we need a third helper? There are cheaper, faster "static" models that are less smart but very quick. The big mystery is: Once you have the strict librarian and the smart translator working together, does adding this cheap third helper actually make the search better, or is it just extra clutter?

This paper dives into that exact mystery, but specifically for the Dutch language. The researchers set up a giant, controlled experiment to see if adding these cheap, static models to a team that already had a word-matcher (BM25) and a smart transformer (Qwen) would actually help find the right answers. They didn't just guess; they tested it on five different types of Dutch text collections, ranging from news articles and Wikipedia pages to government tenders and FAQs. They used a clever scoring method called Weighted Reciprocal Rank Fusion (RRF), which is like a voting system where the three helpers rank their top choices, and the final list is a weighted mix of their opinions.

The results were surprisingly clear and a bit of a reality check for the tech world. After running thousands of tests across 14,500 queries and nearly 800,000 documents, the researchers found that the "cheap third helper" (the static embeddings) never got a vote. In every single test, the best possible mix of results came from just the strict librarian and the smart translator working together. In fact, when they forced the cheap helper to join the team, the results actually got worse. The paper concludes that for Dutch retrieval tasks, adding these static models doesn't add any value; it just adds cost and complexity.

Instead of a three-person team, the study suggests a two-person team is the sweet spot. Interestingly, the perfect mix of the two winners changed depending on the type of text. For news articles, a 50/50 split between the word-matcher and the smart translator worked best. For government tenders, the word-matcher alone was the champion. However, if you didn't know what kind of text you were searching for, a simple 50/50 mix of the two was a very strong, reliable default that worked well everywhere. The study proves that while standalone benchmarks might make these cheap models look okay, they don't actually bring anything new to the table when they are already working alongside a powerful word-matcher and a smart transformer. The lesson? Sometimes, less is more, and a well-chosen duo is better than a crowded committee.

Technical Summary: Do Static Embeddings Add Value to Hybrid Dutch Retrieval?

Problem Statement

Modern retrieval systems increasingly combine lexical methods (e.g., BM25) with dense semantic representations (e.g., transformer-based embeddings). While standalone embedding benchmarks (such as MTEB-NL) effectively measure individual model quality, they fail to answer a critical systems-level question: Does a low-cost retriever contribute complementary ranking information once a lexical–transformer hybrid is already in place?

Specifically, this paper investigates whether modern multilingual static embeddings (which offer lower encoding costs but less contextual expressiveness than transformers) provide marginal value when fused with BM25 and a transformer-based retriever (Qwen/Qwen3-Embedding-0.6B) across Dutch retrieval tasks. The study posits that a weaker standalone model might still improve a fusion if its errors are complementary, but that standalone benchmark performance is insufficient to establish this marginal value.

Methodology

The author conducted a controlled, exhaustive evaluation across five Dutch retrieval datasets from the Massive Text Embedding Benchmark for Dutch (MTEB-NL), comprising 14,500 queries and 786,573 documents.

Retrieval Systems

Four retrievers were evaluated:

BM25: Lexical retrieval with Dutch stopwords and stemming.
Qwen: Transformer-based dense retrieval (1024 dimensions) using query instructions.
Potion: Multilingual static dense embedding (256 dimensions, Model2Vec encoding).
Static Similarity: Multilingual static dense embedding (1024 dimensions, Sentence Transformers static encoding).

Fusion and Evaluation Protocol

Exhaustive Scoring: Every query was scored against every document in its respective dataset (approx. 4.1 billion query-document pairs per retriever), eliminating top- $k$ approximation errors.
Weighted Reciprocal Rank Fusion (RRF): The system combined rankings using the formula:
$RRF(d | q) = \frac{\alpha}{k + r_{BM25}} + \frac{\beta}{k + r_{Qwen}} + \frac{\gamma}{k + r_{static}}$
where $\alpha + \beta + \gamma = 1$ and $k=60$ .
Weight Search: Weights were searched on a simplex in increments of 0.1.
Cross-Validation: A ten-fold query-level cross-validation (CV) procedure was used. Weights were selected on nine folds and evaluated on the held-out fold to estimate out-of-sample performance and prevent model-selection bias.
Statistical Inference: Paired bootstrap confidence intervals and sign-randomisation tests (with Holm correction for multiple comparisons) were used to quantify the significance of performance differences.
Cross-Domain Transfer: A "leave-one-dataset-out" experiment selected a single configuration based on four datasets and evaluated it on the fifth to test transferability.

Key Contributions

Marginal Value Assessment: A controlled test determining whether static retrievers add effectiveness beyond a BM25–Qwen fusion on Dutch tasks.
Exhaustive Evaluation: Complete full-corpus scoring and simplex searches over weighted lexical, transformer, and static rankings, avoiding candidate truncation.
Rigorous Inference: Training-only weight selection with out-of-fold evaluation, paired uncertainty estimates, and multiplicity-corrected hypothesis testing.
Transferable Defaults: Evidence that the optimal configuration always lies on the BM25–Qwen edge, supporting a robust equal-weight default for settings without in-domain tuning data.

Results

1. Static Embeddings Add No Value

The most significant finding is that neither static embedding model (Potion or Static Similarity) received positive weight in any of the 50 held-out fold selections. All optimal configurations lay strictly on the BM25–Qwen edge ( $\gamma = 0$ ).

Forcing a static contribution reduced effectiveness relative to the BM25–Qwen edge.
The static models were rejected even though they are multilingual and modern, suggesting their ranking errors were not complementary to the lexical–transformer hybrid in this setting.

2. Fusion Gains Over Individual Retriever

Weighted RRF outperformed the best individual retriever selected from the same training data on four of the five datasets:

Dutch News: +0.061 MRR (Fusion: 0.658 vs. BM25: 0.597).
VABB: +0.029 MRR (Fusion: 0.752 vs. Qwen: 0.723).
Wikipedia NL: +0.025 MRR (Fusion: 0.873 vs. Qwen: 0.848).
WebFAQ NL: +0.004 MRR (Fusion: 0.677 vs. Qwen: 0.673).
Open Tender: No gain (0.000), as BM25 alone was optimal.

All positive differences remained statistically distinguishable from zero after Holm correction, though the WebFAQ NL gain was noted as operationally modest.

3. Stability and Transferability

Stability: The selected fusion weights were highly stable; the same configuration was chosen in all ten folds for each dataset.
Cross-Domain Transfer: A leave-one-dataset-out selection consistently chose an equal weighting of BM25 and Qwen ( $\alpha=0.5, \beta=0.5, \gamma=0.0$ ). This equal-weight configuration outperformed the individual retriever selected from the remaining four datasets on every held-out task.

Significance and Claims

The paper claims that standalone benchmark performance is insufficient to establish marginal value in hybrid retrieval. A model may perform well in isolation but add no value (or even degrade performance) when fused with other components.

The study supports a two-retriever lexical–transformer architecture (BM25 + Qwen) as a robust, tested default for Dutch retrieval tasks. It argues that:

Static embeddings (specifically the two evaluated models) do not justify the added architectural and computational complexity in this hybrid setting.
Equal weighting (0.5 BM25, 0.5 Qwen) serves as a defensible, domain-agnostic default when in-domain tuning data is unavailable, while dataset-specific tuning yields further gains when representative queries exist.
Exhaustive evaluation is necessary to avoid confounding factors like top- $k$ truncation, which can obscure the true marginal value of fusion components.

The author concludes that for the evaluated models, datasets, and preprocessing choices, the static branch should be removed from subsequent Dutch embedder comparisons, shifting the focus to an effectiveness–compute frontier among transformer families.