Technical Summary: To Case or Not to Case: An Empirical Study in Learned Sparse Retrieval

Problem Statement

Learned Sparse Retrieval (LSR) methods, such as SPLADE, construct sparse lexical representations of queries and documents using pre-trained language models (PLMs) to enable efficient search via inverted indexes. Historically, LSR research has relied almost exclusively on uncased backbone models (e.g., BERT-base-uncased). These models merge case-sensitive distinctions (e.g., "Apple" and "apple") into single tokens, reducing vocabulary mismatch and aligning with traditional Information Retrieval (IR) pre-processing pipelines that typically lower-case text.

However, the landscape of state-of-the-art language models is shifting. Recent powerful architectures, including decoder-only models and newer encoders like ModernBERT, are released only in cased versions. Cased models preserve capitalization, distinguishing tokens based on case, which is beneficial for tasks like Named Entity Recognition (NER) but introduces a vocabulary mismatch risk in retrieval if the query and document casing do not align perfectly. Despite this shift, the impact of backbone model casing on LSR performance remains unexplored. This gap poses a potential risk to the viability of LSR methods as the community adopts newer, cased-only models.

Methodology

To address this gap, the authors conducted a systematic empirical evaluation comparing paired cased and uncased versions of the same backbone models (BERT and DistilBERT) across multiple datasets. The study employed a teacher-student distillation framework using Margin-MSE loss, with FLOPs regularization to induce sparsity.

The experimental design involved three specific intervention strategies to isolate the effects of casing:

1. Pre-processing Strategies:

   • None: Input text is used in its original form (preserving case).
   • Lowercasing: All input tokens are converted to lowercase before encoding.

2. Post-processing Strategies (applied to model outputs):

   • None: Full output embeddings are retained.
   • Uncased Vocabulary Only: Embedding dimensions corresponding to cased tokens are zeroed out; words existing only in cased form are treated as their lowercase equivalents.
   • Cased Regularizer (Training Only): An additional loss term penalizes the L2 norm of activations corresponding to cased dimensions, discouraging reliance on case-specific features during training.

3. Evaluation Benchmarks:

   • In-domain: MSMARCO Passage V1 (Dev set), TREC Deep Learning 2019, and TREC Deep Learning 2020.
   • Out-of-domain (Zero-shot): The BEIR benchmark, covering diverse retrieval tasks.
   • Metrics: MRR@10 and R@1000 for MSMARCO; nDCG@10 and R@1000 for other datasets; FLOPs for efficiency estimation.

Key Findings and Results

1. Impact of Casing on In-Domain Performance (RQ1 & RQ2)

• Uncased Superiority without Pre-processing: When no pre-processing is applied, cased models perform substantially worse than their uncased counterparts. On MSMARCO Dev, BERT-cased (no pre-processing) achieved an MRR@10 of ~31.7%, compared to ~35.8% for BERT-uncased (a ~4–5% drop). This degradation is attributed to vocabulary mismatches where case-sensitive tokens in queries fail to match lowercased tokens in documents (or vice versa), a common issue in open-domain corpora like MSMARCO where queries are typically lowercased.
• Mitigation via Lowercasing: Applying a lowercasing pre-processing step to cased models effectively eliminates this performance gap. For BERT-cased, lowercasing raised MRR@10 to 35.4%, bringing it within 0.3 points of the uncased baseline. Similar results were observed for DistilBERT, where the gap between uncased and lowercased-cased models became negligible (0.3%).
• Token-Level Analysis: The authors analyzed token mappings and found that without pre-processing, cased models frequently map cased inputs to uncased tokens, indicating a systemic preference for the uncased vocabulary even when case is preserved in the input. When lowercasing is applied, cased models almost entirely suppress cased vocabulary items (relying on them <1% of the time), effectively behaving as uncased models.

2. Efficiency vs. Accuracy Trade-offs (RQ3)

• Post-processing for Efficiency: Strategies such as restricting outputs to the uncased vocabulary or applying cased regularization significantly improve efficiency. For instance, restricting logits reduced FLOPs by nearly 50% (e.g., from 4.1 to 2.8 for DistilBERT) with only a marginal decrease in accuracy (<0.2% MRR@10).
• Limitations: While these methods improve efficiency, they introduce implementation complexity and require specific knowledge of the vocabulary structure, limiting their generalizability to new domains or models without known vocabularies. They do not enhance accuracy beyond the baseline lowercased cased model.

3. Zero-Shot Transfer Performance (RQ4)

• Robustness of Uncased Models: Uncased models demonstrated greater robustness in zero-shot transfer across the BEIR benchmark, leading on 9 out of 14 datasets for BERT and 3 for DistilBERT.
• Competitiveness of Lowercased Cased Models: Cased models with lowercasing pre-processing remained competitive, occasionally outperforming uncased models on specific datasets (e.g., NFCorpus and Quora). However, cased models without pre-processing consistently underperformed, barely exceeding BM25 on several datasets (e.g., Climate-FEVER, SciFact).

Significance and Contributions

The paper makes the following contributions to the field of Learned Sparse Retrieval:

1. Empirical Validation of Casing Impact: It provides the first systematic evidence that while cased backbone models default to underperforming in LSR due to casing-induced vocabulary mismatches, this is not an inherent limitation of the architecture.
2. Practical Solution for Modern Architectures: The study demonstrates that simple lowercasing pre-processing allows cased models to achieve performance parity with uncased models. This finding is critical for the adoption of state-of-the-art, cased-only models (like ModernBERT) in sparse retrieval settings, ensuring the long-term viability of LSR.
3. Behavioral Insight: The token-level analysis reveals that cased models, when forced into a lowercased input space, effectively suppress case-sensitive vocabulary and mimic uncased behavior. This explains why the performance gap disappears with pre-processing.
4. Efficiency Optimization: The paper identifies post-processing techniques that can halve computational costs (FLOPs) with minimal accuracy loss, offering a trade-off for resource-constrained applications.

In conclusion, the authors assert that cased models are viable for LSR provided that appropriate pre-processing (specifically lowercasing) is applied. This broadens the applicability of recent, powerful cased architectures to sparse retrieval, facilitating the integration of stronger backbone models without sacrificing retrieval effectiveness.