The Role of Vocabularies in Learning Sparse Representations for Ranking

1. Problem Statement

Learned Sparse Retrieval (LSR) models, such as SPLADE, have emerged as a promising alternative to dense retrieval for the first stage of search ranking. They combine the semantic matching capabilities of neural networks with the efficiency of inverted indices. However, existing LSR models face a trade-off between retrieval effectiveness and efficiency:

• Computational Cost: Standard SPLADE models (using a 32K vocabulary) often result in "small vocabulary set-long posting list" scenarios. Because the vocabulary is coarse-grained (Wordpiece), terms have high co-occurrence, leading to long posting lists and high computational costs (FLOPS) during retrieval.
• Limited Vocabulary Studies: While recent work (ESPLADE) proposed expanding vocabularies to 100K+ using custom pretraining, there is a lack of understanding regarding:
  1. The specific role of vocabulary size in retrieval performance.
  2. The impact of pretrained weights vs. random initialization in expanded vocabularies.
  3. How these factors interact with static term pruning (removing low-scoring terms) to balance efficiency and effectiveness.

2. Methodology

The authors conducted a controlled study comparing three model configurations trained on real-world search click logs from Naver:

A. Model Architectures

1. SPLADE-32K (Baseline): A standard BERT-based model with a 32K Wordpiece vocabulary, pretrained on general corpora.
2. ESPLADE-100K (Pretrained): A BERT model with a 100K vocabulary initialized using the Expanded Masked Language Modeling (EMLM) method. The new vocabulary consists of frequent unigrams from the target corpus, and the output layer weights are constructed by pooling the weights of the corresponding Wordpiece subwords from the original model.
3. Rand-100K (Random): A BERT model with a 100K vocabulary where the output layer weights are randomly initialized, serving as a control to isolate the effect of vocabulary size from pretraining knowledge.

B. Training Strategy

• Pretraining: The EMLM model was pretrained on a large corpus (news, books, wikis) using a custom masking strategy that targets the expanded 100K vocabulary terms.
• Fine-tuning: All models were fine-tuned on a massive dataset of query-document pairs (approx. 262M pairs) using:
  • In-batch Negative Loss: To optimize ranking (N-pair loss).
  • Joint FLOPS Regularization: A loss function ($L_{jFLOPS}$) that minimizes the intersection of query and document embeddings to enforce sparsity and retrieval efficiency.
• Static Pruning: To further balance efficiency, the authors applied logit score-based static pruning. They retained only the top-$k$ terms (highest MLM logit scores) for queries ($q_k$) and documents ($d_k$), discarding the rest.

C. Evaluation Metrics

• Effectiveness: Mean Reciprocal Rank at 10 (MRR@10) and Recall at 10/100 (R@10, R@100).
• Efficiency: A custom FLOPS metric approximating the computational cost of traversing posting lists in an inverted index. Lower FLOPS indicates higher efficiency.

3. Key Contributions

1. Vocabulary Size as Representational Capacity: The paper demonstrates that vocabulary size in LSR is not just about linguistic expressivity but acts as a hyperparameter for representational capacity. A larger vocabulary allows the model to create more fine-grained decision boundaries in the sparse embedding space.
2. The "Latent Term" Hypothesis: The authors propose that during fine-tuning with ranking and FLOPS losses, vocabulary tokens (whether Wordpiece or random) are gradually converted into "latent terms." Their semantic meaning becomes secondary to their role in defining the structure of the inverted index and optimizing the retrieval objective.
3. Impact of Pretraining vs. Random Initialization:
   • Random Initialization: Even with random weights, a 100K vocabulary outperforms a 32K vocabulary in terms of efficiency (lower FLOPS) when pruning is applied, proving that dimensionality itself aids discriminative power.
   • Pretraining Value: The ESPLADE-100K (pretrained) model significantly outperforms the Rand-100K model in effectiveness while maintaining similar efficiency. This highlights that pretraining provides a better initial approximation for the ranking task, allowing the model to leverage the larger dimension more effectively.

4. Experimental Results

The experiments were conducted on a large-scale evaluation set with varying levels of pruning (controlling the number of active terms).

• Unpruned Setting (High FLOPS):

  • The 32K model (SPLADE) achieved the highest effectiveness but at a significantly higher computational cost (nearly 2x FLOPS) compared to 100K models.
  • The 100K models utilized a more diverse vocabulary, resulting in shorter posting lists and lower FLOPS, but initially showed slightly lower effectiveness than the 32K baseline.

• Pruned Setting (Low FLOPS / High Efficiency):

  • Efficiency: As pruning increased (reducing FLOPS to levels comparable to or lower than BM25), the 100K models maintained superior effectiveness compared to the 32K model.
  • The "Sweet Spot": The ESPLADE-100K model achieved the best trade-off. In the "FLOPS under BM25" scenario, it maintained high MRR/R@10 scores while having the same low retrieval cost as the random 100K model.
  • Comparison: The Rand-100K model was more efficient than the 32K model but less effective than ESPLADE-100K. This confirms that while vocabulary size drives efficiency, pretraining drives effectiveness.

• Posting List Analysis:

  • Table 3 in the paper shows that the 32K model has significantly longer posting lists (higher variance and mean) compared to 100K models. This confirms that larger vocabularies distribute term frequencies more sparsely, reducing the traversal cost of the inverted index.

5. Significance and Conclusion

This paper fundamentally shifts the perspective on the role of vocabularies in Information Retrieval (IR):

• Beyond Linguistics: In LSR, vocabularies are not merely tools for linguistic representation but serve as the specification for the retrieval engine's communication protocol. They define the "resolution" of the sparse space.
• Design Implications:
  • Vocabulary Size: Increasing vocabulary size is a viable strategy to improve retrieval efficiency (lower FLOPS) without sacrificing effectiveness, provided the model is pruned.
  • Pretraining: While random initialization works for efficiency, pretraining on the target domain (or a large corpus) is essential to maximize effectiveness within that efficient budget.
• Future Directions: The findings suggest that future LSR research should focus on optimizing vocabulary configuration (size and initialization) as a primary lever for balancing the trade-off between the computational budget and retrieval quality, rather than just treating vocabulary as a fixed linguistic constraint.

In summary, the authors prove that larger vocabularies enable more efficient sparse representations, and pretraining ensures these representations are semantically aligned for ranking, offering a new pathway for high-performance, low-cost search systems.