Technical Summary: Should We Still Pretrain Encoders with Masked Language Modeling?

Problem Statement

Learning high-quality text representations is fundamental to Natural Language Processing (NLP). Traditionally, encoder models have been pretrained exclusively using Masked Language Modeling (MLM), which leverages bidirectional attention to capture contextual dependencies. However, recent evidence suggests that decoder models pretrained with Causal Language Modeling (CLM) can be effectively repurposed as encoders, often surpassing traditional MLM-based encoders on benchmarks like the Massive Text Embedding Benchmark (MTEB).

A critical ambiguity remains: do these gains stem from an inherent advantage of the CLM approach, or are they merely artifacts of confounding factors such as significantly larger model scales and substantially more training data used by recent decoder-based models? It is unclear whether the success of this paradigm is due to the learning objective itself or simply increased scale.

Methodology

To address this question, the authors conducted a large-scale, controlled ablation study designed to isolate the effects of the pretraining objective (MLM vs. CLM) from model size and data scale.

• Experimental Scale: The study trained a total of 38 models ranging from 210 million to 1 billion parameters. This involved over 15,000 fine-tuning and evaluation runs, consuming approximately 110,000 MI250X GPU hours.
• Controlled Variables: All models shared identical architectures (based on EuroBERT), vocabulary (LLaMA 3), and training data (FineWeb-Edu). Crucially, all models were exposed to the exact same sequence of samples to ensure fair comparison.
• Pretraining Paradigms: The authors compared three approaches:
  1. MLM: Standard bidirectional pretraining with varying masking ratios (20%, 30%, 40%, 50%).
  2. CLM: Autoregressive next-token prediction.
  3. Hybrid (CLM+MLM): A two-stage sequential approach where CLM is applied first, followed by MLM.
• Training Setups:
  • Pretraining From Scratch (PFS): Models trained from random initialization.
  • Continued PreTraining (CPT): Models initialized from existing CLM or MLM checkpoints and further trained with MLM.
• Evaluation: Models were evaluated across 12 datasets spanning four key categories: Sequence Classification (SC), Token Classification (TC), Question Answering (QA), and Information Retrieval (IR). Fine-tuning was performed with rigorous hyperparameter searches and multiple random seeds to account for variance.

Key Findings

1. MLM Generally Outperforms CLM on Representation Tasks

When trained from scratch for the same duration, models pretrained with MLM generally achieve superior performance across text representation tasks compared to CLM-only models. The performance gap is most pronounced in Sequence Classification and Question Answering, which appear particularly sensitive to the absence of bidirectional context. While CLM models perform competitively on Token Classification and Information Retrieval (especially as model size increases), MLM remains essential for robust, all-around performance.

2. CLM Offers Data Efficiency and Stability

Despite lower final performance in some tasks, CLM-trained models demonstrate distinct advantages:

• Data Efficiency: In the early stages of training (up to 10,000–20,000 steps), CLM models consistently outperform MLM models in downstream tasks. CLM saturates earlier, suggesting it is more efficient for learning initial representations.
• Fine-Tuning Stability: CLM-pretrained models exhibit lower sensitivity to learning rate choices during fine-tuning, providing a more stable initialization that reduces the need for extensive hyperparameter tuning.

3. Optimal Strategy: Sequential CLM then MLM

Under a fixed computational budget, a biphasic training strategy yields the best results. Specifically, starting pretraining with CLM and transitioning to MLM (CLM+MLM) consistently outperforms pure MLM training.

• In the PFS setting, a split of 25% CLM and 75% MLM steps reliably surpasses the MLM-only baseline.
• This approach combines the data efficiency and stability of CLM with the representation quality of MLM.

4. Leveraging Existing Decoder Models

In the Continued PreTraining (CPT) setting, adapting a model pretrained with CLM using MLM is more effective than continuing MLM training from an MLM-pretrained model.

• Applying MLM CPT to a CLM-initialized model closes the performance gap on QA and IR tasks and significantly outperforms MLM-only models on Sequence Classification.
• This suggests that the most computationally efficient path to state-of-the-art encoders is to initialize from readily available, large-scale pretrained decoder models and continue training with MLM.

Significance and Claims

The paper challenges the long-standing assumption that MLM is the universally optimal approach for encoder pretraining. The authors claim that:

1. Bidirectional training remains essential for achieving robust performance across all tasks, but the path to that state can be optimized.
2. Decoder-only models are not just "accidental" encoders; when properly adapted via MLM, they leverage the data efficiency of causal pretraining to produce superior encoders.
3. Cost Reduction: The proposed strategy (CLM initialization + MLM adaptation) offers a way to train best-in-class encoder models with reduced computational burden, potentially lowering the environmental and economic costs of NLP training.

The authors emphasize that their findings are based on controlled experiments at the 100B token scale, providing robust empirical evidence that the choice of learning objective significantly impacts downstream representation quality, independent of scale confounders. They release all artifacts (checkpoints, code, data) to foster further research into these training dynamics.