Legal Domain Adaptation of Modern BERT Models

Technical Summary: Legal Domain Adaptation of Modern BERT Models

Problem Statement
While BERT-style encoders remain widely used in practice, recent architectural advancements (e.g., flash attention, RoPE embeddings) and massive scale pre-training (trillions of tokens) have led to models like ModernBERT. The central question addressed by this work is whether such modern, heavily pre-trained encoders still benefit from domain adaptation in specialized fields like law. Specifically, the authors investigate if further pre-training on US court opinions improves performance over a "vanilla" ModernBERT baseline, and whether pre-training a ModernBERT architecture from scratch with a domain-specific vocabulary outperforms continuing pre-training on an existing checkpoint.

Methodology
The authors conducted four primary pre-training experiments using the Collaborative Open Legal Data (COLD) corpus, comprising 8.3 million US court opinions (approximately 13 billion words).

1. Continued Pre-training: Existing ModernBERT checkpoints (base and large) were further pre-trained on the COLD corpus using the masked language modeling (MLM) objective.
2. Pre-training from Scratch: New models with the ModernBERT architecture were trained from scratch on the same data, utilizing a customized in-domain legal vocabulary derived from prior work (CaseLawBERT).
3. Data Handling: The authors processed opinions up to 8,192 tokens, discarding those shorter than 500 tokens and splitting longer ones. They experimented with both raw and cleaned versions of the data, ultimately selecting the original COLD data as it yielded slightly better performance on the LexGLUE/SCOTUS task.
4. Hyperparameters: Training followed the ModernBERT paper's specifications: 30% masked ratio, linear learning rate scheduler, starting learning rate of 3e-4, weight decay of 8e-5, and a constant batch size of 512.

Key Contributions

• Investigation of Modern Encoders: This is the first work to investigate the domain adaptation of ModernBERT (trained on 2 trillion tokens) specifically for the legal domain.
• Model Release: The authors publicly released all resulting model checkpoints (LegalModernBERT-base and -large), available on Hugging Face.
• Long-Context Capability: The models are capable of processing sequences up to 8,192 tokens, exceeding the average length of US court opinions and enabling the processing of full documents, a capability limited in prior BERT models (512 tokens).

Results
The models were evaluated on four tasks related to US court opinions: LexGLUE/SCOTUS (classification), LexGLUE/CaseHold (multiple-choice QA), LePaRD (passage retrieval), and BarExam QA (retrieval).

• Continued Pre-training vs. Vanilla: Further pre-training ModernBERT on legal data yielded statistically significant improvements over the vanilla ModernBERT baseline across all datasets. For example, on LexGLUE/SCOTUS, the LegalModernBERT-base achieved a Micro-F1 of 67.5 compared to 65.3 for the vanilla base.
• Continued Pre-training vs. From-Scratch: Contrary to findings in earlier BERT domain adaptation literature, pre-training the ModernBERT architecture from scratch with a specialized vocabulary did not match the performance of continuing pre-training on an existing ModernBERT checkpoint. The authors speculate that the 2 trillion token pre-training of the base ModernBERT offsets the gains typically achieved by from-scratch training, and that ModernBERT's vocabulary is more adaptive to new domains than older BERT/RoBERTa vocabularies.
• Sequence Length Impact: Fine-tuning with a sequence length of 8,192 tokens resulted in substantial performance gains compared to 512 tokens across all experiments, suggesting that previous limitations in legal NLP were partly due to insufficient context lengths.
• Scale: The LegalModernBERT-large models consistently outperformed their base counterparts across all benchmarks.
• Retrieval Tasks: While continued pre-training generally performed best, the authors noted that for retrieval tasks (LePaRD, BarExam QA), the "from scratch" base model sometimes outperformed the continued pre-training base model, suggesting retrieval may reward lexical specialization more than classification tasks for smaller models.

Significance and Claims
The paper claims that despite ModernBERT's massive general pre-training, domain adaptation remains critical for legal NLP tasks, offering improvements consistent with early domain adaptation studies (1–3 percentage point gains). The authors position their work as a step toward increasing access to justice by providing specialized, open-source models capable of embedding, retrieving, and reranking legal documents.

They emphasize the practical utility of these models for "closed universe" legal tech applications, particularly for public defenders, where data privacy and the avoidance of proprietary commercial models are concerns. The authors explicitly state that their models are intended for tasks such as document retrieval and evidence finding, rather than generating legal advice or summaries. They acknowledge limitations, including the focus solely on US court opinions, the lack of bias investigation, and the potential noise in the raw PDF-to-text conversion of the training data.