Technical Summary: A Web Data Recipe for Medical Encoder Pretraining

Problem Statement

While web-scale data curation has become standard for pretraining decoder Large Language Models (LLMs), encoder models for dense-terminology domains like medicine remain constrained. Current medical encoders rely on small, manually curated corpora (e.g., biomedical literature, clinical narratives) assembled from limited canonical repositories. This approach restricts scalability, source heterogeneity, and register diversity, creating a bottleneck particularly severe in non-English clinical settings.

A critical open question remains: does web-scale data curation benefit encoder Masked Language Modeling (MLM) in dense-terminology domains? Standard curation strategies, such as filtering for "educational quality," often prioritize coherent prose and lay explanations, which can dilute specialized vocabulary. For encoder MLM, where the learning signal depends on per-token entity density and per-entity context diversity, these standard signals may be suboptimal.

Methodology

The authors propose a two-lever recipe to adapt web data curation for medical encoder pretraining, instantiated on French medical NLP. The pipeline consists of three stages:

1. Prefiltering and Multi-Axis Annotation

The authors start with three heterogeneous web corpora: FineWeb-2, FinePDFs, and FineWiki.

• Prefiltering: A multilingual domain classifier isolates documents with actual medical content, reducing the raw data volume by over 90%.
• Multi-Axis Annotation: Retained documents are scored along three axes using distilled student models (trained on LLM teachers):
  • Subdomain: A 15-class classifier distinguishing between biomedical, clinical, and consumer-facing content.
  • Educational Quality: A 0–5 regression scorer adapted from general web education rubrics.
  • Medical-Term Density (Novel Lever): A character-ratio metric measuring the concentration of extracted medical entities relative to total document length. This lever targets the specific needs of encoder MLM by ensuring masked tokens frequently fall on domain-specific terms.

2. Signal-Amplifying Rephrasing

To address the limitations of filtering alone (which discards borderline documents and cannot enrich content), the authors introduce Signal-Amplifying Rephrasing, a medical adaptation of Massive Genre–Audience (MGA) reformulation.

• Two-Stage Process:
  • Stage 1 (Gating & Proposal): An LLM assesses if a document has sufficient medical content. If so, it proposes diverse (genre, audience) pairs (e.g., rewriting a drug entry as a pharmacist's reference).
  • Stage 2 (Faithful Densification): The LLM rewrites the document to increase medical-term density and broaden entity contexts. Crucially, the process enforces strict faithfulness: no medical facts, values, or entities are invented. Non-medical Personally Identifiable Information (PII) is replaced with varied fictional values to support downstream de-identification robustness.
• Surface Variation: The rephrasing also varies register (formal vs. telegraphic) and abbreviation density to capture diverse clinical writing styles.

3. Corpus Construction and Training

The recipe produces FineMed (the annotated raw corpus) and FineMed-rephrased. The final training mixture combines filtered raw data (selected via high medical-term density and educational quality thresholds) with rephrased data. This mixture is used to pretrain DoctoBERT, a family of French medical encoders based on RoBERTa and ModernBERT architectures.

Key Contributions

1. Medical-Term Density Filtering: The introduction of a per-document filter based on medical-term richness, which the authors demonstrate is a stronger signal for encoder MLM than educational quality.
2. Signal-Amplifying Rephrasing: A medical adaptation of MGA that increases entity density and context diversity while maintaining strict entity faithfulness, avoiding the hallucination risks common in unconstrained LLM rewriting.
3. FineMed and FineMed-rephrased: The release of a large-scale French medical pretraining corpus (21M documents, 19.2B words) annotated along three axes, alongside a reproducible curation pipeline.
4. DoctoBERT: A state-of-the-art family of French medical encoders pretrained on the proposed recipe.

Results

The authors conducted extensive ablation studies on the DrBenchmark (a French biomedical NLP benchmark) and a proprietary clinical Named Entity Recognition (NER) task.

• Filtering Efficacy: Medical-term density outperformed educational quality as a single-axis filter. The intersection of educational quality and medical-term density yielded the best results, outperforming curated medical corpora (NACHOS, TransCorpus-bio-fr) by significant margins (e.g., +29 Win Probability points over TransCorpus-bio-fr).
• Rephrasing Impact: Signal-amplifying rephrasing alone improved performance over raw web data. Standard MGA (without medical adaptations) performed worse than raw data, highlighting the necessity of the medical-specific constraints.
• Synergy: Mixing rephrased data with filtered raw data produced the largest gains. The best configuration combined Qwen-rephrased data with filtered raw data, achieving a +24 Win Probability point gain over raw data alone.
• Performance: DoctoBERT-fr achieved state-of-the-art performance on DrBenchmark, leading in aggregate metrics and five of seven per-task scores. On the proprietary clinical NER task, DoctoModernBERT-fr achieved the highest precision and F1 scores, demonstrating that broader register coverage translates to better real-world performance.

Significance and Claims

The paper claims to demonstrate that decoder LLM web-data curation methodologies can be successfully adapted for encoder MLM pretraining in dense-terminology domains. By focusing on per-token entity density and per-entity context diversity, the authors show that heterogeneous web data, when filtered and rephrased with specific medical constraints, can outperform labor-intensive, manually curated corpora.

The work challenges the dominance of "educational quality" as a universal curation signal, arguing that for encoders in specialized domains, signals targeting domain-token concentration are more critical. The resulting DoctoBERT models advance the state of the art for French medical NLP, providing a scalable, reproducible alternative to manual corpus assembly without relying on English translation or manual curation.

The authors remain modest regarding limitations, noting that the pipeline is currently instantiated for French medical NLP and that the distilled annotators are language-specific. They also acknowledge that while the recipe is domain-agnostic in design, the specific taxonomies and prompts are medical-by-construction, and cross-domain transfer has not been empirically tested. Furthermore, long-context evaluation remains an open gap due to the short-context nature of current downstream benchmarks.