💬 NLP

BioMamba: Domain-Adaptive Biomedical Language Models

The paper introduces BioMamba, a family of domain-adaptive Mamba2 models developed through balanced continued pretraining on biomedical and general corpora, which successfully enhances performance on biomedical tasks while preserving general language fluency and matching or exceeding baseline performance across various benchmarks.

Original authors: Ling Yue, Mingzhi Zhu, Sixue Xing, Yunning Cao, Yanbo Wang, Shimin Shan, Jinfei Liu, Vijil Chenthamarakshan, Shaowu Pan, Payel Das, Tianfan Fu

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ling Yue, Mingzhi Zhu, Sixue Xing, Yunning Cao, Yanbo Wang, Shimin Shan, Jinfei Liu, Vijil Chenthamarakshan, Shaowu Pan, Payel Das, Tianfan Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-read librarian named Mamba. Mamba is great at reading everything from ancient history to modern pop culture, but if you ask him to explain a complex medical procedure or finish a doctor's hospital note, he sometimes sounds a bit vague or misses the specific jargon doctors use.

The paper introduces BioMamba, which is essentially a "specialized training camp" for this librarian. The goal was to teach Mamba to become a medical expert without making him forget how to speak normal English or understand general news.

Here is how they did it and what they found, explained simply:

The Problem: The "Specialist Trap"

Usually, when you train a general AI to become a medical expert, it's like forcing a general contractor to only study plumbing. They might get really good at pipes, but they might forget how to fix a door or paint a wall. In AI terms, this is called "catastrophic forgetting." The model gets better at medical text but gets worse at general text.

The Solution: The "Balanced Diet" Recipe

The researchers didn't just feed Mamba a diet of only medical books (PubMed). Instead, they created a specific 80/10/10 meal plan:

80% Medical Text (PubMed): To learn the specialized vocabulary and concepts.
10% General Web Text (C4): To keep his general knowledge sharp.
10% Wikipedia: To maintain fluency in how normal people write and explain things.

They tested this recipe on five different sizes of the librarian, ranging from a "junior intern" (130 million parameters) to a "senior chief" (2.7 billion parameters).

The Results: A Win-Win Situation

After this training, the results were surprisingly balanced:

Medical Expertise: The new BioMamba librarians became much better at understanding medical texts. Their "perplexity" (a score that measures how confused a model is; lower is better) on medical papers dropped significantly. The biggest model (2.7B) became very fluent in medical language.
General Fluency: Crucially, they didn't lose their general skills. When tested on general web text and Wikipedia, their performance stayed almost exactly the same. They didn't forget how to be a generalist.
Real-World Tasks: When they put these trained librarians to work on actual hospital tasks, they did better than the untrained versions:
- Finishing Hospital Notes: If you gave them the first half of a patient's discharge note, BioMamba could finish it with more accurate medical terms and better structure than the original.
- Summarizing Discharges: They could take a long list of admission details and write a clean, organized summary of what happened and what medications the patient needs.
- Answering Questions: They got better at answering "Yes/No" medical questions from datasets like BioASQ and PubMedQA.

The "Goldilocks" Finding

The researchers found that the mix of data was the secret sauce.

If they fed the model only medical text, it got great at medicine but started sounding weird and robotic on general topics (forgetting how to speak normally).
If they used the 80/10/10 mix, the model got the best of both worlds: it learned the medical jargon but kept its "general English" voice intact.

What They Are Not Claiming

It is important to note what the paper says this tool is not:

It is not a doctor. The authors explicitly state this model is for workflow support, like helping a human draft a note or screen a document, not for making autonomous decisions about patient care.
It is not perfect. The model still makes mistakes, sometimes repeating itself or missing small details, just like a human intern might.
It is not a replacement for human review. The paper emphasizes that these models need to be used with human oversight and retrieval tools to ensure safety.

The Takeaway

The paper proves that you can turn a general-purpose AI into a medical specialist by giving it a balanced diet of data. You don't have to sacrifice its ability to speak normal language to make it good at medicine. They have released these trained models (BioMamba) for free so other researchers can use them to build better tools for healthcare documentation and research.

Technical Summary: BioMamba: Domain-Adaptive Biomedical Language Models

Problem Statement

Biomedical language models must achieve high performance on specialized medical text while retaining fluency in general language. While Transformer-based models (e.g., BioBERT, PubMedBERT) have demonstrated the value of domain adaptation, they suffer from quadratic computational costs regarding sequence length, which is restrictive for long, terminology-dense biomedical inputs. State Space Models (SSMs), specifically Mamba, offer linear-time scaling but have seen limited systematic study in biomedical adaptation. A critical challenge in this domain is "catastrophic forgetting," where continued pretraining on specialized corpora improves domain performance at the expense of general-language fluency. This trade-off has not been rigorously evaluated for Mamba-based models across both biomedical literature and clinical text.

Methodology

The authors introduce BioMamba, a family of five biomedical Mamba2 models (130M, 370M, 780M, 1.3B, and 2.7B parameters). The architecture remains identical to the released Mamba2 checkpoints; the innovation lies in the adaptation strategy and the resulting open-weight checkpoints.

Training Pipeline

Continued Pretraining (CPT): The models underwent continued pretraining on a balanced mixture of three corpora:
- 80% PubMed: Abstracts indexed in MEDLINE.
- 10% C4: Colossal Clean Crawled Corpus (general web text).
- 10% Wikipedia: English Wikipedia.
  This 80/10/10 mixture was determined via ablation studies to optimize in-domain performance while preserving general fluency.
Optimization: Training utilized AdamW with weight decay (0.1), BF16 mixed precision, and a maximum sequence length of 1,024 tokens. To mitigate catastrophic forgetting, the authors employed a conservative Warmup-Stable-Decay (WSD) learning rate schedule combined with layer-wise learning-rate decay (0.90 for 130M, 0.95 for larger models), updating lower layers more conservatively than higher layers.
Supervised Fine-Tuning (SFT): Following CPT, models were fine-tuned on clinical and biomedical tasks. Notably, BioMamba+SFT utilized lower learning rates and more epochs than the baseline Mamba2+SFT to prevent overwriting the domain knowledge acquired during CPT.

Evaluation Framework

The study employed a tokenizer-controlled internal comparison (using a shared GPT-NeoX tokenizer) to isolate the effects of adaptation, alongside contextual comparisons with public checkpoints using native tokenizers.

Language Modeling: Evaluated on held-out sets of PubMed, C4, and Wikipedia (Wikitext-103).
Out-of-Domain (OOD): Six multiple-choice benchmarks (LAMBADA, HellaSwag, ARC-Easy/Challenge, PIQA, OpenBookQA) to test general reasoning and fluency preservation.
Downstream Tasks:
- Clinical Generation: Note completion and discharge summary generation on MIMIC-IV-Note v2.2.
- Biomedical QA: Yes/No question answering on BioASQ and PubMedQA.

Key Results

1. Language Modeling and Fluency Preservation

In-Domain Gains: BioMamba consistently lowered PubMed perplexity across all scales (e.g., reducing from 9.41 to 8.42 at 130M, and reaching 5.28 at 2.7B).
General Fluency: Wikipedia-style perplexity improved by 1.46–4.72 PPL, while C4 perplexity remained essentially unchanged ( $|\Delta PPL| \le 0.17$ ).
OOD Stability: On six OOD multiple-choice benchmarks, BioMamba stayed within $\pm 3$ percentage points of the base Mamba2 models with no systematic regression, indicating that general commonsense reasoning and reading comprehension were preserved.

2. Downstream Task Performance

Clinical Generation: After SFT, BioMamba+SFT matched or exceeded Mamba2+SFT on MIMIC-IV note completion and discharge summary generation at every scale. At the 2.7B scale, paired-bootstrap tests confirmed the advantage was statistically significant ( $P(\Delta > 0) \ge 0.997$ ). Qualitative analysis showed BioMamba+SFT produced more clinically plausible outputs, better structured discharge sections, and fewer repetition errors compared to the base model.
Biomedical QA: BioMamba+SFT improved PubMedQA accuracy at every scale. BioASQ results were non-monotonic (with gains at smaller scales and a tie at 2.7B), likely due to the small test set size (82 items) and the fact that larger models may already possess sufficient transferable knowledge. The strongest model, BioMamba-2.7B, achieved 90.24% accuracy on BioASQ and 73.00% on PubMedQA.

3. Ablation Insights

Data Mixture: A PubMed-only CPT strategy worsened Wikipedia and C4 perplexity (forgetting). The 80/10/10 mixture was the most balanced, achieving the lowest PubMed perplexity while maintaining general fluency.
Scale Effects: The benefits of continued pretraining were most pronounced in smaller models (130M) for downstream tasks, though the adaptation strategy remained effective across all scales.

Significance and Claims

The paper claims that BioMamba demonstrates that balanced domain-adaptive continued pretraining is a viable strategy for strengthening Mamba2 models on biomedical literature and clinical text without sacrificing general-language fluency.

Key contributions include:

Open Resources: The public release of five open-weight BioMamba checkpoints (130M–2.7B) and full training/evaluation code under the Apache-2.0 license.
Reproducible Recipe: A validated 80/10/10 PubMed/C4/Wikipedia mixture and a conservative fine-tuning schedule that mitigates catastrophic forgetting.
Empirical Evidence: Controlled experiments showing that Mamba-based models can be effectively adapted for biomedical workflows, offering a linear-time alternative to Transformer-based biomedical models.

The authors maintain a modest stance, noting that the models are intended as assistive components for workflow support (e.g., literature screening, note drafting) rather than autonomous clinical decision-making tools. They explicitly state that the study does not establish clinical safety, calibration, or readiness for prospective deployment, emphasizing the need for expert oversight and retrieval-augmented generation in real-world applications.