💻 computer science

MedLLM: An Open Medical Language Model at the Sub-Billion Scale

This paper introduces MedLLM, an open 0.1B-parameter medical language model trained via a fully open three-phase pipeline, which reveals that sub-billion-scale models exhibit a unique dissociation in medical competence—excelling in context-grounded tasks while struggling with knowledge recall due to capacity constraints rather than adaptation issues.

Original authors: Maxx Richard Rahman, Asim Ahmed, Mihan Mohagheghzadeh, Wolfgang Maass

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Maxx Richard Rahman, Asim Ahmed, Mihan Mohagheghzadeh, Wolfgang Maass

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Tiny Brain in a Giant Library

Imagine a world where computers are taught to read and write like humans. These are called Large Language Models (LLMs). Think of them as super-smart students who have read almost everything on the internet. Usually, to make these students really good at a specific job—like being a doctor—you have to make them huge. The bigger they are, the more facts they can remember. But there's a catch: these giant medical students are often locked away in secret labs, or they are so big and expensive that only massive companies can run them.

This leaves a big question: What if we tried to make a tiny medical student? One that is small enough to fit on a regular computer, maybe even a laptop? Scientists have been wondering if a model this small could actually learn medicine, or if it would just be too "dumb" to understand complex medical terms. They also wanted to know: if you shrink a medical brain down, does it forget everything, or does it just forget some things while staying good at others? This paper dives into that tiny, unexplored world to see what a small medical AI can actually do.

Meet MedLLM: The Pocket-Sized Medical Student

The researchers behind this study built MedLLM, a medical language model that is incredibly small—only 0.1 billion parameters. To put that in perspective, the other famous medical AI models they compared it to are 7 billion parameters or even bigger. That means MedLLM is about 70 times smaller than its competitors. It's like comparing a pocket calculator to a massive supercomputer.

But being small isn't just about size; it's about how you teach it. The team didn't just dump random medical books into the model. Instead, they created a special training method called MedFineWeb. Imagine you have a giant library of the entire internet, but you need to find only the pages that look and sound like medical questions and answers. Instead of hiring a librarian to read every single page, they used a "reference guide." They took a few thousand real medical exam questions and told the computer, "Find internet text that sounds similar to these questions." The computer then scooped up the most relevant web pages to create a custom textbook just for MedLLM. This way, the tiny model learned the style and vocabulary of medicine without needing a massive, pre-curated library of medical journals.

The Big Surprise: It's Not About Size, It's About the Task

The most exciting part of the paper is what happened when they tested MedLLM. The researchers expected that because the model was so small, it would be bad at everything. Instead, they found something weird and wonderful: MedLLM didn't fail evenly.

Think of medical knowledge like two different types of games:

The "Context" Game: You are given a short story (a patient's story) and asked to answer a question based only on that story.
The "Memory" Game: You are asked a question, and you have to pull the answer out of your own head without any help.

Here is where the tiny model surprised everyone:

On the "Context" Game: MedLLM was a star! When it was given a patient story to read, it could answer questions almost as well as the giant 7-billion-parameter models. In fact, it beat the instruction-tuned 7-billion models that are usually considered the gold standard. It showed that even a tiny brain can be very smart if it has the right information right in front of it.
On the "Memory" Game: This is where the tiny model hit a wall. When asked to recall facts from its own memory (like complex medical exam questions without a story to help), it struggled. It stayed near the bottom of the scoreboard, barely doing better than random guessing on the hardest USMLE-style questions.

The paper suggests that for a model this small, the limit isn't about how it was trained or how well it learned the language of medicine. The limit is simply capacity. It's like a tiny backpack: it can hold a lot of useful tools if you hand them to it (context), but it physically cannot carry a massive encyclopedia of facts (memory) inside it.

What the Tiny Model Learned (and Didn't Learn)

The researchers also looked at how the model improved during training. They found that as they fed MedLLM more medical text, it got much better at understanding the structure of medical language (its "perplexity" score dropped, meaning it sounded more natural). However, this didn't automatically make it better at answering hard memory-based questions.

This led to a fascinating discovery: Training the model to sound like a doctor didn't teach it to be a doctor. The model learned to speak the language perfectly, but it didn't magically gain the vast database of medical facts that the bigger models have. The paper argues that at this tiny scale, the bottleneck is the model's size, not the training method. If you want a tiny AI to be a medical expert, you probably shouldn't rely on it to memorize facts. Instead, you should give it a "cheat sheet" (the context) to read while it answers.

The Final Verdict

The paper concludes that MedLLM is a successful, open-source experiment that proves small medical models are possible, but they work differently than big ones. They are excellent at using information you give them, but they are not good at remembering things on their own.

The researchers also tested a technique called DPO (Direct Preference Optimization), which is like a teacher correcting a student's answers to make them more confident. They found that this technique made MedLLM much better at distinguishing between right and wrong answers, but it didn't give the model new facts. It just sharpened the model's existing skills.

In short, MedLLM shows us that if we want to build small, accessible medical AIs for everyday computers, we need to stop trying to make them memorize the whole library. Instead, we should design them to be brilliant readers who can instantly understand and use the information right in front of them. The paper doesn't claim this solves all medical AI problems, but it opens a new door: a tiny, open, and surprisingly capable medical student that we can all study and improve.

Technical Summary: MedLLM – An Open Medical Language Model at the Sub-Billion Scale

Problem Statement

The field of medical large language models (LLMs) has converged on a single scale: widely used systems operate at 7 billion parameters or more. This trend leaves the sub-billion regime (models with <1B parameters) uncharacterized. Current medical LLMs are either closed-source with undisclosed training details (e.g., GPT-4, Med-PaLM-2) or open but large (7B–540B), limiting access, reproducibility, and the ability to audit training processes.

A critical gap exists in understanding which medical capabilities survive when model capacity is reduced by an order of magnitude. Adapting general-purpose models to medicine is challenging due to distributional mismatch between web text and medical text, which requires handling long-tail terminology, structured reasoning, and high-precision requirements. Prior work addresses this via continued pretraining on biomedical corpora and instruction tuning, but all such efforts operate at 7B parameters or higher. It remains unknown whether these adaptation recipes transfer to smaller models or if reduced capacity forces specific medical capabilities to collapse before others.

Methodology

Model Architecture: MedLLM

MedLLM is a decoder-only transformer with 0.1 billion (100M) parameters. Key architectural choices include:

Tokenization & Embedding: Uses a SentencePiece vocabulary ( $V=32,000$ ) and embedding dimension $d=512$ . Given the high vocabulary-to-width ratio ( $V/d = 62.5$ ), the authors tie input embeddings to the output projection. This removes a separate $V \times d$ matrix (saving 14% of total parameters) and acts as a regularizer.
Transformer Layers: Stacks 20 layers using RMSNorm placed inside the residual branch (post-attention and post-FFN) rather than pre-norm. This keeps the residual stream at a controlled scale, improving stability for deep, narrow models.
Attention: Uses 8 heads with Rotary Position Embeddings (RoPE) and QK-normalization to bound attention logit magnitudes, preventing training instabilities common in small models.
Feed-Forward Network: Employs SwiGLU with an expansion factor of 4 ( $d_{ff} = 2048$ ). This allocates 62.7% of the model's parameters to the feed-forward sublayer, prioritizing depth and compositional transformation over width.

Data Pipeline: MedFineWeb

To adapt the model without relying on manually curated medical sources (which are often scarce or biased toward abstract-style writing), the authors introduce MedFineWeb, a medical plain-text corpus selected from general web data via reference-guided selection:

Candidate Pool: General web data (RedPajama-V2, C4, OpenWebText, Wikipedia) is split into ~1,000-word chunks.
Reference Sets: Three medical QA datasets (MedMCQA, MedQA, PubMedQA) serve as semantic references. Their prompts and responses are embedded.
Selection: Each candidate chunk is scored by its semantic similarity (cosine similarity via FAISS) to the reference sets. The top 50% of candidates for each reference set are selected, merged, and deduplicated.
Result: A 3B-token corpus that captures medical style and topics without directly copying benchmark questions or answers.

Training Pipeline

The training follows a fully open, three-phase pipeline:

General Pretraining: The model is pretrained on ~29.5B tokens of general web data ( $D_{gen}$ ) using causal language modeling with a curriculum sequence-length schedule (128 $\to$ 256 $\to$ 512 tokens) and a z-loss regularizer to stabilize output logits.
Domain Fine-Tuning (DFT): Continued pretraining on MedFineWeb for 25,000 steps to shift the distribution from general web text to medical text.
Preference Alignment: A two-step process per benchmark:
- Supervised Fine-Tuning (SFT): Fine-tuning on benchmark training sets using label-prediction objectives.
- Direct Preference Optimization (DPO): Optimizing the model to prefer correct answers over incorrect ones without a separate reward model. The KL-penalty strength is scaled by $\beta=0.1$ to amplify the preference margin.

Key Contributions

MedLLM: The first open 0.1B-parameter medical language model, providing a reproducible baseline an order of magnitude smaller than existing open medical LLMs.
MedFineWeb: A novel medical pretraining corpus constructed from general web data using embedding similarity to medical QA. It requires no curated medical sources and can be adapted to any domain given a small set of target-domain QA examples.
Task-Structured Dissociation: The identification that under compression, medical competence does not degrade uniformly. Instead, it splits by task type: the model achieves 7B-competitive accuracy on context-grounded QA but remains capacity-bound on knowledge-recall QA.

Results

Pretrained Evaluation

Even before task-specific fine-tuning, the MedLLM model (evaluated zero-shot via top-token selection) achieved the highest average accuracy (35.1%) among all baselines, including 7B instruction-tuned models (Zephyr-7B-β: 32.4%) and medically adapted 7B models (PMC-LLaMA-7B: 30.6%). It outperformed larger models on recall-oriented benchmarks (MMLU and MedMCQA) despite being 70x smaller.

Fine-Tuned Evaluation

After SFT and DPO, the results revealed a distinct split based on task type:

Knowledge-Recall Benchmarks (MedMCQA, MMLU): MedLLM (DPO) significantly exceeded all 7B and sub-7B baselines. On MedMCQA, it reached 34.9% (vs. 33.0% for Zephyr-7B-β and 26.6% for PMC-LLaMA-7B). On MMLU, it reached 32.4%.
Context-Grounded QA (PubMedQA): MedLLM reached 58.2%, surpassing instruction-tuned 7B baselines (Zephyr-7B-β: 43.0%) but remaining below continued-pretraining baselines (PMC-LLaMA-7B: 72.9%).
Complex Clinical Vignettes (MedQA): The model struggled, reaching only 28.1% (near the 25% four-option floor), significantly behind instruction-tuned 7B models.

Analysis of Training Dynamics

Perplexity vs. Accuracy: During domain fine-tuning, validation perplexity on medical text dropped by half (27.3 $\to$ 13.6), yet benchmark accuracy remained flat. This indicates that at 0.1B scale, the model learns medical form (language distribution) quickly but fails to acquire the retrievable facts required for recall benchmarks.
DPO Effect: DPO increased accuracy only modestly (e.g., +1.0 pp on MedMCQA) but amplified the log-probability margin between correct and incorrect answers by 5.2x. This suggests DPO at this scale refines existing discrimination rather than injecting new knowledge.

Significance and Claims

The paper claims that the sub-billion regime is not merely a degraded version of larger models but exhibits a unique behavior where medical capability is best sourced from retrieval and supplied context rather than parametric recall.

Capacity vs. Adaptation: The results demonstrate that where recall succeeds (MedMCQA), the binding constraint is model capacity, not the adaptation method. Where recall fails (MedQA), the capacity limit is absolute.
Task-Specific Dissociation: The "split" in performance is masked at 7B (where both capabilities exist) and surfaces only when capacity is scarce.
Reproducibility: By releasing MedLLM and MedFineWeb, the authors provide a pathway to train, audit, and deploy medical models on modest hardware, challenging the necessity of massive parameter counts for specific medical tasks.

The authors conclude that at sub-billion scales, the model's knowledge breadth is inherently constrained, and performance relies heavily on the quality of the provided context rather than internalized parametric knowledge.