Disease Risk Prediction Using Structured EHR Data: Can Generalist Large Language Models Match Specialized Clinical Foundation Models? A Comparative Evaluation with Fine-Tuning

This comparative evaluation demonstrates that while fine-tuned generalist large language models generally underperform specialized clinical foundation models on structured EHR disease risk prediction, LLM-generated embeddings paired with lightweight classifiers can achieve superior performance across both AUROC and AUPRC metrics.

The Gist

Imagine you are trying to predict who might get sick in the future by looking at their medical history. For years, doctors and data scientists have used specialized "experts" to do this. Think of these experts as Clinical Foundation Models (CFMs). They are like master chefs who have spent their entire lives cooking only with structured ingredients (like lab codes, diagnosis numbers, and medication lists). They know exactly how to mix these specific ingredients to predict outcomes like heart failure or pancreatic cancer.

Recently, a new type of AI has arrived: Large Language Models (LLMs). These are like generalist geniuses. They have read almost everything on the internet—books, news, code, and conversations. They are incredibly smart at understanding language and context, but they haven't spent their whole lives studying medical charts specifically.

The big question this paper asks is: Can these generalist geniuses beat the specialized master chefs at predicting disease risk using structured medical data?

Here is what the researchers found, broken down simply:

1. The "Fine-Tuning" Race: Specialized vs. Generalist

The researchers took both types of models and gave them a specific task: predict heart failure in diabetic patients and pancreatic cancer in others. They "fine-tuned" them, which is like giving the models a crash course in the specific rules of the game.

• The Result: On large datasets (thousands of patients), the specialized chefs (CFMs) still won, but only by a tiny, almost invisible margin.
  • Analogy: Imagine a race between a Formula 1 car (CFM) and a very fast sports car (LLM). The F1 car finished first, but only by a fraction of a second.
  • The Catch: The F1 car (CFM) was much cheaper and faster to train. The sports car (LLM) took a lot more fuel (computing power) and time to get ready, yet it barely lost.

2. The "Embedding" Trick: The Best Surprise

The researchers tried a third approach. Instead of making the LLMs learn the rules of the game (fine-tuning), they just asked the LLMs to read the patient's history and write a summary (creating an "embedding"). Then, they handed that summary to a very simple, basic calculator (a "lightweight classifier") to make the final prediction.

• The Result: This combination won the race by a landslide.
  • Analogy: Instead of training the genius to be a doctor, they asked the genius to write a perfect, concise biography of the patient. Then, they gave that biography to a smart intern with a simple checklist. The intern, armed with the genius's perfect summary, made better predictions than the specialized chefs or the fine-tuned geniuses.
  • Specifics: Using a model called Qwen3 to write the summary and a simple calculator to read it, they achieved the highest accuracy scores (over 90% in some cases).

3. The "Small" Specialist

They also tested a "Clinical LLM" (Me-LLaMA), which is a generalist genius that has read some medical books.

• The Result: This model performed just as well as the massive generalist models, even though it was much smaller. It proved you don't always need the biggest brain to get the job done if you have the right medical training.

4. The Trade-Off

The paper highlights a major trade-off:

• Specialized Models (CFMs): Fast to train, cheap to run, and very reliable. They are the "workhorses" of the clinic.
• Generalist Models (LLMs): They can match or even beat the specialists, but they are expensive and slow to train. However, if you use them just to "summarize" the data (the embedding trick) rather than training them fully, they become incredibly powerful and efficient.

The Bottom Line

The paper concludes that generalist AI models can definitely match specialized medical models for predicting disease risk. In fact, using a generalist model just to "summarize" the data for a simple calculator was the most successful method of all.

However, the authors warn that because generalist models are so expensive to train and their performance can be a bit "wobbly" (sometimes great, sometimes not), we shouldn't just throw away the specialized models yet. The best future might be a team-up: using the generalist's ability to understand and summarize, combined with the specialized model's efficiency.

In short: The generalist AI is a brilliant student who can ace the medical exam, but the specialized AI is the seasoned doctor who gets there faster and cheaper. The smartest move? Let the student write the notes, and let a simple tool grade them.

Technical Summary

1. Problem Statement

The healthcare industry increasingly relies on predictive models using structured Electronic Health Records (EHR) for clinical decision support. Two distinct classes of AI models have emerged:

• Clinical Foundation Models (CFMs): Specialized models (e.g., Med-BERT, CLMBR) pre-trained on massive, heterogeneous structured EHR datasets. They are optimized for patient trajectory modeling and have shown superior performance in disease prediction.
• Generalist Large Language Models (LLMs): Models (e.g., LLaMA, Mistral) pre-trained on broad, diverse text data. While adaptable, it remains unclear if they can match or surpass specialized CFMs when applied to structured EHR data for disease risk prediction, particularly given the computational costs and data representation challenges.

The study addresses the gap in direct, head-to-head comparisons between these two model classes on real-world, multi-site EHR and claims data.

2. Methodology

Datasets and Tasks

The study evaluated performance on two binary classification tasks:

1. Diabetic Heart Failure (DHF): Predicting heart failure risk in diabetic patients.
2. Pancreatic Cancer (PaCa): Predicting pancreatic cancer diagnosis.

Four datasets were utilized to ensure robustness and reproducibility:

• DHF-EHR & PaCa-EHR: Multi-site structured EHR data from US health systems.
• PaCa-Claims: Patient-level claims data.
• PaCa-EHRSHOT: An open-source, single-institution benchmark dataset (3,810 patients).

Data was processed at three levels of granularity:

• D: Diagnosis codes only.
• DMP: Diagnosis, Medication, and Procedure codes.
• ALL: All available structured data (including demographics, vitals, labs).

Models Evaluated

The study compared three categories of models:

1. Clinical Foundation Models (CFMs):
   • Med-BERT (v2): Encoder-based, pre-trained on >50 million patient trajectories (diagnosis, meds, procedures).
   • CLMBR: Decoder-transformer based, pre-trained on 2.57 million patients.
2. Generalist & Clinical LLMs (Fine-Tuned):
   • Generalist: Mistral-7B, LLaMA-2-13B, LLaMA-3-8B, LLaMA-3.1-8B, LLaMA-3.1-70B.
   • Clinical LLM: Me-LLaMA (fine-tuned on clinical text).
   • Technique: All LLMs were fine-tuned using Parameter-Efficient Fine-Tuning (PEFT/LoRA) and quantization.
3. LLM Embedding Approach (No Fine-Tuning):
   • Using LLMs (DeepSeek, Qwen3, GPT-OSS, MedGemma, etc.) purely as encoders to generate patient trajectory embeddings, followed by lightweight classifiers (Logistic Regression, MLP).

Data Preprocessing & Input

• CFMs: Structured codes were reformatted according to their specific pre-training pipelines.
• LLMs: Structured EHR data was serialized into text prompts. Two formats were tested: CPLLM and LLaMA2-EHR. The latter (incorporating event frequencies) yielded better results.
• Embeddings: For the non-fine-tuned approach, the first 4,096 tokens of patient history were used to generate embeddings.

Evaluation Metrics

• AUROC: Area Under the Receiver Operating Characteristic Curve (primary metric).
• AUPRC: Area Under the Precision-Recall Curve (critical for imbalanced datasets).
• Efficiency: Fine-tuning time and computational resources.

3. Key Contributions

1. First Large-Scale Comparison: Provided the first head-to-head benchmark of open-weight LLMs against CFMs pre-trained at massive scale (tens of millions of patients) for disease risk prediction.
2. Serialization Strategy Analysis: Evaluated the impact of converting structured EHR data into text, finding that the LLaMA2-EHR format (with diagnostic frequency) outperformed other formats.
3. Novel Embedding Paradigm: Demonstrated that using LLMs as frozen encoders to generate patient representations, paired with simple classifiers, outperforms both fine-tuned LLMs and fine-tuned CFMs in specific scenarios.
4. Reproducibility Framework: Released a comprehensive codebase (LLM2Predict) and evaluation framework to support future benchmarking.

4. Key Results

Fine-Tuning Performance (Large Cohorts >30k patients)

• CFMs vs. Fine-Tuned LLMs: On large EHR and claims cohorts, fine-tuned CFMs (Med-BERT) slightly but statistically significantly outperformed fine-tuned LLMs by <1% in AUROC.
  • Example (DHF-EHR): Med-BERT + Bi-GRU (85.39%) vs. LLaMA-3.1-70B (84.73%).
• Clinical LLMs: Me-LLaMA performed comparably to much larger generalist LLMs (e.g., LLaMA-3.1-70B), suggesting domain-specific pre-training is effective even with smaller parameter counts.
• Efficiency: CFMs were significantly faster to fine-tune (<1 hour) compared to LLMs (4 to 190 hours depending on model size).

Open-Source Cohort (PaCa-EHRSHOT)

• AUROC: Fine-tuned LLMs (LLaMA-3.1-70B) achieved slightly higher AUROCs than CFMs, but the difference was not statistically significant (86.1% vs. 85.3%, p=0.27).
• AUPRC: CFMs significantly outperformed LLMs in AUPRC, indicating better handling of class imbalance.
  • Med-BERT v2: 55.9% AUPRC.
  • LLaMA-3.1-70B: 41.1% AUPRC.

The "Embedding + Classifier" Approach (Best Overall)

• Surprising Finding: The best overall performance was achieved without fine-tuning the LLMs.
• Method: Using LLMs (specifically Qwen3 and GPT-OSS) to generate patient trajectory embeddings, followed by a simple Logistic Regression or MLP.
• Performance:
  • AUROC: Exceeded 90% (Qwen3 8B Embedding + Text: 90.8%).
  • AUPRC: Reached 66% (GPT-OSS 20B: 66.1%).
• This approach surpassed both fine-tuned CFMs and fine-tuned LLMs on the EHRSHOT dataset.

5. Significance and Conclusion

• Generalist vs. Specialized: While specialized CFMs remain highly competitive and efficient for fine-tuning on large structured datasets, generalist LLMs have the potential to match or exceed them, particularly when used as feature extractors (encoders) rather than generative models.
• Computational Trade-off: Fine-tuning LLMs is computationally expensive (40x longer than CFMs) and yields marginal gains over CFMs in AUROC, often at the cost of lower AUPRC.
• Optimal Strategy: The study suggests that for structured EHR risk prediction, the most effective and efficient strategy may be using modern LLMs to generate high-quality patient embeddings, which are then fed into lightweight classifiers. This avoids the massive cost of fine-tuning while leveraging the LLM's pre-trained knowledge.
• Future Directions: The authors recommend integrating LLM-generated embeddings into CFM training pipelines to combine the strengths of both approaches. They also highlight the need for further research on multi-class tasks, unstructured data (clinical notes), and hybrid architectures.

Limitations: The study focused on binary classification; results may not generalize to regression or multi-class tasks. Additionally, the evaluation was restricted to structured data, leaving the performance on unstructured clinical notes unexplored.