🤖 machine learning

Scaling Electronic Health Record Foundation Models for Population Health Management

This paper introduces a scalable Electronic Health Record Foundation Model pre-trained on billions of cross-site medical events from over 5 million patients in the US and Taiwan, which demonstrates superior performance and generalization in predicting 11 chronic diseases compared to existing models by leveraging a unified code alignment framework and compute-optimal scaling.

Original authors: Liwen Sun, Hao-Ren Yao, Ophir Frieder, Xiang Qian, Chenyan Xiong

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Liwen Sun, Hao-Ren Yao, Ophir Frieder, Xiang Qian, Chenyan Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Healthcare systems today often operate like a collection of isolated islands. A patient might see a specialist in one city, get a test done in another, and fill a prescription in a third, with each location keeping its own separate records. This fragmentation makes it difficult to see the full picture of a person's health over time. When doctors cannot easily connect these dots, they may miss the early warning signs of serious, long-term illnesses like heart disease or cancer. By the time these conditions are detected, they are often advanced, requiring expensive and intensive treatment. The goal of population health management is to solve this by looking at vast amounts of patient data to find people who are at risk before they ever feel sick, allowing for early, life-saving interventions.

To achieve this, researchers have turned to electronic health records, the digital logs of every visit, test, and medication a patient receives. However, these records are not written in a single, universal language. One hospital might use a specific code for a common procedure, while a hospital in a different country uses a completely different code for the exact same thing. This mismatch has made it nearly impossible to train computer systems to learn from data across different regions. A new study introduces a solution called EHR-FM, a massive computer model designed to read and understand these medical records regardless of where they come from. By learning from billions of medical events involving over five million patients from both the United States and Taiwan, the system has learned to spot the subtle patterns that precede chronic diseases, even when the data comes from two very different healthcare worlds.

The core challenge the researchers faced was not just the sheer volume of data, but the confusion caused by different coding systems. Imagine trying to learn a new language where the word for "heart attack" is spelled differently in every book you open. In medical records, a single drug or procedure might have hundreds of different names depending on the local billing system. To fix this, the team built a translator that converts all these different codes into a single, unified language before the computer ever sees them. They used a combination of official medical dictionaries and advanced artificial intelligence to map thousands of local codes to standard international ones. This allowed them to combine records from Taiwan and the United States into one massive, coherent dataset containing nearly ten billion medical events.

With this unified data, the researchers trained a large foundation model, a type of artificial intelligence that learns general rules from vast amounts of information. They tested how the model's performance changed as they increased its size and the amount of data it saw. They found that the model followed a predictable pattern: as they added more computing power and more data, the model became significantly better at its job. They built versions of the model ranging from very small to over two billion parameters, a measure of its complexity and capacity to learn. The largest version, trained on this cross-border data, proved to be exceptionally skilled at predicting the onset of major diseases.

When tested on real-world scenarios, the model demonstrated a remarkable ability to identify patients who would develop conditions like cancer, heart failure, or stroke within the next year. In the United States, the model achieved an AUC of 0.4 for future heart disease cases while keeping false alarms extremely low. In the Taiwanese data, it achieved an AUC of 0.7 for future cases with the same high level of accuracy. This high sensitivity is crucial for population health; it means the system can flag high-risk individuals for further screening without overwhelming doctors with false warnings. The model outperformed traditional computer programs that rely on simple rules and also surpassed other advanced AI models that had been trained only on data from a single location.

Perhaps the most striking finding was how well the model handled data from places it had never seen before. When tested on a completely different set of patient records from Stanford University, which used different coding systems and covered a different population, the model still performed better than previous state-of-the-art systems. This suggests that by learning from a diverse mix of data, the model developed a deeper understanding of human disease that transcends local differences. The researchers also discovered that simply copying the same data over and over to make a larger dataset did not help the model learn as well as adding new, diverse data from different regions. This indicates that the variety of the information is just as important as the quantity.

The study confirms that it is possible to build powerful, scalable tools for predicting chronic disease by unifying medical records from across the globe. The researchers showed that by aligning different coding systems, they could create a model that learns from the collective history of millions of patients, identifying risks that might otherwise go unnoticed. While the model is not yet a replacement for a doctor's judgment, it offers a powerful new way to prioritize care. By flagging high-risk individuals early, healthcare systems can shift from reacting to illness to preventing it, potentially saving lives and reducing the burden on medical resources. The researchers have made their code available to others, hoping that this approach will help build a future where proactive, data-driven care is accessible to everyone, regardless of where they live.

Technical Summary: Scaling Electronic Health Record Foundation Models for Population Health Management

Problem Statement

Population health management aims to systematically identify and monitor individuals at elevated risk for chronic diseases (e.g., cardiovascular conditions, cancer) to enable proactive interventions. However, current approaches face significant barriers:

Data Fragmentation: Healthcare delivery is often disjointed, with limited coordination across providers and incomplete patient data integration.
Resource Intensity: Preventive strategies frequently rely on resource-intensive screening or episodic care models that are difficult to scale and often fail to reach high-risk or underserved populations.
Coding Heterogeneity: A major technical hurdle for scaling Electronic Health Record (EHR) models is the heterogeneity of coding systems (e.g., different ICD versions, local code sets, and clinical practices) across countries and institutions. This prevents unified representation and cross-site generalization.
Limitations of Existing Models: Prior EHR foundation models are typically limited to single-site settings (e.g., MIMIC-IV focusing on ICU) or fail to account for non-trivial heterogeneity in coding systems, limiting their effectiveness for large-scale, cross-system risk stratification.

Methodology

1. Data Curation and Unified Code Alignment

The authors constructed a unified dataset comprising over 5.7 million patients from two distinct healthcare systems:

Taiwan: Over 3 million patients from the National Health Insurance Research Database (NHIRD) (1996–2013).
United States: Over 2 million patients from multiple states.

To address coding heterogeneity, the authors developed a unified code alignment framework that standardizes heterogeneous medical codes into a shared space using ICD-10 (diagnoses), CPT (procedures), LOINC (labs), and ATC (drugs).

Alignment Process: Unmapped local codes were aggregated and mapped to standardized equivalents using a combination of official crosswalks, Large Language Model (LLM) curation (using Claude), and TF-IDF fuzzy matching.
Result: The final dataset contains over 9 billion standardized medical codes and 225,568 unique tokens, enabling the model to learn from a unified representation across diverse systems.

2. Model Architecture: EHR-FM

EHR-FM is a decoder-only Transformer model designed to operate directly on structured medical code sequences rather than converting them to natural language.

Input Representation: Patient records are represented as sequences of medical event codes (diagnoses, medications, procedures) interleaved with demographic tokens (age, gender) and time tokens (intervals between visits).
Architecture: The model utilizes Rotary Positional Embeddings (RoPE) to encode visit sequences, assigning the same absolute position to events within the same visit to capture intra-visit relationships.
Training Objective:
- Pretraining: Autoregressive next-token prediction on the unified medical code sequences.
- Fine-tuning: Supervised fine-tuning using cross-entropy loss for binary classification of chronic disease onset (predicting disease occurrence one year after a clinical visit based on up to five years of history).

3. Scaling Analysis

The authors conducted an IsoFLOP analysis to characterize the scaling behavior of EHR foundation models. They trained models ranging from 160 million to 2.4 billion parameters across varying compute budgets (FLOPs) and token counts to derive power-law relationships between compute, model size, and training tokens.

Key Contributions

First Scaled Cross-Site EHR Foundation Model: The paper presents the largest and first scaled EHR foundation model trained on longitudinal data spanning two countries (Taiwan and the US) and multiple institutions, addressing the gap in cross-system generalization.
Unified Code Alignment Framework: The authors demonstrate a robust method for aligning heterogeneous medical codes (NHI, ICD-9/10, CPT, etc.) into a unified space, enabling effective pretraining on multi-source data.
Scaling Laws for Healthcare: The study establishes scaling laws for EHR foundation models, showing that optimal model size and token counts follow power-law relationships similar to general-domain LLMs ( $N_{opt} \propto C^{0.34}$ , $D_{opt} \propto C^{0.69}$ ).
Empirical Validation of Cross-System Data: The authors show that aligned cross-system data provides a more effective pretraining signal than duplicating single-site data, particularly in data-limited settings.

Experimental Results

Performance on Chronic Disease Prediction

EHR-FM was evaluated on 11 chronic disease prediction tasks (6 cancers, 5 cardiovascular conditions) across US and Taiwan cohorts.

Superiority over Baselines: EHR-FM significantly outperformed tree-based models (XGBoost, LightGBM) and general/biomedical LLMs (BioMedLM, Qwen2.5) that were fine-tuned on text-converted EHR data.
- AUPRC Gains: Outperformed tree models by >10% AUPRC on US cohorts and >30% on Taiwan cohorts.
- Sensitivity at High Specificity: Achieved over 40% sensitivity at 99% specificity in US cohorts and 70% sensitivity at 99% specificity in Taiwan cohorts.
Scaling Benefits: Larger models (2.4B parameters) consistently outperformed smaller variants (160M), with the largest model showing a 5% sensitivity gain over the smallest on cancer prediction tasks.

Few-Shot Generalization (EHRShot Benchmark)

On the public EHRShot benchmark (Stanford Medicine data), EHR-FM demonstrated strong few-shot generalization:

Despite being trained on data from substantially different populations and coding systems, EHR-FM outperformed the previous state-of-the-art (CLMBR), which was trained on millions of in-site records.
EHR-FM maintained superior performance even with as few as 1–128 positive examples per task, highlighting its robustness to distribution shifts.

Ablation Studies

Code vs. Text: Pretraining directly on medical code IDs outperformed converting codes to natural language text. Text-based pretraining led to rapid loss convergence (memorization of code names) and poorer downstream performance.
Data Mixture: Under data-limited settings, incorporating aligned cross-system data (Taiwan + US) yielded better downstream performance than duplicating single-site data (US only), confirming the value of diverse pretraining signals.

Significance and Claims

The paper claims that EHR-FM provides a scalable, low-cost approach for population health management by enabling the identification of high-risk individuals using routinely collected EHR data.

Clinical Utility: The model's ability to achieve high sensitivity at high specificity suggests it can effectively triage patients for further screening, reducing false positives and unnecessary distress while ensuring high-risk individuals are identified early.
Scalability: By demonstrating that scaling laws hold for EHR data and that cross-system alignment is effective, the work suggests a pathway for building larger, more generalizable healthcare models that are not constrained by single-institution data silos.
Open Science: The authors commit to open-sourcing the code and preprocessing pipelines, though the proprietary healthcare data remains restricted due to privacy regulations.

The authors remain modest regarding limitations, noting that the study is constrained to models up to 2.4 billion parameters and that code alignment, while improved, remains an open challenge requiring future work on ontology-aware and multimodal alignment strategies.