A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records

Technical Summary: A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records

Problem Statement
The development of artificial intelligence (AI) in healthcare is increasingly shifting toward foundation models—large-scale models trained on massive unlabeled data via self-supervised learning. While structured Electronic Health Record (EHR) foundation models have demonstrated benefits such as improved performance with fewer training labels and robustness to distribution shifts, significant questions remain regarding the feasibility of sharing these models across different hospital systems. Specifically, it is unclear how well an off-the-shelf foundation model, trained on data from one institution, can be adapted to local tasks at another institution, and whether such adaptation is more efficient than training models from scratch. Prior work has largely focused on private, site-specific foundation models or encoder-only architectures that perform poorly in few-shot settings. There is a lack of exploration into strategies for improving existing EHR foundation models via continued pretraining and an evaluation of how such training impacts label efficiency in downstream tasks.

Methodology
The authors conducted a multi-center retrospective study utilizing EHR data from two distinct institutions: The Hospital for Sick Children (SickKids) in Toronto, Canada (1.8 million patients), and the Medical Information Mart for Intensive Care (MIMIC-IV) dataset from Beth Israel Deaconess Medical Center in Boston, USA (340,000 patients). Both datasets were mapped to the Observational Medical Outcomes Partnership Common Data Model (OMOP-CDM) to ensure schema compatibility.

The study evaluated the adaptability of CLMBR-T-base (referred to as FMSM), a publicly available, 141-million-parameter, decoder-only Transformer foundation model pretrained on 2.57 million patients from Stanford Medicine. The experimental design compared four modeling approaches:

1. Baseline Gradient Boosting Machines (GBMs): Locally trained using count-based featurization.
2. External Foundation Model (FMSM): Used out-of-the-box as a frozen feature encoder, with linear task heads (logistic regression) trained on task-specific data.
3. External Foundation Model with Continued Pretraining (FMSM+): FMSM was further pretrained on local data (SickKids or MIMIC) using the same next-code prediction task, followed by linear task head training.
4. Local Foundation Models (FMSK/FMMIMIC): Foundation models with the same architecture as FMSM, pretrained from scratch on local data.

The models were evaluated on eight binary clinical prediction tasks (including in-hospital mortality, long length of stay, 30-day readmission, and specific abnormal lab results) using discrimination (AUROC) and calibration (Expected Calibration Error, ECE). The study also assessed label efficiency by varying the number of training examples from 2 to 1,024 and evaluated sample efficiency by varying the size of the pretraining subsamples from 0.1% to 80%.

Key Contributions

1. First Multi-Site Evaluation of Continued Pretraining: The study provides the first external evaluation of continued pretraining for a structured EHR foundation model across three different health systems (Stanford, SickKids, and MIMIC-IV).
2. Label Efficiency Analysis: The authors evaluated the impact of continued pretraining on few-shot performance, demonstrating that continued pretraining significantly improves label efficiency compared to baseline methods and prior encoder-only architectures.
3. Sample Efficiency Comparison: The study quantified the data and compute efficiency of continued pretraining versus training local foundation models from scratch, providing insights into when it is beneficial to adapt an existing model versus pretraining anew.

Results

• Overall Performance: Adapting the off-the-shelf FMSM matched the discrimination performance (AUROC) of locally trained GBMs while providing significantly better calibration (lower ECE). Continued pretraining (FMSM+) further improved both discrimination and calibration compared to GBMs.
• Label Efficiency: In few-shot settings, FMSM and FMSM+ consistently outperformed GBMs. Notably, FMSM+ required fewer than 1% of the total training examples (specifically, 128 samples) to match the performance of GBMs trained on all available data. This represents a substantial improvement over prior encoder-only models, which required hundreds to thousands of examples to outperform baselines.
• Adaptability and Robustness: In ablation experiments, FMSM and FMSM+ achieved significantly higher AUROC than local foundation models (FMSK/FMMIMIC) when tested on external datasets, suggesting that pretraining on a larger, more heterogeneous population (Stanford) improves adaptability across diverse healthcare settings (pediatric vs. adult ICU).
• Sample Efficiency: Continued pretraining was 60% to 90% more sample-efficient than training local foundation models from scratch. For instance, FMSM+ achieved performance comparable to local foundation models trained on all data using only 10% of the pretraining data for SickKids and 40% for MIMIC. Additionally, continued pretraining was significantly faster (70.2% faster for SickKids, 58.4% faster for MIMIC) and processed fewer coded events due to reduced code coverage requirements.

Significance and Conclusion
The authors conclude that adapting shared EHR foundation models across hospitals offers a cost-effective route to improved prediction performance. By leveraging a base foundation model as a modular component, institutions can streamline the development of healthcare AI, reducing the need for extensive labeled data and massive compute resources required for training from scratch. The findings underscore the utility of sharing pretrained model weights, which facilitates research into mitigating biases and improving robustness, while highlighting that continued pretraining is a highly efficient strategy for local adaptation. The study advocates for a paradigm shift from training bespoke models for each site to developing and sharing general-purpose base foundation models with site-specific refinement recipes.