🤖 machine learning

LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models

The paper introduces LLM4EHR, a clinical foundation model that enhances outcome prediction in ICU settings by aligning Electronic Health Record time series with medical event sequences through a domain-adapted large language model and a regularized contrastive objective, thereby improving performance on downstream tasks and enabling effective k-shot adaptation to new patient cohorts.

Original authors: Jingteng Li, Alexander Capstick, Louise Rigny, Iona Biggart, Neil J Sebire, Payam Barnaghi

Published 2026-07-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingteng Li, Alexander Capstick, Louise Rigny, Iona Biggart, Neil J Sebire, Payam Barnaghi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a hospital as a giant, bustling library where every patient's story is written in two very different languages. One language is a steady, rhythmic drumbeat: the time series. This is the continuous stream of data like heart rates, blood pressure, and oxygen levels, ticking away second by second, creating a smooth, flowing melody of a patient's condition. The other language is a series of scattered, irregular notes: the medical events. These are the specific, one-off actions like a doctor prescribing a pill, a nurse inserting a tube, or a machine being turned on. For decades, computer scientists trying to predict how a patient will do have struggled to read these two languages together. They often treated the steady drumbeat and the scattered notes as separate stories, missing the crucial clues that happen when the two intersect. This is the heart of the problem in "clinical machine learning": how do you teach a computer to understand that a specific medication (an event) given at a specific moment changes the rhythm of the heart (the time series) in a way that predicts the future?

Enter LLM4EHR, a new approach that acts like a master translator and conductor for these two languages. The researchers built a system that uses a "Large Language Model" (LLM)—a type of AI famous for understanding human text—to read the scattered medical notes and turn them into a rich, semantic summary. Think of this LLM as a wise librarian who can instantly summarize a complex medical order into a clear sentence. Then, the system uses a special "contrastive" technique to force the computer to listen to the steady drumbeat of the patient's vital signs and match it perfectly with the librarian's summary of the events happening at that exact same time. It's like training a musician to play a melody that perfectly syncs with a spoken poem, ensuring that every note matches the emotion of the words. By doing this, the AI learns a much deeper, more connected understanding of what is happening to the patient, rather than just memorizing patterns in isolation.

The paper, titled LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models, proposes that by forcing these two data streams to "dance" together during the training phase, the AI learns a much better representation of a patient's health. The authors tested this on data from Intensive Care Units (ICUs), specifically using the massive MIMIC-IV dataset and the Physionet2012 challenge. They found that their new method, which aligns the time series with the event summaries using a "semantic regularised contrastive objective," consistently outperformed existing methods. In plain terms, the AI became better at predicting critical outcomes like whether a patient would pass away, if they would develop a specific condition (phenotyping), if their health would suddenly crash (decompensation), or how long they would stay in the hospital.

One of the most exciting findings is that this "musical conductor" approach is incredibly flexible. The paper demonstrates that the AI learns transferable skills; it can be taught on one set of patients and then quickly adapted to a completely new group of patients with very few examples (a technique called "k-shot adaptation"). This is like teaching a musician a new song with just a few notes and having them play the whole piece perfectly. The researchers also ran "ablation studies," which are like taking apart a machine to see which gears are essential. They found that the specific way they aligned the data—using the LLM's understanding of medical language to guide the time series—was the secret sauce. Without this alignment, or without the specific "semantic" guidance from the language model, the performance dropped.

However, the paper is careful not to claim this is a magic wand that solves everything. The authors note that while the model is excellent at distinguishing between short and long hospital stays, predicting the exact number of days a patient will stay remains difficult, with the model's accuracy (measured by R²) staying relatively low across all models tested. They also point out that the system currently relies on data from the ICU, and it's not yet clear how well it would work for patients moving through different hospitals over years. Furthermore, the system requires significant computing power, especially because it uses large language models, which can be heavy on memory. The authors suggest that future work could look at making these models more efficient or applying them to longer-term patient journeys beyond the ICU.

In essence, LLM4EHR suggests that the future of medical AI lies not in treating data as isolated facts, but in weaving together the continuous story of a patient's vitals with the specific, meaningful events that shape their care. By teaching computers to understand the relationship between the "what" (the event) and the "when" (the time series) simultaneously, the researchers have created a tool that is more robust, more accurate, and more adaptable to the chaotic, complex reality of human health.

Technical Summary: LLM4EHR

Problem Statement

Electronic Health Records (EHRs) are inherently multimodal, comprising two distinct temporal data types: EHR events (irregularly sampled, discrete clinical interactions like prescriptions or procedures) and EHR Time Series (TS) (routinely collected, continuous physiological measurements like vital signs). While recent clinical foundation models have shifted toward using Large Language Models (LLMs) to process EHR event sequences, existing methods often fail to sufficiently explore the shared temporal structures between these events and TS observations.

Current multimodal approaches typically employ "stationary" alignment, where the embedding of an entire episode (or a coarse-grained note) is aligned with the embedding of the corresponding TS. This approach ignores the fine-grained, time-step-level interdependencies between specific events and physiological changes. Furthermore, standard contrastive learning objectives (e.g., InfoNCE) suffer from "class collision," where semantically similar instances are incorrectly treated as negatives, pushing them apart in the embedding space. The paper argues that these limitations result in less robust and adaptive clinical foundation models, reducing performance on downstream tasks such as mortality prediction and phenotyping.

Methodology: LLM4EHR

The authors propose LLM4EHR, a multimodal fusion framework designed to temporally align LLM-encoded EHR event sequences with a dedicated EHR Time Series encoder.

1. Data Representation and Aggregation

EHR Events: Discrete events (e.g., medication orders, line insertions) are aggregated into fixed time windows (5-hour bins in MIMIC-IV) and converted into natural language summaries using a fixed schema. These summaries are embedded using a domain-adapted LLM (specifically BioClinical ModernBERT). The LLM is frozen during pre-training, with a learnable projection layer added for stability.
EHR Time Series: Multivariate physiological variables (e.g., lab values, vitals) are aggregated within the same time windows using statistical features (mean, min, max, std, count). These are processed by a trainable Transformer-based TS encoder consisting of 1D convolutional embedding layers and autoregressive attention blocks.

2. Semantic-Regularised Contrastive Objective

The core innovation is a pre-training objective that aligns the TS embeddings ( $v_t$ ) with the LLM-generated event embeddings ( $z_t$ ) at each time step $t$ . The total loss function ( $\mathcal{L}_{total}$ ) combines a contrastive alignment loss and a reconstruction loss:

Modified InfoNCE ( $\mathcal{L}_{NCE}$ ): Aligns $v_t$ and $z_t$ by pulling them closer while pushing them apart from other time steps within the same episode.
Semantic Regularisation ( $\mathcal{L}_{\omega}$ ): To mitigate class collision, the authors introduce a weight matrix $\omega$ derived from the semantic similarity of the LLM event embeddings ( $z_t$ ). If two time steps $t$ and $t^*$ have similar event embeddings (high semantic similarity), the contrastive loss is adjusted to prevent the corresponding TS embeddings ( $v_t$ and $v_{t^*}$ ) from being pushed apart. This effectively uses the LLM's learned semantic knowledge to guide the TS representation learning.
Reconstruction Loss ( $\mathcal{L}_{recon}$ ): An auxiliary autoregressive reconstruction task ensures the TS encoder retains sufficient information to reconstruct the input time series.

3. Training and Adaptation

The model is pre-trained on ICU data (MIMIC-IV) using the combined objective. For downstream tasks, the TS encoder is adapted using task-specific linear heads. The authors demonstrate that the model can be deployed to new cohorts via k-shot adaptation, where only a small number of labeled samples are required for fine-tuning.

Key Contributions

LLM4EHR Framework: A novel foundation model that jointly models EHR events and TS via a time-and-semantic-aware contrastive pre-training objective, moving beyond stationary alignment to capture fine-grained temporal dynamics.
Semantic-Regularised Contrastive Learning: The proposal of a $\omega$ -regularised InfoNCE objective that leverages LLM-derived semantic similarities to mitigate class collision, resulting in more consistent and robust TS embeddings.
Empirical Validation of Transferability: Demonstration that the learned TS embeddings are transferable to new datasets (Physionet2012) via k-shot adaptation, outperforming both supervised baselines and other self-supervised methods.

Results

The model was evaluated on four downstream tasks using MIMIC-IV and Physionet2012 datasets:

Phenotyping: LLM4EHR achieved state-of-the-art performance across all phenotype groups, with a 2.7% gain in AUROC and 5.1% gain in AUPRC over the next best baseline. Notably, it showed significant robustness in low-prevalence phenotypes.
Rolling Predictions (Decompensation & LoS): The model achieved the best performance in rolling decompensation prediction (AUROC 0.876), significantly outperforming ablated baselines. While performance gains in Length of Stay (LoS) prediction were marginal compared to some baselines (consistent with the difficulty of the task), LLM4EHR still outperformed ablated versions, confirming the value of the pre-training objective.
Mortality Prediction: LLM4EHR achieved the highest AUROC (0.870 on MIMIC-IV, 0.856 on Physionet2012) in full-shot settings. In few-shot settings, it demonstrated superior adaptability, outperforming other models at $k \ge 10$ and showing less sensitivity to data shift compared to models using stationary alignment.
Ablation Studies:
- Removing the semantic regularisation ( $\mathcal{L}_{\omega}$ ) resulted in consistent performance drops, confirming its role in stabilizing embeddings.
- Using domain-adapted LLMs (BioClinical ModernBERT) yielded significantly better results than general-purpose LLMs, highlighting the importance of domain-specific semantic knowledge.

Significance and Claims

The paper claims that LLM4EHR represents a step toward building more generalizable and performant clinical foundation models. By explicitly aligning the temporal structures of discrete clinical events with continuous physiological time series, the model learns representations that are both clinically meaningful and adaptable to new patient cohorts with limited labeled data.

The authors emphasize that their approach addresses the "temporal mismatch" found in prior multimodal works (which often align coarse notes with fine-grained TS) and the "class collision" inherent in standard contrastive learning. The results suggest that leveraging the semantic knowledge of pre-trained LLMs to guide time-series representation learning is a viable and effective strategy for improving clinical predictive modeling. The paper concludes modestly, acknowledging limitations in scalability to larger LLMs and the need for future work on modeling extended patient trajectories beyond single ICU stays.