🤖 machine learning

X-FEMR: A Token-level Explainable Approach for Electronic Health Records Foundation Models using Transformer-based Models

This paper introduces X-FEMR, the first token-level explainability framework for Electronic Health Records Foundation Models, which employs a Transformer-based surrogate model to approximate predictions and validate clinical relevance through a novel alignment metric, thereby enhancing the interpretability and trustworthiness of black-box clinical AI.

Original authors: Jie Huang, Pengfei Yin, Zihan Xu, Daniel Capurro, Mike Conway, Ting Dang

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jie Huang, Pengfei Yin, Zihan Xu, Daniel Capurro, Mike Conway, Ting Dang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Black Box" Doctor

Imagine a super-smart AI doctor trained on millions of patient records. This AI, called a Foundation Model for Electronic Health Records (FEMR), is incredibly good at predicting things like "Will this patient need to stay in the hospital for more than a week?" or "Will they need to go to the ICU?"

However, there is a big problem: It's a black box. You give it a patient's history, and it gives you an answer, but it won't tell you why. It's like a wizard casting a spell and saying, "It's done," without explaining the ingredients or the magic words used. Doctors are hesitant to trust a "magic" doctor because they can't see the reasoning, and they worry the AI might be making mistakes based on hidden biases.

The Solution: The "Shadow Puppeteer"

The researchers wanted to open this black box without breaking the AI. To do this, they built a surrogate model.

Think of the original AI (the FEMR) as a master chef who makes a complex, secret dish. You can taste the dish (the prediction), but you can't see the recipe.
The researchers created a shadow puppeteer (the surrogate model). They watched the master chef cook thousands of times, noting every ingredient added and the final taste. Then, they trained the shadow puppeteer to mimic the chef's cooking style perfectly.

Once the shadow puppeteer learned to cook the exact same way as the master chef, the researchers could ask the puppeteer: "Hey, which specific ingredient made this dish taste salty?" Because the puppeteer is simpler and more transparent, it can point to the exact spice (or in this case, the specific data point) that influenced the decision.

How They Did It: The "Token" Detective

The paper introduces a new way to look at the data called Token-Level Explainability.

The Data: A patient's medical record isn't just a paragraph of text; it's a long timeline of events. Each event (like a lab test, a diagnosis, or a vital sign) is a "token."
The Detective Work: The researchers used a tool called SHAP (which acts like a magnifying glass) to look at the shadow puppeteer's work. They asked: "Which specific tokens (words, numbers, or codes) did the model pay the most attention to when making its prediction?"

For example, if the model predicted a long hospital stay, the "magnifying glass" might light up on specific tokens like "High Heart Rate" or "Low Oxygen," showing that these were the main reasons for the decision.

The "Clinical Alignment" Test

Just because the AI points to "Heart Rate" doesn't mean it's right; it could be pointing to the wrong thing by accident. To check if the AI is actually thinking like a human doctor, the researchers invented a new score called the Clinical Validated Events Ratio.

Imagine a checklist of "Real Medical Facts" that doctors know are important (like blood pressure, temperature, and heart rate).

The researchers counted how many times the AI's "magnifying glass" landed on these real medical facts.
They found that about 30% of the time, the AI's most important clues matched the things human doctors know are critical.

This is like a student taking a test. If the student gets the right answer, we want to know if they got it because they studied the right chapters (clinical facts) or just guessed. This paper shows that the AI is mostly studying the right chapters.

The Results

The Shadow Mimics the Master: The shadow puppeteer (surrogate model) was able to predict patient outcomes almost as well as the original master chef (the FEMR). This proves the shadow is a faithful copy.
The Clues Make Sense: When the researchers looked at what the AI was focusing on, the top clues were things like heart rate, blood pressure, and body temperature. These are exactly the things doctors look at.
Numbers Matter Most: The AI paid the most attention to specific numbers (like lab results) and the timing of events, rather than just the text descriptions.

The Bottom Line

This paper presents the first method to break down the "black box" of advanced medical AI at the level of individual data points. By building a transparent "shadow" version of the AI and checking its clues against real medical knowledge, the researchers showed that these powerful models are making decisions based on clinically relevant information, not just random noise. This helps build trust, showing doctors that the AI is reasoning in a way that aligns with human medical expertise.

Technical Summary: X-FEMR

Problem Statement
Foundation Models for Electronic Health Records (FEMRs), such as ETHOS and CLMBR-T-Base, have demonstrated strong performance in converting longitudinal patient trajectories into generalizable representations for clinical prediction tasks. However, these models function as "black boxes," raising significant concerns regarding bias, interpretability, and clinical trust. Existing Explainable AI (XAI) techniques, including feature attribution (e.g., SHAP) and gradient-based methods (e.g., saliency maps), are often ill-suited for foundation-scale architectures. These methods struggle with high-dimensional distributed representations, complex long-context interactions, and the task-agnostic nature of FEMR latent spaces. Furthermore, current surrogate models (e.g., linear regression, decision trees) often assume smooth, linear data relationships, which fails to capture the irregular, heterogeneous, and non-linear temporal dynamics inherent in EHR data. Consequently, there is a lack of token-level explainability frameworks specifically designed for FEMRs that can align with clinical reasoning.

Methodology
The authors propose X-FEMR, the first token-level explainability approach for FEMRs, utilizing a transformer-based surrogate model. The methodology consists of three primary components:

Data Collection and Preprocessing:
- The study utilizes the EHRSHOT dataset, organizing patient records into time-ordered sequences of clinical events following the OMOP Common Data Model (OMOP-CDM).
- Each event is characterized by four features: a medical code (e.g., SNOMED, LOINC), a numeric value (e.g., lab results), a text field, and a timestamp.
- Input-output pairs are extracted from the target FEMR (CLMBR-T-Base) for two specific prediction tasks: predicting Length of Stay (LOS) and ICU transfer.
Surrogate Model Development:
- A Transformer-based surrogate model is trained to approximate the input-output behavior of the FEMR. Unlike traditional linear surrogates, this architecture captures complex temporal dependencies and non-linear patterns.
- Event Encoding: Each clinical event is embedded using learned code embeddings, text embeddings, and a gated injection mechanism for numeric values. Temporal dynamics are captured by calculating the time-to-prediction interval ( $\Delta t$ ), which is log-transformed and normalized.
- Supervision Strategies: The surrogate is trained under two settings:
  - Hard-label Supervision: Predicting binary outputs (e.g., transfer vs. non-transfer) to capture the final decision boundary.
  - Soft-label Supervision: Predicting raw probability outputs to capture the model's confidence and nuanced decision logic.
- The study compares these settings to determine which better reflects the underlying decision logic of the FEMR.
Token-Level Explainability and Evaluation:
- SHAP Analysis: The trained surrogate model is analyzed using SHAP (SHapley Additive exPlanations) to quantify the contribution of individual tokens (clinical events) to the model's predictions.
- Clinical Alignment Metric: To evaluate the clinical relevance of these explanations, the authors introduce a novel metric called the Clinical Validated Events Ratio (R). This metric quantifies the correspondence between the tokens identified as important by the surrogate and a set of clinically validated features (e.g., specific ICD-10 codes, vital signs like heart rate and blood pressure) established in medical literature.
- $R = \frac{\sum_{f_j \in F_c} c(f_j; D)}{\sum_{f_j \in F} c(f_j; D)}$ , where $F_c$ is the subset of clinically relevant tokens and $c(f_j; D)$ is the occurrence frequency of token $f_j$ in the dataset.

Key Results

Surrogate Performance: The transformer-based surrogate trained with hard-label supervision closely approximated the FEMR's predictive behavior, outperforming the original CLMBR-T-Base on the LOS task (AUPRC: 0.4434 vs. 0.4173) and achieving comparable performance on the ICU transfer task. Soft-label supervision yielded lower performance, likely due to the difficulty in modeling weak positive signals near the decision threshold.
Token Importance: SHAP analysis revealed that numerical values (e.g., laboratory measurements) and time deltas were the most influential features, contributing more significantly than text or codes.
Clinical Alignment: The identified important tokens aligned well with clinical knowledge.
- For the LOS task, the top 5 most frequent tokens (Heart rate, SBP, Body temperature, DBP, Respiratory rate) accounted for 29% of total events, with a Clinical Validated Events Ratio ( $R$ ) of 0.3093.
- For the ICU Transfer task, the top 5 tokens (Heart rate, Body temperature, SBP, DBP, Oxygen saturation) accounted for 24% of total events, with an $R$ of 0.2717.
- These results indicate that the surrogate model's explanations are concentrated on clinically validated predictors of patient outcomes.

Significance and Claims
The paper claims to present the first token-level explainability approach specifically for FEMRs. Its significance lies in:

Bridging the Interpretability Gap: It moves beyond traditional feature-level explanations to provide granular, token-level insights into how FEMRs leverage specific patient history elements (e.g., specific vital signs at specific times) for predictions.
Validated Framework: By introducing the Clinical Validated Events Ratio, the authors provide a quantitative method to assess whether model explanations align with established medical knowledge, addressing the lack of evaluation metrics for FEMR explainability.
Practical Utility: The approach offers a practical framework for generating interpretable and trustworthy clinical AI, helping clinicians understand the reasoning behind AI recommendations and potentially improving clinical scoring methods or practice guidelines.

The authors acknowledge limitations, including the complexity of imitating long, noisy patient trajectories and the potential loss of fine-grained information when combining multiple feature channels into single event tokens. They suggest future work could explore alternative tokenization strategies that treat time as a distinct modeling component.