💬 NLP

Equitable Electronic Health Record Prediction with FAME: Fairness-Aware Multimodal Embedding

This paper introduces FAME, a fairness-aware multimodal embedding framework that optimizes both prediction performance and equity in Electronic Health Record tasks by explicitly weighting modalities based on their fairness contributions and utilizing a sign-agnostic aggregation method to balance subgroup outcomes.

Original authors: Nikkie Hooman, Zhongjie Wu, Eric C. Larson, Mehak Gupta

Published 2026-03-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nikkie Hooman, Zhongjie Wu, Eric C. Larson, Mehak Gupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a hospital's digital filing cabinet, known as an Electronic Health Record (EHR). This cabinet is packed with different types of information about a patient:

The "Numbers" (Structured Data): Lab results, age, blood pressure, and insurance codes.
The "Stories" (Unstructured Data): Doctors' handwritten notes, nurse observations, and radiology reports.
The "Background" (Demographics): Race, gender, and age.

For years, computer programs (AI) trying to predict patient outcomes (like "Will this patient survive?") have mostly looked at just one of these things. Some only looked at the numbers; others only read the stories.

The Problem:
When these AI programs try to look at everything at once, they often get the math right but the ethics wrong. They might become very good at predicting who gets sick, but they accidentally learn to treat patients differently based on their race or insurance type. It's like a judge who gives harsher sentences to people from a specific neighborhood, even if the crime is the same. This is bias.

The Solution: FAME
The authors of this paper created a new system called FAME (Fairness-Aware Multimodal Embedding). Think of FAME as a super-smart, fair-minded team captain who manages a group of three experts:

The Statistician: Reads the numbers.
The Storyteller: Reads the doctors' notes.
The Demographer: Knows the patient's background.

How FAME Works (The Analogy)

In the past, when these three experts gave their opinions to the team captain, the captain just took the average of all their votes. "Okay, the Statistician says 50%, the Storyteller says 50%, so the final answer is 50%."

FAME changes the rules. It asks a crucial question before making a decision: "Who is being unfair?"

The "Fairness Score" (EDDI): FAME constantly checks the team. It asks, "Is the Storyteller making mistakes mostly with Black patients? Is the Statistician making mistakes mostly with older patients?" It gives each expert a Fairness Score.
The "Volume Knob": If the Storyteller is being unfair to a specific group, FAME turns down their volume. If the Statistician is being fair, FAME turns up their volume.
The "Microphone Filter": FAME also looks at specific words or specific numbers within the notes. If a certain word in a doctor's note is usually associated with bias, FAME mutes that specific word while keeping the rest of the sentence loud and clear.

The "Sign-Agnostic" Secret Sauce

The paper mentions a fancy term called "sign-agnostic aggregation." Here's a simple way to think about it:

Imagine you are judging a contest.

Contestant A makes a mistake that helps Group X but hurts Group Y.
Contestant B makes a mistake that hurts Group X but helps Group Y.

If you just add the mistakes up, they might cancel each other out, making it look like the contestant is "fair." But they aren't! They are just swinging the pendulum back and forth.

FAME doesn't let the mistakes cancel out. It looks at the size of the mistakes regardless of which way they swing. It ensures that no one is being left behind, whether the bias is "too high" or "too low."

What Happened When They Tested It?

The researchers tested FAME on real hospital data to predict three things:

Will the patient die in the ICU?
Will they stay in the hospital for more than 7 days?
Will they need a breathing machine (ventilator)?

The Results:

Better Accuracy: FAME was actually better at predicting the outcomes than the old methods. By listening to both the numbers and the stories, it got a fuller picture.
Much Fairer: Most importantly, FAME made the predictions equally accurate for everyone, regardless of their race, age, or insurance. It reduced the "unfair gap" between different groups significantly more than any other method they tried.

The Big Takeaway

The paper teaches us a valuable lesson about AI in healthcare: You don't have to choose between being smart and being fair.

In fact, by being intentional about fairness—by actively listening to which data sources are causing bias and turning them down—you can actually build a system that is smarter for everyone. It's like tuning a radio: when you stop the static (bias), the music (accurate predictions) comes through much clearer.

1. Problem Statement

Electronic Health Records (EHRs) contain diverse data modalities, including structured data (e.g., lab values, vitals, medical codes), unstructured text (e.g., clinical notes), and demographic information. While Multimodal AI (MAI) models have emerged to fuse these data types for improved clinical decision-making, they often optimize solely for predictive performance. This optimization frequently reinforces existing biases across patient subgroups defined by sensitive attributes such as race, ethnicity, age, and insurance status.

Current bias mitigation techniques often apply debiasing after modality fusion or treat all data sources equally, failing to leverage the unique fairness contributions of individual modalities. There is a lack of frameworks that explicitly weigh modalities based on their specific contribution to fairness during the fusion process to achieve equitable outcomes without sacrificing accuracy.

2. Methodology: FAME Framework

The authors propose FAME (Fairness-Aware Multimodal Embeddings), a framework that explicitly weights each modality according to its fairness contribution before fusion. The methodology consists of three core components:

A. Multimodal Architecture

The model utilizes a Joint Fusion approach, combining three specific modalities:

Demographic Data ( $X_d$ ): Sensitive attributes (Age, Ethnicity, Insurance).
Structured Clinical Data ( $X_s$ ): Longitudinal numerical/categorical data (labs, vitals). Processed using BEHRT (a transformer model for structured EHRs).
Unstructured Clinical Notes ( $X_n$ ): Free-text physician notes. Processed using BioClinicalBERT.

Each modality is encoded into latent embeddings ( $z_m$ ) via unimodal encoders.

B. Fairness-Aware Weighting Mechanisms

FAME introduces two distinct weighting mechanisms to modulate embeddings:

EDDI-Weighted Fusion (Modality Level):
- Metric: The framework uses the Error Distribution Disparity Index (EDDI) to measure fairness. EDDI quantifies the disparity in error rates between subgroups and the overall dataset.
- Sign-Agnostic Aggregation: To prevent positive and negative disparities from canceling each other out, the authors propose a sign-agnostic aggregation method (root mean square of subgroup EDDIs) to calculate a single fairness score per sensitive attribute.
- Dynamic Weighting: A gating vector ( $\hat{w}_m$ ) is computed for each modality at every training iteration. Modalities with lower EDDI (higher fairness) receive higher weights, while those with higher disparity (e.g., demographic data) are down-weighted.
- Formula: The fused embedding is calculated as $z_{fuse} = \sum \hat{w}_m \cdot (z_m \cdot W_m)$ , where weights are updated based on the gradient of the EDDI loss.
Sigmoid-Weighted Feature Selection (Feature Level):
- An extension where a sigmoid activation layer learns to gate specific features within the concatenated multimodal embedding. This allows the model to suppress specific biased features within a modality while retaining others.

C. Loss Function

The model is optimized using a combined loss function:
$L_{total} = L_{BCE} + \lambda \cdot L_{EDDI}$
Where $L_{BCE}$ is the binary cross-entropy loss for prediction accuracy, $L_{EDDI}$ is the aggregated fairness loss, and $\lambda$ is a hyperparameter balancing the trade-off between performance and fairness.

3. Key Contributions

FAME Framework: A novel method for fusing EHR modalities using weighted aggregation based explicitly on fairness metrics rather than just performance.
Sign-Agnostic Aggregation: A mathematical approach to aggregate EDDI values across subgroups to ensure equitable fairness estimation regardless of the direction of disparity.
Dual-Stage Weighting: The integration of both modality-level weighting (EDDI) and feature-level gating (Sigmoid) to create a granular control over bias.
Empirical Validation: Demonstration that leveraging complementary modalities (structured + unstructured) while down-weighting sensitive demographic inputs yields superior fairness-accuracy trade-offs compared to existing baselines.

4. Experimental Results

The framework was evaluated on the MIMIC-III dataset across three binary prediction tasks: In-ICU Mortality, Length of Stay (LOS) $\ge$ 7 days, and Mechanical Ventilation.

Baselines: Compared against Demographic-free Classification (DfC), Adversarial Debiasing (AdvDebias), Fair Patient Model (FPM), and FairEHR-CLP.
Performance Metrics: AUROC, AUPRC, EDDI (lower is better), and Equalized Opportunity (EO) (lower is better).

Key Findings:

Superior Fairness-Accuracy Trade-off: FAME achieved the best results in both accuracy and fairness across all tasks. For example, in the "LOS $\ge$ 7" task, FAME achieved an AUPRC of 1.00 and an EDDI of 0.02%, significantly outperforming baselines (e.g., FairEHR-CLP had an EDDI of 3.85%).
Modality Analysis: Unimodal BioClinicalBERT (text) outperformed BEHRT (structured) in fairness, but the multimodal FAME model outperformed both, proving that combining modalities with fairness weighting captures complementary signals.
Ablation Study: Removing the EDDI weighting (using Average Fusion) or the Sigmoid gating resulted in higher bias (higher EDDI/EO), confirming that both components are necessary for optimal equity.
Sensitivity Analysis: A hyperparameter $\lambda = 0.8$ provided the optimal balance, where the model learned to assign higher weights to structured and unstructured clinical data while consistently down-weighting demographic data.

5. Significance and Implications

Paradigm Shift in Multimodal AI: The paper challenges the standard "equal-weight" fusion approach in healthcare AI. It demonstrates that not all data modalities contribute equally to fairness; in fact, demographic data often introduces bias that should be actively suppressed during fusion.
Clinical Relevance: By reducing disparities across race, age, and insurance status without sacrificing predictive power, FAME offers a pathway to more equitable AI-assisted clinical decision support systems.
Interpretability: The dynamic weighting mechanism provides insight into which data sources drive predictions. The study found that clinical notes and structured data are robust predictors, while demographic data, while present, should be minimized to ensure fairness.
Future Directions: The authors suggest that future work could involve extracting non-clinical social determinants of health (SDoH) from unstructured text and converting them into structured formats, potentially further reducing the need to rely on raw demographic inputs.

In conclusion, FAME represents a significant advancement in healthcare AI by treating fairness as a first-class citizen in the multimodal fusion process, rather than an afterthought, resulting in models that are both highly accurate and equitable.