📊 statistics

Contrastive Learning on Multimodal Analysis of Electronic Health Records

This paper proposes a theoretically grounded, privacy-preserving multimodal contrastive learning framework that effectively integrates structured codes and unstructured notes from electronic health records to outperform single-modality approaches and demonstrate clinical utility.

Original authors: Tianxi Cai, Feiqing Huang, Ryumei Nakada, Linjun Zhang, Doudou Zhou

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianxi Cai, Feiqing Huang, Ryumei Nakada, Linjun Zhang, Doudou Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a patient's health story, but the story is written in two very different languages at the same time. One language is Structured Codes: short, strict labels like "Rheumatoid Arthritis" or "Methotrexate" that doctors use for billing and records. The other language is Unstructured Notes: the messy, free-flowing paragraphs doctors write, like "Patient reports morning stiffness and swollen joints."

For a long time, researchers tried to read these stories by looking at the codes or the notes, but rarely both together. It was like trying to understand a movie by only reading the subtitles or only listening to the soundtrack. Some tried to glue the subtitles and soundtrack together into one giant, messy pile, but that often confused the distinct rhythms of each language.

Enter CLAIME (Contrastive Learning Algorithm for Integrated Multimodal Electronic health records). Think of CLAIME as a super-smart translator that doesn't just glue the two languages together; it learns how they dance with each other.

The Big Idea: The "Same Patient" Dance

The core secret of CLAIME is a concept called Contrastive Learning. Imagine you have a room full of people (patients). Each person has a unique "health fingerprint" made of both their codes and their notes.

CLAIME plays a game of "Spot the Match":

The Good Match (Positive Pair): It takes the codes and notes from the same person and says, "Hey, these two belong together! They are describing the same person's reality." It pulls their digital representations closer, like magnets snapping together.
The Bad Match (Negative Pair): It takes the codes from Person A and the notes from Person B and says, "Nope! These don't belong together." It pushes them apart.

By doing this millions of times, CLAIME learns a map where similar medical concepts (like "swollen joints" in a note and "Rheumatoid Arthritis" in a code) end up sitting right next to each other, even if they look totally different on the surface.

What CLAIME Beats (And Why)

The paper explicitly argues against two common ways of handling this data, showing they are like using a sledgehammer to crack a nut:

The "Glue" Method (Concate): This is when researchers just mash the codes and notes into one giant list and run standard analysis. The paper shows this is a bad idea because it ignores the fact that codes and notes have different "noise" (errors). It's like trying to mix oil and water and expecting a smooth smoothie; the result is biased and misses the subtle connections.
The "Standard Contrast" Method (CL): This is a popular technique used in other fields (like matching images to text). The paper argues that applying this directly to medical data without special adjustments also fails. It gets confused by the specific way medical data is generated and ends up learning the noise (the mistakes in the records) instead of the signal (the actual health truth).

In simulations, where the authors created fake patient data to test their theories, the "Glue" and "Standard" methods stayed stuck with high error rates, no matter how much data they saw. CLAIME, however, got sharper and more accurate as it saw more data, successfully ignoring the noise.

The Privacy Superpower

Here is the most playful part: CLAIME is a privacy ninja. Most AI models need to see every single patient's raw file to learn, which is a huge privacy risk. CLAIME, however, only needs summary statistics.

Think of it like this: Instead of asking to read everyone's diary, CLAIME just asks for a list of how often certain words appear together across the whole hospital. It's like learning the rules of a game by watching the scoreboard, without ever needing to see the players' faces. This means different hospitals can team up to build a smarter model without ever sharing a single patient's private data.

How Sure Are We?

The authors are very confident in their math. They didn't just guess; they built a rigorous theoretical framework proving that CLAIME works better than the other methods under specific conditions. They showed that their method connects to a mathematical concept called Singular Value Decomposition (SVD), which is a fancy way of finding the most important patterns in a messy matrix.

They tested this in two ways:

Simulations: They created 100 different fake scenarios with varying amounts of data and noise. In every single test, CLAIME outperformed the others, especially when the data was noisy (which real medical data always is).
Real-World Test: They applied CLAIME to real data from 53,716 patients with Rheumatoid Arthritis at Mass General Brigham. They asked the model to find connections between medical codes and clinical notes.
- The Result: CLAIME was better at finding "related" concepts (like a drug that causes a side effect) than the other methods. For example, it improved the ability to spot "causative" relationships by about 3.3% compared to the next best method, and by 8% compared to a method called KPCA. Interestingly, for the task of detecting simple "Similar" pairs, KPCA actually performed best, but CLAIME's true strength shines in uncovering the complex, nuanced relationships that matter most for clinical reasoning.

The Bottom Line

The paper suggests that by treating structured codes and unstructured notes as two partners in a dance rather than two separate piles of data, we can build a much clearer picture of patient health. CLAIME is a new tool that learns this dance, respects patient privacy by only needing summary data, and has been shown to be more accurate than current methods in both computer simulations and real-world hospital data. It's not a magic wand that solves everything, but it's a significant step toward understanding the complex, multimodal stories hidden in electronic health records.

Technical Summary: Contrastive Learning on Multimodal Analysis of Electronic Health Records

Problem Statement

Electronic Health Records (EHRs) contain rich multimodal data, including structured clinical codes (e.g., diagnoses, medications, procedures) and unstructured clinical notes. While these modalities offer complementary views of patient health, existing research often treats them in isolation or combines them using simplistic methods (e.g., direct concatenation). This approach overlooks the intrinsic synergy between modalities and fails to account for their distinct statistical forms and noise structures.

Current multimodal approaches face several limitations:

Lack of Theoretical Understanding: While multimodal contrastive learning has succeeded in vision-language tasks (e.g., CLIP), its theoretical underpinnings for sparse, discrete EHR data remain under-explored.
Privacy Constraints: Many deep learning methods require patient-level data, raising privacy concerns and hindering collaboration across institutions.
Bias and Noise: Ignoring modality-specific differences can obscure specific information, discard complementary signals, and amplify systematic bias. Existing methods often fail to distinguish between true clinical signals and noise inherent to specific data collection processes (e.g., billing codes vs. NLP-extracted concepts).

Methodology

The authors propose CLAIME (Contrastive Learning Algorithm for Integrated Multimodal Electronic health records), a framework designed to learn robust, privacy-preserving feature embeddings for multimodal EHR data.

1. Generative Model

The method is grounded in a noisy multimodal log-linear generative model (Equation 2.1).

Latent State: Each patient $i$ is associated with a latent clinical state $c_i \sim N(0, I_p)$ .
Feature Generation: Structured codes ( $w^{(1)}$ ) and unstructured concepts ( $w^{(2)}$ , e.g., CUIs) are generated conditionally from $c_i$ but are subject to modality-specific noise vectors $\epsilon^{(M)}_i$ .
Key Assumption: While features share a common latent structure, they are distorted independently by noise specific to their modality (e.g., billing practices vs. NLP extraction errors).

2. The CLAIME Algorithm

CLAIME utilizes a multimodal contrastive loss function to learn embeddings $V_1$ and $V_2$ for the two modalities.

Contrastive Objective: The algorithm treats code-CUI pairs from the same patient as positive pairs and pairs from different patients as negative pairs.
Loss Function: The loss $L_{CLAIME}$ encourages high similarity for positive pairs and low similarity for negative pairs, regularized to control embedding magnitude.
Closed-Form Solution: A key theoretical insight is that minimizing the contrastive loss is equivalent to finding a rank-constrained approximation of an empirical Pointwise Mutual Information (PMI) matrix ( $\widehat{PMI}_{CLAIME}$ ). This allows the algorithm to be solved efficiently via Singular Value Decomposition (SVD) without iterative gradient descent.
Privacy Preservation: The algorithm operates solely on aggregated co-occurrence statistics (summary-level data) rather than raw patient-level records, enabling secure collaboration across institutions.

3. Theoretical Analysis

The paper provides rigorous theoretical guarantees establishing the consistency of CLAIME:

Connection to PMI: The solution to the loss function connects directly to the SVD of a population-level PMI matrix.
Convergence Rates: Under specific assumptions regarding the number of patients ( $n$ ), feature dimensions ( $d$ ), and latent dimension ( $p$ ), the authors prove that $\widehat{PMI}_{CLAIME}$ is an asymptotically unbiased estimator of the true cross-modal PMI matrix.
Superiority over Baselines: Theoretical analysis demonstrates that baseline methods (simple concatenation "Concate" and standard contrastive learning "CL" on concatenated data) suffer from irreducible bias and noise contamination. Specifically, under certain noise covariance conditions, baselines may fail to capture the true signal (converging to the noise subspace), whereas CLAIME successfully recovers the core representations.

Key Contributions

Novel Algorithm (CLAIME): A privacy-preserving algorithm for multimodal EHR representation learning that requires only summary-level co-occurrence data.
Generative Model: A novel latent variable model that explicitly accounts for modality-specific noise and patient heterogeneity, distinguishing it from previous word vector models.
Theoretical Framework: The first theoretical analysis connecting multimodal contrastive learning in healthcare to the SVD of a PMI matrix, providing convergence rates and proving the optimality of the cross-modal approach over unimodal or concatenated baselines.
Empirical Validation: Extensive simulations and real-world applications demonstrating robustness to noise and superior performance in capturing complex clinical relationships.

Results

Simulation Studies

The authors evaluated CLAIME against baselines (Concate, CL, KPCA) and gradient-descent variants under varying sample sizes, feature dimensions, and signal-to-noise ratios (SNR).

Noise Robustness: CLAIME consistently outperformed baselines, particularly in low-SNR settings where noise covariance matrices were orthogonal to the signal. Baselines often failed to recover the true embedding subspace, while CLAIME maintained accuracy.
Scalability: The SVD-based approach performed comparably to gradient-based methods (CLAIME-GD) but with significantly lower computational overhead and without requiring patient-level data.

Real-World Application (Mass General Brigham RA Cohort)

The method was applied to a cohort of 53,716 patients with Rheumatoid Arthritis (RA), integrating 3,477 codified features and 1,048 CUIs.

Relationship Detection: CLAIME was evaluated on its ability to detect "Similar" and "Related" (e.g., causative, treatment) relationships between codes and CUIs.
Performance: While all methods performed well on "Similar" pairs, CLAIME significantly outperformed baselines on "Related" pairs. Notably, it showed an ~8% AUC improvement over KPCA in detecting causative relationships.
Case Studies: Using Kendall's tau correlation with LLM-generated relevance scores (GPT-3.5/4), CLAIME demonstrated superior alignment with clinical semantics for specific concepts (e.g., distinguishing relevant drug-disease pairs) compared to concatenation and standard contrastive learning, which showed negative correlations for certain concepts.

Significance and Claims

The paper claims that CLAIME addresses a critical gap in the statistical analysis of multimodal EHR data. By moving beyond simplistic concatenation and deep "black-box" models, CLAIME offers:

Interpretability: The method is grounded in a generative model with clear statistical properties.
Privacy: It enables multi-institutional learning without sharing sensitive patient-level data, relying instead on aggregated statistics.
Theoretical Rigor: It provides the first theoretical justification for why multimodal contrastive learning outperforms single-modality or naive fusion approaches in the context of sparse, noisy EHR data.
Clinical Utility: The empirical results suggest that the learned representations capture nuanced, clinically relevant associations (e.g., causality) that other methods miss, potentially improving downstream tasks like patient profiling and disease prediction.

The authors note that while the current framework focuses on two modalities, the approach is conceptually extensible to more modalities, though theoretical guarantees for higher-order interactions remain an open challenge for future work.