💻 computer science

Is Clinical Text Enough? A Multimodal Study on Mortality Prediction in Heart Failure Patients

This study demonstrates that supervised multimodal fusion of clinical text with entity-level representations and structured data outperforms both text-only and current large language model approaches for short-term mortality prediction in heart failure patients.

Original authors: Oumaima El Khettari, Virgile Barthet, Guillaume Hocquet, Joconde Weller, Emmanuel Morin, Pierre Zweigenbaum

Published 2026-04-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Oumaima El Khettari, Virgile Barthet, Guillaume Hocquet, Joconde Weller, Emmanuel Morin, Pierre Zweigenbaum

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict whether a patient with heart failure will be okay in the next three months. You have two main sources of information:

The "Scorecard" (Structured Data): Numbers like age, blood pressure, and lab results. It's clean, organized, and easy to read, like a spreadsheet.
The "Story" (Clinical Text): The messy, handwritten (or typed) notes doctors write about the patient's symptoms, history, and how they are feeling. It's full of nuance but hard to organize.

This paper is a race to see which method works best at predicting the outcome: relying on the Scorecard, the Story, or a mix of both. They also tested if "Super-AI" (Large Language Models) could do the job better than traditional methods.

Here is the breakdown of their findings using simple analogies:

1. The Old Way vs. The New Way (Text Only)

The Old Way (The "Summary"):
Imagine you ask a student to read a 50-page novel and then write a one-sentence summary to guess the ending. That's what older AI models did with doctor's notes. They took the whole text and compressed it into a single "summary vector" (called a CLS embedding).

Result: It was okay, but it missed the details.

The New Way (The "Highlighter"):
The researchers tried a smarter approach. Instead of just summarizing the whole book, they asked the AI to highlight specific keywords like "heart attack," "high blood pressure," or "taking medication." They then fed both the summary and these highlighted keywords into the model.

Result: This was much better! It's like giving the student the summary plus a list of the most important plot points. The model could see the specific details that matter most.

2. The Scorecard vs. The Story (Structured vs. Text)

When they tested just the "Scorecard" (numbers) against just the "Story" (text):

The Scorecard was surprisingly strong. Numbers like age and kidney function are very reliable predictors.
The Story was good, but not as consistent on its own.
The Winner: When they combined them, the prediction became the most accurate. It's like a doctor looking at the blood test results while listening to the patient's story. The numbers give the hard facts, and the story gives the context.

3. The "Super-AI" Experiment (Large Language Models)

The researchers also tried using "Super-AIs" (like GPT-4 or specialized medical AIs) to act as the doctor. They asked these AIs: "Based on this data, will the patient die in 3 months? Say 0 or 1."

The Result: A Disappointment.
The Super-AIs were inconsistent and often confused.

The "Yes-Man" Problem: Sometimes the AI would just guess "Yes" (Class 1) for everyone because it was scared to say "No." Other times, it would guess "No" for everyone.
The Format Issue: When given a spreadsheet of numbers, the AI got lost. It didn't know how to read a table. When given the text story, it did slightly better, but still made mistakes.
The Takeaway: These powerful AIs are great at writing poems or chatting, but they are currently terrible at being precise medical calculators. They lack the discipline to follow strict rules when the stakes are high.

4. The Final Verdict

The paper concludes that the best tool for the job isn't a "Super-AI" that tries to think like a human. Instead, the winner is a specialized, supervised model that acts like a hybrid detective:

It reads the doctor's notes and uses a "highlighter" to find specific medical terms (like "edema" or "diuretics").
It looks at the hard numbers (lab results).
It uses a smart "fusion" mechanism to weigh how much the story matters compared to the numbers for that specific patient.

In a nutshell:
Don't rely on a chatbot to save lives yet. The most reliable way to predict heart failure outcomes right now is to combine the hard numbers with a smart reading of the doctor's notes, focusing on the specific medical details rather than just the general vibe.

Why does this matter?

Hospitals have mountains of data. This study shows us how to build a tool that actually helps doctors make better decisions by respecting both the facts (numbers) and the context (stories), while warning us that we shouldn't just blindly trust the newest, flashiest AI models for critical medical tasks.

1. Problem Statement

The paper addresses the challenge of predicting short-term (3-month) mortality in patients hospitalized for heart failure (HF). While Electronic Health Records (EHRs) contain rich data, current approaches often rely solely on structured variables (e.g., demographics, lab results) or unstructured clinical text, failing to leverage the complementary nature of both. Furthermore, existing studies predominantly use English datasets (like MIMIC), ignoring linguistic and structural differences in French medical records. The authors investigate whether:

Enriching text embeddings with entity-level information improves prediction over standard document-level embeddings.
Multimodal fusion of text and structured data outperforms unimodal approaches.
Large Language Models (LLMs) can effectively perform this prediction task via prompting compared to supervised transformer models.

2. Dataset and Resources

Source: A closed dataset from the cardiology department of Hôpital Saint Joseph in Paris, France.
Cohort: 2,254 patient hospitalizations.
Task: Binary classification (Death within 3 months post-discharge vs. Survival).
Class Imbalance: ~11% positive class (deceased), 89% negative class.
Data Modalities:
- Structured: 115 initial variables (demographics, biology, comorbidities, medications) reduced to 41 predictive features via LASSO feature selection.
- Unstructured: Free-text clinical notes in French.
Annotation: A hybrid approach was used for Named Entity Recognition (NER). Heuristic rules (pattern matching, UMLS, BDPM) were refined by expert and non-expert manual annotation. A supervised BERT-based NER model was trained on this hybrid data to extract 23 clinical entity types (e.g., Pathology, Treatment, Anatomy).

3. Methodology

The study compares four main families of models:

A. Text-Only Models

Baselines: Used [CLS] token embeddings from pre-trained French biomedical models (CamemBERT-bio and DrBERT).
Entity-Enhanced: Combined the [CLS] embedding with averaged embeddings of extracted medical entities.
Fusion Strategies:
- Static: Average, Sum, Concatenation.
- Learnable: Weighted Fusion (learnable scalars) and Gated Fusion (sigmoid gate to modulate entity influence).
Classifier: Logistic Regression on top of embeddings.

B. Structured-Only Models

Reference Model: LASSO regression for feature selection followed by Logistic Regression (L2 regularization).
Baseline Model: Logistic Regression on the 41 selected features (mean imputation, standardization).

C. Multimodal Fusion Models

Simple Fusion: Direct concatenation of text (CLS or CLS+Entities) and structured data; Late fusion (averaging or stacking predictions).
Learnable Interaction:
- Gated Fusion: Gates entity embeddings before concatenation with structured data.
- Dual Cross-Attention (Dual-XAttn): Bidirectional attention between text and structured representations.
- Gated Dual-XAttn & Bi-Gated Dual-XAttn: Combines gating mechanisms with cross-attention for dynamic balancing.

D. Generative LLM-Based Experiments

Models: Mistral-7B-Instruct, Qwen2.5-7B-Instruct, and MedGemma-4B.
Prompting: Tested three input modalities: Text only, Structured data only (tabular), and Multimodal (Text + Structured).
Decoding: Compared Greedy decoding vs. Constrained decoding (vocabulary restricted to "0" or "1").
Textification: Structured data was also converted to natural language summaries to test if format, rather than content, hindered LLM performance.

4. Key Results

Text-Only Performance

Entity-Level Superiority: Incorporating entity-level embeddings consistently outperformed using [CLS] embeddings alone.
Best Model: Gated Fusion with CamemBERT-bio achieved the highest F1 (39.5%) and AUC (75.37%).
Observation: Learnable fusion strategies (Gated/Weighted) generally outperformed simple static fusion (Average/Sum) for CamemBERT-bio.

Structured-Only Performance

Structured data alone yielded strong results, with the Baseline Logistic Regression achieving an AUC of 79.04, outperforming all text-only models.
Feature analysis confirmed clinically plausible predictors (e.g., Age, Urea, CRP positively correlated with mortality; Beta-blockers negatively correlated).

Multimodal Performance (The Best Approach)

Fusion Wins: Multimodal models significantly outperformed unimodal baselines.
Top Performer: Gated Fusion (CamemBERT-bio + Selected Structured Data) achieved the best F1 (48.5%) and AUC (81.46%).
Statistical Significance: Paired t-tests confirmed that the Multimodal Late Fusion (Stacking) significantly improved discrimination over the best structured-only and text-only models ( $p < 0.05$ ).
Key Insight: The performance gap between CamemBERT-bio and DrBERT narrowed in multimodal settings, suggesting structured data compensates for text model weaknesses.

LLM Performance

Poor Predictive Ability: LLMs performed significantly worse than supervised models (F1 scores generally < 26%).
Modality Bias: LLMs performed best with Text-only prompts and worst with Structured-only inputs.
Instruction Adherence Issues:
- Models often failed to output only the class label, generating explanations or variable lists.
- Mistral on structured data outputted only the negative class (Recall = 0).
- Qwen with constrained decoding outputted only the positive class (Precision = 10.92%, Recall = 100%).
Textification: Converting structured data to text improved Mistral's interpretability (F1 rose from 0 to 19.42) but did not significantly boost overall predictive power.

5. Key Contributions

Entity-Aware Multimodal Framework: Demonstrated that enriching clinical text with entity-level representations (via NER) significantly boosts mortality prediction compared to global document embeddings.
Systematic Multimodal Comparison: Provided a rigorous evaluation of various fusion strategies (static vs. learnable, early vs. late) on a French HF cohort, identifying Gated Fusion as the optimal strategy.
LLM Limitations Analysis: Highlighted that current instruction-tuned LLMs are unreliable for direct clinical outcome prediction, exhibiting inconsistent behavior, poor instruction adherence, and a strong bias toward text over structured data.
French Clinical Dataset: Created and annotated a specialized French heart failure dataset, addressing the gap in non-English, condition-specific medical NLP resources.

6. Significance and Conclusion

The study concludes that clinical text alone is insufficient for reliable short-term mortality prediction in heart failure. The most robust solution is a supervised multimodal transformer model that fuses structured variables with entity-aware text embeddings.

While LLMs show promise in reasoning over clinical narratives, they currently lack the reliability, consistency, and precision required for clinical decision support in high-stakes prediction tasks. The authors emphasize that for clinical deployment, supervised models with explicit fusion mechanisms remain superior to generative prompting strategies.

Limitations noted: Single-center data (limiting generalizability), dependence on NER quality, hardware constraints limiting LLM size, and the "black box" nature of multimodal models regarding interpretability.