💻 computer science

Multimodal Fusion of Histopathology Images and Electronic Health Records for Early Breast Cancer Diagnosis

This paper presents a multimodal framework integrating histopathology images from the BreCaHAD dataset and structured EHR data from MIMIC-IV, demonstrating that an intermediate-fusion model significantly outperforms unimodal baselines in early breast cancer diagnosis while maintaining high interpretability.

Original authors: Aditya Shribhagwan Khandelwal, Mohammad Samar Ansari, Asra Aslam

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aditya Shribhagwan Khandelwal, Mohammad Samar Ansari, Asra Aslam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very tricky mystery: Is this specific spot in a patient's body cancerous, and if so, how aggressive is it?

In the real world, doctors solve this by acting like detectives who use two different sets of clues:

The Microscope Clues: They look at a tiny slice of tissue under a microscope to see what the cells look like (are they messy? are they dividing too fast?).
The Patient History Clues: They look at the patient's medical file (age, past illnesses, blood test results) to get the bigger picture.

Usually, computer programs (AI) have been trained to look at only one of these clues at a time. Some are great at reading the microscope slides, while others are great at reading the medical files. But they rarely talk to each other.

This paper introduces a new "Super Detective" AI that learns to use both clues simultaneously to diagnose breast cancer earlier and more accurately.

Here is a simple breakdown of how they built it and what they found:

1. The Two "Detectives" (The Data)

The researchers trained two separate AI teams:

The Eye Team (Images): They fed an AI thousands of tiny, high-resolution pictures of breast tissue cells. The AI learned to spot three things: normal cells, cancer cells, and "mitosis" (cells that are actively dividing, which is a sign of aggressive cancer).
- The Result: This team became incredibly good at looking at the pictures. It was like a master art critic who could spot a fake painting instantly.
The File Team (Medical Records): They fed a different AI structured data from hospital records (like age, weight, and past health issues).
- The Result: This team became very good at spotting patterns in the numbers. It was like a seasoned insurance adjuster who knows exactly which factors predict risk.

2. The "Handshake" (Multimodal Fusion)

The big innovation here wasn't just having two good detectives; it was making them hold hands.

Instead of letting the "Eye Team" and the "File Team" give separate opinions, the researchers built a system where they combine their thoughts before making a final decision.

The Analogy: Imagine the Eye Team sees a suspicious stain on a wall. It thinks, "That looks like a leak!" But the File Team says, "Wait, I know this building; the pipes are old and leaky." When they combine their thoughts, they are much more confident it's a leak than if they were working alone.

In technical terms, they took the "brain" of the image AI and the "brain" of the medical record AI and mashed them together to create a single, super-smart decision-maker.

3. The Big Surprise (Where the Magic Happened)

You might think, "If the Eye Team is already 99% perfect at looking at pictures, why do we need the File Team?"

The answer lies in the rare and tricky cases.

The Problem: The "Eye Team" was great at spotting obvious cancer and obvious healthy cells. But it struggled with mitosis (the rapidly dividing cells). These are the most dangerous cells, but they are also the hardest to spot because they look very similar to dying healthy cells. It's like trying to tell the difference between a sleeping cat and a dead cat from a blurry photo.
The Solution: When the "File Team" joined in, it provided context. If the patient is older or has specific blood markers, the combined AI became much better at saying, "Okay, this blurry cell looks like a dying cell, but given the patient's history, it's actually a dangerous dividing cell."

The Result: The combined team didn't just get slightly better; they became significantly better at catching the dangerous, rare cases that the image-only AI missed.

4. Trusting the AI (Interpretability)

Doctors are skeptical of "black box" computers that just give an answer without explaining why. This paper made sure the AI could explain itself:

For Images: The AI highlighted exactly which part of the cell it was looking at (like a teacher circling the answer on a test).
For Records: The AI listed which medical facts mattered most (e.g., "I flagged this patient because they are 60 and have high blood sugar").

This transparency is crucial. It proves the AI isn't just guessing; it's reasoning like a human doctor.

The Bottom Line

This paper shows that in healthcare, 1 + 1 = 3.

By teaching AI to look at the microscope slides and read the patient's history at the same time, we get a diagnostic tool that is not only more accurate but also safer for patients. It catches the tricky, dangerous cases that a single method might miss, potentially saving lives by catching cancer earlier.

In short: We stopped asking the AI to choose between "seeing" and "reading," and started asking it to do both at once. And the result is a much smarter doctor's assistant.

1. Problem Statement

Breast cancer diagnosis relies heavily on histopathological examination of biopsy tissues (the gold standard) and structured Electronic Health Records (EHR). However, current computational approaches typically treat these data modalities in isolation:

Histopathology: Convolutional Neural Networks (CNNs) excel at classifying cellular features in images but lack access to patient-specific clinical context (demographics, comorbidities, lab results).
EHR: Machine learning models on tabular data provide risk stratification but ignore the microscopic morphological evidence required for definitive diagnosis.
The Gap: Existing methods fail to replicate the integrative reasoning of clinicians who synthesize visual tissue evidence with longitudinal patient history. Furthermore, class imbalance (particularly for rare but critical mitotic figures) often leads to poor performance in minority classes, even when overall accuracy is high.

2. Methodology

The authors propose a systematic intermediate-fusion framework that integrates patch-level histopathology features from the BreCaHAD dataset with structured clinical data from MIMIC-IV.

A. Data Preprocessing

Image Data (BreCaHAD):
- Source: High-resolution Whole Slide Images (WSI) with expert dot-level annotations for three classes: Tumour Nuclei, Non-Tumour Nuclei, and Mitosis.
- Pipeline: A sliding window extracts $64 \times 64$ pixel patches centered on annotated nuclei. Patches are normalized, resized, and augmented (rotations, flips, color jitter) to handle staining variability.
- Output: Millions of labeled patches organized for efficient batching.
Tabular Data (MIMIC-IV):
- Source: De-identified ICU records filtered for breast cancer diagnoses (ICD-9/10).
- Pipeline: Features include demographics, 29 Elixhauser comorbidity indicators, lab values, and treatment history.
- Handling: Missing values >80% were excluded. Continuous variables were standardized; categorical variables were one-hot encoded. Binary missingness indicators were added for sparse fields.

B. Model Architectures

The study evaluates three unimodal baselines and one multimodal fusion model:

Unimodal Image Models:
- Simple CNN: A shallow 3-block CNN (32 $\to$ 64 $\to$ 128 channels) trained from scratch.
- ResNet-18: A pre-trained (ImageNet) ResNet-18 fine-tuned on the patches. It uses a staged unfreezing strategy and class-weighted loss to address imbalance.
Unimodal Tabular Models:
- MLP (Multilayer Perceptron): A 2-layer feed-forward network (256 $\to$ 128 units) with BatchNorm and Dropout.
- XGBoost: A gradient boosting model configured with max depth 6–8 and scale_pos_weight to handle class imbalance.
Intermediate Fusion Model:
- Architecture: Two parallel branches.
  - Image Branch: ResNet-18 (frozen backbone, fine-tuned head) extracts a 512-dimensional latent embedding from the penultimate layer.
  - Tabular Branch: The MLP extracts a 128-dimensional patient embedding.
- Fusion: The 512D and 128D vectors are concatenated to form a 640-dimensional multimodal representation.
- Classification: The fused vector passes through two fully connected layers (256 $\to$ 128) with ReLU and Dropout, followed by a Softmax layer for 3-class prediction.

C. Interpretability

Images: Grad-CAM is used to generate heatmaps, verifying that the model attends to biologically relevant nuclear structures rather than background stroma.
Tabular: SHAP (SHapley Additive exPlanations) is used to provide global and local feature attributions, validating that the model relies on clinically plausible risk factors (e.g., age, comorbidities).

3. Key Contributions

Systematic Multimodal Framework: A novel architecture integrating fine-grained cellular annotations (BreCaHAD) with large-scale EHR data (MIMIC-IV) via intermediate fusion.
Strong Unimodal Baselines: Establishment of high-performance baselines for both image (ResNet-18) and tabular (XGBoost) modalities, demonstrating that XGBoost outperforms MLPs on structured clinical data.
Performance on Minority Classes: Demonstration that fusion significantly improves recall and AUC for the mitosis class, which is diagnostically critical but class-imbalanced.
Dual-Modality Interpretability: A combined explanation framework (Grad-CAM + SHAP) that aligns model decisions with established pathological and clinical criteria, enhancing trust and transparency.

4. Key Results

Unimodal Performance:
- ResNet-18 (Image): Achieved near-perfect accuracy (1.000) and AUC (1.000) on the test set. However, class-level analysis revealed a lower AUC for the mitosis class (0.927) compared to majority classes.
- XGBoost (Tabular): Achieved 98% accuracy and 0.94 AUC, outperforming the MLP baseline.
Multimodal Fusion Performance:
- Overall: The fusion model achieved a macro-average AUC of 0.997, outperforming all unimodal baselines.
- Minority Class Improvement: The most significant gain was in the mitosis category. The fusion model improved the AUC from 0.927 (ResNet-only) to 0.994 (a 6.7 percentage point increase).
- Error Analysis: The fusion model reduced misclassifications of mitotic patches as "tumour nuclei" (dropping from 15 to 0 errors in the test set), indicating that clinical context helps disambiguate morphologically similar cases.
Interpretability:
- Grad-CAM heatmaps confirmed attention on nuclear structures.
- SHAP plots confirmed that age, comorbidity count, and lab biomarkers were the dominant drivers for clinical predictions, matching oncological knowledge.

5. Significance and Limitations

Significance:

Clinical Impact: The study proves that integrating EHR data with histopathology images improves diagnostic accuracy specifically for rare, high-stakes conditions (mitosis) where image-only models plateau.
Methodological Insight: It highlights that overall accuracy can be misleading; per-class minority metrics (like mitosis AUC) are essential for evaluating healthcare AI.
Transparency: The dual-interpretability approach bridges the gap between "black box" AI and clinical reasoning, offering explanations that clinicians can verify.

Limitations:

Data Alignment: BreCaHAD and MIMIC-IV are not patient-paired; the fusion simulates a clinical scenario by using MIMIC-IV as contextual augmentation rather than true paired data.
Dataset Scope: BreCaHAD is single-institution, and MIMIC-IV represents ICU patients, potentially limiting generalizability to outpatient screening populations.
Architecture: The study did not explore Vision Transformers or multimodal attention mechanisms due to computational constraints.
Validation: Results are based on internal splits; external validation on independent cohorts with different staining protocols is required before clinical deployment.