Multi-Stain Fusion of Histopathology Images Using Deep Learning for Pediatric Brain Tumor Classification

This study demonstrates that multi-stain deep learning fusion of H&E and Ki-67 whole slide images, particularly through intermediate and late fusion strategies, significantly outperforms single-stain models in classifying pediatric brain tumor grades and types by leveraging complementary histological information.

The Gist

Imagine a pediatric brain tumor diagnosis is like trying to identify a specific type of fruit in a dark room. You have two flashlights: one shines a standard white light (H&E stain), and the other shines a special "growth" light that only highlights cells that are multiplying rapidly (Ki-67 stain).

For a long time, doctors had to rely on just one flashlight, or look at them separately, to guess what kind of tumor they were dealing with. This paper is about a team of researchers who decided to try fusing the beams of both flashlights together using a super-smart computer brain (Deep Learning) to see if they could identify the fruit much more accurately.

Here is the story of their experiment, broken down simply:

1. The Problem: A Crowded, Dark Room

Pediatric brain tumors are tricky. There are many different "families" of tumors (like Low-Grade Gliomas, High-Grade Gliomas, Medulloblastomas, etc.).

• The H&E Flashlight: This is the standard view. It shows the shape and structure of the cells, like seeing the outline of a fruit. It's good, but sometimes different fruits look very similar.
• The Ki-67 Flashlight: This is a special stain that highlights cells that are growing fast. Think of it like a "glow-in-the-dark" marker on the parts of the fruit that are ripening or expanding quickly. High-grade (dangerous) tumors usually have a lot of this "glow."

The challenge? Pathologists (the experts looking at the slides) are busy, and looking at two separate slides for every patient is slow. Also, the two slides often don't line up perfectly (they are "unregistered"), making it hard to compare them pixel-by-pixel.

2. The Solution: The "Super-Brain" Detective

The researchers used a powerful AI tool called Deep Learning. They didn't just ask the AI to look at the pictures; they taught it to be a detective that combines clues from both flashlights.

They tested three different ways to combine the information:

• Early Fusion (The Smoothie): They blended the raw data from both flashlights together before the AI looked at it. It's like blending the fruit and the glow-stick into a smoothie and then trying to taste it.
• Intermediate Fusion (The Team Huddle): The AI looked at the H&E slide and the Ki-67 slide separately first, formed an opinion, and then the two opinions "huddled" to discuss and combine their insights before making a final decision.
• Late Fusion (The Jury Vote): The AI made a guess based on the H&E slide, made a separate guess based on the Ki-67 slide, and then a final "judge" looked at both guesses and decided the winner.

3. The Results: Two Heads (and Two Lights) Are Better Than One

The team tested this on over 1,600 brain tumor slides from children. Here is what they found:

• The "Glow" Light is Powerful: When looking at just the Ki-67 slide, the AI was actually better at telling the difference between "low-grade" (slow-growing) and "high-grade" (fast-growing) tumors than the standard H&E slide. This makes sense because the "glow" directly shows how fast the tumor is growing.
• The Combination Wins: However, the fusion models (especially the "Team Huddle" and "Jury Vote" methods) were the champions. By combining the structural details of the H&E slide with the growth speed of the Ki-67 slide, the AI got the most accurate diagnosis.
  • For telling "Good vs. Bad" tumors, the combined approach was significantly better than using either light alone.
  • For sorting the 5 different specific types of tumors, the combined approach again beat the single-light models.

4. Did the AI Know What It Was Looking At?

A big worry with AI is that it might be a "black box"—it gives an answer, but we don't know why. The researchers wanted to make sure the AI wasn't just guessing.

They checked the AI's "attention maps" (heatmaps showing where the AI was looking). They compared these maps to the actual "growth maps" (Ki-67 labeling index).

• The Result: The AI was looking exactly where the fast-growing cells were! There was a strong correlation between where the AI paid attention and where the "glow" was strongest. This proved the AI was actually learning real biological features, not just memorizing patterns.

5. The Takeaway

This study is like discovering that while a single flashlight helps you see in the dark, two different colored flashlights help you see the whole picture.

• Why it matters: In the real world, not every hospital has expensive molecular tests. But many have these two types of slides. This research shows that by using AI to fuse these two existing slides, doctors could get a more accurate diagnosis for children with brain tumors, potentially leading to better treatment plans.
• The Future: The researchers suggest that in the future, we could add even more "flashlights" (like genetic data or other stains) to make the AI even smarter, turning a good diagnosis into a great one.

In short: By teaching a computer to look at two different types of brain tumor slides at the same time, the researchers created a diagnostic tool that is more accurate than looking at either slide alone, helping to catch dangerous tumors earlier and treat them better.

Technical Summary

1. Problem Statement

Pediatric brain tumor diagnosis relies heavily on histopathological analysis, traditionally using Hematoxylin and Eosin (H&E) stains. However, accurate grading and subtyping often require additional immunohistochemical (IHC) markers, such as Ki-67 (a marker for cellular proliferation), to distinguish between low-grade (LGG) and high-grade gliomas (HGG) or to classify specific tumor types (e.g., Medulloblastoma, Ependymoma).

• Challenges: Manual histological assessment is labor-intensive and requires scarce expert neuropathologists. While deep learning (DL) has advanced computational pathology (CPath), most models rely on single-stain (H&E) data.
• Gap: There is limited research on fusing unregistered H&E and Ki-67 Whole Slide Images (WSIs) using deep learning to improve diagnostic accuracy in pediatric cases. Existing fusion methods often require pixel-perfect registration, which is difficult to achieve in clinical practice where slides are scanned separately.

2. Methodology

The study employs a Weakly-Supervised Multiple Instance Learning (MIL) framework combined with Multi-Stain Fusion strategies.

Dataset

• Source: Children's Brain Tumor Network (CBTN).
• Cohort: 529 pediatric subjects (ages 0.15–36.5 years) with 1,662 unregistered WSIs (1,047 H&E and 615 Ki-67).
• Classes:
  • Binary Task: Low-Grade Glioma (LGG) vs. High-Grade Glioma (HGG).
  • 5-Class Task: LGG, HGG, Medulloblastoma (MB), Ependymoma (EP), Ganglioglioma (GG).
• Preprocessing: Tissue segmentation, extraction of non-overlapping $224 \times 224$ pixel patches, and exclusion of artifacts.

Feature Extraction

• Foundation Model: The CONCHv1_5 histology foundation model (a Vision Transformer initialized from UNI and fine-tuned on histopathology) was used to extract 768-dimensional feature vectors for each patch.
• Strategy: Features were extracted independently for H&E and Ki-67 modalities.

Classification Framework (CLAM)

The Clustering-Constrained Attention Multiple Instance Learning (CLAM) model was used for patient-level classification.

• Mechanism: It uses an attention mechanism to weigh the importance of individual patches (instances) within a slide (bag) and incorporates a clustering loss to guide attention toward informative regions.
• Single-Stain Baselines: CLAM was trained separately on H&E and Ki-67 features.

Multi-Stain Fusion Strategies

The study investigated three fusion paradigms to combine H&E and Ki-67 information:

1. Early Fusion: Concatenation of raw patch-level features from both stains before feeding them into a single CLAM model.
2. Intermediate Fusion: Fusion of patient-level attention vectors (outputs of the single-stain CLAM models) before the final classification layer.
   • Methods tested: Cross-attention, Concatenation, and Element-wise Multiplication.
   • Note: Since CONCH features were frozen, traditional intermediate fusion (updating feature extractors) was not possible; instead, the study fused the learned attention vectors.
3. Late Fusion: Independent training of single-stain models, followed by aggregation of their predictions (logits or softmax scores) or feeding these scores into a secondary "learning model" (Linear, 1-Hidden Layer, 2-Hidden Layer, or Attention-based).

Evaluation & Interpretability

• Metrics: Balanced Accuracy, Matthews Correlation Coefficient (MCC), AUC-ROC, and Weighted F1-score.
• Statistical Analysis: Non-parametric bootstrapping (50 replicates) and permutation tests with Bonferroni correction.
• Explainability: Spearman's rank correlation ($\rho$) was calculated between the model's Ki-67 attention heatmaps and ground-truth Ki-67 Labeling Index (LI) maps, as well as positive/negative cell density maps.

3. Key Results

Binary Classification (LGG vs. HGG)

• Single-Stain Performance: Ki-67 (Balanced Acc: $0.86 \pm 0.05$) outperformed H&E ($0.84 \pm 0.05$), confirming Ki-67's clinical utility for grading.
• Fusion Performance:
  • Best Model: Intermediate Concatenation Fusion achieved the highest Balanced Accuracy of $0.88 \pm 0.05$ ($p < 0.005$ vs. single stains).
  • Most fusion approaches (Early, Intermediate, and specific Late Fusion models) significantly outperformed single-stain models.
  • Linear and Attention-based Late Fusion models performed worse than single-stain baselines.

5-Class Tumor Type Classification

• Single-Stain Performance: H&E ($0.77 \pm 0.05$) outperformed Ki-67 ($0.74 \pm 0.05$), suggesting H&E provides better morphological features for distinguishing tumor types.
• Fusion Performance:
  • Best Model: One-Hidden Layer Late Fusion Learning Model achieved the highest Balanced Accuracy of $0.83 \pm 0.04$ ($p < 0.005$).
  • Fusion strategies generally improved performance over single stains, particularly in distinguishing complex subtypes like Ganglioglioma and HGG.

Interpretability

• Correlation: The Ki-67 attention heatmaps showed moderate to strong positive correlations ($\rho = 0.576 - 0.823$) with Ki-67 LI maps and cell density maps.
• Implication: This validates that the model's attention mechanisms are biologically relevant, focusing on proliferating tumor cells rather than random artifacts.

4. Key Contributions

1. Unregistered Multi-Stain Fusion: Demonstrated that deep learning can effectively fuse unregistered H&E and Ki-67 WSIs without requiring pixel-level alignment, a significant practical advantage for clinical deployment.
2. Foundation Model Integration: Successfully utilized the CONCHv1_5 histology foundation model as a robust feature extractor for pediatric tumors, addressing data scarcity and domain shift issues.
3. Fusion Strategy Comparison: Systematically compared Early, Intermediate, and Late fusion strategies, identifying that Intermediate Concatenation is optimal for binary grading, while Late Fusion with a shallow neural network is best for multi-class tumor typing.
4. Explainability Validation: Quantitatively linked model attention maps to biological proliferation markers (Ki-67 LI), providing evidence that the model learns clinically meaningful features.

5. Significance and Conclusion

• Clinical Impact: The study proves that H&E and Ki-67 provide complementary information. H&E captures morphological structure, while Ki-67 captures proliferation rates. Fusing them improves diagnostic accuracy for both grading and subtyping pediatric brain tumors.
• Resource Efficiency: This approach offers a pathway to improve diagnosis in low-resource settings where molecular profiling is unavailable, by leveraging standard histology and IHC slides with AI.
• Future Directions: The authors suggest that while current fusion methods are effective, future work should explore fine-tuning foundation models specifically on pediatric data and incorporating additional modalities (e.g., molecular data, other IHC stains) to further enhance performance.

In summary, this paper establishes that multi-stain fusion using deep learning is a superior strategy for pediatric brain tumor classification compared to single-stain analysis, offering a robust, interpretable, and clinically viable decision-support tool.