CoRe-BT: A Multimodal Radiology-Pathology-Text Benchmark for Robust Brain Tumor Typing

The paper introduces CoRe-BT, a multimodal benchmark comprising 310 patients with MRI, histopathology, and pathology reports, designed to evaluate robust brain tumor typing under realistic conditions of missing clinical data.

The Gist

Imagine you are trying to identify a specific type of intruder in a house. You have three different ways to gather clues:

1. The Security Cameras (MRI): These show you the layout of the house and where the intruder is hiding, but they can't tell you exactly who the intruder is or what their personality is like.
2. The Fingerprint Analysis (Pathology): This gives you a microscopic look at the intruder's DNA and habits, which is very precise, but you can only get this after you've caught them and sent them to the lab.
3. The Detective's Notebook (Text Reports): This is the written story of what happened, combining the camera footage and the lab results into a final conclusion.

In the real world, doctors often have to make life-or-death decisions about brain tumors using only the "Security Cameras" (MRIs) because the "Fingerprint Analysis" (biopsy results) takes days or weeks to arrive. This is risky because the cameras alone might not be enough to tell the difference between a dangerous tumor and a less dangerous one.

Enter CoRe-BT: The Ultimate Detective Training Ground

The paper introduces CoRe-BT, a new "training gym" for artificial intelligence (AI) designed to solve this exact problem. It's a massive dataset that combines all three types of clues (MRI scans, microscope slides, and doctor's notes) for 310 brain tumor patients.

Here is the simple breakdown of what they did and why it matters:

1. The Problem: The "Missing Clue" Scenario

Usually, AI models are trained like students who only study one subject. Some only look at X-rays; others only read lab reports. But in a real hospital, a doctor might have an X-ray today, but the lab report won't be ready until next week.

• The Goal: Create an AI that is smart enough to use all clues when they are available, but also smart enough to make a good guess using only the X-ray if the other clues are missing. It needs to be "robust," meaning it doesn't crash or get confused when information is incomplete.

2. The Solution: A "Super-Brain" AI

The researchers built a system called CoRe-BT-Fusion. Think of this AI as a detective who has two specialized assistants:

• Assistant A (The Radiologist): An AI trained on millions of MRI scans to understand the shape and size of the tumor.
• Assistant B (The Pathologist): An AI trained on millions of microscope slides to understand the cellular details of the tumor.

Instead of letting these assistants work separately, the researchers built a Fusion Module. This is like a team captain who listens to both assistants.

• If the Pathologist's report is ready, the captain weighs that heavily.
• If the report is missing, the captain knows to rely more on the Radiologist's visual clues, using what it learned from working with the Pathologist to make a better guess.

3. The "Grading" System

Brain tumors aren't just "good" or "bad." They are like a complex family tree. The researchers organized the tumors into a clear hierarchy:

• Level 1 (The Big Categories): Is it a Glioblastoma? Is it an Astrocytoma?
• Level 2 (The Subtypes): Which specific version of that tumor is it?
  The AI was trained to recognize these specific "family members" to help doctors choose the right treatment.

4. The Results: Teamwork Makes the Dream Work

When they tested the AI, they found some fascinating things:

• The Power of Combination: When the AI had both the MRI and the Pathology report, it was the most accurate at identifying the specific type of tumor. It was like having a full puzzle instead of just half of it.
• The "Missing Piece" Magic: Even when the AI was forced to guess without the Pathology report, it performed surprisingly well. Why? Because during training, it learned how the two types of clues relate to each other. It learned to "read between the lines" of the MRI scan using the knowledge it gained from the pathology slides.
• The Surprise: In some cases, removing the MRI data actually made the AI better at a specific task (distinguishing low-grade vs. high-grade tumors). This suggests that sometimes, the microscopic details are so powerful that they can override the visual noise of the MRI.

Why This Matters

Before this, most AI research assumed doctors would always have all the data ready at once. CoRe-BT changes the game by simulating the messy reality of real-life medicine.

It proves that we can build AI systems that are flexible. They can act as a "second opinion" that helps doctors make better decisions even when the full picture isn't available yet. This could lead to faster diagnoses, more accurate treatments, and ultimately, better outcomes for patients with brain tumors.

In short: CoRe-BT is teaching AI to be a detective that never gives up, even when some of the clues are missing, by learning how to connect the dots between different types of medical evidence.

Technical Summary

1. Problem Definition

The paper addresses the challenge of brain tumor typing (specifically gliomas), a critical task in neuro-oncology that informs treatment and prognosis. Current diagnostic workflows rely on integrating heterogeneous data sources:

• Radiology: Pre-operative MRI (T1, T1c, T2, FLAIR).
• Pathology: H&E-stained whole-slide images (WSI) and free-text diagnostic reports.
• Clinical Reality: These modalities are rarely available simultaneously. Imaging is often available at presentation, while histopathology and finalized reports are delayed or incomplete.

The Gap: Existing benchmarks (e.g., BraTS, TCGA) are either unimodal (MRI only) or assume complete modality availability at inference time. There is a lack of standardized, clinically grounded benchmarks that evaluate robust multimodal learning under conditions of missing modalities and fine-grained, hierarchical tumor classification.

2. Methodology

A. The CoRe-BT Dataset

The authors introduce CoRe-BT, a new benchmark comprising 310 patients from the University of Washington.

• Modalities:
  • MRI: Multi-sequence (T1, T1c, T2, FLAIR) with expert-annotated tumor masks (edema, enhancing tumor, necrotic core).
  • WSI: 597 gigapixel H&E-stained whole-slide images (avg. >5 slides/patient).
  • Text: Free-text neuropathology reports.
• Labeling: A pathologist-validated, two-level hierarchical classification schema:
  • Level 1 (Coarse): 6 clinically relevant classes (e.g., GBM/IDH-wt HGG, IDH-mutant Astrocytoma, Oligodendroglioma, etc.).
  • Level 2 (Fine): Specific subtypes (e.g., Pilocytic, BRAF-fusion).
• Data Split: The dataset includes cases with full MRI/pathology pairs ($n=95$) and cases with incomplete modalities, designed to test robustness.

B. Model Architecture: CoRe-BT-Fusion

The proposed framework is a dual foundation model designed for variable modality availability.

1. MRI Embedding: Uses NeuroVFM, a volumetric neuroimaging foundation model trained on 5.24M scans. It employs a Joint-Embedding Predictive Architecture (JEPA) to generate subject-level embeddings from 3D volumes.
2. Histopathology Embedding: Uses Prov-GigaPath, a whole-slide foundation model trained on 1.3B tiles. It processes gigapixel WSIs via tiling, DINO-v2 encoding, and a LongNet-adapted Vision Transformer for ultra-long context modeling.
3. Fusion Mechanism:
   • Linear Probes: Pre-trained linear classifiers are trained on each modality's foundation embeddings.
   • Dynamic Weighting: The fusion module combines embeddings using instance-specific weights ($w_i$) to determine the contribution of each modality.
   • Residual Gating: A learnable scalar gate $\sigma(\alpha)$ weights a residual branch ($\Phi$). The residual branch is zero-initialized, ensuring the model starts as an ensemble of modality experts and learns to improve upon this baseline.
   • Handling Missing Data: The architecture is designed to operate even when one modality is missing (e.g., $m_{WSI} = 0$), allowing the model to rely on available inputs.

3. Key Contributions

1. CoRe-BT Benchmark: The first clinically grounded benchmark integrating MRI, WSI, and text for glioma typing, explicitly supporting variable modality availability to reflect real-world workflows.
2. Hierarchical Labeling: A pathologist-validated schema enabling both coarse-grained (WHO Grade) and fine-grained (molecular subtype) evaluation.
3. Robust Evaluation Framework: Systematic assessment of multimodal models in missing-modality settings, moving beyond the assumption of complete data availability.
4. Baseline Performance: Standardized baselines demonstrating the feasibility of fusing large-scale foundation models for cross-modal reasoning.

4. Experimental Results

The authors evaluated the model on three tasks: WHO Grade prediction (3-class), LGG vs. HGG (binary), and Level-1 Tumor Grouping (4-class).

• LGG vs. HGG (Binary):
  • MRI-only and Pathology-only baselines achieved 81.2% Macro Accuracy.
  • The Fusion model achieved 79.2%.
  • Insight: Interestingly, the Fusion model with Pathology ablated (relying only on MRI) achieved 83.3%, suggesting the multimodal training improved the model's ability to interpret MRI features even when pathology was missing.
• Level-1 Tumor Classification (Fine-grained):
  • MRI-only and Pathology-only baselines struggled (~47% accuracy).
  • The Fusion model significantly outperformed unimodal baselines, achieving 55.6% Macro Accuracy.
  • Insight: Multimodal integration provides the most significant benefit for granular subtype classification where complementary information is crucial.
• WHO Grade:
  • Fusion models generally matched or slightly exceeded unimodal baselines (60% vs 56-60%), with the Pathology-ablated fusion performing best (66.7%).

5. Significance and Conclusion

• Clinical Relevance: CoRe-BT bridges the gap between controlled research challenges and real-world clinical practice, where data is often incomplete.
• Multimodal Synergy: The results demonstrate that while unimodal models are strong for binary tasks, multimodal fusion is essential for fine-grained tumor typing. The complementary nature of radiologic (macroscopic) and histopathologic (microscopic) features improves predictive performance for complex subtypes.
• Foundation for Future AI: By providing a testbed for robust learning under missing data, CoRe-BT facilitates the development of AI systems that can assist clinicians at various stages of the diagnostic pathway, not just when all data is present.

In summary, CoRe-BT establishes a new standard for evaluating multimodal brain tumor AI, proving that integrating radiology and pathology foundation models yields superior performance for complex, fine-grained diagnostic tasks, even when data availability is variable.