💻 computer science

DS@GT ARC at MEDIQA-CORE-Task-1 2026: Trimodal Model Fusion with Task-Specific Gates for Brain Tumor Subtype Classification

The DS@GT ARC team presents a trimodal model fusion approach for the MEDIQA-CORE 2026 Brain Tumor Subtype Classification task that integrates MRI, histopathology, and radiology report embeddings to achieve a second-place ranking with a macro-F1 score of 0.801, demonstrating strong performance when all modalities are available but highlighting a dependency on histopathology data.

Original authors: Hoang Thanh Thanh Truong, Charles R. Clark

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hoang Thanh Thanh Truong, Charles R. Clark

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the clues are scattered across three different worlds. One world is a series of high-tech photos (MRI scans) showing the inside of a patient's brain. Another world is a microscopic view of the actual tissue (histopathology), like looking at the bricks of a building under a microscope. The third world is a written story (the radiology report) where a doctor describes what they see in plain language. In the real world, getting all these clues together takes time—sometimes weeks. For a patient with a brain tumor, every day of waiting is a day the tumor can grow, potentially doubling in size. Scientists are trying to build a "super-detective" computer that can look at whatever clues are available right now and instantly guess what kind of tumor it is, helping doctors start treatment faster. This paper is about a team of researchers who built such a computer and tested how well it works when they give it different combinations of these clues.

The team, known as DS@GT ARC, entered a competition called MEDIQA-CORE 2026 to see if their computer could correctly identify three different things about brain tumors: the specific molecular type, whether it is low-grade (slower growing) or high-grade (faster growing), and its official medical grade. They fed their computer three types of data: the MRI photos, the tissue slides, and the written reports. Instead of just mashing all this information into one giant pile, they tried a clever trick. They built two different versions of their computer brain. The first version tried to blend all three clues together into one shared understanding. The second version, which turned out to be their strongest system, gave each of the three classification tasks its own special "gatekeeper." Think of it like a restaurant with three different chefs. Instead of one head chef deciding how much salt, pepper, and garlic to put in every single dish, each chef (one for molecular type, one for low-vs-high grade, and one for the official grade) has their own hand on the spice rack. They decide for themselves how much of the MRI, the tissue slide, or the written report they need to make the perfect dish.

The results showed that this "specialized chef" approach was very effective. When the computer had access to all three types of clues, it achieved a score of 0.801, beating the competition's standard baseline of 0.796 and ranking second among the teams whose code was verified. The biggest win was in identifying the specific molecular type of the tumor, where their system scored 0.867 compared to the baseline's 0.672. However, the paper reveals a crucial twist: this success was heavily dependent on having the tissue slide (histopathology) data. When the researchers simulated a situation where the tissue slides were missing, the computer's performance dropped significantly, falling behind the baseline system. This suggests that while the computer learned to use the written reports well, it became so reliant on the tissue slides that it struggled when they weren't there. Interestingly, when the computer was tested on a group of patients who never had tissue slides collected at all, it performed much worse than the baseline, indicating that its "specialized gates" had learned to lean on the tissue data so heavily that it couldn't adapt to a world without it.

The paper concludes that while giving each task its own way of mixing the clues works better than forcing them to share a single mix, the system still needs to learn how to solve the mystery without the most important clue (the tissue slides). The authors suggest that in the future, they might need to train the computer to be less dependent on the tissue slides, perhaps by practicing more often with cases where those slides are missing, so it can become a truly robust detective for every patient, not just the lucky ones who have all three types of clues available.

Technical Summary: DS@GT ARC at MEDIQA-CORE-Task-1 2026

Problem Statement

The paper addresses the critical need to reduce diagnostic delays for brain tumor patients, where the time between symptom onset and definitive pathology reports can extend to weeks. During this interval, aggressive tumors like glioblastoma can grow significantly. The authors tackle the MEDIQA-CORE 2026 Task 1: Brain Tumor Subtype Classification, which requires automated systems to classify glioma subtypes using a trimodal dataset comprising:

MRI embeddings (pre-extracted via NeuroVFM).
Histopathology embeddings (pre-extracted via Prov-GigaPath from whole-slide images).
Free-text radiology reports.

The system must predict three distinct classification targets simultaneously:

Level-1 Molecular Type (3 classes).
LGG vs. HGG (Low-Grade vs. High-Grade glioma; binary).
WHO Grade (3 classes aligned with the 2021 WHO CNS classification).

A key challenge is modality robustness: the system must perform effectively even when one or more modalities are missing, as not all patients have complete data (e.g., some lack histopathology entirely).

Methodology

The DS@GT ARC team proposed a trimodal fusion framework that integrates pre-extracted embeddings with free-text report encodings. They explored two primary architectures, with the Task-Specific Gates model serving as their primary submission.

1. Data Preprocessing and Encoders

MRI & Histopathology: Utilized the 768-dimensional embeddings provided by the challenge organizers.
Radiology Reports: The team experimented with two encoders:
- RadBERT: Used in the cross-modal self-attentive model.
- Llama-3.1-8B-Instruct: Used in the primary task-specific gating model. Reports were passed through as frozen feature extractors, with final-layer hidden states mean-pooled to create 4096-dimensional vectors.
Subject Filtering: Subjects with the "Other/NEC" molecular subtype were excluded from training to improve macro-F1 performance.

2. Fusion Architectures

The team compared two strategies for combining the three modalities:

Cross-Modal Self-Attentive Fusion:
- A shared transformer trunk processes the three modality latents ( $z_{MRI}, z_{Histo}, z_{Report}$ ) via cross-modal self-attention.
- A single shared learned gate produces a fused representation used by all three classification heads.
- This model uses a shared projection layer and parallel linear heads for the three tasks.
Trimodal Fusion with Task-Specific Gates (Primary System):
- Instead of a shared representation, this architecture maintains three independent learned gates, one for each classification task.
- Each gate takes the concatenation of the three modality latents and outputs softmax weights, determining how much each modality contributes to that specific task's fused representation.
- Missing Modality Handling: If a modality is missing for a subject, its corresponding gate weight is forced to zero, preventing it from contributing to the fusion.
- This design allows each task to learn its own optimal modality mixing strategy rather than forcing a single fusion strategy across all biological questions.

3. Training and Post-Processing

Training: The models were trained using 5-fold stratified cross-validation with logit averaging for ensembling.
Regularization: Modality dropout was applied during training (50% probability per modality per sample) to simulate missing data and improve robustness.
Loss Function: A sum of class-weighted cross-entropy losses with label smoothing ( $\epsilon=0.1$ ).
Biologically Motivated Post-Processing: A rule-based stage was applied to the ensemble's averaged probabilities to enforce known biological constraints:
- If Level-1 predicts "Astrocytoma" with high confidence, the LGG/HGG prediction is overridden to "LGG."
- If LGG is predicted with high confidence, the WHO Grade is forced to 0.
- If HGG is predicted with high confidence, the WHO Grade is forced to 1 or 2 (masking Grade 0).

Key Results

The system was evaluated on two test sets: Test Set 1 (36 subjects with all modalities, used for ablation studies) and Test Set 2 (18 subjects with MRI only, used to test generalization to populations without histopathology).

Performance on Test Set 1 (Fully Multimodal)

Overall: The Task-Specific Gates model achieved a mean macro-F1 of 0.801 in the post-verification stage, ranking 2nd among verified teams and exceeding the organizers' baseline (0.796). In the pre-verification stage, the system ranked 4th of 5 teams.
Task Breakdown:
- Level-1 Molecular Type: The system significantly outperformed the baseline (0.867 vs. 0.672), demonstrating the value of the task-specific gating and report integration for molecular subtyping.
- Grade-Based Tasks (LGG vs. HGG, WHO Grade): The baseline outperformed the proposed system on these tasks (e.g., 0.920 vs. 0.800 for LGG vs. HGG). The authors note that their fusion strategy and added report modality helped molecular subtyping more than tumor grading.

Modality Ablation and Robustness

Dependence on Histopathology: The system's advantage over the baseline was entirely conditional on the availability of histopathology.
- When Pathology was dropped, the system's mean macro-F1 dropped to 0.642, falling below the baseline (0.744).
- When Reports were dropped, the system still outperformed the baseline (0.720 vs. 0.512), indicating that Pathology is the dominant signal for their model, not the added text modality.
Test Set 2 (Histopathology-Absent Cohort):
- In the absence of histopathology, the system performed significantly worse than the baseline across all conditions (Mean F1: 0.493 vs. 0.797 in MRI+Reports).
- This suggests the task-specific gates learned to rely heavily on histopathology-correlated patterns that do not transfer to a population where histopathology was never collected.

Significance and Claims

The paper claims that task-specific gating is a superior strategy for multi-task medical classification compared to shared fusion representations. By allowing each biological task (molecular type vs. grade) to independently weigh the importance of MRI, pathology, and text, the model achieved second place among verified teams and a higher mean macro-F1 than the baseline on the overall task, driven primarily by a large gain on the Level-1 Molecular Type task.

However, the authors are modest about the system's generalizability. They explicitly acknowledge that:

The performance gain is driven primarily by the histopathology modality, not the added radiology reports.
The system fails to generalize to cohorts lacking histopathology, performing worse than a baseline that never used pathology.
The current approach does not yet solve the problem of robust classification when the most informative modality (histopathology) is missing, a critical limitation for real-world deployment where such data may be unavailable.

The work demonstrates that while multimodal fusion can significantly improve molecular subtyping, the "one-size-fits-all" fusion approach is insufficient, and the reliance on specific modalities must be carefully managed to ensure robustness across diverse patient populations.