💻 computer science

Hierarchical MoE for Multi-Modal ILD Diagnosis

This paper proposes a hierarchical multimodal Mixture-of-Experts model that integrates frozen imaging features with structured EHR data via a two-stage gating mechanism to achieve state-of-the-art interstitial lung disease classification (0.8750 AUC) while enhancing interpretability across anatomical regions and clinical feature groups.

Original authors: Alec K. Peltekian, Gorkem Durak, Halil Ertugrul Aktas, Carrie Lynn Richardson, Mary Carns, Kathleen Aren, GR Scott Budinger, Anthony J. Esposito, Alexander Misharin, Alok Nidhi Choudhary, Ankit Agrawa

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alec K. Peltekian, Gorkem Durak, Halil Ertugrul Aktas, Carrie Lynn Richardson, Mary Carns, Kathleen Aren, GR Scott Budinger, Anthony J. Esposito, Alexander Misharin, Alok Nidhi Choudhary, Ankit Agrawal, Ulas Bagci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the complex landscape of modern medicine, diagnosing difficult conditions often requires piecing together clues from different sources. A doctor might look at a detailed image of a patient's lungs, but they also rely on a history of blood tests, breathing measurements, and personal health records. These pieces of information do not always tell the same story, and they do not always carry the same weight for every single person. For some patients, a scan might reveal the problem clearly, while for others, the answer lies in a subtle change in their blood work or a specific symptom recorded years ago. The challenge for computer systems designed to help doctors is to learn how to weigh these different types of information correctly, adapting to the unique needs of each individual rather than applying a single, rigid rule to everyone.

This is the central problem tackled by a team of researchers at Northwestern University, who have developed a new way for artificial intelligence to diagnose interstitial lung disease, a condition that causes scarring and stiffness in the lungs. Instead of forcing the computer to treat every patient the same way, they built a system that mimics a team of specialists. In this digital team, some experts focus solely on the images of the lungs, while others specialize in the patient's medical history. The innovation lies in a "gatekeeper" mechanism that decides, for each specific patient, how much attention to pay to the images versus the medical records. It is a flexible approach that allows the system to listen more closely to the scan when the picture is clear, or to lean heavily on the clinical history when the scan is ambiguous.

The researchers tested this system on data from nearly 600 patients who had been monitored over many years. These patients had undergone hundreds of chest scans and had extensive medical records containing information like blood counts, lung function tests, and demographic details. The team organized the medical records into logical groups, such as "lung function tests" or "blood markers," and assigned a specific digital expert to analyze each group. Similarly, the lung scans were broken down by the five distinct lobes of the lung, with a dedicated expert examining each section. This structure ensured that the system did not just see a blur of pixels or a list of numbers, but rather understood the specific anatomy and the specific clinical context of the disease.

When the researchers put this system to the test, they found that it performed better than other methods that relied only on images or used simpler ways to combine data. The most successful version of their model achieved a high level of accuracy in distinguishing between patients with the disease and those without it. Crucially, the system showed that it could learn to adapt. For patients where the lung scans were very clear, the system relied more on the images. For others, where the scans were less definitive, the system naturally shifted its focus to the clinical data, such as pulmonary function tests, to make its decision. This behavior mirrors how a human doctor works, shifting their attention based on what evidence is most available and reliable for the person in front of them.

The study also revealed that not all medical data is equally useful for every diagnosis. The researchers found that the system performed best when it focused on a specific, carefully chosen set of clinical variables, such as lung function tests and specific disease markers, rather than trying to process every single piece of information in a patient's file. By narrowing its focus to the most relevant clinical clues, the system avoided getting distracted by less important details. This suggests that in the future, artificial intelligence tools for medicine might not need to be massive, all-consuming databases, but rather smart, focused systems that know exactly which pieces of the puzzle to look at for each specific case.

While the results are promising, the researchers are careful to note that this work is a step forward, not a final solution. The system was tested on a specific group of patients with a particular type of lung disease linked to a condition called scleroderma, and it has not yet been proven to work for all types of lung issues or in different hospitals. The team also observed that the system is not perfect; there were still some cases where it made mistakes, particularly when the evidence from the images and the medical records seemed to contradict each other. However, the ability of the system to explain its own reasoning—showing exactly which parts of the scan or which specific medical tests influenced its decision—offers a new level of transparency that is essential for building trust between doctors and the tools they use. This work demonstrates that by respecting the unique structure of medical data and allowing the computer to choose its own path of reasoning, we can create diagnostic tools that are not only accurate but also understandable.

Technical Summary: Hierarchical MoE for Multi-Modal ILD Diagnosis

Problem Statement
The paper addresses the challenge of Interstitial Lung Disease (ILD) classification, specifically within the context of systemic sclerosis. Current deep learning approaches often treat the lung as a single global volume, diluting region-specific pathological signals and lacking interpretability. Furthermore, existing Mixture-of-Experts (MoE) architectures typically assume homogeneous expert capacity and domain-agnostic routing, which fails to account for the strong anatomical constraints of medical imaging and the heterogeneous, patient-specific relevance of clinical data.

The authors frame multimodal ILD classification as a structured conditional computation problem. They argue that the optimal predictive function depends on patient-specific modality relevance, where fixed fusion strategies (e.g., simple concatenation) assume uniform utility across modalities, leading to suboptimal bias. The goal is to develop a model that adaptively weights imaging and Electronic Health Record (EHR) data while preserving the interpretability required for clinical deployment.

Methodology
The proposed solution is a Hierarchical Multimodal MoE architecture featuring a two-stage gating mechanism:

Imaging Branch (Lobe-Aware Experts):
- Architecture: The imaging branch consists of five lobe-specific experts (Left Upper, Left Lower, Right Upper, Right Middle, Right Lower), each implemented as a 3D SwinUNETR backbone.
- Preprocessing: CT volumes are segmented into five anatomical lobes using LungMask R231.
- Routing Strategy: Instead of learning lobe importance dynamically, the model uses radiomics-guided fixed weights. For each cross-validation fold, 107 PyRadiomics features are extracted per lobe, and XGBoost classifiers determine lobe-specific AUCs. These AUCs are mapped to fixed attention weights (Eq. 2) to aggregate the expert representations. This decouples anatomical importance estimation from gradient-based optimization, acting as a structured inductive bias.
- Training: The imaging expert is trained independently on CT data and frozen during the multimodal training phase to preserve lobe-specific specialization.
EHR Branch (Hierarchical Sub-Gating):
- Feature Grouping: Structured EHR variables are partitioned into clinically coherent groups. The authors evaluate two configurations:
  - Full Hierarchy: 7 groups (69 variables) including CBC, Pulmonary Function Tests (PFTs), Serum Chemistries, etc.
  - Selective Hierarchy: 3 groups (12 variables) focusing on PFTs, Disease-Specific Demographics (e.g., scleroderma subtype, autoantibodies), and Key Biomarkers.
- Architecture: Each group is processed by a dedicated sub-expert (MLP with residual projection).
- Routing: A learned sub-gating network ( $\pi_g$ ) dynamically aggregates the outputs of these group-specific sub-experts. This reduces the effective interaction dimensionality from $O(|X_E|^2)$ to $O(G)$ , controlling variance while maintaining conditional flexibility.
Modality-Level Gating:
- A final modality gate ( $g_{mod}$ ) adaptively balances the frozen imaging representation ( $\phi_{img}$ ) and the learned EHR representation ( $\phi_{ehr}$ ).
- The fused representation is passed to a lightweight MLP classifier for binary ILD prediction.
Training Protocol:
- The model is trained in three stages: (1) radiomics-based lobe importance estimation, (2) imaging expert training, and (3) multimodal training with the imaging expert frozen.
- Evaluation uses strict patient-level 5-fold cross-validation, ensuring all longitudinal CT scans for a single patient remain in the same fold to prevent data leakage.

Key Results

Performance: The Selective Hierarchical MoE achieved the highest mean AUC of 0.8750 ± 0.0443.
Comparisons:
- Outperformed the imaging-only SwinUNETR baseline (0.7685) significantly ( $p < 0.001$ ).
- Outperformed the imaging-only Radiomics-Guided REN (0.8646) with an absolute increase of 0.0104. However, the authors note this specific difference was not statistically significant ( $p = 0.804$ ) in their paired t-test, suggesting the improvement requires confirmation in larger evaluations.
- Outperformed the Full Hierarchy MoE (0.8496), indicating that restricting the EHR branch to clinically proximal variables (PFTs, demographics, biomarkers) yields better performance than including all 69 variables.
Robustness: The model showed negligible performance changes when varying the temporal linkage window for EHR-CT matching (0 to 90 days).
Interpretability:
- Modality Routing: Imaging experts received higher weights for ILD-negative predictions (suggesting clear parenchyma is sufficient to exclude disease), while EHR experts were weighted more heavily for ILD-positive predictions, particularly at higher confidence levels.
- Sub-Expert Routing: In the selective hierarchy, routing concentrated heavily on the PFT sub-expert, followed by disease demographics and biomarkers.
- Error Analysis: Misclassified cases (17.4% of test set) exhibited lower EHR reliance compared to correct predictions, suggesting high-confidence errors were often driven by imaging-dominant routing in clinically ambiguous cases.

Significance and Claims
The paper claims to present a framework that extends interpretability across anatomical regions, imaging-EHR utilization, and clinically defined feature groups. The primary significance lies in the structured conditional computation approach, which:

Preserves Stable Representations: By freezing the imaging expert, the model maintains stable, lobe-specific imaging features while allowing the EHR branch and gating networks to adapt to patient-specific clinical contexts.
Enables Explicit Specialization: The hierarchical gating allows the model to decompose EHR data into clinically meaningful groups, avoiding the "dense entanglement" of variables often seen in standard fusion strategies.
Provides Case-Specific Summaries: The learned gate activations offer a mechanism to generate case-level summaries of how the model utilized anatomical regions, modalities, and specific clinical feature groups.

The authors remain modest regarding the statistical significance of the improvement over the imaging-only REN, acknowledging that the numerical gain (0.0104 AUC) needs validation in larger cohorts. They also note limitations, including the single-institution origin of the data (Northwestern Scleroderma Registry), the reliance on manually defined (rather than learned) EHR groupings, and the lack of explicit missingness indicators in the EHR preprocessing. The framework is proposed as a generalizable approach for multimodal diagnostic tasks involving structured imaging and clinical data, though it was specifically developed and validated for systemic sclerosis-associated ILD.

Technical Summary: Hierarchical MoE for Multi-Modal ILD Diagnosis

More like this