Technical Summary: Modality-Missing Robust Representation Network (MMR-Net) for Multi-Sequence Brain MRI

1. Problem Statement

Multi-sequence brain magnetic resonance imaging (MRI) offers complementary anatomical and physiological information essential for robust brain characterization. However, in both clinical and research settings, missing modalities are frequent due to acquisition constraints, patient motion, or varying study designs. Existing deep learning models typically assume complete input modalities; when even a single sequence is absent, these models suffer substantial performance degradation. Common mitigation strategies, such as zero-padding or mean imputation, often introduce artifacts and spurious correlations, further compromising prediction accuracy. Furthermore, current architectures rarely enforce representational stability across different modality availability scenarios, meaning the feature representation of a subject changes significantly depending on which sequences are present.

2. Methodology

The authors propose the Modality-Missing Robust Representation Network (MMR-Net), a framework designed to maintain stable performance under both complete and incomplete modality conditions using the publicly available IXI dataset. The architecture consists of five primary components:

• Modality-Specific Encoders: A set of 3D convolutional neural networks (CNNs) with shared backbones but modality-specific batch normalization and early layers. These encoders preserve the distinct imaging characteristics of T1-weighted, T2-weighted, proton density (PD), and time-of-flight magnetic resonance angiography (TOF-MRA) sequences. The first two convolutional blocks are modality-specific, while the final two blocks share weights to enforce commonality in high-level feature extraction.
• Modality Tokenization: High-dimensional feature maps from each encoder are partitioned into non-overlapping regions and pooled to generate a compact set of tokens (64 tokens per modality), projected into a common embedding dimension of 256.
• Cross-Modal Token Fusion: A core innovation utilizing an availability-aware attention mechanism. This module dynamically aggregates information based on the set of currently available modalities. It computes queries, keys, and values only from present modalities, preventing missing inputs from contributing to the fusion process. This allows the model to adapt to arbitrary missing patterns without architectural changes at test time.
• Shared Latent Space Projection: A multi-layer perceptron (MLP) maps the fused cross-modal tokens into a 128-dimensional feature vector. This representation is designed to be invariant to modality availability, ensuring that the latent vector for a subject remains consistent whether modalities are missing or complete.
• Task-Specific Prediction Heads: Separate heads handle age regression (linear layer), sex classification (sigmoid activation), and adversarial site debiasing. The debiasing head employs a gradient reversal layer to predict the acquisition center, penalizing the encoder for learning site-specific information to improve cross-center generalizability.

Training Strategy and Objectives:
The network is trained using a modality dropout strategy, where random subsets of modalities are omitted during each epoch to simulate realistic missing scenarios. The optimization objective is a weighted sum of:

1. Age Regression Loss: Smooth L1 loss for robustness against outliers.
2. Sex Classification Loss: Binary cross-entropy.
3. Adversarial Site Loss: Cross-entropy for site classification, reversed during backpropagation.
4. Representation Consistency Loss: Squared Euclidean distance between latent representations of complete and missing-modality passes for the same subject.
5. Prediction Consistency Loss: Absolute difference between age predictions from complete and missing-modality passes.

3. Key Contributions

• Robust Architecture: The introduction of a cross-modal token fusion module with availability-aware attention that dynamically handles arbitrary missing patterns without requiring imputation or architectural modification at inference.
• Consistency Constraints: The explicit enforcement of representational and prediction consistency between complete and missing-modality inputs, ensuring stable feature extraction regardless of data availability.
• Modality Dropout Training: A training strategy that forces the network to learn robust representations by simulating all possible non-empty subsets of modalities during training.
• Adversarial Site Debiasing: Integration of a gradient reversal layer to reduce scanner-specific biases, enhancing generalizability across different acquisition centers.
• Reproducibility: The study utilizes only the public IXI dataset, avoiding new ethical approval requirements and ensuring full reproducibility of methods and results.

4. Experimental Results

The model was evaluated on the IXI dataset (approx. 600 healthy subjects) across four settings: complete modality, single-modality missing, two-modality missing, and single-modality only. Tasks included age regression (primary) and sex classification (auxiliary).

• Complete Modality: MMR-Net achieved a Mean Absolute Error (MAE) of 3.42 years for age regression, outperforming the next best baseline (Transformer Fusion) at 3.78 years.
• Single-Modality Missing: The method demonstrated superior robustness. For example, when T1 was missing, MMR-Net's MAE increased by only 8.5% (to 3.71 years), compared to a 19.6% drop for the Transformer Fusion baseline and a 28.7% drop for the zero-filling baseline.
• Two-Modality Missing: In the most challenging scenario (missing T1 and T2), MMR-Net achieved an MAE of 4.45 years (30.1% relative drop), significantly outperforming the Transformer Fusion baseline (62.0% drop) and the zero-filling baseline (95.2% drop).
• Single-Modality Only: Even when tested with only one modality available, MMR-Net outperformed baselines trained exclusively on that single modality (e.g., 4.18 years MAE with only T1 vs. 4.85 years for the T1-only baseline).
• Cross-Center Generalization: In leave-one-center-out experiments, the adversarial debiasing component reduced the MAE on the held-out center from 4.67 years to 3.98 years (a 14.8% improvement).

Statistical analysis (paired t-tests) confirmed that MMR-Net significantly outperformed all baselines across missing modality settings (p < 0.01), with effect sizes ranging from medium to very large.

5. Significance and Claims

The paper claims that MMR-Net provides a generalizable tool for multi-sequence brain MRI analysis that effectively addresses the critical, often overlooked challenge of missing modalities. By maintaining stable performance under arbitrary missing patterns, the framework offers practical value for clinical settings where acquisition protocols vary or are incomplete.

The authors explicitly state that while the study focuses on healthy brain data from the IXI dataset and does not involve acupuncture interventions, the methodology is relevant to future research on non-pharmacological neuromodulation, including longitudinal multi-sequence follow-up investigations of acupuncture effects. In such studies, where subjects may miss sequences due to acquisition challenges, MMR-Net could enable comprehensive analyses without excluding subjects with incomplete data. The use of public data ensures the work aligns with open science principles and avoids the need for new ethical approvals.

The authors acknowledge limitations, including the restriction to healthy subjects, the underutilization of diffusion-weighted imaging in the main experiments, and the assumption of random missingness rather than non-random missingness correlated with patient characteristics. Future work is proposed to address these by extending validation to disease cohorts and incorporating diffusion metrics.