Technical Summary: LONGMOE

Problem Statement

Multimodal clinical learning faces two fundamental, intersecting challenges that existing methods fail to address jointly:

1. Modality Missingness: In real-world clinical cohorts, arbitrary subsets of modalities (e.g., imaging, text, genetic data) are often unavailable at a given patient visit due to heterogeneous acquisition protocols and resource constraints.
2. Longitudinal Dynamics: The diagnostic significance of an observation is inseparable from the patient's evolving disease trajectory. A biomarker's value depends on its position along an individual's trajectory of decline rather than its absolute value at a single static snapshot.

Existing approaches treat these challenges in isolation. Missing-modality frameworks (e.g., Flex-MoE, FuseMoE) typically treat each visit as an independent static snapshot, discarding temporal context. Conversely, longitudinal models (e.g., GRU-based approaches) often assume complete modality availability at every visit and degrade under systematic missingness. No existing model effectively handles the joint challenge of irregular visit spacing, arbitrary modality combinations, and population heterogeneity.

Methodology: LONGMOE Architecture

The authors propose LONGMOE (Longitudinal Mixture-of-Experts), a unified framework designed to jointly address modality missingness and longitudinal dynamics. The architecture processes a patient's longitudinal visit sequence through four sequential modules:

1. Modality-Specific Embeddings

Each modality $m_l$ is processed by a dedicated encoder $\phi_l$ tailored to its structural properties:

• MRI: A 3D convolutional encoder followed by an MLP projection.
• Clinical Scores, Biomarkers, and Genetics: Modality-specific MLPs.
• Temporal Embedding: Visit timestamps are encoded as learned embeddings capturing absolute time and inter-visit intervals.
  Unobserved modalities are initially set to zero vectors.

2. Modality Imputation and Tokenization

This module handles missing data and creates a unified visit token:

• Context-Aware Imputation:
  • Carry-Forward: If a modality was observed in a previous visit, the most recent embedding is replicated (motivated by strong short-term autocorrelation of biomarkers).
  • Contextual Imputation: If no prior observation exists, a learned network conditions on available modalities at the current visit to generate an embedding, using zero-masking to prevent contamination from missing signals.
• Attentional Tokenization:
  • Modality Fusion: Multi-head self-attention is applied over the imputed modality embeddings to learn inter-modality dependencies.
  • Viscode Encoding: A continuous-time multi-frequency sinusoidal positional encoding maps visit timestamps to a fixed-dimensional vector. This encoding is provably injective for non-uniformly sampled sequences, capturing both slow longitudinal trends and rapid rate changes without assuming uniform visit spacing.
  • Fusion: The modality embedding and viscode encoding are fused via element-wise addition to produce the visit token $z_t$.

3. Trajectory-Aware Longitudinal Modeling

A stack of Transformer encoder blocks processes the sequence of visit tokens $\{z_t\}$.

• Causal Masking: A causal attention mask ensures that the representation at visit $t$ depends only on history $\{v_1, \dots, v_t\}$, preventing future information leakage.
• Output: This produces a contextualized embedding $h_t$ that integrates the patient's full disease-progression trajectory, serving as a rich input for the routing module.

4. Context-Aware Sparse MoE Routing

The final layer employs a Sparse Mixture-of-Experts (SMoE) architecture to handle patient-specific heterogeneity and modality patterns:

• Expert Architecture: The layer contains $K=8$ experts partitioned into:
  • Individual Experts: Specialized sub-networks for distinct patient subpopulations.
  • Shared Experts: Accessible to all inputs to capture population-level common patterns and prevent over-specialization.
• Routing: A router network maps the trajectory embedding $h_t$ to a score vector. Hard top-1 routing selects the single most relevant expert (plus the always-activated shared experts).
• Aggregation: Final predictions are obtained via attentional aggregation of expert outputs across all visits, weighted by the router's confidence.
• Regularization: A load-balancing loss prevents router collapse, ensuring uniform expert utilization.

Key Contributions

1. Unified Framework: LONGMOE is the first model to jointly address modality missingness and longitudinal temporal dynamics in clinical multimodal learning.
2. Novel Architectural Components:
   • A context-aware imputation module that leverages both historical data and current context.
   • An attentional tokenization module combining cross-modal attention with a provably injective, multi-frequency temporal encoding for irregular sequences.
   • A trajectory-aware encoder that models disease progression.
   • A context-conditioned SMoE routing layer that dynamically selects experts based on the observed modality combination and patient trajectory.
3. Robustness: The framework demonstrates robustness under missing or weak contemporaneous modalities while remaining competitive in full-modality settings.

Experimental Results

The authors evaluated LONGMOE on three datasets: ADNI (Alzheimer's Disease Neuroimaging Initiative), OASIS-3, and MIMIC-IV.

• Performance: LONGMOE achieved the best AUC-ROC across all modality combinations on ADNI and MIMIC-IV. On OASIS-3, it led in four of seven combinations, particularly in imaging- and genetics-dominant settings where other baselines collapsed.
• Missing Modality Robustness:
  • On ADNI, LONGMOE maintained high performance (AUC $\approx$ 0.94–0.97) even when only a single modality was present, whereas baselines like TF collapsed to random chance (0.50) and others degraded significantly.
  • The trajectory-aware encoder effectively substituted longitudinal context for missing contemporaneous modality information, resulting in near-flat degradation curves as modalities were removed.
• Ablation Studies: Removing the Longitudinal Encoder (LE) or Viscode Encoding (VE) resulted in the largest performance drops, confirming their critical role. The Shared Experts and Load Balancing mechanisms were also shown to be vital, particularly in settings with severe missingness or smaller cohorts.
• Complexity: LONGMOE has higher parameter counts and forward-pass latency compared to static baselines (e.g., 4.4M parameters vs. 183K for Flex-MoE) due to the trajectory-aware Transformer. However, the authors note this is practical for offline batch inference, though it precludes real-time bedside use.

Significance and Claims

The paper claims that LONGMOE establishes a strong foundation for longitudinally-aware multimodal clinical learning. Its primary significance lies in bridging the gap between handling missing data and modeling disease progression. By treating the patient's history as a continuous, irregular trajectory rather than a series of independent snapshots, the model can extract discriminative signals even when current visits lack critical modalities.

The authors explicitly state that while the model is highly effective for offline batch inference and handling complex, incomplete clinical records, the increased computational cost of the trajectory-aware encoder limits its applicability to real-time bedside support. Additionally, the model cannot leverage longitudinal context for first-contact patients with no prior visit history, as the encoder requires at least one past observation to generate meaningful embeddings.