Handling and Interpreting Missing Modalities in Patient Clinical Trajectories via Autoregressive Sequence Modeling

1. Problem Statement

Clinical data is inherently temporal, multimodal, and sparse. Patients present with irregular information availability (e.g., missing lab results, absent imaging, or incomplete notes) during their stay. Current multimodal machine learning (ML) approaches face two critical limitations:

1. Static Architectures: Most models treat patient data as a "bag-of-features" (concatenating all data into a single vector), ignoring the sequential nature of clinical reasoning and preventing clinicians from auditing the evolution of diagnostic beliefs.
2. Handling Missingness: Existing methods for missing modalities typically rely on imputation (hallucinating data) or zero-padding/sample-dropping. Imputation risks introducing false clinical signals, while dropping data destroys the latent space geometry, leading to unsafe diagnostic models that may fail unpredictably when data is missing.

The authors aim to bridge the gap between clinical reality and ML methodology by developing a framework that handles missing modalities robustly while maintaining model explainability and transparency.

2. Methodology

The proposed framework consists of two main stages: Missingness-Aware Contrastive Pre-training and Autoregressive Sequence Modeling.

A. Missingness-Aware Contrastive Pre-training

The authors introduce a novel pre-training objective called "Masked Global Alignment" to learn robust latent representations from sparse, multimodal data.

• Learnable Missing Tokens: Instead of zero-padding, the model uses a set of independently learnable tokens $\{t_i\}$ for each modality. If a modality is missing, the learnable token is injected directly into the latent space.
• Modality Dropout: During pre-training, modalities are randomly dropped (probability $p=0.15$) to simulate real-world sparsity and force the model to learn robust representations.
• Unified Masked Centroid Alignment Loss:
  • Positive Target: For a given modality $i$, the positive target is the Complementary Centroid ($c_{i,k}$), which is the aggregated, normalized representation of all other available modalities for that patient.
  • Negative Targets: Representations of other patients in the batch.
  • Fallback Mechanism: If a patient has only one modality, a "Stable Global Representation" is used to ensure they can still serve as valid negative examples without destabilizing the loss.
  • Objective: The InfoNCE loss forces the embedding of a single modality to align with the context of the patient's other modalities, creating a modality-invariant latent space.

B. Autoregressive Sequence Modeling

The authors re-frame clinical diagnosis as an autoregressive sequence modeling task, utilizing causal decoders from Large Language Models (LLMs).

• Architecture: They fine-tune six pre-trained LLM backbones (e.g., Llama-3, Mistral, DeepSeek, BioMistral, Meditron).
• Input Processing: Multimodal events are chronologically sorted. A special [PREDICT] token is injected after every event, forcing the model to update its "belief state" and output a prediction at every time step as new evidence arrives.
• Fusion: Modality-specific encoders (e.g., ClinicalBERT for text, CNN for X-rays, LSTM for time-series) generate embeddings that are projected into the LLM's hidden dimension.

C. Interpretability Analysis

To ensure safety, the authors perform mechanistic interpretability by analyzing the self-attention matrices of the transformer. They trace how the model routes attention when specific modalities are removed (ablation studies) to identify if the model relies on valid clinical signals or falls back to spurious correlations (e.g., demographics).

3. Key Contributions

1. Novel Hybrid Contrastive Loss: Introduction of "Masked Global Alignment," which learns explicit missingness tokens and aligns incomplete multimodal states into a global latent representation, benefiting datasets with high missingness rates.
2. Sequential Modeling Framework: Fusion of contrastive pre-training with transformer-based autoregressive architectures to model patient trajectories, demonstrating superior performance over static baselines (MLP, LSTM).
3. Mechanistic Safety Evidence: Demonstration that contrastive pre-training prevents "safety failures" (such as attention collapse onto demographic data) under missing modality conditions, providing a transparent view of the model's reasoning process.

4. Experimental Results

The framework was evaluated on MIMIC-IV (paired with MIMIC-CXR) and eICU datasets across three tasks: Phenotyping, In-Hospital Mortality, and Length of Stay (LOS).

• Performance Gains:
  • MIMIC-IV (High Sparsity): Sequential models with contrastive pre-training significantly outperformed static baselines. For example, on the Mortality task, the DeepSeek-LLM-7B model improved its AUROC from 0.5720 (scratch) to 0.8409 (contrastive).
  • eICU (Lower Sparsity): Sequential models generally outperformed static models. However, in this denser dataset, training from scratch sometimes outperformed contrastive pre-training, suggesting the benefit of contrastive learning is most pronounced when modality missingness is high.
• Latent Space Quality:
  • Silhouette Scores: Low scores (e.g., 0.009) indicated successful integration of modalities into a shared space.
  • Cross-Modal Retrieval: Massive improvements over random baselines (e.g., 600x improvement in R@1 for eICU), confirming that embeddings from different modalities (e.g., X-ray vs. Notes) are semantically aligned for the same patient.
• Interpretability & Safety (Ablation Studies):
  • Non-Contrastive Models: When a modality (e.g., Radiology Notes) was removed, these models exhibited attention collapse, shifting focus entirely to demographic tokens (an "attention sink"), leading to incorrect predictions based on demographic priors rather than physiology.
  • Contrastive Models: Under the same ablation, these models maintained distributed attention across the remaining clinical history, preserving predictive accuracy and avoiding demographic bias.

5. Significance and Conclusion

This work addresses a critical bottleneck in clinical AI: the reliability and transparency of models when data is incomplete.

• Safety: By preventing models from defaulting to demographic proxies when clinical data is missing, the proposed method mitigates a major source of bias and unsafe decision-making.
• Explainability: The autoregressive approach allows clinicians to audit the sequence of evidence and belief updates, moving beyond "black box" static predictions.
• Framework: The combination of missingness-aware contrastive pre-training and autoregressive sequence modeling offers a robust pathway for deploying safe, transparent, and high-performing multimodal AI in healthcare.

The authors conclude that while static models are limited by performance ceilings in sparse settings and lack interpretability, their proposed framework enables explicit auditing of diagnostic reasoning, making it a viable candidate for real-world clinical deployment.