Technical Summary: Guarantees That Survive a Missing Scan

Problem Statement

Multimodal medical diagnosis increasingly relies on fusing medical imaging with structured tabular data (e.g., labs, demographics). In real-world deployment, these models frequently encounter incomplete inputs where entire modalities are missing (e.g., a patient arrives without a scan or without lab results). While Conformal Prediction (CP) is the standard method for providing distribution-free, finite-sample coverage guarantees for such predictors, its validity relies on the assumption of exchangeability between calibration and test data.

The paper identifies a critical failure mode: when an entire modality is absent at test time, the exchangeability assumption breaks. If calibration is performed on modality-complete patients but tested on modality-incomplete patients, the non-conformity scores are no longer exchangeable. Consequently, standard marginal split CP silently loses conditional validity on the modality-incomplete subgroup. The paper demonstrates that this failure manifests in two distinct, opposing ways depending on the dataset and training regime:

1. Valid but Uninformative: The method satisfies the coverage target (e.g., 90%) only by outputting prediction sets that contain all possible labels (degenerate sets), offering no actionable clinical insight.
2. Informative but Invalid: The method produces small, confident prediction sets but fails to meet the coverage target, missing the true diagnosis significantly more often than the error rate allows.

Methodology: Modality-Conditional Conformal Prediction (MC2)

To address this, the authors introduce Modality-Conditional Conformal Prediction (MC2). This approach modifies the calibration process to account for missing modalities without requiring retraining of the underlying predictor.

• Mondrian Calibration: MC2 treats the modality-availability pattern (e.g., "image+tabular," "image-only," "tabular-only") as a grouping function. Instead of computing a single global threshold for the entire dataset, MC2 stratifies the calibration set by these patterns.
• Per-Pattern Thresholds: For each modality pattern $m$, a separate threshold $\hat{q}_m$ is computed as the $(n_m + 1)(1-\alpha)/n_m$-quantile of the non-conformity scores for that specific subgroup.
• Implementation Details:
  • The predictor $f$ is trained to handle incomplete inputs using a learnable mask token and modality dropout during training.
  • The image encoder is frozen to isolate the contribution to the calibration layer and ensure feasibility on commodity hardware.
  • If a specific modality pattern has insufficient calibration data (below a threshold $\tau$), the method falls back to the pooled (marginal) threshold, explicitly reporting this fallback.
  • The method uses standard non-conformity scores, primarily LAC (Least Ambiguous set-valued Classifier) ($s = 1 - p_y$), though Adaptive Prediction Sets (APS) are also considered.

Key Contributions

1. Formalization of Conditional Validity Loss: The paper formally demonstrates that standard split CP loses conditional validity under whole-modality absence and empirically characterizes the "two faces" of this failure (valid-but-uninformative vs. informative-but-invalid).
2. Introduction of MC2: The authors propose MC2, a Mondrian calibration strategy that stratifies by modality-availability patterns. This restores valid per-pattern coverage with finite-sample guarantees, requiring no changes to the trained predictor.
3. Mechanism Identification: Through controlled experiments, the paper identifies class prevalence (not modality redundancy) as the primary driver determining which "face" of failure appears. It shows that low prevalence leads to under-coverage, while high prevalence leads to degenerate sets.

Experimental Results

The method was evaluated on two public image+tabular datasets:

• CBIS-DDSM: Mammography ROIs with tabular metadata (Binary: Benign/Malignant; Prevalence ~40.8%).
• OL3I (Stanford AIMI): Single L3 CT slices with clinical risk factors (Binary: 1-year Ischemic Heart Disease; Prevalence ~4.4%).

Findings:

• CBIS-DDSM (High Prevalence): Marginal CP was valid but uninformative. At 200 epochs, it achieved 100% coverage on the image-missing subgroup but output the full label set for every case (0% actionable rate). MC2 restored the coverage to ~90% while doubling the actionable rate (from ~13% to ~26% in early stopping, and recovering from 0% in longer training).
• OL3I (Low Prevalence): Marginal CP was informative but invalid. At 200 epochs, it achieved a high actionable rate (~89%) but violated coverage, dropping to ~73% (target 90%). MC2 restored coverage to ~91% at the cost of a lower, but trustworthy, actionable rate.
• Synthetic Multi-Class Study: A controlled study with $K=6$ classes confirmed that marginal CP under-covers the image-missing subgroup severely (~45% coverage vs. 90% target), while MC2 maintains the target.
• Prevalence Swap: By artificially swapping the prevalence rates between the datasets (keeping modality structures fixed), the authors showed the failure mode "flips." Reducing CBIS prevalence to 4.4% induced under-coverage, while increasing OL3I prevalence to 40% induced degeneracy. This confirms prevalence drives the threshold mismatch.

Significance and Claims

The paper claims that MC2 provides a simple, post-hoc, and computationally cheap fix for a failure mode ubiquitous in multimodal clinical deployment. Its significance lies in:

• Restoring Trust: It ensures that the uncertainty quantification remains valid specifically for the patients with degraded inputs (missing scans), who are often the most vulnerable and in need of reliable decision support.
• Preserving Utility: It recovers the trade-off between validity (coverage) and informativeness (actionable set size), preventing the model from either lying about its confidence or being useless.
• No Retraining Required: The method operates as a calibration layer on top of existing predictors, making it immediately applicable to deployed systems.

The authors remain modest regarding limitations, noting that the results rely on a frozen encoder, the datasets are binary (with multi-class results shown on synthetic data), and the mechanism relies on the assumption that missingness is conditionally at random within each pattern. They also note that the performance of MC2 with APS scores can be regime-dependent and less robust than with LAC scores.