📊 statistics

Conformal Calibration for Multi-Modal Regression with Missing Modalities

This paper introduces a modality-aware conformal calibration layer that improves prediction interval reliability for multi-modal regression with missing or conflicting data by leveraging a disagreement score to either dynamically scale interval widths or stratify calibration, thereby achieving superior performance and robust coverage across diverse datasets and missingness regimes.

Original authors: Ilia Azizi

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ilia Azizi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the future using a team of experts. Some experts look at the weather, others at the stock market, and some at social media trends. When they all agree, you feel confident. But what happens when the weather expert says "sunny" while the stock expert screams "crash"? Or what if one expert suddenly goes on vacation and stops talking? This is the daily struggle of modern artificial intelligence, specifically a field called machine learning. AI models are great at making guesses, but they often forget to tell us how sure they are.

To fix this, scientists use a clever trick called conformal prediction. Think of it like a safety net. Instead of just giving a single number (like "the rent will be $2,000"), the AI gives a range (like "between $1,800 and $2,200"). The goal is to make sure that, over time, the real answer falls inside that range 95% of the time. This is called a "coverage guarantee." It's like a weather forecast that promises to be right 19 out of 20 times. But here's the catch: traditional safety nets are "one-size-fits-all." They stretch the same amount for every prediction, whether the experts are in total agreement or screaming at each other. This paper tackles the messy reality where AI inputs (like text, images, and numbers) sometimes clash or go missing, and asks: Can we make the safety net smarter so it stretches only when it really needs to?

The Problem: The "One-Size-Fits-All" Safety Net

In the world of multi-modal AI, a computer doesn't just look at one thing; it looks at many things at once. It might read a house listing (text), look at a photo of the room (image), and check the neighborhood stats (numbers). Usually, these sources help each other. But sometimes, they disagree. Maybe the text says the apartment is "cozy," but the photo shows a tiny closet. Or maybe the photo is missing entirely because the file got corrupted.

The old way of handling this was to use a global quantile. Imagine a teacher who gives every student in the class the same size safety net, regardless of whether they are an A+ student or struggling. If the AI's sources are in total agreement, the safety net is too loose (wasteful). If the sources are fighting each other, the safety net might be too tight (dangerous), leading to the real answer falling outside the range. The paper argues that this "average" approach fails when the inputs are messy or incomplete.

The Solution: A Smart, Shape-Shifting Safety Net

The author, Ilia Azizi, proposes a new layer of intelligence called a modality-aware conformal calibration layer. Think of this as a smart supervisor standing over the AI's team of experts.

The Disagreement Score: The supervisor listens to the experts. If the text expert, image expert, and number expert all say "Rent is $2,000," the supervisor notes, "Great, they agree!" (Low disagreement). If the text says "$2,000" and the image suggests "$3,500," the supervisor notes, "Uh oh, big conflict!" (High disagreement).
The Two Tools: The paper introduces two ways to use this score to fix the safety net:
- The "Stretchy" Method (Disagreement-Scaled): This is like a magical rubber band. When the experts agree, the rubber band shrinks, giving a tight, precise prediction. When they fight, the rubber band stretches wide to catch the real answer, ensuring safety. This method keeps the overall promise of being right 95% of the time but makes the predictions much sharper.
- The "Group" Method (Mondrian Calibration): This is like sorting students into different classrooms based on their situation. If a photo is missing, the student goes to the "Missing Photo" classroom, which has its own specific safety net size. If the text is missing, they go to the "Missing Text" classroom. Each group gets a safety net sized exactly for their specific trouble.

What They Found: Smarter Nets, Fewer Mistakes

The team tested this idea on four different real-world datasets, including Swiss apartment rentals, pet photos, and movie reviews. They ran the experiments 60 times with different settings to be sure.

The "Stretchy" Method Wins: In 59 out of 60 tests, the new disagreement-scaled method produced better prediction intervals than the old "one-size-fits-all" method. It managed to make the intervals narrower (more precise) without losing accuracy. In fact, it kept the "coverage" (the rate of being right) very close to the target of 95%.
Fixing Missing Data: The real magic happened when they simulated missing data (like deleting the photo or text). In the hardest cases, where the AI was missing a whole type of information, the old method failed miserably, only getting it right about 76% of the time. The new "mask-matched" method fixed this, boosting the success rate up to 95.3%. That's a massive recovery of 19.5 percentage points.
The Trade-off: To get that extra safety when data is missing, the safety net had to get wider. For example, when only tabular data was available, the width of the prediction interval increased by 125%. But the author argues this is a fair trade: it's better to have a wide, safe net than a tight net that misses the target.

Why It Matters

This paper doesn't just suggest a new math trick; it offers a practical, "plug-and-play" layer that can be added to almost any AI system. It doesn't care if the AI uses trees, neural networks, or anything else. The key takeaway is that uncertainty should change based on the situation. When the AI is confused or missing pieces of the puzzle, it should admit it by widening its guess. When it's confident, it should be precise.

By treating disagreement and missing data as signals rather than errors, the author shows that we can build AI systems that are not just smarter, but also more honest about what they know and what they don't. It's a step toward AI that doesn't just guess, but knows when to say, "I'm not sure, so here's a big range just in case."

Technical Summary: Conformal Calibration for Multi-Modal Regression with Missing Modalities

Problem Statement
Multi-modal regression systems, which integrate diverse input sources such as tabular data, text, and images, face significant challenges in generating reliable uncertainty estimates. While fusing these sources often improves point prediction accuracy, standard conformal prediction methods typically rely on a single global quantile to construct prediction intervals. This global approach averages across different regimes of input data, failing to account for systematic differences in interval reliability when modalities agree, conflict, or when specific modalities are missing at test time. Consequently, a single marginal coverage guarantee can mask under-coverage in difficult subgroups (e.g., high disagreement or missing data) and inefficiency in easy cases. The paper addresses the need for a calibration layer that adapts interval width based on the observed modality pattern and source disagreement while maintaining rigorous finite-sample coverage guarantees.

Methodology
The proposed method introduces a modality-aware conformal calibration layer that operates around a fixed base predictor and a set of per-source auxiliary predictors. The core mechanism involves two complementary strategies driven by a disagreement score and modality availability:

Disagreement Score Construction:
For each example $x$ , per-source predictors $g_k$ generate predictions $\hat{y}_k$ from their respective source blocks $x^{(k)}$ . The disagreement score $d(x)$ is defined as the standard deviation of these per-source predictions:
$d(x) = \left( \frac{1}{K_x} \sum_{k \in A(x)} (g_k(x^{(k)}) - \bar{g}_{A(x)})^2 \right)^{1/2}$
where $A(x)$ is the set of available sources. If only one source is available, $d(x)=0$ .
Disagreement-Scaled Calibration (Continuous Approach):
This method reallocates interval width continuously based on the disagreement score. A scaling function $a_\gamma(x) = \sqrt{1 + \gamma d(x)^2}$ is applied to the Conformalized Quantile Regression (CQR) scores. The parameter $\gamma$ is selected on a tuning split and fixed before calibration.
- Mechanism: The non-conformity score is normalized: $s_i = \max\{e_i, 0\} / a_\gamma(X_i)$ .
- Effect: High disagreement leads to a larger scaling factor, widening the interval; low disagreement narrows it.
- Guarantee: This preserves the standard marginal split-conformal coverage guarantee ( $1-\alpha$ ) under the condition that the scaling rule is fixed prior to calibration.
Mondrian (Stratified) Calibration (Discrete Approach):
This method constructs separate calibration quantiles for pre-specified strata defined by either the disagreement score (binned) or modality availability regimes (e.g., "text missing," "image missing").
- Mechanism: The calibration set is partitioned into strata $h$ . A distinct quantile $\hat{q}_h$ is computed for each stratum.
- Guarantee: This provides finite-stratum conditional coverage guarantees, provided the stratum definition is fixed before calibration and calibration/test examples are jointly exchangeable conditional on the pre-calibration data.
Handling Missing Modalities:
The framework explicitly handles missing modalities by defining "regime" strata. In stress tests, a "mask-matched" recalibration is used where the calibration examples are transformed using the same missingness mask as the test examples. This allows the system to recover coverage for specific missing-data regimes without inventing predictions for absent sources.

Key Contributions
The paper makes four primary contributions:

Definition of a Disagreement Score: It defines a model-agnostic disagreement score derived from per-source regression predictions, serving as a calibration covariate constructed before conformal scores are computed.
Disagreement-Scaled CQR: It incorporates this score into a normalized split-conformal CQR method, selecting a source-disagreement scale before calibration and stating the exact split conditions required for validity.
Mondrian Construction: It provides a formal construction for coverage within pre-specified strata based on modality availability or disagreement levels.
Comprehensive Evaluation: The method is evaluated across four multi-modal datasets (Swiss Real Estate, Mercari, Pawpularity, IMDB-WIKI) using various base predictors (XGBoost point/quantile, SEMF ensembles) and includes stress tests with missing modalities.

Experimental Results

Efficiency and Coverage: Across 60 paired runs (4 datasets $\times$ 3 predictors $\times$ 5 seeds), the disagreement-scaled method matched or improved the marginal conformal baseline in 59 of 60 cases for Interval Continuous Ranked Probability Score (CRPS) and 52 of 60 cases for interval width. Empirical coverage remained near the 95% target (0.949 vs. 0.949).
Disagreement Regimes: The method successfully narrows intervals for low-disagreement examples and widens them for high-disagreement examples, correcting the under-coverage observed in high-disagreement bins under standard marginal calibration.
Missing Modalities: In stress tests with missing modalities, the "mask-matched recalibration" (a special case of Mondrian calibration) significantly recovered coverage. In the most challenging fixed-mask regime (IMDB-WIKI with only tabular data), coverage was recovered by 19.5 percentage points (from 0.758 to 0.953), albeit with a corresponding increase in interval width (125%).
Robustness: The results held across different base predictors, including gradient-boosted trees and Monte Carlo ensembles derived from SEMF decoders.

Significance and Claims
The paper claims to provide a simple, model-agnostic reliability layer for multi-modal regression systems. Its significance lies in:

Decoupling Calibration from Fusion: The layer works with any fixed base predictor (tree, neural, ensemble) and does not require retraining the fusion model.
Adaptive Uncertainty: It moves beyond a "one-size-fits-all" global quantile, allowing the uncertainty estimate to reflect the specific reliability of the input regime (agreement vs. conflict vs. missingness).
Rigorous Guarantees: Unlike heuristic adjustments, the method maintains strict finite-sample coverage guarantees (marginal or conditional) by adhering to a strict split protocol where all data-driven choices (including the disagreement scale and strata definitions) are fixed before the calibration step.
Practical Utility: It offers a practical solution for deployed models where modality availability is dynamic, ensuring that uncertainty estimates remain valid even when the input distribution shifts (e.g., missing sensors or corrupted text).

The author remains modest, noting that the guarantees rely on the strict separation of fitting, tuning, and calibration data, and that the scalar disagreement score treats all sources equally, potentially missing cross-modal dependencies or source reliability differences. The method is presented as a wrapper to improve existing systems rather than a new fusion architecture.

The Problem: The "One-Size-Fits-All" Safety Net

The Solution: A Smart, Shape-Shifting Safety Net

What They Found: Smarter Nets, Fewer Mistakes

Why It Matters

Technical Summary: Conformal Calibration for Multi-Modal Regression with Missing Modalities

More like this