📊 statistics

Pattern-Calibrated Multimodal Prediction under Blockwise Missingness

This paper proposes MOSAIC, a pattern-calibrated framework for multimodal prediction under blockwise missingness that improves accuracy by learning shared and modality-specific representations and applying inter-pattern calibration to avoid collapsing distinct prediction rules across different observed-modality patterns.

Original authors: Junhan Yu, Kejian Zhang, Doudou Zhou, Guojun Zhu

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junhan Yu, Kejian Zhang, Doudou Zhou, Guojun Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the outcome of a medical patient's health journey. You have three types of clues (modalities) available: structured codes (like a checklist of diagnoses), clinical notes (doctors' written stories), and medical images (X-rays).

In the real world, you rarely get all three clues for every single patient.

Some patients only have the checklist.
Some have the checklist and the notes.
A few lucky ones have all three.

This is called blockwise missingness: entire chunks of information are missing, not just random words here and there.

The Problem: The "One-Size-Fits-All" Trap

Most current AI methods try to solve this by pretending the missing clues are there. They might:

Guess (Impute): Try to invent what the missing X-ray looks like.
Fill with Zero: Pretend the missing note is a blank page.
Mix Everything: Train one giant brain to handle all patients, regardless of what clues they have.

The authors argue that mixing everything together is dangerous. It's like trying to teach a chef to cook a steak using only a recipe book, but then forcing that same chef to cook a steak using only a picture of the steak and a smell description. The "rules" for cooking based on a book are different from the rules based on a picture. If you force the chef to use one single rule for both, they will get confused and make mistakes.

The Solution: MOSAIC

The paper proposes a new method called MOSAIC (Multimodal Overlap-aware Shared-specific Alignment and Inter-pattern Calibration). Think of MOSAIC as a two-step translation and correction process.

Step 1: The "Universal Translator" (Representation Learning)

Before predicting anything, MOSAIC looks at all the data (even the parts without labels) to learn a common language.

It identifies the Shared Language: The core facts that are true whether you have an X-ray, a note, or just a code (e.g., "the patient has a fever").
It identifies the Specialized Dialects: The unique details only found in specific clues (e.g., the texture of an X-ray or the tone of a doctor's note).

By separating these, MOSAIC ensures that when it looks at a patient with only codes, it knows exactly which "Shared Language" facts it has and which "Specialized Dialects" are missing. It doesn't try to fake the missing dialect; it just acknowledges the gap.

Step 2: The "Smart Borrower" (Overlap-Based Prediction)

Now, MOSAIC needs to predict the outcome for a patient who only has Codes.

The Borrowing: Instead of ignoring the patients who have Codes + Notes + Images, MOSAIC looks at the "Shared Language" parts of those rich records. It asks: "What did the patients with full data tell us about the 'Shared Language' facts?" It uses this extra information to make a first guess.
The Correction (Calibration): The authors know this first guess isn't perfect because the "full data" patients had extra clues (Notes/Images) that the "Code-only" patient doesn't. So, MOSAIC takes a small group of "Code-only" patients and asks: "How much does our first guess need to be tweaked to fit just the Code-only reality?"
It applies this specific tweak (calibration) to the final prediction.

The Analogy: The Detective Team

Imagine a detective team trying to solve a crime.

The "Pooled" Method: One detective tries to solve every case using a single notebook. If a case has a fingerprint, a witness, and a weapon, they use all three. If a case only has a witness, they try to force the witness story to fit the same rules as the fingerprint case. It's messy and often wrong.
The MOSAIC Method:
1. Training: The team learns a "Universal Crime Logic" (Shared) and "Specialized Skills" (Fingerprint analysis, Witness interviewing, Weapon analysis).
2. Case A (Witness Only): The detective uses the "Universal Crime Logic" learned from cases that had all evidence to make a strong initial theory.
3. The Tweak: They then look at other cases that also had only a witness. They see how the "Universal Logic" was slightly off for witness-only cases and apply a specific correction.
4. Result: They get the benefit of the team's total experience (borrowing) without losing the specific nuance of the "Witness Only" case.

Why It Works (The Results)

The paper tested this on three real-world scenarios:

ICU Mortality: Predicting if a patient will die in the hospital using codes, notes, and X-rays.
Emotion Recognition: Detecting emotions from text, audio, and video.
Glaucoma Classification: Diagnosing eye disease.

The Findings:

When a specific type of patient (e.g., those with only codes) is rare, MOSAIC shines. It borrows strength from the common patient types but corrects for the differences.
It outperforms methods that just "fill in the blanks" or "mix everything together."
The math proves that the error in the prediction comes from three things: how well the "Universal Translator" worked, how much data was borrowed, and how big the "tweak" (calibration) needed to be.

In Short

MOSAIC is a smart way to use data from different groups without forcing them to be the same. It learns what they have in common, borrows insights from the "rich" data groups, and then carefully adjusts the answer to fit the specific "poor" data group it is trying to predict for. It's about borrowing without blending.

Technical Summary: Pattern-Calibrated Multimodal Prediction under Blockwise Missingness

1. Problem Statement

In multimodal prediction tasks (e.g., Electronic Health Records containing codes, text, and images), data often suffer from blockwise missingness, where entire modalities are unobserved for certain subjects due to acquisition costs, workflow constraints, or privacy.

Existing approaches typically treat this as an incomplete-input problem, aiming to make all records usable by a single prediction model via imputation, zero-filling, or fusion architectures. However, the authors argue that these methods often fail to address the fundamental estimand problem:

Pattern-Specific Tasks: The set of observed modalities defines the information available for prediction. A rule conditioned on "codes only" is fundamentally different from a rule conditioned on "codes, notes, and images."
Bias from Pooling: Training a single pooled predictor on all patterns (even with missingness indicators) forces distinct observable prediction tasks to share a common rule. This induces bias because the pooled rule minimizes a prevalence-weighted average of risks rather than the risk specific to a target pattern.
Inefficiency of Local Fitting: Conversely, fitting a separate model for each missingness pattern (fully local strategy) is inefficient when certain patterns are rare (e.g., fully observed records are less common than code-only records).

The core challenge is how to borrow information across different missingness patterns to improve efficiency without collapsing distinct pattern-specific prediction rules into a single biased rule.

2. Methodology: MOSAIC

The authors propose Multimodal Overlap-aware Shared-specific Alignment and Inter-pattern Calibration (MOSAIC), a framework designed to borrow strength across patterns while preserving pattern-specific rules. The method operates in three stages:

2.1 Shared and Modality-Specific Representation Learning

MOSAIC first learns a representation space from unlabeled multimodal data (or a separate representation-learning sample). This space decomposes information into:

Shared Representations ( $z^{(0)}$ ): Information comparable across modalities.
Modality-Specific Representations ( $z^{(l)}$ ): Information unique to modality $l$ .

The learning objective minimizes three terms:

Alignment Loss ( $L_{shared}$ ): Ensures shared representations extracted from different observed modalities of the same subject are close.
Orthogonality/Decoupling Loss ( $L_{orth}$ ): Prevents shared and specific representations from carrying the same variation (regularizing the empirical second moment of the concatenated representation).
Reconstruction Loss ( $L_{recon}$ ): Ensures the representations retain enough information to reconstruct the original modality.

2.2 Overlap-Based First-Stage Prediction

For a target missingness pattern $k$ , MOSAIC constructs a first-stage predictor $h^{(k)}$ by borrowing samples from all other patterns $j$ that share overlapping representation coordinates.

It retains coordinates in $S_j \cap S_k$ (shared and observed in both) and zero-fills coordinates in $S_k \setminus S_j$ .
This allows the model to utilize a larger effective sample size ( $e n(k)$ ) derived from the overlap of source patterns, rather than being limited to the small number of samples $n_k$ in the target pattern.
Crucially, this first-stage predictor is generally not the optimal rule for the target pattern because source samples lack the full target representation.

2.3 Target-Pattern Calibration

To correct the bias introduced by the first-stage predictor, MOSAIC estimates a calibration gap $\Delta^{(k)} = g^{(k)} - h^{(k)}$ using only the labeled samples from the target pattern $k$ .

The final predictor is $g^{(k)} = h^{(k)} + \Delta^{(k)}$ .
This step is a residual correction that is less demanding than fitting the full rule from scratch, especially when the shared and specific representations are well-separated.

3. Key Contributions

Problem Formulation: The paper formalizes the "pattern-specific cross-pattern borrowing" problem, distinguishing between the bias of pooling distinct prediction rules and the inefficiency of local fitting. It defines the estimand as the best prediction rule conditional on the specific observed modality set.
MOSAIC Framework: It introduces a two-stage, pattern-calibrated framework that:
- Learns decoupled shared and modality-specific representations.
- Borrows samples via overlap in the representation space.
- Applies a target-pattern specific calibration to correct the borrowed predictor.
Theoretical Guarantees: The authors establish non-asymptotic error bounds that decompose the prediction error into three components:
- Overlap Effective Sample Size: The benefit of borrowing from overlapping patterns.
- Calibration Gap: The error due to the mismatch between the borrowed predictor and the target rule.
- Representation Learning Error: The error introduced by estimating the latent representations.
- The theory clarifies that borrowing improves over local fitting when the effective overlap sample size is significantly larger than the local sample size, provided the representation learning error and calibration gap are controlled.

4. Experimental Results

4.1 Simulations

Representation Recovery: Experiments demonstrate that MOSAIC successfully recovers shared and modality-specific representations, outperforming ablations that lack orthogonal decoupling or reconstruction losses. Recovery improves with larger unlabeled sample sizes and higher co-observation probabilities.
Prediction Performance: MOSAIC achieves the lowest balanced Mean Squared Error (MSE) across various missingness patterns compared to external baselines (e.g., M3Care, MedFuse, IF-MMIN) and internal ablations.
- Ablation Studies: Removing the calibration step leads to performance degradation as residual dependence between modalities increases, confirming the necessity of the correction step. Removing representation learning (using PCA instead) slows convergence, highlighting the value of the learned shared structure.

4.2 Real-World Applications

MIMIC (ICU Mortality Prediction): Applied to next-visit ICU mortality prediction using structured codes, clinical notes, and chest X-rays.
- MOSAIC achieved the highest overall AUROC (0.861), AUPRC, and F1 scores.
- Pattern-Specific Gains: The method showed the most pronounced improvements in the "code-only" pattern, where cross-pattern borrowing from richer modalities (via shared code representations) compensated for the lack of notes and images. It also maintained top performance in the fully observed pattern.
IEMOCAP and FairDomain: Additional evaluations on emotion recognition and glaucoma classification confirmed MOSAIC's advantages over leading methods under controlled modality masking.

5. Significance and Claims

The paper claims that MOSAIC provides a principled solution to the trade-off between bias (from pooling distinct prediction tasks) and efficiency (from limited local samples).

Mechanism: By separating representation learning from target-pattern calibration, MOSAIC allows models to "borrow strength" across missingness patterns without forcing them to share a single, potentially biased, prediction rule.
Theoretical Insight: The decomposition of error bounds offers a clear explanation of when cross-pattern borrowing is beneficial: specifically when the overlap effective sample size is large relative to the local sample size, and when the representation learning and calibration gaps are small.
Practical Impact: The method is particularly effective in scenarios where target-pattern samples are limited (e.g., rare fully observed records) or where pattern-specific rules differ significantly.

The authors note that the current theoretical analysis relies on simplified linear representation models and Missing Completely at Random (MCAR) assumptions. Future work is suggested to extend these guarantees to nonlinear encoders and more general missingness mechanisms (MAR/MNAR), as well as to address challenges in rare patterns with weak overlap.