🤖 machine learning

MuteBench: Modality Unavailability Tolerance Evaluation for Incomplete Multimodal Fusion

MuteBench is a comprehensive benchmark evaluating the robustness of multimodal fusion architectures against modality and within-modality missing data across diverse clinical datasets, revealing that model architecture family is the primary determinant of tolerance and offering practical insights for selecting and designing resilient clinical AI systems.

Original authors: Wugeng Zheng, Ziwen Kan, Tianlong Chen, Chen Chen, Song Wang

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wugeng Zheng, Ziwen Kan, Tianlong Chen, Chen Chen, Song Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to diagnose a patient's heart condition using a high-tech medical kit. This kit doesn't just use one tool; it uses a symphony of sensors: an ECG for heart rhythm, a microphone for heart sounds, a camera for skin color, and a text log for the patient's history. In a perfect world, all these sensors work perfectly together.

But in the real world, things go wrong. Sometimes a sensor falls off completely (like a loose wire). Other times, a sensor works, but it gets interrupted by a burst of static noise for a few seconds (like a bad phone connection).

This paper, MuteBench, is like a giant "stress test" for the AI doctors that use these sensor kits. The researchers wanted to see: When sensors fail, which AI designs keep working, and which ones crash?

Here is the breakdown of their findings using simple analogies:

1. The Two Ways Things Break

The paper identifies two specific ways data gets lost, and they are very different:

Modality Missing (The "Missing Instrument" Scenario): Imagine a band playing, but the drummer suddenly leaves the room. The whole drum track is gone. The AI has to guess what the rhythm was without hearing it at all.
Within-Modality Missing (The "Static Noise" Scenario): The drummer is still there, but for 10 seconds, the microphone picks up only static. The AI has to figure out the rhythm from the gaps in the sound.

2. The Stress Test (The Benchmark)

The researchers built a massive testing ground called MuteBench.

They gathered 9 different "medical scenarios" (like ICU monitoring, sleep tracking, and heart sound analysis).
They tested 6 different AI "architectures" (different ways of building the AI brain).
They simulated 125,000+ cases where sensors failed at different levels of severity (20% and 50% failure rates).

3. The Big Discoveries

A. The "Team Structure" Matters More Than the "Team Size"
You might think a bigger AI with more "brain power" (parameters) would handle failures better. The paper says no.

The Winner: The most robust AIs were the ones where each sensor had its own dedicated "translator" (Channel-Independent models). If the heart sensor fails, the translator for the breathing sensor keeps working perfectly because they don't rely on each other.
The Loser: The "Big Brains" that tried to mix all sensors together early on (Mixture-of-Experts) often crashed harder when data was missing. It's like a conductor who relies on every musician playing together; if one stops, the whole song falls apart.
Key Takeaway: How you build the team (the architecture) is more important than how many people are on it.

B. The "Shape" of the Data Changes the Danger
Which failure is worse? Losing a whole sensor or losing a chunk of time? It depends on the data:

Short, Dense Data: If you have a short recording with many sensors (like a quick ICU snapshot), losing a whole sensor is a disaster. It's like trying to solve a puzzle with half the pieces missing.
Long, Continuous Data: If you have a long recording (like a sleep study), losing a chunk of time is worse. It's like missing a crucial chapter in a long novel; you lose the story's flow.

C. The "Training Cheat Code" Has Limits
One of the AI models was trained with a special trick called "Curriculum Dropout." This is like a student practicing with one eye closed, then two, then three, to get used to blindness.

The Result: It worked great! But only up to the level it practiced. If the model practiced with up to 40% missing data, but the real test had 50% missing, the model panicked and failed. You can't train for a disaster you haven't simulated.

D. The "Magic Fix" (Diffusion Imputation)
The researchers tried using a "magic eraser" (Diffusion Imputation) to fill in the missing gaps in the data before the AI looked at it.

For "Static Noise" (Within-Modality): It worked! It was like using Photoshop to fix a blurry photo. The AI could see the picture clearly again.
For "Missing Instrument" (Modality Missing): It failed. You can't Photoshop a whole missing instrument into a song if you have no recording of what it sounded like. The AI tried to guess, but the guess was so bad it made the diagnosis worse.

Summary

The paper concludes that if you are building an AI for medical sensors, don't just make it bigger. Instead, design it so that if one sensor fails, the others can keep working independently. Also, be careful about how you train it; if you don't train it for the worst-case scenario, it won't survive the real world.

They released all their code and data (MuteBench) so other scientists can run these same stress tests to build better, safer medical AI.

Technical Summary: MuteBench

Problem Statement

Multimodal physiological data (e.g., ECG, EEG, PPG, structured records) is foundational to clinical AI systems, yet sensor failures are routine in practice. These failures manifest in two distinct modes:

Modality Missing: An entire channel or sensor stream is absent (e.g., a detached lead).
Within-Modality Missing: A contiguous time segment is lost within an active channel (e.g., motion artifacts or transmission errors).

Despite the prevalence of these issues, existing benchmarks fail to jointly evaluate multiple fusion architectures under both failure modes at controlled severity levels across diverse clinical datasets. Most models are trained assuming complete modality availability, leading to a gap in understanding how architectural choices interact with data incompleteness and signal structure.

Methodology

The authors introduce MuteBench, a comprehensive benchmark designed to evaluate the robustness of multimodal fusion architectures against incomplete data.

Benchmark Scope

Datasets: 9 datasets spanning 7 clinical domains (ICU, cardiology, wearable activity, sleep, affect detection), totaling over 125,000 samples.
Data Diversity: The datasets vary significantly in channel counts (2 to 78), sequence lengths (48 to 3,000 time steps), and sampling rates (hourly observations to 4,000 Hz waveforms). They are categorized into three modality alignment types:
- Type 1: Homogeneous and Aligned (e.g., 12-lead ECG).
- Type 2: Heterogeneous and Aligned (e.g., EEG + Respiration).
- Type 3: Heterogeneous and Unaligned (e.g., time series + static metadata).
Models: 6 representative fusion architectures from three families:
- Channel-Independent: CLIMB, MIRA.
- Shared-Specific Decomposition: ShaSpec.
- Mixture-of-Experts (MoE) Fusion: Flex-MoE, FuseMoE, Maestro.
Experimental Protocol:
- Missingness Injection: A framework-agnostic library injects deterministic missingness patterns.
  - Modality Missing: Channels are dropped independently with probability $p \in \{0.2, 0.5\}$ .
  - Within-Modality Missing: Non-overlapping contiguous blocks (5–10% of sequence length) are masked per channel to cover fractions $b \in \{0.2, 0.5\}$ .
- Evaluation: Models are tested on 270 configurations (6 models $\times$ 9 datasets $\times$ 5 conditions) replicated across 3 random seeds, totaling 810 runs. Metrics include Macro-AUROC and Macro-F1.
- Imputation Study: A case study on PTB-XL evaluates diffusion-based imputation as a potential remedy.

Key Contributions

First Unified Benchmark: MuteBench is the first benchmark to simultaneously cross multiple axes: multiple clinical datasets, diverse fusion architectures, and both modality and within-modality missingness at controlled severity levels.
Empirical Insights on Robustness Drivers:
- Architecture Family > Parameter Count: The choice of architecture family is the strongest predictor of robustness, outweighing model scale.
- Failure Mode Sensitivity: Channel-independent models (e.g., CLIMB, MIRA) are most stable under modality missing. Shared-specific models (e.g., ShaSpec) handle within-modality missing best on homogeneous signals.
- Dataset Structure: The severity of degradation depends on the interaction between dataset properties (channel count, sequence length) and the failure mode. Short, high-channel datasets suffer more from modality missing, while long-sequence datasets are more vulnerable to within-modality missing.
- Curriculum Dropout Limits: Models trained with curriculum modality dropout (e.g., Maestro) show robustness only up to the maximum dropout rate encountered during training; performance degrades sharply if test-time missingness exceeds this bound.
Imputation Analysis: A case study on PTB-XL reveals that diffusion-based imputation improves downstream classification under within-modality missing (gains scale with severity and are highest for models sensitive to corrupted inputs). However, it offers negligible or negative benefits under modality missing, where the absence of an entire channel provides insufficient conditioning signal for reconstruction.

Results

Clean Data: No single model dominates across all datasets; performance varies by task and data structure.
Modality Missing: Channel-independent models exhibit the smallest performance drops (e.g., CLIMB on Chapman at 50% drop: $\Delta$ AUROC $\approx -0.014$ ). MoE-fusion models (e.g., FuseMoE) and shared-specific models often suffer larger degradation, with some experiencing catastrophic failure (e.g., ShaSpec on Challenge-2012 at 20% drop).
Within-Modality Missing: Shared-specific models (ShaSpec) demonstrate high stability on ECG datasets (e.g., PTB-XL at 50% block rate: $\Delta$ AUROC $\approx -0.007$ ). However, models relying on curriculum dropout (Maestro) show significant degradation on long sequences (e.g., Sleep-EDF) when the missing blocks disrupt temporal continuity.
Imputation: Diffusion imputation successfully recovers performance for within-modality missing on PTB-XL (e.g., Flex-MoE gains +0.038 AUROC at 50% missing), but fails to improve performance when entire modalities are missing.

Significance and Claims

The paper claims that MuteBench provides concrete, dataset-aware guidance for practitioners selecting fusion architectures for real-world clinical deployment. By identifying structural properties (modality alignment, channel count, sequence length) that govern sensitivity to specific failure modes, the benchmark informs the design of future robust multimodal systems.

The authors emphasize that robustness is not a generic property but is contingent on the alignment between architectural design and the specific nature of the data and expected sensor failures. They conclude that treating dataset properties as first-class design inputs is essential, rather than relying solely on post-hoc generic missing-data strategies. The release of the benchmark suite, missingness library, and pretrained checkpoints aims to establish a reproducible foundation for developing the next generation of robust clinical AI.