📊 statistics

DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passing

The paper proposes DAIF, a data-driven intermediate fusion framework that leverages random matrix theory and approximate message passing to dynamically learn modality-specific fusion structures from estimated intermodal dependence, thereby improving predictive performance in multimodal supervised learning tasks compared to state-of-the-art methods.

Original authors: Sagnik Nandy, Samriddha Lahiry, Pragya Sur, Subhabrata Sen

Published 2026-08-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Sagnik Nandy, Samriddha Lahiry, Pragya Sur, Subhabrata Sen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of just one witness, you have a whole room full of them. Some witnesses are sharp and remember every detail perfectly, while others are a bit fuzzy and only recall the general vibe. Some of them are telling the exact same story, while others are talking about completely different things. In the world of data science, this "room of witnesses" is called multimodal learning. Scientists collect different types of information about the same subject—like a person's DNA, their protein levels, and their medical history all at once. The goal is to combine these clues to make a better prediction than any single clue could provide on its own.

However, there is a tricky problem: how do you decide which witnesses to listen to together? If you force everyone to speak in unison (a strategy called "early fusion"), the noisy, confused witnesses might drown out the smart ones. But if you let everyone shout their own story separately and only try to combine the answers at the very end ("late fusion"), you might miss the subtle connections between the smart witnesses who are actually telling the same story. For a long time, scientists had to guess the best way to mix these clues, often picking a strategy based on gut feeling rather than what the data actually said. This paper introduces a new, clever way to let the data itself decide the perfect mix.

The researchers behind this study, Sagnik Nandy and his team, have built a framework called DAIF (Data-Adaptive Intermediate Fusion). Think of DAIF as a super-smart party host who doesn't just tell everyone to sit at one big table or stay in separate rooms. Instead, the host listens to the conversations, figures out which groups of people are actually talking about the same topic, and then gently guides those specific groups to share their stories.

Here is how it works in practice:

The Listening Phase: DAIF looks at all the different types of data (the "modalities") and uses a special mathematical tool called Centered Kernel Alignment (CKA) to measure how much they depend on each other. It's like checking if two people are whispering the same secret or talking about totally different things.
The Grouping Phase: Based on these measurements, DAIF automatically clusters the data. If the RNA data and the protein data are strongly linked, it puts them in the same group. If the DNA data is totally independent, it keeps that in its own group. This happens without the researchers needing to tell the computer which groups to make; the computer figures it out from the numbers.
The Cleaning Phase: Once the groups are formed, DAIF uses a technique called Approximate Message Passing (AMP). Imagine this as a game of "telephone" where the signal gets clearer with every round. The algorithm takes the noisy data from each group and "denoises" it, borrowing strength from the related clues to fill in the gaps. It's like if one witness forgot a detail, but the other witness in their group remembered it, so the group can reconstruct the full story together.
The Prediction Phase: Finally, these cleaned-up, smart summaries are used to predict the outcome, whether that's how a patient will survive a disease or how a specific protein will behave.

The authors tested this idea in two main ways. First, they ran thousands of computer simulations where they knew the "true" answer. In these tests, DAIF consistently outperformed other popular methods. When the data was messy and the signals were weak, DAIF was about 10 times more accurate at finding the hidden patterns than methods that didn't use this smart clustering approach. It was also better at predicting outcomes than methods that forced all data together or kept it completely separate.

Then, they took DAIF into the real world using two massive biological datasets. In the first case, they used a dataset called TEA-seq, which contains information on 8,213 cells, including 36,601 genes, 66,828 chromatin peaks, and 48 proteins. They tried to predict the level of a specific protein marker (CD45RA). The results were striking: depending on the type of cell, the best way to combine the data changed. For some cells, mixing everything together worked best; for others, keeping them separate was better. DAIF automatically figured this out and achieved a prediction error (RMSE) of 2.6867, beating other top methods like MOFA+ and JAFAR.

In the second real-world test, they looked at the TCGA-BRCA dataset, which involves 769 patients with breast cancer, tracking their RNA, DNA methylation, and copy number variations to predict survival. Here, DAIF again showed its flexibility. While a standard method might force a single strategy, DAIF found that an intermediate approach worked best for the training data, though it noted that the "all-together" approach performed slightly better on the final test set, likely because the dataset was small. Even so, DAIF's predictions were more reliable than deep learning models that tended to overfit (memorize the training data too closely) and fail on new patients.

The paper argues against the old way of doing things, where scientists just pick a fusion strategy (early, late, or intermediate) and stick with it regardless of the data. The authors show that this "one-size-fits-all" approach can be inefficient, sometimes mixing unrelated data and drowning out the signal. They also demonstrate that while some existing methods try to learn from data, they often rely on rigid assumptions or require the data to be perfectly balanced, which isn't true in the messy real world.

In short, DAIF suggests that the best way to solve a mystery isn't to force everyone to agree or to let everyone shout alone. It's to listen carefully, figure out who is actually on the same team, and let those teams collaborate to tell the clearest story possible. The authors are confident in these findings because they backed them up with rigorous mathematical proofs showing that their method works as well as if they had known the perfect groups from the start, plus extensive simulations and real-world tests that showed it beating the competition.

Technical Summary: DAIF – A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning

Problem Statement
Multimodal supervised learning aims to leverage heterogeneous data sources (modalities) to improve predictive performance. A central challenge in this domain is determining the optimal fusion granularity: how to integrate information across modalities.

Over-integration (e.g., Early Fusion) may amplify noise by forcing unrelated modalities into a shared representation, leading to high-dimensional estimation problems with insufficient data.
Under-integration (e.g., Late Fusion) fails to exploit cross-modal dependencies, resulting in weak or uninformative representations.
Existing approaches typically rely on pre-specified, fixed fusion architectures (Early, Intermediate, or Late) guided by domain expertise rather than data. This rigidity is problematic because the optimal granularity often depends on the specific composition of the dataset (e.g., varying cell-type abundances in single-cell biology). The paper posits that a data-adaptive framework is required to learn the appropriate fusion structure directly from the data.

Methodology: The DAIF Framework
The authors propose DAIF (Data-Adaptive Intermediate Fusion), a framework that combines random matrix theory, non-parametric dependence measures, and Approximate Message Passing (AMP). The methodology operates under a Bayesian multimodal factor model where latent factors mediate cross-modal dependence. The pipeline consists of five key phases:

Spectral Initialization: The algorithm begins by computing the top singular vectors of each modality's data matrix (after appropriate scaling).
Modality Clustering: To determine fusion granularity, the method computes pairwise dependence between modality-specific representations using non-parametric measures, specifically Centered Kernel Alignment (CKA). A dissimilarity matrix is constructed, and hierarchical clustering (UPGMA) is applied to group modalities.
- The number of clusters ( $K$ ) determines the fusion strategy: $K=1$ implies Early Fusion, $K=$ total modalities implies Late Fusion, and intermediate $K$ values yield Partial/Intermediate Fusion.
Cluster-wise Empirical Bayes (EB) Prior Estimation: Within each estimated cluster, the method estimates a prior distribution ( $\mu_k$ ) for the latent factors using the observed data. This step mitigates the curse of dimensionality by restricting prior estimation to strongly dependent modalities. The authors use Gaussian Mixture Models (GMM) for this estimation.
Orchestrated Approximate Message Passing (AMP): The estimated clustered priors are used to construct denoisers within an AMP framework. The algorithm alternates between:
- Power Iteration: To recover low-rank factors.
- Bayesian Shrinkage: To denoise the iterates by shrinking them toward the mean of the estimated cluster-specific priors.
- Onsager Correction: A debiasing step to ensure the iterates behave asymptotically like true latent variables corrupted by Gaussian noise.
  This process yields denoised, low-dimensional embeddings that borrow strength across related modalities while preserving modality-specific signals.
Downstream Prediction:
- Training: The rescaled AMP embeddings are used to train a supervised predictor (linear or neural network). The loss function explicitly accounts for the Gaussian perturbations inherent in the AMP iterates (errors-in-variables setting).
- Testing: For test data, where singular value decomposition is infeasible due to small sample sizes relative to dimensionality, the method projects test features onto the estimated loading matrices (via Ordinary Least Squares). These projected features are then fed into the pre-trained predictor.

Key Contributions

Bayesian Modeling of Varying Granularity: The authors propose a multimodal factor model with a structured prior that decomposes features into components corresponding to strongly dependent modalities. This formulation flexibly interpolates between complete dependence and independence, accommodating heterogeneous latent dimensions across modalities.
Data-Adaptive Intermediate Fusion Pipeline: DAIF learns the fusion architecture directly from data via hierarchical clustering of non-parametric dependence measures. Theoretical results (Theorem 6.1) demonstrate that this data-adaptive clustering yields consistent prior estimation, provided the true prior exhibits a clustered structure.
Theoretical Guarantees: The paper establishes rigorous theoretical properties, including:
- Consistency of the clustered empirical Bayes prior estimator.
- Asymptotic characterization of AMP iterates (State Evolution), showing they behave like oracle estimators.
- Proof that the resulting iterates are Bayes-optimal estimators for the true signals under mild conditions (Theorem 6.3).
- Justification for the test-time projection and prediction strategy (Theorem 6.4).
Empirical Performance: The framework is benchmarked against state-of-the-art methods (EB-PCA, OrchAMP, AJIVE, MCCA, GCCA, MOFA+, Multigrate) on both synthetic data and real biological datasets.

Results

Synthetic Experiments: DAIF consistently outperforms AMP-based benchmarks (EB-PCA, OrchAMP) and non-AMP methods (AJIVE, MCCA, GCCA) in latent factor reconstruction and downstream prediction accuracy. It demonstrates particular robustness in regimes with low signal strength and large sample sizes, where it is approximately 10 times more accurate than non-AMP integration benchmarks.
TEA-seq Dataset (Swanson et al., 2021): In predicting T-cell differentiation marker protein expression, DAIF adapts to the data structure. While Early and Late fusion strategies performed comparably in aggregate, DAIF's ability to select the optimal granularity (via the gap statistic) allowed it to compete with or outperform specialized methods like JAFAR and MOFA+ across different cell types.
TCGA-BRCA Dataset (Goldman et al., 2020): In survival analysis, DAIF (using intermediate fusion selected by the gap statistic) achieved a test C-index of 0.7375, outperforming Multigrate (0.6313), which suffered from overfitting, and competing with MOFA+ (0.7335).

Significance and Claims
The paper claims that DAIF provides a principled, data-driven mechanism for determining the appropriate granularity of information fusion, avoiding the inefficiencies of fixed-architecture approaches. By learning the dependence structure from the data, DAIF effectively balances signal integration and noise suppression. The framework is presented as a versatile tool that subsumes existing pipelines (Early and Late fusion) as special cases and offers rigorous statistical guarantees in high-dimensional regimes. The authors emphasize that their method is particularly suited for settings where modalities are only partially dependent, a common scenario in computational biology that existing "complete fusion" methods fail to address optimally.

More like this