MESSI: Multimodal Experiments with SyStematic Interrogation using nextflow

The paper introduces MESSI, a reproducible Nextflow-based framework that standardizes the benchmarking of multimodal integration methods across diverse biomedical datasets, revealing that while no single method is universally optimal, choices should balance predictive accuracy, biological interpretability, and computational efficiency.

The Gist

Imagine you are trying to solve a massive, complex mystery: Why do some people get sick while others stay healthy?

In the past, doctors and scientists looked at just one clue at a time. Maybe they only checked your blood sugar, or maybe they only looked at your family history. But modern medicine is like a detective who finally gets access to the entire crime scene: your DNA, your proteins, your brain scans, your lifestyle habits, and your medical history, all at once.

This is called Multimodal Data. It's a goldmine of information, but it's also a mess. Trying to make sense of all these different clues together is like trying to build a house while five different architects are shouting instructions in five different languages. Some say "use wood," others say "use steel," and no one agrees on the blueprint.

The Problem: The "Taste Test" Disaster

Scientists have invented many different "recipes" (algorithms) to mix these clues together to predict disease. But here's the problem: Nobody was tasting the dishes fairly.

Some chefs (researchers) used different ovens, some used different ingredients, and some tasted the food at different times. Because of this, it was impossible to know which recipe was actually the best. Was the "Steel House" recipe better, or was it just that the chef who made it had a better oven?

The Solution: Enter MESSI

The authors of this paper built a new kitchen called MESSI (Multimodal Experiments with SyStematic Interrogation).

Think of MESSI as a super-fair, automated taste-testing competition.

• The Rules: It forces every chef (every computer method) to use the exact same ingredients, the exact same oven temperature, and the exact same tasting schedule.
• The Setup: It's built on a system called "Nextflow," which is like a robotic kitchen manager. It doesn't matter if a chef speaks "R" or "Python" (two different programming languages); the robot manager translates everything so they can all compete on a level playing field.
• The Safety Net: The most important rule is "No Cheating." In the past, chefs might have peeked at the answer key before cooking. MESSI uses a strict "Nested Cross-Validation" rule. This means the chefs have to cook a dish, taste it, and then guess the answer, without ever seeing the final result until the very end. This ensures the results are real and not just lucky guesses.

The Big Taste Test

The researchers put 19 different "dishes" (real medical datasets) into the MESSI kitchen. These included data from cancer patients, people with Alzheimer's, heart transplant recipients, and even people with COVID-19. They tested various "recipes" (algorithms) to see which one could best predict who would get sick or how severe the disease would be.

Here's what they found:

1. There is no "Magic Bullet": Just like there is no single spice that makes every dish taste perfect, there is no single computer method that wins every time.

   • For some diseases, DIABLO (a method that looks for shared patterns) was the star chef.
   • For others, RGCCA or Multiview took the crown.
   • Some methods, like MOGONET, struggled to find the flavor in the data.

2. Taste vs. Health: Sometimes the method that predicted the disease best wasn't the one that gave the most useful biological clues.

   • Imagine a chef who makes a cake that tastes amazing (great prediction) but uses mystery ingredients you can't identify (bad for science).
   • Other chefs made cakes that tasted okay but used clear, healthy ingredients that explained why the cake was good (great for biology).
   • MESSI showed us that the best method depends on what you want: Do you just want to know if someone is sick, or do you want to understand why?

3. Speed and Cost: Some methods were like a slow-cooked stew—delicious but took all day and used a lot of gas (computing power). Others were like a quick stir-fry—fast and efficient.

   • DIABLO and MOFA were the "fast and efficient" chefs.
   • Multiview was the "slow and expensive" chef, taking forever to cook.

The Takeaway

This paper is a game-changer because it stopped scientists from arguing about whose oven was better. Instead, they built a standardized, fair kitchen where everyone competes under the same rules.

The main lesson? Don't look for one "best" method to solve all medical mysteries. Instead, pick the right tool for the job. If you need speed, pick the fast chef. If you need to understand the biology, pick the chef who explains their ingredients.

MESSI is the tool that helps doctors and scientists choose the right chef for their specific mystery, ensuring that the answers they get are reliable, fair, and actually helpful for curing diseases.

Technical Summary

1. Problem Statement

Multimodal biomedical studies are increasingly profiling diverse molecular and clinical modalities (e.g., transcriptomics, proteomics, imaging, clinical data) from the same samples to improve disease prediction and biological discovery. However, benchmarking multimodal integration methods faces significant challenges:

• Inconsistency: Studies often use inconsistent preprocessing pipelines, unequal hyperparameter tuning strategies, and non-comparable evaluation schemes.
• Bias: Performance differences often reflect evaluation design (e.g., data leakage, fixed train/test splits) rather than inherent methodological strengths.
• Reproducibility: Many existing benchmarks lack containerization and standardized workflows, making it difficult to reproduce results or fairly compare methods across different programming languages (R vs. Python) and integration paradigms (early, intermediate, late).

2. Methodology: The MESSI Framework

The authors developed MESSI (Multimodal Experiments with SyStematic Interrogation), a reproducible, Nextflow-based benchmarking framework designed to standardize the evaluation of multimodal integration methods.

Core Architecture

• Workflow Engine: Built on Nextflow, allowing for modular, scalable, and parallel execution across local machines, HPC clusters (SLURM, PBS), and cloud platforms (AWS, Google Cloud).
• Containerization: Uses Docker and Singularity to ensure identical operating system configurations and software versions, guaranteeing reproducibility and portability.
• Interoperability: Supports both R (via MultiAssayExperiment) and Python (via MuData) data structures, enabling seamless integration of methods written in either language.

Pipeline Stages

The framework executes four main stages:

1. Data Preparation: Standardizes heterogeneous inputs into unified formats, handling missing values, variance filtering, and binary encoding.
2. Index Splitting: Generates stratified $k$-fold cross-validation splits (default $k=5$) using Python's scikit-learn. Indices are saved as text files to ensure all methods use identical train/test partitions, eliminating randomness in splitting.
3. Model Assessment (Nested Cross-Validation):
   • Outer Loop: Evaluates generalization performance on held-out test sets.
   • Inner Loop: Performs hyperparameter tuning and model selection within the training set.
   • Leakage-Free: This nested structure ensures that hyperparameter tuning does not leak information from the test set, providing unbiased estimates of generalization error.
4. Model Selection & Reporting: Aggregates feature weights, coefficients, and performance metrics (AUC, runtime, memory) for downstream analysis.

Benchmarked Methods

MESSI evaluated a representative set of intermediate and late integration methods:

• Intermediate: DIABLO, RGCCA, MOFA+, IntegrAO, Multiview (Cooperative Learning).
• Late: caretMultimodal (Ensemble of glmnet), MOGONET (Deep Learning).
• Note: The study focused on vertical integration (same samples, different features) for binary classification tasks.

3. Key Contributions

• Standardized Framework: MESSI is the first framework to enforce a consistent, leakage-free nested cross-validation strategy across diverse multimodal methods, regardless of their underlying language or integration philosophy.
• Scalability & Flexibility: The Nextflow implementation allows for easy scaling, parallel processing of method-dataset combinations, and flexible configuration via profiles (e.g., skipping specific methods, adjusting resources).
• Comprehensive Benchmarking: The study covers 19 real-world datasets (16 bulk multimodal, 3 single-cell multiomics) and simulation studies, spanning cancer, neurodegenerative, infectious, and transplant diseases.

4. Key Results

A. Simulation Studies

• Calibration: All methods were well-calibrated (AUC $\approx$ 0.5) in the absence of signal, indicating no systematic false positives.
• Signal Strength: Under low-signal conditions (mimicking real biology), unsupervised methods (MOFA) and weakly supervised approaches performed worse than supervised methods.
• Feature Recovery: DIABLO, RGCCA, and IntegrAO showed high sensitivity in recovering true signal variables. However, some methods (e.g., Multiview) had higher false positive rates in feature selection.
• Efficiency: MOFA and DIABLO were computationally efficient; Multiview and IntegrAO were the most time and memory-intensive, respectively.

B. Bulk Multimodal Benchmarking (16 Datasets)

• Predictive Performance: Differences in classification accuracy (AUC) were generally modest across methods. DIABLO and Multiview tended to rank highest, while MOFA+glmnet and MOGONET were weaker.
• Biological Interpretability: This was the primary differentiator.
  • DIABLO, RGCCA, MOFA, and IntegrAO consistently recovered the most significant Reactome and oncogenic pathways.
  • RGCCA, MOFA, and DIABLO were superior in identifying tissue-relevant cell-marker signatures (PanglaoDB).
• Computational Scalability: DIABLO and MOFA demonstrated the most favorable runtime and memory profiles. Multiview was the most resource-intensive.

C. Single-Cell Multiomics Benchmarking (3 Datasets)

• Dataset Dependency: Method rankings were highly dataset-dependent.
• Robustness: DIABLO performed consistently well across all three single-cell case studies (Heart Transplant, COVID-19 Multiomics, COVID-19 Multi-organ), suggesting its supervised regularized framework is robust to heterogeneous single-cell structures.
• Biological Insights: Pathway enrichment varied by method. For example, in the heart transplant dataset, RGCCA variants identified the most cell-specific pathways related to allograft rejection.
• Resources: DIABLO remained the most computationally efficient, while IntegrAO was the most memory-demanding.

5. Significance and Conclusions

• No "One-Size-Fits-All": The study concludes that no single method is uniformly optimal. Method choice must balance predictive performance, biological interpretability, and computational efficiency based on the specific research goal.
• Interpretability vs. Accuracy: Methods with similar predictive accuracy can differ significantly in the biological plausibility of the signals they recover. For biomarker discovery, methods like DIABLO and RGCCA offer a strong balance of performance and interpretability.
• Future Foundation: MESSI establishes a transparent, reproducible foundation for future benchmarking. It can be extended to other tasks (survival analysis, clustering) and new integration methods, addressing the critical need for standardized evaluation in the rapidly evolving field of multimodal data integration.

Availability:

• Pipeline: Available on GitHub (CompBio-Lab/MESSI-pipeline).
• Data/Scripts: Available on Zenodo and GitHub (CompBio-Lab/MESSI_manuscript).