Benchmarking resting state fMRI connectivity pipelines for classification: Robust accuracy despite processing variability in cross-site eye state prediction

This study demonstrates that resting-state fMRI connectivity-based models can robustly classify eye states with approximately 80% accuracy across different acquisition sites, achieving consistent results despite substantial variability in preprocessing pipelines and connectivity metrics.

The Gist

Imagine you are trying to teach a computer to tell the difference between a person's brain when their eyes are open and when their eyes are closed. It sounds simple, right? But in the world of brain scanning (fMRI), the data is messy, like trying to hear a whisper in a crowded stadium.

This paper is essentially a massive "cooking competition" to see which recipe (or pipeline) works best for turning that messy brain data into a clear answer.

Here is the breakdown of their experiment using simple analogies:

1. The Problem: Too Many Ways to Cook the Same Dish

In brain research, scientists have to process raw data before they can use it. They have to:

• Clean the noise: Remove static caused by head movements or breathing (like cleaning mud off a camera lens).
• Divide the brain: Break the brain into smaller chunks (like slicing a pizza into slices).
• Measure connections: See how different slices talk to each other (like measuring how much traffic flows between city districts).

The problem is that there are thousands of ways to do these steps. One scientist might slice the brain into 100 pieces; another might slice it into 400. One might use a specific math trick to clean the noise; another might use a different one.

The Question: Does it matter which "recipe" you use? Or will the computer get the right answer (Eyes Open vs. Eyes Closed) no matter what?

2. The Experiment: The Ultimate Taste Test

The researchers didn't just try one recipe. They cooked 256 different dishes (pipelines) using two different sets of ingredients (data from two different labs in China and Russia).

They tested these 256 recipes using two different challenges:

• Challenge A (The Blind Taste Test): Train the computer on data from Lab A, then test it on Lab B. Can it generalize?
• Challenge B (The Little Helper): Train on Lab A, but give the computer just a tiny taste of Lab B's data to help it adjust, then test it on the rest of Lab B.

3. The Big Surprise: The "Robustness" Effect

Usually, in science, if you change the recipe slightly, the taste changes completely. But here, the researchers found something amazing: The computer got it right about 80% of the time, almost no matter which recipe they used.

It's like if you tried to identify a friend's voice over the phone. Even if you changed the phone model, the background music, or the volume, you could still recognize them. The signal (the difference between eyes open and closed) was so strong that it survived almost any processing method.

4. The Winners: The "Golden Recipes"

While most recipes worked well, some were clearly better chefs than others. The researchers found the "Gold Standard" combination:

• The Brain Map (Atlas): Using the Brainnetome atlas (a map that divides the brain into 246 specific regions) worked best. Think of this as using a detailed, high-quality map of a city rather than a rough sketch.
• The Math Trick (Connectivity): Using Tangent Space Parametrization was the winner.
  • Analogy: Imagine you are trying to describe the shape of a crumpled piece of paper.
  • Standard Math (Pearson Correlation): Just measures how much the paper moves up and down.
  • Tangent Space: It's like uncrumpling the paper onto a flat table to see the true shape more clearly. It handles the "wrinkles" (noise) in the data better, making the computer's job easier.
• The Cleaning (Denoising): Using CompCor (a method that removes noise from white matter and fluid in the brain) was the best cleaner. Interestingly, removing the "Global Signal" (the average noise of the whole brain) didn't help much and sometimes made things worse.

5. What Did They Learn? (The Takeaway)

• Don't Panic About Perfection: If you are studying a very clear difference (like eyes open vs. closed), you don't need to stress about finding the perfect data processing method. The signal is strong enough that most standard methods will work.
• The "Secret Sauce": If you do want the absolute best results, use the Brainnetome map and the Tangent Space math trick.
• Cross-Site Success: The computer learned from one lab and worked perfectly in another. This is huge news for the medical world. It means we can share brain data between different hospitals and countries without needing to perfectly harmonize every single machine setting.

Summary Analogy

Imagine you are trying to identify a specific song playing in a room.

• The Old Fear: "If the room has different walls, different speakers, or different background noise, we won't be able to tell what song it is!"
• This Study's Finding: "Actually, the song is so loud and distinct that we can identify it even if the walls are made of wood, concrete, or glass, and even if the speakers are cheap or expensive. We just need to make sure we aren't listening to a broken speaker (bad data cleaning)."

The Bottom Line: The brain's "open vs. closed eye" signal is incredibly robust. We can trust these brain scans to tell us what state a person is in, even if different scientists process the data in slightly different ways.

Technical Summary

1. Problem Statement

The application of machine learning (ML) to resting-state functional MRI (rs-fMRI) data for brain state classification faces significant challenges regarding reproducibility and generalizability.

• Methodological Variability: The fMRI analysis pipeline involves numerous arbitrary choices (preprocessing, denoising, brain parcellation, connectivity metrics) that can drastically alter results, making cross-study comparisons difficult.
• Site Variability: Differences in scanners, acquisition parameters, and sample populations across sites often introduce confounds that prevent models trained on one dataset from generalizing to another.
• Clinical vs. Controlled Designs: Many existing studies rely on heterogeneous clinical populations or complex cognitive tasks, introducing uncontrolled inter-subject variability that obscures the true impact of methodological choices.

The authors aim to systematically assess how these pipeline variations affect ML classification performance by using a highly controlled within-subject design (Eyes Open vs. Eyes Closed) across two independent, multi-site datasets.

2. Methodology

Datasets

The study utilized two independent datasets from healthy participants:

1. IHB RAS (Russia): 84 subjects, 3T Philips scanner. Sessions included Eyes Closed (EC) and Eyes Open with Fixation (EO-F).
2. China Dataset: 48 subjects, 3T Siemens scanner. Sessions included EC and EO.
   Total subjects: 132. All subjects underwent multiple scanning sessions, allowing for repeated-measures analysis.

Experimental Design

The study benchmarked 256 distinct processing pipelines by systematically varying four critical factors:

1. Data Acquisition: Two different sites/scanners.
2. BOLD Signal Denoising: 16 strategies, including:
   • Head Motion Parameters (12 or 24 regressors).
   • CompCor variants (aCompCor, tCompCor, with 10 or 50% variance components).
   • ICA-AROMA (Aggressive and Non-Aggressive).
   • Global Signal Regression (GSR) vs. No GSR.
3. Brain Parcellation: 4 atlases with varying resolutions and origins:
   • AAL: 116 regions (Anatomical).
   • Schaefer200: 200 regions (Functional).
   • Brainnetome: 246 regions (Anatomical + Functional).
   • HCPex: 426 regions (High-resolution multimodal).
4. Connectivity Estimation: 4 metrics:
   • Pearson Correlation.
   • Tangent Space Parametrization.
   • Partial Correlation.
   • Regularized Partial Correlation (Graphical Lasso).

Validation Strategy

To test robustness and generalizability, two validation protocols were employed:

1. Direct Cross-Site Validation: Train on Site A, test on Site B (no overlap).
2. Few-Shot Domain Adaptation: Train on Site A + a small subset of Site B, test on the remainder of Site B.
   Note: Subject-level splitting was enforced (all scans from one subject stayed in either train or test) to prevent data leakage.

Machine Learning Model

• Classifier: Logistic Regression with L2 regularization.
• Feature Engineering: Functional Connectivity (FC) matrices were vectorized (upper triangle), and Principal Component Analysis (PCA) was applied to retain 95% of variance before classification.
• Metrics: Classification Accuracy, ROC-AUC, Brier Score, QC-FC (correlation between motion and FC), and Intraclass Correlation Coefficient (ICC) for reliability.

3. Key Contributions

1. Comprehensive Benchmarking: The first large-scale evaluation of 256 pipeline permutations specifically for a binary brain state classification task across two distinct sites.
2. Demonstration of Robustness: The study proves that for well-defined physiological states (EO vs. EC), FC-based classifiers achieve high accuracy (~80%) regardless of significant variations in preprocessing pipelines.
3. Identification of Optimal Pipelines: The authors identified specific combinations that maximize both accuracy and stability:
   • Best FC Metric: Tangent Space Parametrization (followed by Pearson Correlation).
   • Best Atlas: Brainnetome (for accuracy) and Schaefer200 (for reliability/ICC).
   • Best Denoising: CompCor-based strategies (specifically aCompCor with 24 motion parameters) without Global Signal Regression (GSR).
4. Cross-Site Generalization: Successful transfer of models between the Russian and Chinese datasets, validating the utility of FC features for multi-site biomarker discovery.
5. Stable Biomarker Identification: Through repeated subsampling and Haufe transformation, the study identified stable functional connections distinguishing EO and EC states, specifically highlighting:
   • EO State: Strengthened coupling between Control, Default Mode, and Somatomotor networks.
   • EC State: Heightened coupling between Visual, Somatomotor, and Dorsal Attention networks.

4. Key Results

• Classification Accuracy: Despite pipeline variability, the best pipelines achieved an average cross-site accuracy of ~80% and an ROC-AUC of ~0.92.
• Factor Importance (Linear Mixed-Effects Model):
  • Functional Connectivity (FC) Type: The most significant determinant of performance ($p < 10^{-15}$). Tangent space significantly outperformed partial correlations.
  • Atlas: Significant effect, with Brainnetome and AAL performing well for accuracy, while Schaefer200 showed superior reliability (ICC).
  • Denoising Strategy: Significant effect. ICA-AROMA (Aggressive) consistently performed worse than CompCor strategies.
  • Global Signal Regression (GSR): Had a negligible and inconsistent effect on performance, suggesting it is not a critical factor for this specific task.
• Reliability (ICC): Masking near-zero edges significantly improved ICC scores. Tangent space FC combined with the Schaefer200 atlas yielded the highest reliability (ICC $\approx$ 0.67).
• Motion Artifacts: QC-FC analysis showed that precision-regularized estimators (Glasso) and CompCor strategies without GSR resulted in the lowest correlation between motion and connectivity, indicating better noise removal.

5. Significance and Implications

• Resilience of rs-fMRI Biomarkers: The findings suggest that for strong, physiologically distinct brain states, the core information in rs-fMRI is robust to methodological noise. This supports the feasibility of developing reproducible neuroimaging biomarkers even without perfect pipeline harmonization across sites.
• Guidance for Future Studies:
  • Researchers should prioritize Tangent Space Parametrization over partial correlations for classification tasks to balance accuracy and stability.
  • CompCor is recommended over ICA-AROMA aggressive strategies for this specific application.
  • The choice of Atlas matters less for accuracy than the choice of FC metric, but Schaefer200 is preferred for reliability studies.
• Limitations & Future Work: The study focused on a high-contrast task (EO/EC). The authors caution that these results may not extend to subtle cognitive tasks or clinical populations with smaller effect sizes. Future work should investigate more complex ML architectures and clinical cohorts.

In conclusion, this paper provides a rigorous validation that rs-fMRI functional connectivity is a reliable feature for brain state classification, provided that appropriate connectivity metrics (like Tangent Space) and denoising strategies (CompCor) are selected, even when data is acquired across different sites and scanners.