Clinical profile impacts the replicability of multivariate brain-behaviour associations

This study using UK Biobank data demonstrates that while multivariate brain-behaviour associations generally require sample sizes of approximately 500 for replicability, targeting specific clinical subpopulations, such as individuals with a history of psychoactive substance use, can yield robust results with even smaller samples.

The Gist

Imagine you are trying to figure out how a car's engine (the brain) relates to how fast it can drive on a track (the behavior).

For a long time, scientists have been trying to find the perfect formula to link these two things. But they've been running into a big problem: Replication.

The Problem: The "Small Sample" Trap

In the past, researchers would grab a small group of people (say, 30 or 50), run their brains through a scanner, give them a few tests, and try to find a pattern.

The paper argues that this is like trying to predict the weather by looking at the sky for only 10 seconds. You might see a cloud and guess "rain," but you'd be wrong most of the time. When other scientists tried to repeat these small studies with new groups of people, the patterns disappeared. The results were just noise, not real signals.

A recent famous study suggested that to get a real answer, you need thousands of people. It's like needing to watch the sky for a whole week to know if it's actually going to rain.

The New Discovery: It's About Who You Ask

This new paper asks a clever question: "Do we really need thousands of random people, or do we just need the right people?"

They used a massive database called the UK Biobank, which has data on over 40,000 people. They didn't just look at everyone; they sliced the group into four different "cohorts" (teams):

1. The Full Team: A random mix of everyone.
2. The Healthy Team: People with no major health issues.
3. The Hypertension Team: People with high blood pressure.
4. The "Psychoactive" Team: People with a history of using substances like alcohol, opioids, or cocaine.

The Experiment: The "Taste Test"

Think of the researchers as chefs trying to find the perfect recipe. They took small batches of ingredients (sample sizes) and tried to cook a dish (a brain-behavior model).

• The Result with Random People: If you use a random mix of people, you need a huge pot (about 500 people) before the dish tastes consistent and reliable. If you use fewer than that, the flavor is all over the place.
• The Result with the "Psychoactive" Team: Here is the magic. When they used the group of people with a history of substance use, they found a strong, clear pattern with much fewer people (around 500, but the signal was much stronger than in the other groups).

The Analogy:
Imagine you are trying to find a specific type of bird.

• The Healthy Group: You are looking in a giant forest with thousands of different birds. You need to look at thousands of birds to be sure you've found the one you want.
• The Targeted Group: You go to a specific swamp where that specific bird always hangs out. You only need to look at a few birds to find it. The "effect" (the bird's presence) is so obvious in that specific group that you don't need a massive crowd to see it.

What Did They Actually Find?

1. You still need a decent number: You can't get away with just 30 people. Even the best groups need about 500 people to get a reliable answer. The "thousands" rule is a bit too strict for targeted groups, but the "dozens" rule is definitely too loose.
2. Targeted groups are powerful: People with specific clinical histories (like substance use) showed a much stronger link between their brain structure and their cognitive test scores than healthy people did. This means smaller studies focusing on specific clinical groups can actually be very successful.
3. The "Loading" Stability: The researchers also checked which brain parts were important. They found that once you hit that 500-person mark, the list of "important brain parts" stopped changing. It became stable. Below 500, the list was chaotic and unreliable.

The Takeaway for Everyone

This paper is a relief for scientists who can't afford to scan 10,000 people.

It tells us: "Don't just throw more people at the problem. Be smarter about who you study."

If you are studying a specific condition (like addiction, depression, or a specific disease), you might not need a massive, expensive study. You just need a focused group of about 500 people who actually have that condition. By narrowing the focus, you make the signal louder and the results more reliable, even with a smaller sample size.

In short: To find the truth about the brain, you don't always need a bigger crowd; sometimes, you just need the right crowd.

Technical Summary

1. Problem Statement

Multivariate brain-behaviour analyses, particularly Canonical Correlation Analysis (CCA), have faced a "replication crisis." Recent literature (e.g., Marek et al., 2022) suggests that thousands of participants are required to achieve replicable results, as smaller samples lead to overfitting and unstable variable loadings. However, it remains unclear whether targeted clinical subpopulations (which may exhibit larger effect sizes) can achieve replicability with moderate sample sizes (hundreds rather than thousands). Additionally, the specific impact of cohort composition (e.g., healthy vs. clinical) on the stability of CCA results has not been systematically tested using structural brain imaging data.

2. Methodology

Data Source and Preprocessing:

• Dataset: UK Biobank (N = 40,514 participants).
• Brain Features: 162 diffusion-weighted MRI (dMRI) phenotypes derived from 27 white matter tracts. Metrics included Fractional Anisotropy (FA), Mean Diffusivity (MD), and Neurite Orientation Dispersion and Density Imaging (NODDI) metrics (ICVF, ISOVF, OD).
• Behavioural Features: 43 measures from nine cognitive tests (e.g., fluid intelligence, reaction time, memory).
• Confounding: 18 variables (sex, blood pressure, handedness, etc.) were regressed out. Age was excluded as a confound due to its role as an effect modifier.
• Dimensionality Reduction: Principal Component Analysis (PCA) reduced both Brain and Behavioural datasets to 20 principal components each.

Cohort Definition:
Four distinct cohorts were defined based on clinical profiles:

1. Full: All available participants (N=40,514).
2. Healthy: Participants with no diagnosis codes (N=6,676).
3. Psychoactive: Participants with a history of psychoactive substance use (N=4,725).
4. Hypertension: Participants with essential hypertension (N=7,768).

Experimental Design:

• Sample Size Variation: Subsamples were generated at 30 logarithmically spaced sizes (ranging from N=50 to N=20,257).
• Model Types:
  1. Standard CCA: A single model fitted on the entire training set.
  2. Cross-Validated (CV) CCA: An ensemble model consisting of 100 CCA instances (5-fold CV repeated 20 times) to reduce overfitting.
• Validation: For each sample size, models were trained on a subset and tested on a held-out "Test set" (50% of the cohort) to assess generalizability ($r_{test}$).
• Metrics:
  • Canonical Correlation: Strength of the relationship between brain and behaviour latent factors.
  • Variable Loadings: The contribution of specific features to the canonical axes.
  • Null Models: Permutation tests (shuffled data) to establish significance thresholds.

3. Key Contributions

1. Cohort-Specific Replicability: Demonstrated that clinical cohort composition significantly alters the sample size required for replicable multivariate associations.
2. Targeted Cohort Efficiency: Identified that a specific clinical subgroup (Psychoactive substance use) achieves high replicability with significantly fewer samples compared to healthy controls.
3. Stability Thresholds: Established that while canonical correlations require large samples to stabilize, variable loadings (interpretability) become stable at moderate sample sizes (~500) for targeted cohorts.
4. Cross-Validation Utility: Clarified that cross-validation sampling effectively reduces effect size inflation (overfitting) only at very small sample sizes (N < 100), but does not compensate for the lack of power in moderate-sized studies.

4. Key Results

• Sample Size Requirements:

  • Across all cohorts, sample sizes below N=213 produced effect sizes indistinguishable from zero.
  • A minimum of N=487 was required to produce statistically significant canonical correlations ($r_{test}$) and stable variable loadings across all cohorts.
  • Standard CCA models without cross-validation showed severe overfitting at small sample sizes (e.g., $r_{train} \approx 1.0$ at N=50), while CV models mitigated this only for N < 100.

• Impact of Clinical Profile (The "Psychoactive" Effect):

  • The Psychoactive cohort consistently outperformed others. At N=487, it achieved a mean $r_{test}$ of 0.24, whereas the Healthy cohort only reached this level at N=1,116 (more than double the sample size).
  • The Psychoactive cohort required the lowest sample size (487–907) to reach a target correlation of 0.24, compared to Full (907–1116), Hypertension (907–1372), and Healthy (1372–2077).

• Variable Loading Stability:

  • For sample sizes N $\ge$ 500, the order and magnitude of variable loadings (feature contributions) stabilized and remained consistent across cohorts.
  • Below N=100–200, loadings were unstable with high variance and no clear pattern.
  • Interpretation: Strongest positive brain loadings were associated with thalamo-cortical ICVF; strongest negative loadings with MD. Strongest behavioural loadings were linked to symbol digit substitution (positive) and reaction time/trail making (negative).

• Effect Size Inflation:

  • Cross-validation reduced the gap between training and testing correlations, but only for very small samples (N < 100). For N > 100, both CV and non-CV models showed similar levels of overfitting relative to sample size.

5. Significance and Implications

• Redefining Sample Size Standards: The study challenges the notion that "thousands" are universally required for multivariate brain-behaviour studies. It suggests that moderate sample sizes (hundreds) are sufficient if the cohort is clinically targeted to maximize effect sizes.
• Experimental Design: Researchers studying specific clinical populations (e.g., substance use disorders) can achieve robust, replicable findings with smaller, more focused datasets, making large-scale neuroimaging more accessible for specific research questions.
• Methodological Rigor: The paper emphasizes the importance of examining variable loadings alongside correlation strength. A model may show a significant correlation, but if the loadings are unstable (N < 500), the biological interpretation of the features is unreliable.
• Future Directions: The authors call for a re-evaluation of what constitutes a "significant" CCA model, arguing that statistical reliability of parameter estimates is crucial for moving from descriptive models to predictive ones.

In conclusion, this paper provides empirical evidence that cohort composition is a critical determinant of replicability in multivariate neuroimaging, offering a pathway for smaller, targeted studies to yield scientifically robust brain-behaviour associations.