🧬 biology

The Variance Brain Foundation Models Forgot: Third-Order Statistics Predict Cognition Where Billion-Parameter Models Fail

Despite their scale, current brain foundation models fail to predict cognitive performance because their pretraining objectives discard crucial third-order statistical structures (co-skewness), a limitation that can be overcome by a simple, pretraining-free linear pipeline that explicitly preserves these higher-order features to outperform both the models and standard functional connectivity baselines.

Original authors: Giovanni Marraffini, Gabriel Mahuas, Trinidad Borrell, Victoria Shevchenko, Demian Wassermann

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giovanni Marraffini, Gabriel Mahuas, Trinidad Borrell, Victoria Shevchenko, Demian Wassermann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Picture: The "Smart" Brain Model That Missed the Point

Imagine you have a super-smart student (a Brain Foundation Model) who has read every single textbook on how the brain works. You give them a test: "Look at this brain scan and tell me how smart the person is."

You expect this student to ace the test because they are huge, complex, and have been trained on massive amounts of data. But instead, they fail miserably. A simple, old-fashioned calculator (a linear regression) that just looks at the raw brain scan data actually gets a much better score.

Even stranger: when the researchers made the "smart student" even bigger and more powerful, they got worse at the test.

This paper explains why this happened and how they fixed it.

1. The Problem: Listening to the Wrong Noise

Think of a brain scan (fMRI) like a recording of a busy city street.

The Noise: The recording is dominated by loud, obvious sounds: the rumble of a bus (heartbeats), the wind (breathing), and people shuffling their feet (head movement). These are the "dominant" sounds.
The Signal: The actual conversation between two people (cognitive thinking) is quiet and subtle. It's hidden in the complex, rhythmic patterns of how three or more people talk over each other at once.

What the AI did: The AI models were trained to "reconstruct" the recording. Their goal was to make the output sound exactly like the input. Because the loud noises (heartbeats, breathing) were so much louder than the quiet conversation, the AI learned to be perfect at copying the noise. It became an expert at predicting the bus rumble, but it completely ignored the quiet conversation.

The Result: The AI models were so good at copying the "noise" that they forgot the "signal." They failed to predict human cognition because they were too busy focusing on the loud, boring parts of the data.

2. The "Inverse Scaling" Paradox

Usually, in AI, if you make a model bigger (give it more brain power), it gets smarter.

The Paper's Finding: Here, the opposite happened. The researchers tested a "small" AI (111 million parameters) and a "huge" AI (650 million parameters).
The Twist: The huge AI was worse at predicting cognition than the small one.
Why? The bigger AI was even better at copying the loud noise. It had more capacity to memorize the heartbeats and breathing, so it drowned out the subtle cognitive signals even more effectively. It was like hiring a giant, super-accurate microphone that only amplified the traffic noise and drowned out the conversation.

3. The Solution: Looking at "Third-Order" Patterns

The researchers realized that human thinking isn't just about two brain regions talking to each other (which is what standard brain scans usually measure). It's about three regions interacting in a specific, complex way at the same time.

The Analogy:
- Second-Order (Standard): Two friends, Alice and Bob, talking. You can measure how often they speak.
- Third-Order (The Secret Sauce): Alice, Bob, and Charlie are in a room. The interesting part isn't just Alice talking to Bob; it's the specific moment when all three react to a joke simultaneously. This complex, three-way interaction is where the "thinking" lives.

The standard AI models were blind to this "third-order" interaction because their training method (trying to copy the raw sound) destroyed it.

4. The Fix: The "Denoising Filter"

Instead of trying to build a bigger AI, the researchers built a simple, clever filter.

The Method: They used a mathematical technique called Tucker decomposition. Think of this as a special pair of glasses that filters out the loud bus noises and the wind, leaving only the complex, three-way conversations.
The Result:
1. They took the raw brain data.
2. They passed it through this "third-order filter."
3. They measured the connections inside this filtered space.
4. Boom: This simple, non-AI method predicted human cognition better than any of the massive, pre-trained AI models. It was so good that it beat the previous best results in the field, all without needing a supercomputer or expensive training.

5. The "Fine-Tuning" Miracle

The researchers then asked: "Can we teach the big AI to see this?"

They took the massive AI and gave it a new homework assignment. Instead of just "copy the sound," they told it: "Copy the sound, but make sure you keep the complex three-way conversations intact."

The Outcome: When they did this, the AI suddenly became as good as the simple filter. It went from failing the test to acing it.
The Lesson: The AI wasn't "dumb" or "too small." It was just trained with the wrong goal. The problem wasn't the architecture (the brain of the model); the problem was the objective (what it was told to learn).

Summary of Key Takeaways

Bigger isn't always better: In brain modeling, making the AI bigger just made it better at copying the "noise" (heartbeats/motion) and worse at finding the "signal" (thinking).
The signal is hidden in the complex: Human cognition lives in complex, three-way interactions (third-order statistics), which standard AI training methods accidentally delete.
Simple is powerful: A simple mathematical filter that focuses on these complex interactions works better than billion-parameter AI models.
The goal matters most: If you teach an AI the right thing to look for (preserving those complex interactions), even a standard model can outperform the "state-of-the-art" giants.

In a nutshell: The researchers found that the "smart" AI models were too busy listening to the traffic noise to hear the conversation. By building a filter that mutes the traffic and amplifies the complex group conversations, they found a way to predict human thinking better than any giant AI model could.

Technical Summary: The Variance Brain Foundation Models Forgot

Problem Statement
Brain Foundation Models (BFMs) are self-supervised Transformers pretrained on fMRI data with the goal of learning rich representations of brain activity to predict cognitive performance. The prevailing hypothesis suggests that deep Transformers, with their capacity to model complex interactions beyond pairwise correlations, should outperform classical Functional Connectivity (FC) baselines. However, this paper demonstrates that current state-of-the-art BFMs (specifically BrainLM, Brain-JEPA, and BrainMass) fail to capture the variance necessary for cognition prediction. Instead, they underperform a simple linear regression on raw FC matrices (approx. 80K parameters). Furthermore, the paper identifies an "inverse scaling" phenomenon where larger models (e.g., BrainLM's 650M parameters) perform worse than smaller ones (111M parameters), suggesting that increased capacity is being wasted on fitting dominant, non-cognitive noise rather than task-relevant structure.

Methodology
The authors employ a rigorous evaluation protocol based on Ooi et al. [2022], utilizing replicated nested cross-validation (20 repetitions × 10 folds) across two datasets: AOMIC-ID1000 (N=876) and HCP (N=955), with two parcellations (AAL-424 and Schaefer-400).

Statistical Analysis via Cumulants: The paper frames the analysis using the joint cumulant expansion.
- $\kappa_2$ (Second-order): Represents pairwise covariance (standard FC).
- $\kappa_3$ (Third-order): Represents three-way co-skewness, capturing non-Gaussian structure invisible to FC.
  The authors analyze BFM reconstructions to determine which statistical orders are preserved. They find that while $\kappa_2$ is partially preserved, $\kappa_3$ is largely destroyed by the Mean Squared Error (MSE) pretraining objective, which prioritizes high-variance physiological noise (cardiac, respiratory, motion) over synergistic cognitive signals.
Proposed Linear Pipeline (Tucker Decomposition): To recover the lost $\kappa_3$ structure without deep learning, the authors propose a linear pipeline:
- Compute the empirical co-skewness tensor ( $S \in \mathbb{R}^{P \times P \times P}$ ) from the training data.
- Perform a Tucker decomposition (Higher-Order SVD) to extract a low-rank spatial subspace ( $U$ ) that maximizes third-order structure.
- Project the raw ROI timeseries into this subspace.
- Compute FC within this $\kappa_3$ -informed subspace.
- Feed these features into a Kernel Ridge Regression (KRR) predictor.
Finetuning Strategy: To test if the BFM architecture itself is the bottleneck or merely the pretraining objective, the authors finetune BrainLM using a "dual-moment" loss. This loss aligns the reconstruction not just with the raw signal, but with the FC computed inside the optimal $\kappa_3$ -Tucker subspace (using a Log-Cholesky distance metric).

Key Results

BFM Underperformance: Under the Ooi et al. protocol, all three BFMs achieve negligible cognition prediction scores (Pearson $r \approx 0.00$ to $0.18$), significantly worse than raw FC baselines ( $r \approx 0.31$ to $0.39$).
Inverse Scaling: In the BrainLM family, the 650M model predicts cognition worse than the 111M model, confirming that scaling up parameters without changing the objective exacerbates the failure to capture cognitive variance.
Cumulant Preservation Failure: Analysis of BFM reconstructions shows that $\kappa_2$ error is moderate (29–47% relative error), but $\kappa_3$ error is catastrophic (NMSE 2.6–6.2 $\times$ input energy), indicating the models discard the specific statistical order where cognition-relevant information resides.
Tucker Subspace Superiority: The proposed linear pipeline (FC computed in the $\kappa_3$ $κ_{3}$ -optimal Tucker subspace) sets a new state-of-the-art (SOTA) for resting-state fMRI cognition prediction. It outperforms raw FC and all pretrained BFMs across all datasets and parcellations without requiring pretraining or GPUs.
- On HCP Schaefer-400, the method achieves $r = 0.571 \pm 0.089$ , surpassing prior SOTA results (e.g., $r = 0.520$ for principal gradients).
Finetuning Recovery: When BrainLM is finetuned with the cumulant-informed loss (dual-moment), its reconstruction-based FC prediction recovers the raw-FC ceiling ( $r \approx 0.39$ on HCP, matching the baseline). This proves the bottleneck is the pretraining objective, not the Transformer architecture or model size.
Embedding Failure: Despite the success of the reconstruction output, the CLS embeddings of the BFMs remain near-zero in predictive power even after finetuning, suggesting the latent geometry is fixed during pretraining and not reorganized by the decoder-level loss.

Significance and Claims
The paper claims that the failure of brain foundation models is not due to a lack of capacity but a "variance allocation problem" inherent to reconstruction-based pretraining objectives. These objectives prioritize dominant physiological noise ( $\kappa_2$ ) and discard higher-order synergistic information ( $\kappa_3$ ) that predicts cognition.

The authors argue that:

Evaluation Standards: Foundation models must be evaluated against classical domain-knowledge baselines (like raw FC) before scaling further.
Explicit Decomposition: When the task-relevant subspace is known (in this case, the $\kappa_3$ -optimal subspace), explicit linear decompositions (Tucker) are cleaner and more effective than learning latent representations via self-supervision.
Objective vs. Architecture: The bottleneck in BFM performance is the pretraining objective, not the model architecture. Aligning the loss function with the specific statistical structure of the task (third-order co-skewness) can recover performance without changing the model structure.

The paper concludes that cognition-relevant structure in resting-state fMRI lives at least at the third cumulant, and that simple, GPU-free linear pipelines exploiting this structure currently outperform billion-parameter foundation models.