Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG

Brain-OF is the first omnifunctional foundation model that unifies fMRI, EEG, and MEG data through novel components like the Any-Resolution Neural Signal Sampler and Dual-Domain Masked Modeling to overcome modality limitations and achieve superior performance across diverse neuroscience tasks.

The Gist

Imagine your brain is a massive, bustling city. To understand how this city works, scientists use different types of "cameras" to take pictures and record sounds. But here's the problem: each camera sees the city differently.

• fMRI is like a high-resolution satellite photo. It shows you exactly where things are happening in the city (great detail on location), but the photo is blurry in time—it's like taking a picture every few seconds, so you miss the fast-moving cars.
• EEG and MEG are like super-fast security cameras on the street corners. They capture every split-second movement and sound (great speed), but they struggle to tell you exactly which building the sound came from (blurry location).

For a long time, scientists built AI models that could only look at one type of camera at a time. If you wanted to understand the city's traffic, you had to hire a specialist for the satellite photos and a different specialist for the street cameras. They couldn't talk to each other, and they often missed the big picture.

Enter Brain-OF.

What is Brain-OF?

Think of Brain-OF as a super-intelligent "City Manager" who has been trained to watch all the cameras at once. It is the first AI model designed to understand fMRI, EEG, and MEG data simultaneously. Instead of hiring separate experts, Brain-OF is a single "omnifunctional" brain that learns from all three sources together.

How Does It Work? (The Secret Sauce)

The researchers had to solve three big puzzles to make this City Manager work:

1. The "Language Barrier" (Any-Resolution Neural Signal Sampler)
The satellite photos and street cameras speak different "languages" and have different speeds. Brain-OF uses a special translator called ARNESS.

• Analogy: Imagine ARNESS is a universal translator booth. No matter if you feed it a slow, detailed satellite image or a rapid-fire street video, ARNESS converts them all into the same "secret code" (a shared semantic space). Now, the AI can read them all without getting confused by the differences in speed or detail.

2. The "Crowded Room" Problem (Sparse Mixture of Experts)
When you put all this data into one brain, it can get chaotic. The AI might get distracted by the noise.

• Analogy: Imagine Brain-OF is a giant meeting room with many specialized consultants (Experts).
  • Some consultants are Generalists who know the basics of all cities (Modality-Invariant).
  • Other consultants are Specialists who only know about satellite photos or only about street cameras.
  • Brain-OF uses a smart Traffic Cop (DINT Attention) to decide which consultant to listen to for each specific question. If the question is about location, it calls the satellite expert. If it's about speed, it calls the street expert. This prevents the AI from getting overwhelmed and ensures it listens to the right voice at the right time.

3. The "Two-Part Puzzle" (Masked Temporal-Frequency Modeling)
To learn deeply, the AI plays a game of "fill in the blanks."

• Analogy: Imagine you are listening to a song, but someone mutes parts of the melody (time) and parts of the harmony (frequency).
  • Old AI models would just try to guess the missing melody.
  • Brain-OF is forced to guess both the missing melody and the missing harmony at the same time. This forces it to understand how the rhythm and the pitch work together, giving it a much deeper understanding of how the brain actually "sings."

Why Does This Matter?

1. It's Smarter and More Accurate
Because Brain-OF combines the "where" (from fMRI) with the "when" (from EEG/MEG), it creates a much clearer picture of brain activity. In tests, it beat all previous models at tasks like:

• Detecting seizures (epilepsy).
• Diagnosing Alzheimer's disease.
• Predicting a person's age based on brain activity.
• Recognizing emotions.

2. It's a "Zero-Shot" Hero
Because it learned from so many different types of data, Brain-OF is very good at figuring out new tasks it hasn't seen before. It's like a chef who has cooked with every ingredient in the world; if you give them a new recipe, they can probably figure it out immediately.

3. It Helps Everyone
The researchers released the biggest version of this model (Brain-OF Huge) for free. This is like giving every neuroscientist and doctor a super-powered microscope that they don't have to build from scratch. It lowers the barrier for researchers who don't have millions of dollars to train their own AI, potentially speeding up cures for brain diseases.

The Bottom Line

Brain-OF is a breakthrough because it stopped treating brain signals as separate, isolated puzzles. Instead, it built a single, unified system that understands the brain's full story—combining the high-definition location of fMRI with the lightning-fast speed of EEG and MEG. It's the first step toward a truly "all-seeing" AI for neuroscience.

Technical Summary

1. Problem Statement

Current brain foundation models face three critical limitations:

• Modality Silos: Existing models are typically restricted to a single functional modality (e.g., only fMRI or only EEG), failing to exploit the complementary spatiotemporal dynamics between different imaging techniques.
• Data Scarcity & Heterogeneity: Neuroimaging data is orders of magnitude smaller than NLP or CV corpora. Furthermore, integrating data is difficult due to significant structural variability (different channel counts, sampling rates, acquisition protocols) and fundamental physical differences in signal origins (hemodynamic vs. electromagnetic).
• Semantic Misalignment: Standard attention mechanisms often struggle with the low signal-to-noise ratio (SNR) of brain signals, attending to irrelevant fluctuations rather than biologically meaningful patterns. Additionally, existing pretraining objectives often treat frequency information as a static auxiliary input rather than a target for active reconstruction.

2. Methodology: Brain-OF Architecture

Brain-OF is a unified framework designed to jointly pretrain on fMRI, EEG, and MEG. It addresses heterogeneity through a novel architecture and pretraining strategy.

A. Any-Resolution Neural Signal Sampler (ARNESS)

To handle inputs with diverse spatial, temporal, and spectral resolutions:

• Standardization: Raw signals are segmented into non-overlapping patches and flattened into sequences.
• Cross-Attention Projection: ARNESS uses a Perceiver-style cross-attention mechanism to project heterogeneous sequences into a shared semantic space using a fixed number of learnable latent tokens ($Z$).
• Function: This allows the model to handle arbitrary resolutions uniformly, reduces token counts for high-dimensional signals (lowering compute costs), and enables seamless multimodal fusion by serially resampling multiple unimodal sequences into a single representation without dedicated fusion branches.

B. Backbone Architecture

The core backbone integrates two advanced mechanisms to manage semantic diversity and noise:

1. DINT Attention (Differential-Integral Attention):
   • Differential Component: Computes the difference between two attention distributions to suppress "attention noise" and irrelevant context, which is crucial for noisy neuroimaging data.
   • Integral Component: Computes a global importance score to capture long-range dependencies.
   • Result: A robust attention mechanism that amplifies global relevance while suppressing local noise.
2. Sparse Mixture of Experts (Sparse MoE):
   • Problem Solved: Prevents the "modality seesaw effect" where optimizing for one modality degrades performance on others.
   • Mechanism: Uses a combination of Shared Experts (to capture modality-invariant representations) and Routed Experts (to specialize in modality-specific semantics).
   • Routing: Employs an auxiliary-loss-free load-balancing strategy to dynamically route tokens to the top-$k$ experts, ensuring balanced utilization without collapsing.

C. Pretraining Objective: Masked Temporal-Frequency Modeling (MTFM)

Unlike standard Masked Autoencoders (MAE) that only reconstruct time-domain signals, Brain-OF introduces a dual-domain pretraining objective:

• Process:
  1. Transform signals into the frequency domain (Spatial FFT for fMRI; Temporal FFT for EEG/MEG).
  2. Apply random masking in both the frequency domain ($M_F$) and the time domain ($M_T$).
  3. Reconstruct the signal in both domains simultaneously.
• Loss Function: Combines reconstruction losses for the time domain and the magnitude of the frequency domain, weighted by $\alpha=0.8$.
• Goal: Forces the model to internalize coupled time-frequency dynamics, leveraging the high temporal resolution of EEG/MEG to compensate for fMRI latency, and vice versa.

3. Key Contributions

1. First Omnifunctional Model: Brain-OF is the first foundation model to jointly pretrain on fMRI, EEG, and MEG, creating a unified framework for cross-modal neuroscience.
2. Heterogeneity-Aligned Architecture: The introduction of ARNESS for resolution alignment and Sparse MoE for semantic specialization allows the model to scale effectively across diverse datasets.
3. Dual-Domain Pretraining: The MTFM objective explicitly enforces joint reconstruction in time and frequency, leading to richer neural representations than time-only approaches.
4. Large-Scale Corpus: The model is pretrained on a curated corpus of 37 public datasets comprising over 32,000 unique participants and 5.87 million samples.

4. Experimental Results

Brain-OF was evaluated on 7 downstream tasks (emotion recognition, seizure detection, abnormality detection, AD diagnosis, ADHD identification, brain age prediction, and aging classification) across 11 experimental settings.

• State-of-the-Art Performance: The Brain-OF Huge variant (1.7B total parameters, 500M active) achieved SOTA results in 7 out of 9 tasks, with an average rank of 1.8.
• Scaling Laws: Performance improved monotonically with model size (Base $\to$ Large $\to$ Huge). For example, on the CamCAN brain-age regression task, MAE decreased from 8.99 (Base) to 7.87 (Huge).
• Cross-Modal Generalization: Brain-OF outperformed single-modality foundation models (e.g., LaBraM, BrainHarmonix-F) even when tested on modalities different from their primary training domain, demonstrating robust transferability.
• Multimodal Fusion: Serial fusion of modalities (e.g., fMRI + MEG) consistently outperformed unimodal baselines, proving the model's ability to integrate complementary spatial and temporal information.
• Ablation Studies:
  • Removing MTFM degraded performance, confirming the necessity of dual-domain learning.
  • Removing specific modalities (e.g., EEG) did not cause catastrophic failure on EEG tasks, indicating strong modality-invariant representation learning.
  • Replacing Sparse MoE with dense FFNs caused significant performance drops, especially on fMRI tasks, validating the need for expert specialization.

5. Significance and Impact

• Unified Neuroscience Framework: Brain-OF bridges the gap between hemodynamic (fMRI) and electromagnetic (EEG/MEG) imaging, enabling a holistic characterization of brain dynamics.
• Clinical and Research Utility: By learning shared representations, the model can transfer knowledge from capital-intensive modalities (fMRI/MEG) to accessible, wearable ones (EEG), potentially accelerating the deployment of neurotechnologies in real-world settings.
• Open Source: The release of Brain-OF Huge as an open-source model lowers the barrier to entry for researchers, providing a pre-validated backbone for diverse neuroimaging applications without the need for massive computational resources to train from scratch.
• Interpretability: Visualization of attention maps and reconstruction quality confirms that the model learns biologically plausible features (e.g., specific brain regions associated with Alzheimer's) rather than spurious correlations.

In summary, Brain-OF represents a paradigm shift in brain AI, moving from single-modality, task-specific models to a scalable, omnifunctional foundation model capable of understanding the complex, multi-faceted nature of human brain activity.