🤖 machine learning

EEG-FM-Audit: A Systematic Evaluation and Analysis Pipeline for EEG Foundation Models

The paper introduces EEG-FM-Audit, a systematic evaluation pipeline comprising ASHA-driven benchmarking, paradigm-level ablation, and neurophysiological probing to address transparency issues in EEG Foundation Models, revealing that optimized supervised baselines often outperform these models while providing deeper insights into their reliance on physiological features.

Original authors: Xianheng Wang, Yige Yang, Damien Coyle

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xianheng Wang, Yige Yang, Damien Coyle

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge who is the best chef in a city. You have a group of famous, high-tech "Foundation Model" chefs who claim they can cook anything because they've tasted millions of dishes from around the world. Then, you have a group of local "Supervised" chefs who only cook for their specific neighborhood.

The problem is, the famous chefs are always winning the competitions, but the local chefs are often being judged with a rusty spoon and a broken oven, while the famous chefs get the best equipment. This makes the famous chefs look better than they actually are.

This paper, EEG-FM-Audit, is like a new, super-fair judging system designed to fix this mess. The authors built a three-step "audit" to see if these high-tech brain-computer models are actually geniuses or just lucky.

Here is how their system works, explained simply:

1. The "Fair Play" Check (ASHA-Driven Benchmarking)

The Analogy: Imagine the local chefs are given a brand-new, professional kitchen and a team of experts to help them tune their recipes perfectly before the competition starts.
What they did: The authors realized that previous studies compared the fancy "Foundation Models" against "Supervised Models" that were barely tuned. So, they used a smart algorithm (called ASHA) to give the simple models the best possible settings, just like giving them the perfect oven temperature and fresh ingredients.
The Result: When they gave the simple models a fair shot, they often performed just as well as, or even better than, the massive, complex Foundation Models. The fancy models didn't always win; sometimes, a simple, well-tuned recipe was enough.

2. The "Deconstruction" Test (Paradigm-Level Ablation)

The Analogy: Imagine taking apart a fancy robot to see which parts are actually doing the work. Is the robot smart because of its giant brain (the pre-training), or is it just the body (the architecture) that's good?
What they did: They took the big Foundation Models and started removing their "superpowers" one by one:

Removed the "Big Data" training: They tested if the models needed to have eaten millions of brain signals to work, or if they could learn just from the specific task at hand.
Removed the "Language" brain: Some models use a language AI (like a chatbot) to understand brain waves. They removed this to see if the language part was actually helping or just adding weight.
The Result: It depends on the situation. If the data is small, the "Big Data" training is essential. But if the data is huge, the fancy pre-training isn't always necessary. Also, for some models, the "language brain" was critical; without it, the model collapsed. For others, it didn't matter much.

3. The "Brain Truth" Check (Neurophysiological Probing)

The Analogy: Imagine asking a detective, "How did you solve the crime?" If they say, "I looked at the muddy footprints," that's a good answer. If they say, "I looked at the color of the sky," that's suspicious.
What they did: They wanted to know if these AI models were actually looking at the right parts of the brain. They did three things:

Time Scramble: They mixed up the timing of the brain signals (like shuffling a deck of cards) to see if the model cared about the order of events.
Region Noise: They added static noise to specific parts of the brain (like the frontal lobe) to see if the model got confused.
Frequency Removal: They blocked out specific brain wave rhythms (like Alpha or Delta waves) to see if the model relied on them.
The Result: The models passed the test! They seemed to focus on the right brain areas and rhythms that real scientists know are important. For example, when detecting seizures, they focused on the frontal and temporal lobes, which matches real medical knowledge.

The Big Takeaway

The paper concludes that while these massive "Foundation Models" are impressive, they aren't a magic bullet yet.

Fairness matters: If you don't tune the simple models properly, you will overestimate how good the big models are.
Context is key: The fancy training methods only help when you have a lot of data.
They are learning the right things: The models are actually paying attention to real brain biology, not just random noise.

In short, the authors built a "truth detector" to ensure that when we say a new AI model is great, it's actually great, and not just because the competition was rigged.

Technical Summary: EEG-FM-Audit

Problem Statement

While Large EEG Foundation Models (FMs) have demonstrated potential for decoding EEG signals across diverse cognitive tasks, the field suffers from three critical limitations in existing literature:

Opaque Baseline Tuning: Comparisons between FMs and traditional supervised models often lack transparency in the optimization of supervised baselines. Traditional deep learning models are typically trained from scratch and are highly sensitive to hyperparameter configurations, yet they are frequently compared against FMs using suboptimal or default settings. This leads to an "optimization gap" where reported FM advantages may stem from poor baseline tuning rather than inherent architectural superiority.
Unverified Learning Paradigms: Existing studies often focus on model layers rather than the effectiveness of the underlying learning paradigms (e.g., large-scale pretraining, integration of pretrained language model backbones). It remains unclear whether the increased complexity of FM architectures yields tangible performance gains or if specific components are merely redundant.
Lack of Neurophysiological Interpretability: Current evaluations often treat FMs as "black boxes," lacking rigorous analysis of whether these models leverage valid temporal, spatial, and spectral EEG properties consistent with established neuroscientific findings.

Methodology: EEG-FM-Audit Pipeline

To address these gaps, the authors propose EEG-FM-Audit, a systematic evaluation and analysis pipeline comprising three primary components:

1. ASHA-Driven Benchmarking Protocol

To ensure fair comparisons, the authors implement a two-stage benchmarking protocol driven by the Asynchronous Successive Halving Algorithm (ASHA):

Stage 1 (Hyperparameter Search): ASHA is used to search the hyperparameter space ( $\Omega$ ) for each supervised baseline model, identifying the optimal configuration ( $\Theta^*$ ) based on validation balanced accuracy. Crucially, test data is strictly excluded from this search to prevent data leakage.
Stage 2 (Full Re-Training & Benchmarking): Baselines are retrained from scratch using both the optimized configuration ( $\Theta^*$ ) and default configurations ( $\Theta_{def}$ ). The final reported performance reflects the superior result between these two versions, ensuring the baseline represents its peak possible implementation.

2. Paradigm-Level Ablation Studies

The authors deconstruct FMs to isolate the contributions of specific learning paradigms through three ablation studies:

Pretraining-Source Ablation: Compares full FMs against variants pretrained only on the target dataset to evaluate the necessity of large-scale heterogeneous pretraining.
LLM-Rep Ablation: Reconstructs the FM encoder into a traditional supervised framework (removing pretraining) to determine if gains stem from the pretraining strategy or the underlying architecture.
No-GPT Ablation: Replaces pretrained language model backbones (e.g., GPT-2) with randomly initialized self-attention layers to assess the specific contribution of linguistic priors to neural decoding.

3. Neurophysiological Probing (NPP) Framework

This framework utilizes surrogate signals to quantify model reliance on specific physiological features:

Temporal Probing: Uses Fourier Phase Randomization (FPR) to destroy phase-locked temporal structures while preserving spatial covariance and power spectra.
Spatial Probing: Injects signal-aware Gaussian noise into predefined Regions of Interest (ROIs) corresponding to specific brain lobes to assess spatial dependency.
Spectral Probing: Performs spectral ablation by selectively removing standard EEG frequency bands (e.g., Alpha, Delta, Beta) to measure reliance on frequency-specific features.

Experimental Setup

The pipeline was applied to:

Four State-of-the-Art EEG-FMs: NeuroGPT, LaBraM-Base, NeuroLM-B, and EEGPT-Large.
Five Supervised Baselines: EEGNet, CSPNet, TS-SEFFNet, MSCFormer, and CTNet.
Three Public Datasets: TUEV (event detection), TUAB (abnormality detection), and BCI Competition IV-2b (motor imagery).

Key Results

1. Fair Benchmarking Outcomes

Competitive Baselines: Properly tuned supervised baselines can match or outperform advanced FMs, particularly on smaller datasets (e.g., BCI IV 2b) and specific clinical tasks (TUAB). For instance, on BCI IV 2b, a simple EEGNet with ASHA optimization outperformed the best FM by approximately 0.035 in balanced accuracy.
Parameter Efficiency: These high-performing baselines require significantly fewer parameters (e.g., ~1.6k–337k) compared to FMs (e.g., ~5.8M–253M).
Optimization Impact: Systematic hyperparameter optimization via ASHA yielded substantial performance gains for supervised baselines (e.g., improvements of 0.028–0.074 in balanced accuracy), highlighting that previous studies may have underestimated baseline capabilities.

2. Paradigm-Level Insights

Pretraining Necessity: The effectiveness of large-scale pretraining is highly dependent on model architecture and dataset scale. Discrete tokenization models (LaBraM, NeuroLM) rely heavily on large-scale pretraining, whereas continuous-representation models (NeuroGPT, EEGPT) can achieve competitive performance with target-only pretraining. However, all models degraded on small datasets when restricted to target-only pretraining.
LLM Backbones: The contribution of pretrained linguistic models varies by architecture. For generation-based FMs (NeuroLM), removing the linguistic backbone caused prediction collapse. For direct classification FMs (NeuroGPT), the pretrained GPT component improved generalization on out-of-domain tasks but offered marginal or slightly negative impacts on in-domain tasks.

3. Neurophysiological Validity

Temporal Dependence: All evaluated FMs showed performance degradation when temporal phase structures were randomized, confirming they utilize temporal dependencies. Continuous-representation models exhibited greater sensitivity to phase randomization than discrete models.
Spatial and Spectral Alignment: FMs demonstrated task-congruent regional focus (e.g., frontal/temporal lobes for seizure detection) and spectral sensitivity (e.g., Delta bands for epilepsy, Alpha/Beta for motor imagery) that align with established neuroscientific literature.

Significance and Claims

The paper claims that EEG-FM-Audit provides the community with a rigorous, established pipeline for the fair and comprehensive evaluation of EEG Foundation Models. Its primary contributions are:

Correcting Evaluation Bias: By demonstrating that optimized supervised baselines can rival FMs, the work challenges the assumption that FMs inherently possess superior performance, suggesting that many reported gains may be artifacts of suboptimal baseline tuning.
Disentangling Complexity: The ablation studies reveal that the "signatures" of FMs (temporal, spatial, spectral) are not uniformly beneficial; their effectiveness is contingent on dataset scale and specific architectural choices.
Establishing Interpretability: The NPP framework establishes a method for verifying that FMs learn meaningful neural representations consistent with physiological findings, moving beyond black-box performance metrics.

The authors conclude that while FMs show promise, they do not yet hold a definitive advantage in EEG classification across all scenarios, and their utility is strongly dependent on the scale of available data and the specific design of the learning paradigm.