🤖 machine learning

NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces

The paper introduces NeuroAtlas, the largest EEG benchmark to date comprising 42 datasets and 260k hours of data, to demonstrate that current foundation models do not yet consistently outperform supervised baselines or generic time-series models in clinical applications, highlighting the critical need for specialized clinical evaluation metrics over standard machine learning scores.

Original authors: Konstantinos Kontras, Trui Osselaer, Stylianos G. Mouslech, Angeliki-Ilektra Karaiskou, Guido Gagliardi, Thomas Strypsteen, Mohammad Hossein Badiei, Anku Rani, Maarten Vanmarcke, Miguel Bhagubai, Chan

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Konstantinos Kontras, Trui Osselaer, Stylianos G. Mouslech, Angeliki-Ilektra Karaiskou, Guido Gagliardi, Thomas Strypsteen, Mohammad Hossein Badiei, Anku Rani, Maarten Vanmarcke, Miguel Bhagubai, Chanakya Ekbote, Jaedong Hwang, Christos Chatzichristos, Paul Pu Liang, Maarten De Vos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of brainwave recordings (EEG) from thousands of people. For years, scientists have been trying to build a "universal translator" for these brainwaves—a single, super-smart AI model (called a Foundation Model) that can read any brainwave, understand what's happening, and help doctors diagnose epilepsy, analyze sleep, or even control computers with thoughts.

The paper "NeuroAtlas" is like a massive, rigorous "report card" for these AI models. The authors built the biggest test suite ever created for brainwave AI to see if these models are actually living up to their hype or if they are just overhyped.

Here is the breakdown of their findings using simple analogies:

1. The Setup: A Giant "Taste Test"

Think of the researchers as food critics. Instead of tasting one dish, they prepared 42 different "meals" (datasets) representing four main courses:

Epilepsy: Detecting seizures (like spotting a sudden storm in the brain).
Sleep: Analyzing sleep stages (like sorting a night's sleep into deep, light, and dream phases).
Brain Age: Estimating how "old" a brain looks based on its activity (like checking if a car's engine sounds like it belongs to a new or old vehicle).
BCI (Brain-Computer Interfaces): Trying to read thoughts to move a cursor (like translating "I want to move left" into a mouse click).

They tested 20 different AI models (including the new "brain-specific" ones and generic time-series models) on all these courses. Crucially, they didn't let the models "study" for the test; they froze the models and just asked them to do the job immediately, simulating a real-world "out-of-the-box" scenario.

2. The Big Surprises (The Findings)

Surprise #1: The "Brain-Specific" Chefs aren't always the best.
You might think a chef who only cooks brainwaves (an EEG Foundation Model) would be better than a chef who cooks everything (a General Time-Series Model).

The Reality: The paper found that the "brain-specialist" chefs didn't consistently beat the "generalist" chefs. Sometimes the generalists did just as well, or even better. It turns out, just because a model was trained specifically on brain data doesn't mean it automatically becomes a master of the brain.

Surprise #2: The "Report Card" grades were misleading.
In the past, scientists graded these models using standard math scores (like "Accuracy" or "AUROC").

The Analogy: Imagine grading a doctor who diagnoses heart attacks. If you only count how many times they guessed "sick" or "healthy" correctly, you might miss the fact that they are sending healthy people to the ER (false alarms) or missing real heart attacks.
The Reality: The authors found that standard math scores often hid the real problems. When they switched to clinical metrics (like "How many seizures did we actually catch without raising false alarms?" or "Did the model correctly calculate the total time spent in deep sleep?"), the rankings changed completely. A model that looked like an "A" student on paper turned out to be a "C" student in a real hospital setting.

Surprise #3: No Single "Super-Model" Exists Yet.
The dream was to have one model that works perfectly for everything.

The Reality: The results were messy. A model that was great at detecting seizures in one hospital dataset often failed miserably in another hospital's dataset. There was no single "King of the Hill." The best model for epilepsy wasn't the best for sleep, and the best for sleep wasn't the best for brain age.
The Conclusion: Currently, these models are not ready to be "plug-and-play" tools for doctors. They are still too sensitive to the specific type of equipment used or the specific group of patients.

3. The "Brain Age" Twist

For the "Brain Age" task (predicting how old a brain is), the researchers found something interesting:

Predicting the exact age of a person was hard for the fancy AI models.
However, the models were somewhat okay at spotting if a brain looked "older" than it should be (a sign of cognitive issues).
But: Even here, the "out-of-the-box" models struggled to reliably separate healthy brains from those with cognitive impairment. The "gap" between a healthy brain and a sick brain wasn't clear enough for the AI to spot consistently.

4. The "Eye-Tracking" Trap (BCI)

In the Brain-Computer Interface tests, the researchers discovered that some models were cheating.

The Analogy: Imagine a student taking a test who isn't reading the questions but is just looking at where the teacher is pointing.
The Reality: Some models were picking up on eye movements or muscle artifacts (noise) rather than the actual brain signals. When the researchers cleaned the data to remove these "cheats," the models' performance dropped significantly, proving they weren't actually "thinking" the way we hoped.

The Final Verdict

NeuroAtlas is a reality check. It says:

"We have built the biggest, most honest test for brainwave AI. The results show that while these models are promising, they are not yet the magic bullet we hoped for. They don't consistently beat simpler models, they often fail when you look at the real clinical questions, and they struggle to work across different hospitals and patients without heavy customization."

The paper concludes that we need to stop just looking at standard math scores and start testing models with real-world clinical metrics if we ever want them to help doctors in the future. Until then, the "out-of-the-box" unified brain model remains a promise, not a product.

Technical Summary: NeuroAtlas

Problem Statement

Foundation models (FMs) promise to extract unified representations that generalize across downstream tasks, a capability that has seen notable success in fields like computer vision and natural language processing. In electroencephalography (EEG), several FMs have been proposed with the goal of deriving cross-domain generalizable embeddings from massive, diverse datasets. However, existing evaluations of EEG-FMs suffer from three critical limitations that obscure their true clinical utility and generalization capabilities:

Intra-domain Generalization: Most benchmarks reduce a clinical domain to a single dataset, failing to account for substantial performance variations caused by differences in acquisition devices, electrode layouts, referencing schemes, and population phenotypes.
Clinical Relevance of Metrics: Standard aggregate metrics (e.g., AUROC, accuracy) often hide failure modes critical for real-world deployment, such as high false-alarm rates in seizure detection or the inability to recover clinically meaningful hypnogram features in sleep staging.
Comparison Against Alternatives: Existing benchmarks typically compare EEG-FMs only against each other, leaving the value of EEG-specific pretraining unverified against general-purpose time-series foundation models (TS-FMs) and models pretrained with supervision on specific EEG tasks (Supervised-Pre).

Consequently, it remains unclear whether current EEG-FMs truly fulfill their potential or if their performance stems from generic temporal pretraining rather than EEG-specific architectural choices.

Methodology

The authors introduce NeuroAtlas, the largest EEG benchmark to date, designed to rigorously evaluate foundation models across four distinct clinical domains: Epilepsy, Sleep, Brain Age, and Brain-Computer Interfaces (BCI).

Benchmark Scope

NeuroAtlas spans 42 datasets and approximately 260,000 hours of EEG data:

Epilepsy: 7 datasets (~58k hours) covering seizure detection across neonatal, pediatric, and adult cohorts.
Sleep: 15 datasets (~201k hours) including sleep staging, microarousal detection, respiratory events, and cognitive impairment diagnosis.
Brain Age: 10 cohorts (~193k hours) focusing on chronological age prediction and the "Brain-Age Gap" (BAG) as a marker for cognitive impairment.
BCI: 17 datasets (~170 hours) covering Motor Imagery (MI), Event-Related Potentials (ERP), Steady-State Visually Evoked Potentials (SSVEP), and cognitive/affective tasks.

Model Families Evaluated

The benchmark compares three distinct families of models under a unified frozen-encoder probe protocol to isolate the quality of pretrained representations from fine-tuning or architecture-specific tuning:

EEG Foundation Models (EEG-FMs): 10 models (e.g., BIOT, LaBraM, EEGPT, NeuroLM, REVE) pretrained on EEG data via self-supervised objectives (masked reconstruction, contrastive learning).
Time-Series Foundation Models (TS-FMs): 3 families (Chronos, Moirai, MOMENT) pretrained on diverse non-EEG temporal corpora with no EEG-specific architecture.
Supervised-Pre Models: 7 task-specific models pretrained on specific EEG tasks (e.g., Seizure-TF, SleepTransformer, EEGNet) serving as specialized baselines.

Random Initialization: A randomly initialized version of CBraMod is included to isolate the contribution of pretraining from architecture.

Evaluation Protocol

Unified Linear Probing: All models are evaluated as fixed feature extractors. A linear probe (logistic regression or ridge regression) is fit per fold on the frozen embeddings without fine-tuning the backbone.
Domain-Specific Metrics: Beyond standard metrics, the benchmark employs clinically grounded measures:
- Epilepsy: Event-level sensitivity at fixed false-alarm rates (Event-Sens@FA) rather than just window-level AUROC.
- Sleep: Hypnogram-derived macrostructural features (e.g., REM latency, total sleep time) and event detection (microarousals).
- Brain Age: Prediction of chronological age and the Brain-Age Gap (BAG) to assess cognitive impairment separation.
- BCI: Normalized balanced accuracy with strict statistical significance testing (binomial tests) to account for finite sample sizes and varying class counts, alongside confound filtering analysis to detect reliance on non-neural artifacts.

Key Results

The evaluation yields three primary findings:

Inconsistency Across Datasets: Model rankings and absolute performance vary substantially within the same domain across different datasets. A model that ranks first on one dataset often performs poorly on another, making single-dataset benchmarks unreliable predictors of general behavior.
EEG-FMs vs. TS-FMs: EEG-specific foundation models do not consistently outperform generic time-series foundation models. In many cases, TS-FMs (which lack EEG-specific architectures or pretraining) achieve performance on par with EEG-FMs. This suggests that reported gains in EEG benchmarks may stem from generic large-scale temporal pretraining rather than domain-specific design.
Limitations of Aggregate Metrics: Standard metrics often fail to capture clinical utility.
- In Epilepsy, models with high AUROC can generate unacceptable false-alarm rates; event-level metrics reveal different model rankings.
- In Sleep, high epoch-wise agreement (Cohen's $\kappa$ ) does not guarantee accurate recovery of clinically vital hypnogram features (e.g., REM latency).
- In Brain Age, models that predict chronological age well do not necessarily produce a robust Brain-Age Gap capable of separating cognitively impaired patients from healthy controls.
- In BCI, performance is often driven by non-neural confounds (e.g., eye gaze); when these are filtered, many models drop to or below chance levels.

Only a few specific models (notably NeuroLM and REVE) consistently rank in the top tier across multiple datasets and domains, but even they do not demonstrate a universal "out-of-the-box" capability.

Significance and Claims

The paper claims that NeuroAtlas exposes a gap between the promise of unified EEG foundation models and their current reality. The authors conclude that:

Pretrained models perform largely on par with each other, with only narrow advantages for a few specific models.
Current models do not yet deliver on the promise of a unified, out-of-the-box EEG model that generalizes robustly across diverse clinical tasks and datasets.
The benchmark provides the necessary datasets, preprocessing pipelines, and clinically grounded metrics to guide the next generation of unified EEG foundation models.

The work emphasizes that future progress requires moving beyond single-dataset evaluations and aggregate metrics toward rigorous, multi-dataset testing with measures that reflect real-world clinical decision-making.