🤖 machine learning

OOD Detection for EEG-based Machine Learning in High-Risk Environments

This paper introduces a benchmark for EEG-based out-of-distribution (OOD) detection, evaluates various methods across clinical downstream tasks, and demonstrates how combining OOD detection with model uncertainty estimation creates a robust safety net for deploying EEG models in high-risk environments.

Original authors: Philipp Bomatter, Henry Gouk

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Philipp Bomatter, Henry Gouk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The human brain speaks in electricity. Tiny bursts of electrical activity ripple across the scalp, carrying the complex signals of thought, movement, and sensation. Doctors and researchers have long used electroencephalography, or EEG, to listen to this language, placing sensors on the head to record these patterns. In recent years, computer programs trained on this data have shown great promise, learning to spot signs of epilepsy, sleep disorders, or cognitive decline with impressive speed. However, these programs are fragile. They are like students who have memorized a textbook perfectly but fail completely when asked a question phrased in a slightly different way. If the data they receive comes from a different hospital, a different machine, or a patient with a unique physiology that the computer has never seen before, the program may not just be unsure; it may confidently give a completely wrong answer. This is a dangerous problem in medicine, where a confident mistake can lead to a missed diagnosis or unnecessary treatment. The core challenge is teaching these systems to recognize when they are out of their depth, to admit when the data they are seeing is strange, and to step aside for a human expert before a disaster occurs.

A team of researchers at the University of Edinburgh set out to solve this specific problem for brainwave analysis. They wanted to know if current computer programs could reliably tell the difference between familiar brainwaves and unfamiliar ones, and if they could do so without needing to see the strange data beforehand. To test this, they created a rigorous experiment using real patient data from two large hospital collections. They took normal brainwave recordings and deliberately altered them in ways that mimic real-world errors and variations. They changed the speed at which the signals were recorded, shuffled the order of the sensors on the scalp, and filtered out specific frequencies, creating a wide range of "out-of-distribution" data. These altered recordings represented the kind of unexpected shifts a computer might encounter in a busy clinic. The researchers then pitted two different types of computer programs against these challenges: one type that simply learned to classify brainwaves as normal or abnormal, and another type that learned to understand the underlying structure of the brainwaves themselves.

The results revealed a sharp divide between these two approaches. The programs that relied solely on classification, which are the most common type used today, failed almost entirely at spotting the strange data. When presented with altered brainwaves, these systems continued to make predictions with high confidence, often failing to distinguish the altered data from the real thing. They were effectively blind to the fact that the input had changed. In contrast, the programs designed to understand the structure of the data performed remarkably well. These systems, which build a model of what "normal" brain activity looks like, could easily detect when the input deviated from that model. One specific method, which checks if the noise in the signal matches what the system expects, proved to be the most effective, correctly identifying the altered data with high accuracy even when the changes were subtle. This finding suggests that for brainwave analysis, simply training a computer to sort data is not enough; the computer must also understand the shape of the data it is sorting.

However, the researchers discovered that the story was more nuanced than a simple victory for one side. While the structural models were excellent at spotting unfamiliar data, they were not very good at telling the computer how confident it should be when the data was familiar. When the brainwaves were normal but the case was difficult or ambiguous, the structural models did not signal that the computer was struggling. The classification models, despite their failure to spot the strange data, were actually quite good at this second task. They could sense when a specific brainwave pattern was confusing, even if that pattern came from a patient they had seen before. This led the researchers to a crucial conclusion: the two types of programs measure different things. One measures whether the data is new and unknown, while the other measures how difficult the current task is.

The most significant finding of the study was that these two capabilities are not rivals, but partners. By combining the two approaches, the researchers created a safety net that was far stronger than either method alone. In their tests, they set up a system where the computer would only make a diagnosis if it passed two checks: first, the structural model confirmed the data was familiar enough to trust, and second, the classification model confirmed it was confident enough in its answer. When they tested this combined system, it successfully prevented the computer from making mistakes on both the strange, altered data and the difficult, familiar cases. The system learned to step back and defer to a human doctor whenever either condition was met. This approach offers a practical path forward for using artificial intelligence in high-risk medical environments. It shows that by understanding the limits of what different computer programs can do, and by combining their strengths, we can build systems that are not just smart, but also safe and reliable enough to be used in real hospitals.

Technical Summary: OOD Detection for EEG-based Machine Learning in High-Risk Environments

Problem Statement
Machine learning models for electroencephalography (EEG) analysis face significant barriers to deployment in high-risk domains, such as clinical diagnostics, due to their vulnerability to distribution shifts. When models encounter out-of-distribution (OOD) data—arising from factors like hardware differences, preprocessing mismatches, or patient population variations—they often produce catastrophic, overconfident failures. While OOD detection methods exist, they remain under-explored for EEG. Furthermore, existing literature frequently conflates two distinct concepts: true OOD detection (identifying data outside the training distribution) and model uncertainty estimation (assessing confidence on in-distribution data). Current evaluation paradigms often isolate OOD detection performance from downstream clinical utility, failing to reflect how detection mechanisms impact real-world diagnostic reliability.

Methodology
The authors introduce a systematic benchmark framework designed to bridge the gap between OOD detection capabilities and downstream clinical performance. The methodology comprises three core components:

OOD Perturbation Suite: To create controlled, reproducible OOD data, the authors developed a suite of parametric perturbations applied to in-distribution (ID) test data. These include:
- Resampling: Altering the sampling rate (e.g., 125 Hz to 200 Hz) to simulate hardware mismatches.
- Channel Shuffling: Permuting subsets of EEG channels to simulate electrode misplacement.
- Re-referencing: Changing the reference scheme (e.g., to Cz, linked-temporal, or bipolar montages).
- Filtering: Applying high-pass and low-pass filters with varying cutoff frequencies to simulate preprocessing errors.
Model Architectures and Detection Methods: The framework evaluates two distinct model paradigms:
- Discriminative Models: A Temporal Convolutional Network (TCN) trained via supervised learning for the prediction task. OOD detection methods based on this model include Maximum Softmax Probability (MSP), Energy Scores, ODIN, and Activation Shaping (ASH).
- Generative Models: A UNet architecture trained via flow matching (unsupervised) on EEG data without labels. OOD detection methods based on this model include Log-likelihood, Typicality, Density of State Estimation (DoSE), and Signal in the Noise (SITN).
Evaluation Strategy: The study utilizes two large clinical datasets (TUAB for EEG normality and CAUEEG for dementia/MCI diagnosis). Evaluation occurs in two stages:
- Isolated OOD Detection: Measuring the ability to distinguish ID from perturbed OOD samples using AUROC.
- Downstream Impact: Analyzing how OOD scores correlate with the classification accuracy of the discriminative model across different percentile bins. This assesses whether flagging specific samples actually improves clinical reliability.

Key Contributions

Benchmark Framework: The introduction of a structured OOD detection evaluation framework for EEG that includes controlled perturbations and explicitly links detection performance to downstream clinical prediction tasks.
Comprehensive Method Benchmarking: The first extensive evaluation of generative-model-based OOD detection methods (Typicality, DoSE, SITN) in an EEG context, alongside standard discriminative approaches.
Conceptual Disentanglement: Empirical evidence distinguishing between true OOD detection and model uncertainty estimation, demonstrating that these are frequently conflated but functionally distinct capabilities.
Actionable Guidance: A demonstration of how combining complementary methods (generative for OOD detection and discriminative for uncertainty) creates a robust safety net for high-risk deployments.

Results

OOD Detection Performance: Discriminative methods (MSP, Energy, ODIN, ASH) performed largely at chance levels (AUROC $\approx$ 0.5) across most perturbations, failing to reliably separate ID from OOD data. In contrast, generative methods achieved significantly better performance, with Signal in the Noise (SITN) emerging as the state-of-the-art method. SITN showed consistent performance improvements as perturbation severity increased.
Downstream Impact:
- Generative Methods (e.g., SITN): Effectively identified OOD samples where model accuracy dropped to near-chance levels. These methods serve as a critical safety net to prevent misclassifications on unfamiliar data. However, they provided little insight into model performance within the in-distribution data.
- Discriminative Methods (e.g., MSP): While ineffective at detecting OOD data, MSP scores were highly predictive of model accuracy on in-distribution samples, serving as a proxy for model uncertainty. They identified difficult ID cases where the model was likely to fail, though they exhibited a failure mode at very high confidence levels.
Combined Approach: A decision system combining SITN (to reject OOD samples) and MSP (to reject high-uncertainty ID samples) yielded the highest accuracy across both ID-only and ID+OOD scenarios. This combination maintained high coverage on ID data while effectively protecting against performance degradation on OOD data.

Significance and Claims
The paper claims that standard discriminative methods are insufficient for OOD detection in EEG under realistic covariate shifts, supporting recent theoretical critiques that such methods "answer the wrong question" for OOD detection. The authors argue that the field must move beyond semantic-shift-focused benchmarks (common in computer vision) toward evaluations that address covariate shifts and practical utility.

The study emphasizes that true OOD detection and model uncertainty are distinct capabilities. Generative models are required to detect unfamiliar data distributions, while discriminative models are better suited for estimating uncertainty on familiar data. The authors conclude that a robust safety net for clinical EEG applications requires the integration of both approaches: generative methods to prevent failures on OOD data and discriminative methods to manage uncertainty on difficult in-distribution cases. This work provides a foundation for safer deployment of EEG-based machine learning by disentangling these concepts and offering a validated framework for their combined use.

Technical Summary: OOD Detection for EEG-based Machine Learning in High-Risk Environments

More like this