📊 statistics

Making Conformal Predictors Robust in Healthcare Settings: a Case Study on EEG Classification

This paper addresses the challenge of distribution shifts in healthcare by demonstrating that personalized calibration strategies significantly improve the coverage of conformal predictors for EEG seizure classification, thereby enhancing the reliability of uncertainty quantification in high-stakes clinical settings.

Original authors: Arjun Chatterjee, Sayeed Sajjad Razin, John Wu, Siddhartha Laghuvarapu, Jathurshan Pradeepkumar, Jimeng Sun

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Arjun Chatterjee, Sayeed Sajjad Razin, John Wu, Siddhartha Laghuvarapu, Jathurshan Pradeepkumar, Jimeng Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor looking at an EEG scan (a recording of brain waves) to see if a patient is having a seizure. In the real world, this isn't always a clear "Yes" or "No." Sometimes, even expert doctors disagree on what the squiggly lines mean, or the machine picks up noise that looks like a seizure but isn't.

This paper tackles a specific problem: How do we make computer programs admit when they aren't 100% sure?

The Problem: The "One-Size-Fits-All" Mistake

Standard computer models usually try to give a single answer: "This is a seizure" or "This is normal." But in healthcare, being overconfident is dangerous. If the model is wrong, the patient could get unnecessary treatment or miss out on needed care.

To fix this, researchers use a technique called Conformal Prediction. Instead of giving one answer, it gives a "safety net" or a list of possible answers. For example, instead of saying "It's a seizure," it might say, "It's either a seizure OR just noise."

The Catch: These safety nets are built on a mathematical rule that assumes every patient is exactly like the next one (like flipping a coin where every flip is identical). But in the real world, every patient is different. Their brain waves, their age, and how the test was recorded all vary. This is called distribution shift. Because of this, the standard safety nets often fail to catch the errors they are supposed to catch.

The Solution: A Personalized Safety Net

The authors tested a new method called Neighborhood Conformal Prediction (NCP) on EEG data.

Think of the standard method like a generic raincoat given to everyone in a city. It might work okay for a light drizzle, but if you are in a heavy storm or a different climate, it might leak.

The new method (NCP) is like a custom-tailored raincoat. Before the computer makes a prediction for a specific patient, it looks at that patient's "neighbors"—other patients who look very similar to them (similar brain wave patterns, similar history). It calibrates its safety net based on those specific neighbors rather than the whole city.

What They Found

The researchers tested this on two different scenarios:

The "Random" Mix (Easy Mode): When they mixed patients from different groups together randomly, the personalized method (NCP) was a huge success. It caught 20% more errors than the standard method without making the list of possible answers any longer. It was like upgrading from a leaky umbrella to a perfect one.
The "Real World" Split (Hard Mode): When they tested the model on completely new patients it had never seen before (simulating a real hospital where the next patient is totally different), things got harder. Even the best methods struggled to catch every error. However, the personalized method still managed to keep its "lists" of possibilities smaller and more efficient than the others, even if it couldn't perfectly solve the problem of total uncertainty.

The Takeaway

The paper concludes that while we can't perfectly predict the future or eliminate all uncertainty in brain wave analysis, personalizing the prediction based on similar patients makes the computer much more honest about its confidence.

They also made their tools available for free (via a framework called PyHealth) so other researchers can easily try this "custom raincoat" approach in their own medical AI projects.

In short: Standard AI tries to guess for everyone the same way and often fails when patients are different. This new method asks, "Who does this patient look like?" and adjusts its guess accordingly, making it much more reliable in tricky medical situations.

1. Problem Statement

The paper addresses the critical challenge of uncertainty quantification in high-stakes healthcare diagnostics, specifically focusing on EEG (electroencephalogram) seizure classification.

Intrinsic Uncertainty: EEG labeling is inherently uncertain due to measurement noise and inter-expert disagreement among neurologists. Single-point predictions are often overconfident or misleading.
Distribution Shift: Standard Conformal Prediction (CP) methods rely on the assumption that calibration and test data are independent and identically distributed (i.i.d.). In healthcare, this assumption is routinely violated due to patient-level distribution shifts (variations in physiology, recording conditions, and demographics).
The Gap: Under these non-i.i.d. conditions, conventional CP methods fail to provide the theoretical coverage guarantees required for safe clinical deployment, often resulting in prediction sets that are either too small (under-coverage) or too large (uninformative).

2. Methodology

The authors evaluated several conformal prediction approaches using the ContraWR model (a ResNet-based 2D CNN trained on multi-channel spectrograms) as the underlying predictor.

Datasets:

TUAB: A binary normal/abnormal detection task.
TUEV: A six-class EEG event classification task (SPSW, GPED, PLED, EYEM, ARTF, BCKG).
Splitting Regimes:
- Random Split: Samples are pooled and split globally (60/10/15/15 for train/val/cal/test). This tests performance under mild shifts.
- Patient Split: Patients are partitioned first, ensuring calibration and test sets contain disjoint patients. This simulates realistic cross-patient distribution shifts.

Evaluated Conformal Approaches:

Naive CP: Standard split conformal prediction assuming i.i.d. data.
Covariate CP: Uses likelihood-ratio reweighting via Kernel Density Estimation (KDE) to adjust for covariate shifts.
K-means CP: Derives thresholds based on the nearest cluster's calibration samples.
Neighborhood Conformal Prediction (NCP): A personalized approach using $k$ -nearest neighbors with relevance weighting to perform sample-specific calibration.

3. Key Contributions

The paper makes four primary contributions:

NCP Approach: Introduction of a Neighborhood Conformal Prediction method that significantly improves coverage (up to 20 percentage points) over non-personalized baselines.
Theoretical Guarantees: Provision of theoretical results demonstrating that NCP yields provably better coverage under covariate shift compared to standard methods.
Efficiency: Demonstration that NCP maintains relatively stable and compact prediction set sizes, avoiding the trade-off where higher coverage leads to uninformative, massive sets.
Open-Source Implementation: Integration of these methods into PyHealth, an open-source healthcare AI framework, to facilitate replication and adoption in other clinical settings.

4. Experimental Results

Performance under Random Splits:

NCP vs. Baselines: On the TUEV dataset, NCP achieved approximately 34% greater coverage than Naive CP at $\alpha = 0.2$ (0.87 vs. 0.53).
Set Size: This improvement came with only a modest increase in average prediction set size (1.22 vs. 0.90).
Covariate CP Failure: The Covariate CP method (KDE-based) performed similarly to Naive CP, failing to improve coverage. The authors attribute this to the unreliability of likelihood-ratio estimation in high-dimensional EEG feature spaces.

Performance under Patient Splits (Distribution Shift):

General Failure: When calibration and test patients were disjoint, all methods fell short of the target coverage ( $1-\alpha$ ), highlighting the severity of cross-patient distribution shifts.
NCP Behavior: While NCP still struggled to meet target coverage in this extreme shift scenario, it produced smaller prediction sets compared to other methods. This suggests that while personalization helps, the distribution shift is so strong that even personalized calibration yields overly tight (under-covering) sets.
Variance: High variance across random seeds was observed, indicating instability in current CP methods for this specific task under patient splits.

5. Significance and Future Directions

Clinical Relevance: The study underscores that standard uncertainty quantification tools are insufficient for real-world healthcare applications where patient heterogeneity is the norm. Personalized calibration is a necessary step toward robustness.
Limitations: The paper acknowledges that current methods cannot fully solve the cross-patient distribution shift problem; empirical coverage remains imperfect in disjoint patient settings.
Future Work:
- Applying personalized CP to other healthcare domains to identify which types of shifts are most manageable.
- Integrating stronger EEG foundation model embeddings (e.g., TFM-Tokenizer) to better characterize signal structure and improve NCP coverage.
- Promoting the adoption of conformal prediction in safety-critical AI via the PyHealth framework.

In conclusion, the paper argues that while standard conformal prediction fails under patient-level distribution shifts, personalized calibration strategies (specifically NCP) offer a significant, theoretically grounded improvement in coverage and efficiency, representing a vital step toward deployable, uncertainty-aware healthcare AI.