Quantifying the Generalization Gap in Seizure Detection: A Large-Scale Empirical Benchmark via the SzCORE Challenge

Technical Summary: Quantifying the Generalization Gap in Seizure Detection

Problem Statement
Reliable automatic seizure detection from long-term electroencephalogram (EEG) recordings remains an unsolved challenge in clinical practice. Despite the proliferation of machine learning models, current algorithms frequently fail to generalize across different patients and clinical settings. Consequently, manual EEG review by clinicians remains the gold standard. A significant barrier to progress is the lack of standardized evaluation practices; existing literature often reports high efficacy on specific datasets, but these results are not directly comparable due to heterogeneity in data, metrics, and methodologies. Furthermore, there is a documented "generalization gap" where models perform well on self-selected or public datasets but fail when deployed on unseen, diverse patient populations.

Methodology
To address these issues, the authors organized a large-scale empirical benchmark, the SzCORE Challenge, in conjunction with the 2025 International Conference on Artificial Intelligence in Epilepsy and Neurological Disorders.

• Dataset: The evaluation utilized a strictly held-out private dataset comprising continuous EEG recordings from 65 subjects (398 annotated seizures) totaling 4,360 hours of data. The data was collected from the Filadelfia Danish Epilepsy Centre between 2018 and 2020. It includes 19-channel scalp EEG (10-20 system) sampled at 256 Hz. Ground truth was established by three board-certified neurophysiologists, with consensus used to resolve disagreements. The dataset includes a mix of adults and children (median age 34, range 5–66).
• Algorithm Collection: The challenge solicited submissions from the community, resulting in 28 successfully evaluated algorithmic executions derived from 20 unique architectural approaches. These ranged from classical feature engineering (e.g., Random Forest, Gradient Boosting) to modern Deep Learning (e.g., Transformers, CNNs, STFT-based models).
• Standardized Framework: All algorithms were evaluated using the SzCORE framework, which enforces:
  • Input/Output Standardization: Inputs are .edf files; outputs are HED-SCORE compliant .tsv files.
  • Reproducibility: Algorithms were packaged as isolated Docker images to ensure a blind, reproducible evaluation environment on high-performance computing infrastructure (utilizing NVIDIA A100 and V100 GPUs).
  • Metrics: Performance was measured using event-based metrics: Sensitivity, Precision, F1-score, and False Positives per day (FP/24h). Scoring relied on specific overlap tolerances (30s pre-ictal, 60s post-ictal) and merging rules (90s minimum separation).
• Statistical Analysis: The study employed non-parametric statistical testing, including the Friedman test for omnibus differences, Wilcoxon signed-rank tests with Holm-Bonferroni correction for pairwise comparisons, and the Nemenyi test for critical difference diagrams. Effect sizes were quantified using Cliff's Delta.

Key Results

• Performance Ceiling: The top-performing algorithm, Sz Transformer, achieved an event-based F1-score of 32% (Sensitivity: 37%, Precision: 29%) with an average of 1.34 false positives per day.
• Stability vs. Peak Performance: A critical finding was the discordance between peak aggregate performance and population-level stability. While Sz Transformer had the highest aggregate F1-score, HySEIZa v1 achieved the best mean rank across subjects, indicating greater consistency. Statistical analysis showed no significant difference between the top-ranked algorithm and the F1-leader, yet seven leading architectures were statistically indistinguishable from the rank-leader.
• Generalization Gap: There was a stark discrepancy between self-reported performance by developers on their chosen datasets and the actual performance on the independent held-out dataset. Nearly all algorithms significantly overestimated their efficacy, highlighting a pervasive generalization gap.
• Failure Modes:
  • False Positives: False positives were largely idiosyncratic to individual algorithms, with less than 25% of algorithms agreeing on 91% of false detections.
  • Missed Seizures: 15 of the 65 subjects (23%) had an F1-score of 0 across all top-5 algorithms simultaneously, suggesting patient-specific electrographic patterns were not captured by current models.
  • Seizure Duration: Hard-to-detect seizures were substantially shorter (median 48s) compared to easy ones (median 118s), suggesting current processing windows (1–600s) may miss brief ictal patterns.
• Comparison to Prior Work: The best algorithm in this study surpassed prior methods in scalp-EEG challenges (e.g., the Neureka challenge) and approached the performance of established commercial systems, though commercial systems often prioritize higher sensitivity at the cost of increased false positives.

Key Contributions

1. Large-Scale Empirical Benchmark: This study provides the first large-scale, direct comparison of seizure detection algorithms using a standardized methodology on a substantial, expertly annotated private dataset.
2. Quantification of the Generalization Gap: The work rigorously documents the failure of state-of-the-art models to generalize to unseen patient populations and exposes the unreliability of self-reported metrics.
3. Open Benchmarking Platform: The evaluation infrastructure has been transitioned into a continuously open benchmarking platform (hosted on GitHub). This allows for ongoing submissions, reproducible research, and the integration of additional private datasets from other clinical centers, moving beyond a one-time competition to a living resource.
4. Statistical Rigor: The application of robust non-parametric statistical testing provides a nuanced view of algorithm performance, distinguishing between aggregate scores and patient-level stability.

Significance and Claims
The paper claims that while the top algorithms demonstrate competitive performance and tangible progress over prior challenges, the field still faces significant hurdles in achieving robust, generalizable seizure detection. The primary significance lies not merely in the snapshot of current performance, but in the establishment of a standardized, transparent, and reproducible evaluation paradigm.

The authors argue that the persistent generalization gap and the instability of high-capacity models on outlier patients pose critical safety risks for clinical deployment. They posit that the shift from isolated competitions to a continuous, open benchmarking platform is essential to:

• Mitigate self-reporting bias.
• Objectively track field progress over time.
• Facilitate the development of models that are robust across diverse patient demographics (including pediatric and rare seizure types).
• Accelerate the translation of research into clinically viable decision support tools.

Ultimately, the work advocates for a paradigm shift where rigorous, independent benchmarking on diverse clinical data becomes the standard for validating machine learning models in epilepsy care, rather than relying on aggregate metrics from curated datasets.