Cross-Dataset Generalization of Quantitative EEG Spectral Biomarkers for Alzheimer’s Disease: A Pre-registered Two-Cohort External Validation and What Does Not Transfer

Technical Summary: Cross-Dataset Generalization of Quantitative EEG Spectral Biomarkers for Alzheimer's Disease

Problem Statement
Current machine learning research on quantitative EEG (qEEG) biomarkers for Alzheimer's Disease (AD) predominantly relies on single-acquisition setups. While these studies often report high accuracy, they frequently fail to address a fundamental question: do these markers remain valid when the recording equipment or electrode montage changes? Most existing validation schemes leak subject information (e.g., epoch-level cross-validation), inflating performance metrics. Genuine external validation—training a model on one acquisition setup and testing it on another—is rare in medical AI (under 4% of studies). This study addresses the "cross-configuration generalization" gap identified in previous benchmarks, specifically investigating whether spectral features derived from one dataset can robustly separate AD from cognitively normal (CN) participants in a completely independent dataset recorded with different hardware.

Methodology
The study employed a pre-registered design to ensure analytical rigor, fixing the feature set, model, validation scheme, and interpretation criteria prior to data analysis.

• Datasets: Two independent, public, resting-state (eyes-closed) EEG cohorts were utilized:
  • AHEPA: 65 participants (36 AD, 29 CN) recorded using a Nihon Kohden 2100 system with a 19-channel 10–20 montage (Cz reference).
  • Meghdadi: 81 participants (26 AD, 55 CN) recorded using an Advanced Brain Monitoring system with a 19-channel 10–20 montage. The Meghdadi dataset was filtered to include only the eyes-closed condition and a specific age-matched control stratum to minimize demographic variance.
• Feature Extraction: Three scale-invariant spectral features were computed for each of the 19 common channels, yielding 133 features per subject:
  1. Theta/Alpha Ratio (TAR): The ratio of theta (3–7 Hz) to alpha (8–13 Hz) power.
  2. Relative Band Power: Normalized power across $\delta$, $\theta$, $\alpha$, $\beta$, and $\gamma$ bands.
  3. Alpha Peak Frequency.
     Note: Power spectral density (PSD) values from the Meghdadi dataset (stored in log10) were linearized ($10^x$) before ratio calculation to ensure comparability with AHEPA.
• Model and Validation:
  • Classifier: Logistic regression (primary) and linear SVM (robustness check).
  • Internal Validation: Leave-One-Subject-Out (LOSO) cross-validation within each dataset to prevent subject leakage.
  • External Validation: Transfer learning in both directions (AHEPA $\to$ Meghdadi and Meghdadi $\to$ AHEPA).
  • Statistical Control: A permutation null hypothesis test was conducted (100 shuffles for internal, 500 for external) to verify that observed results were not artifacts of chance or data leakage. Bootstrap 95% confidence intervals were calculated for all AUC estimates.

Key Results
The study revealed a stark asymmetry in cross-dataset generalization:

1. Internal Validity (Within-Cohort): Both datasets successfully separated AD from CN under strict LOSO validation.

   • AHEPA: 0.785 Accuracy, 0.802 AUC.
   • Meghdadi: 0.753 Accuracy, 0.799 AUC.
   • Permutation nulls confirmed these results were significantly above chance (0% of shuffles matched observed AHEPA values).

2. External Validity (Cross-Cohort): Performance was fragile and highly direction-dependent.

   • Meghdadi $\to$ AHEPA: The model trained on Meghdadi achieved a moderate 0.728 AUC on AHEPA, indicating successful transfer in this direction.
   • AHEPA $\to$ Meghdadi: The model trained on AHEPA collapsed on the Meghdadi dataset, achieving a 0.329 AUC. This is significantly below chance (99.2% of permutation shuffles yielded equal or better results), indicating the classifier was actively misclassifying the groups.

3. Diagnosis of Failure:

   • Class imbalance and scaling errors were ruled out as primary causes.
   • The failure was attributed to a domain shift in the absolute scale of the features, which is tied to specific device and montage configurations.
   • Feature Ablation: When individual features were tested in isolation during cross-dataset transfer, the Theta/Alpha Ratio (TAR) was the only feature that maintained robustness. While absolute and relative band powers failed to generalize (leading to the inversion of class probabilities in the AHEPA $\to$ Meghdadi direction), the TAR preserved the correct ranking of subjects (0.516 AUC in the weak direction, 0.724 in the strong direction).

Significance and Claims
The paper claims to provide a "measured statement" on the limits of qEEG biomarker generalization rather than a new high-accuracy benchmark. Its primary contributions are:

• Empirical Evidence of Asymmetry: It demonstrates that a model trained on one acquisition setup can perform significantly worse than random chance when applied to another, challenging the assumption that qEEG markers are universally portable.
• Identification of Robust Features: The study isolates the Theta/Alpha Ratio (TAR) as the most reproducible marker across different acquisition configurations. Because TAR is a unit-invariant ratio, it is less susceptible to the absolute scale shifts caused by different hardware and montages compared to absolute or relative power measures.
• Methodological Rigor: By adhering to a pre-registered protocol with strict subject-level validation and permutation testing, the study offers a reproducible case of external validation in neurodiagnostics.
• Practical Recommendation: For research aiming to develop montage-independent EEG biomarkers, the author suggests favoring relative, ratio-based measures (specifically TAR) over absolute power features to mitigate acquisition-specific domain shifts.

The author explicitly states that the "headline" is a verified non-generalization in one direction, framing this negative observation as an honest and necessary step toward understanding the boundaries of current qEEG diagnostic tools.