Temporal Context and Architecture: A Benchmark for Naturalistic EEG Decoding

Technical Summary: Temporal Context and Architecture for Naturalistic EEG Decoding

Problem Statement
Decoding brain activity from EEG during naturalistic tasks, such as movie-watching, presents a significant challenge due to the need to capture long-range temporal dependencies within noisy, continuous recordings. While deep learning has been widely applied to this domain, traditional architectures face fundamental trade-offs: Convolutional Neural Networks (CNNs) are computationally efficient but constrained by local receptive fields, whereas Transformers offer global context via attention mechanisms but suffer from quadratic scaling in time and memory. Recently, Structured State-Space Models (SSMs) like S4 and S5 have emerged as a potential middle ground, offering long-range modeling with near-linear complexity. However, a systematic comparison of how these modern architectures perform against varying temporal context lengths, and how they generalize across unseen tasks, frequencies, and subjects, has been missing.

Methodology
The authors benchmark five distinct architectures on a 4-class naturalistic EEG decoding task using data from the Healthy Brain Network (HBN) initiative. The dataset includes recordings from movie-watching tasks ("Despicable Me," "Diary of a Wimpy Kid," "The Present") and a resting-state baseline.

• Data Processing: Raw data was downsampled to 250 Hz, filtered (1-50 Hz), re-referenced (common average), and segmented into windows ranging from 8 s to 128 s with 50% overlap. Crucially, data splitting occurred before segmentation to prevent leakage.
• Architectures Evaluated:
  1. Temporal CNN: A standard local receptive field baseline.
  2. Bidirectional LSTM: A recurrent baseline.
  3. Vanilla Transformer: A global attention baseline.
  4. EEGXF: A stabilized Transformer baseline developed by the authors, featuring simplified input projections, BatchNorm layers, norm-first encoding, and learned attention pooling to address training instability on noisy EEG.
  5. S5: A Structured State-Space model utilizing diagonalized state matrices for efficient parallel scanning.
• Evaluation Metrics: The study evaluates performance across three dimensions:
  1. Temporal Scaling: Accuracy vs. segment length (8s–128s).
  2. Robustness: Zero-shot cross-frequency shifts (testing on 128 Hz and 64 Hz signals trained at 250 Hz) and Leave-One-Subject-Out (LOSO) generalization.
  3. Out-of-Distribution (OOD) Behavior: Zero-shot cross-task generalization on unseen active cognitive tasks, analyzed alongside calibration metrics (NLL, Brier, ECE).

Key Results

• Temporal Context Scaling: Accuracy generally improves with longer context. At 64 s, S5 achieved 98.7% ± 0.6 accuracy, and CNN achieved 98.3% ± 0.3. Notably, S5 reached this performance with approximately 20× fewer parameters (0.2M vs. 4.4M) than the CNN.
• Efficiency: CNN remained the fastest in wall-clock time (9.8 min at 64 s), while S5 and EEGXF had comparable training times (~25 min). S4 and LSTM were found non-competitive; LSTM failed to converge, and S4 was prohibitively slow.
• Cross-Frequency Robustness: A significant trade-off emerged. S5's accuracy dropped significantly (>18 percentage points) when tested on downsampled signals (128 Hz/64 Hz), indicating high sensitivity to temporal resolution. Conversely, EEGXF remained remarkably stable, with an accuracy drop of less than 2%.
• Subject Generalization (LOSO): In leave-one-subject-out tests, S5 demonstrated superior generalization (55.9% mean accuracy) compared to EEGXF (48.4%), though both scores were substantially lower than subject-mixed results, highlighting the difficulty of cross-subject decoding.
• OOD and Calibration:
  • S5: Exhibited "overconfident specialization." While well-calibrated on in-distribution data, it consistently misclassified unseen OOD tasks as a specific movie class with high confidence (~60%), failing to express uncertainty.
  • EEGXF: Demonstrated "conservative collapse." It correctly identified OOD tasks as unfamiliar by defaulting to a neutral class ("Resting State") with low confidence (~26%), indicating robust uncertainty quantification despite being less calibrated in-distribution.

Key Contributions

1. Systematic Benchmarking: The paper provides a comprehensive comparison of CNN, RNN, Transformer, and SSM architectures across a wide range of temporal contexts (8s–128s) for naturalistic EEG decoding.
2. Stabilized Transformer Baseline: The introduction of EEGXF, a modified Transformer architecture designed to handle the instability of noisy EEG data, serves as a robust baseline for future comparisons.
3. Generalization Suite: The study moves beyond simple accuracy by introducing a suite of challenging generalization tests, including cross-frequency shifts, subject-independent generalization, and zero-shot cross-task evaluation.
4. Architecture Trade-off Characterization: The work explicitly characterizes a practical efficiency–robustness trade-off, identifying S5 as optimal for parameter-efficient peak accuracy in controlled settings, and EEGXF as preferable when robustness and conservative uncertainty estimation are critical.

Significance and Claims
The paper claims that the interplay between model architecture and temporal context is a critical factor in EEG decoding. It demonstrates that while S5 offers a compelling balance of high accuracy and parameter efficiency, particularly in subject-mixed and long-context scenarios, it lacks robustness to frequency shifts and exhibits dangerous overconfidence on out-of-distribution data. Conversely, EEGXF offers greater stability and safer uncertainty quantification at the cost of peak accuracy and computational efficiency.

The authors conclude that the choice of architecture depends on the application requirements: S5 is ideal for speed and accuracy in controlled environments, whereas EEGXF is preferable for real-world deployment where robustness to signal variations and reliable uncertainty estimation are paramount. The study acknowledges its reliance on the HBN dataset and suggests future validation on standard benchmarks like Sleep-EDF.