Pretraining on Sleep Data Improves non-Sleep Biosignal Tasks

Technical Summary: Pretraining on Sleep Data Improves non-Sleep Biosignal Tasks

Problem Statement
Sleep foundation models have recently demonstrated strong performance on in-domain polysomnography (PSG) tasks, such as sleep staging and apnea detection. However, it remains an open question whether sleep biosignals can serve as an effective pretraining distribution for learning representations that transfer to adjacent, non-sleep domains recorded during wakefulness. While clinical PSG typically utilizes low-density EEG montages and single-lead ECG due to acquisition constraints, wake-based clinical recordings often employ higher-density channels for fine-grained analysis. The core challenge is determining if the unique properties of sleep data—specifically long continuous contexts, diverse pathological waveforms, and tight cross-modal alignment—can overcome the limitations of sparse sensor configurations to improve performance on downstream wake-based EEG and ECG tasks.

Methodology
The authors build upon the SleepFM architecture (Thapa et al., 2026) to pretrain a model exclusively on sleep-only multimodal data.

• Pretraining Strategy: The model undergoes sleep-only multimodal contrastive pretraining using a leave-one-out objective. The dataset comprises 48,093 full-night PSG recordings (432,000 hours) from four cohorts (Stanford Sleep, Bioserenity, MESA, MrOS).
• Architectural Adaptations: To accommodate non-sleep biosignal tasks, the authors made minimal architectural changes to isolate the effect of the pretraining distribution:
  • Patch Size: Reduced from 5 seconds to 1-second patches to preserve transient dynamics (e.g., sharp waves, R-peaks) crucial for non-sleep tasks.
  • Window Size: Reduced from 5 minutes to 2 minutes for pretraining to reduce memory requirements while maintaining multi-cycle physiological context.
  • Positional Encoding: A channel positional encoding (PE) was introduced during fine-tuning for EEG tasks. This injects 3D electrode geometry (x, y, z coordinates based on the 10-20 system) into the embeddings, enabling flexible inference with varying channel counts.
• Downstream Evaluation: The pretrained encoder was fine-tuned on eight downstream tasks across five EEG datasets (TUAB, TUEV, TUSL, CHB-MIT, BrainCapture) and three ECG datasets (PTB-XL, G12EC, CPSC). Tasks included abnormality detection, seizure detection, event classification, and arrhythmia detection.
• Baselines: Performance was compared against models trained from scratch (random initialization) and against state-of-the-art (SOTA) foundation models pretrained on in-domain wake EEG (e.g., LaBraM, BIOT, CBraMod, REVE) or large ECG corpora. Notably, the sleep-pretrained model had no domain overlap with the wake EEG test sets, whereas many SOTA baselines were pretrained on corpora containing the test data.

Key Contributions

1. Cross-Domain Transfer Validation: The paper demonstrates that sleep biosignals can effectively serve as a pretraining distribution for wake-based EEG and ECG tasks, consistently outperforming training from scratch.
2. Competitive Performance: Despite being pretrained only on sleep data with low-density montages, the model achieves performance competitive with or surpassing specialized SOTA foundation models pretrained directly on wake EEG data.
3. Channel Robustness: The study introduces a channel-agnostic approach where the model maintains 95% of its full-channel performance using only 16–50% of available EEG channels and 41–75% of ECG leads.
4. Methodological Rigor: The work includes systematic channel ablation experiments and evaluates transfer across heterogeneous datasets and protocols, distinguishing between in-distribution benchmarks and external validation transfer.

Results

• EEG Tasks: Sleep pretraining consistently improved performance across all five EEG datasets.
  • On TUAB (abnormality detection), the 3.8M parameter SleepFM1 (with PT+PE) achieved a balanced accuracy of 0.825, outperforming LUNA-Huge (311M parameters, 0.816) and LaBraM-Large (46M, 0.823), despite lacking the direct domain overlap of the latter.
  • On TUSL (slowing) and CHB-MIT (seizure), the model achieved SOTA results, outperforming FEMBA, EEGFormer, and BIOT.
  • On BrainCapture, the model substantially outperformed BENDR and LaBraM.
  • Linear probe results indicated that the pretrained encoder already produced informative representations without fine-tuning, with full fine-tuning providing additional gains.
• ECG Tasks: In an external-validation transfer experiment (fine-tuned on Code-15, evaluated on PTB-XL, G12EC, CPSC), sleep pretraining consistently improved performance over random initialization. While gains were smaller than in EEG tasks, the model outperformed supervised baselines like Inception Time and 1DResNet on PTB-XL.
• Channel Efficiency: Performance degradation was minimal when reducing channel counts. For EEG, 16–50% of channels were sufficient to reach 95% of full-channel performance. For ECG, 41–75% of leads were sufficient.

Significance and Claims
The paper claims that the physiological richness of sleep data—characterized by long continuous contexts capturing various physiological states, broad representations of abnormal waveforms (e.g., epileptic patterns, arrhythmias), and tight cross-modal alignment between brain, heart, and breathing—provides a rich self-supervised structure for learning transferable representations.

The authors argue that these properties allow a model trained exclusively on sleep PSG to generate representations competitive with models trained directly on wake EEG, even without domain overlap. This suggests that sleep data can serve as a scalable, information-dense pretraining corpus for biosignal foundation models, potentially reducing the need for extensive labeled wake datasets, particularly in resource-limited settings. The findings support the viability of sleep as a pretraining distribution for general biosignal modeling, with particularly strong evidence in EEG and encouraging, though more limited, evidence in ECG external-transfer settings.