Technical Summary: A Foundation Model for Wearable Movement Data in Mental Health Research

Problem Statement

Wearable movement data, specifically minute-level actigraphy collected from smartwatches and fitness trackers, offers a rich, longitudinal resource for mental health research. It captures fine-grained behavioral trends such as circadian fragmentation and psychomotor slowing. However, the development of foundation models for this modality lags behind clinical image and text analysis. Existing analytical approaches face significant limitations:

• Classical methods rely on handcrafted features and rule-based detectors, which are task-specific and sensitive to cohort and device variations.
• Deep learning approaches (CNNs and RNNs) often struggle with long temporal windows; CNNs have limited receptive fields, while recurrent models face gradient degradation over long sequences.
• Current foundation models for wearables are often proprietary, trained on short temporal segments (seconds to minutes) rather than multi-day patterns, or focused on general activity recognition rather than psychiatric and sleep outcomes.
• There is a lack of open-source, generalizable models capable of modeling week-long dependencies at minute-level resolution to predict mental health outcomes.

Methodology

Dataset and Preprocessing

The study utilized data from the National Health and Nutrition Examination Survey (NHANES), a nationally representative U.S. cohort.

• Pretraining: Data from 21,538 participants across the 2003–2004, 2005–2006, and 2011–2012 survey cycles. This included hip-mounted uniaxial and wrist-worn triaxial accelerometers. To ensure comparability, triaxial data was summed to match uniaxial counts, and data was standardized using z-scores per device type to mitigate amplitude discrepancies.
• Fine-tuning: Data from 7,769 participants in the 2013–2014 cycle was used for downstream supervised tasks.
• Input: Minute-level actigraphy sequences spanning 7 days (10,080 time steps).

Model Architecture: Pretrained Actigraphy Transformer (PAT)

The authors designed a Transformer-based architecture specifically for long-sequence actigraphy:

1. Patch Embeddings: To handle the quadratic complexity of processing 10,000+ time steps, the input sequence is divided into non-overlapping patches (e.g., 10-minute patches). Each patch is mapped to a fixed dimension via a trainable linear projection or 1D convolution.
2. Positional Embeddings: Fixed sine-cosine positional embeddings are added to encode temporal order.
3. Transformer Encoder: A stack of transformer blocks (multi-headed self-attention, layer normalization, feed-forward networks) processes the patch embeddings.
4. Masked Autoencoder (MAE) Pretraining:
   • Strategy: An asymmetric autoencoder approach where a high ratio of patches (90%) is randomly masked and removed before the encoder.
   • Encoder: Processes only the unmasked patches.
   • Decoder: A lightweight decoder receives the encoder's output along with mask tokens to reconstruct the original standardized actigraphy signal.
   • Loss: Mean Squared Error (MSE) calculated between the reconstructed and original data across all patches.
5. Fine-tuning: The patch embedding layer and pretrained encoder are extracted. A task-specific feed-forward classification layer with sigmoid activation is added for binary classification tasks.

Experimental Design

• Tasks: Five binary classification tasks were defined: Benzodiazepine use, SSRI use, Depression (PHQ-9 ≥10), Sleep disorder history, and Sleep abnormality detection.
• Baselines: Comparisons included 1D CNN, LSTM, ConvLSTM, Wavelet-based classifiers, and a "ScratchTransformer" (identical architecture to PAT but trained from random initialization without pretraining).
• Evaluation: Performance was measured using the Area Under the Receiver Operating Characteristic Curve (AUC), averaged across training set sizes of 500, 1,000, 2,500, and the full available set to assess robustness in low-data regimes.

Key Contributions

1. Open-Source Foundation Model: PAT is the first open-source foundation model for wearable movement data that combines week-long, minute-level temporal modeling with explicit evaluation on mental health outcomes. It releases pretrained weights, code, and training pipelines.
2. Long-Range Temporal Modeling: Unlike existing models operating on short segments (seconds to minutes), PAT models 10,080-minute (week-long) sequences, capturing circadian structures and multi-day behavioral patterns.
3. Generalizability: The model demonstrates consistent performance across five diverse clinical tasks (medication use, depression, sleep) and shows superior performance in low-data regimes (n ≤ 1,000).
4. Interpretability: The model provides built-in attention maps, highlighting specific temporal periods (e.g., morning activity transitions) that drive predictions, offering transparency without requiring post-hoc explanation tools.

Results

Performance Metrics

• Benzodiazepine Prediction: PAT-L (the largest variant) achieved an average AUC of 0.767. This represented a 55.6% improvement over the LSTM baseline (0.493), a 21.4% improvement over the 1D CNN (0.632), and a 14.8% improvement over the ConvLSTM (0.668).
• Low-Data Robustness: PAT consistently outperformed all baselines, including the ScratchTransformer, across all training set sizes. This indicates that the self-supervised pretraining effectively learned generalizable representations that transfer well even with limited labeled data.
• Multi-Task Performance: PAT outperformed baselines on SSRI use, sleep abnormality detection, and depression classification. For sleep disorder history, PAT performed comparably to other models but did not show independent association after adjusting for covariates (see below).

Hyperparameter Insights

• Masking Ratio: A 90% masking ratio yielded the best downstream performance (AUC 0.773), encouraging the encoder to learn broader contextual dependencies.
• Smoothing: Models trained on raw actigraphy signals outperformed those trained on smoothed data (AUC 0.773 vs. 0.741), suggesting that fine-grained temporal information is critical.
• Loss Function: Computing MSE across all patches (reconstructed and original) significantly outperformed computing loss only on masked patches.

Interpretability Findings

Attention heatmaps revealed that PAT focuses on specific behavioral transitions rather than random noise.

• Benzodiazepine Users: High attention was assigned to extended early-morning inactivity and delayed wake-up times.
• Non-Users: High attention was assigned to sharp, consistent early-morning physical activity.
• Population Level: Attention patterns exhibited strong 24-hour periodicity, with peaks concentrated around wake transitions and morning activity, confirming the model learns recurrent daily rhythms.

Confounder Analysis

Multivariable logistic regression showed that PAT predictions were independently associated with benzodiazepine use, SSRI use, depression, and sleep abnormalities after adjusting for age, sex, BMI, and depression severity. However, for sleep disorder history, the association was not significant after adjustment, suggesting the model may be predicting underlying risk factors (e.g., BMI, depression) rather than the diagnosed condition itself, or that treated conditions attenuate the actigraphy signal.

Significance and Claims

The authors position PAT as a practical, scalable solution to advance clinical insights from wearable sensor data.

• Accessibility: By being open-source and capable of running on consumer-grade GPUs (e.g., NVIDIA T4), PAT lowers the barrier to entry for researchers with limited computational resources.
• Scalability: The model is designed to be adaptable to variable input lengths and different devices, making it suitable for diverse research settings.
• Clinical Utility: The model's ability to predict medication use and psychiatric conditions from passive movement data offers a non-invasive method for risk stratification and treatment monitoring.
• Modesty: The authors acknowledge limitations, noting that labels are derived from self-reports rather than gold-standard clinical diagnoses, which may introduce noise. They also caution that actigraphy alone provides only a partial signal for complex psychopathology and should be viewed as a complementary tool in multimodal models rather than a standalone diagnostic instrument. Furthermore, the cross-sectional nature of the data precludes causal inference.

In conclusion, PAT represents a step toward generalizable, data-efficient, and interpretable digital phenotyping for behavioral health, bridging the gap between large-scale wearable data collection and clinically relevant mental health modeling.