A Comparative Study of Deep Learning Architectures for Multi-Horizon Behavioural Forecasting for Mobile Health

Technical Summary: A Comparative Study of Deep Learning Architectures for Multi-Horizon Behavioural Forecasting for Mobile Health

Problem Statement
Wearable devices and smartphones generate rich behavioural time series (e.g., step counts, screen time, sleep duration) that hold potential for proactive health interventions. However, systematic comparisons of modern deep learning forecasting architectures for these specific data types are lacking. Critical gaps identified include: (1) the lack of evaluation of modern architectures (Transformers, State-Space Models) specifically for continuous behavioural forecasting, as prior work focuses heavily on classification; (2) the challenge of cross-population generalisation, where models often fail to transfer to new users; and (3) the absence of multi-horizon analysis, despite practical applications requiring predictions over extended periods (1–8 days) rather than single-step forecasts.

Methodology
The study benchmarks nine forecasting approaches across three public datasets (GLOBEM, CAPTURE-24, and Continuous Wearables) encompassing over 800 participants. The evaluation covers three distinct modelling paradigms:

1. Global: A single model trained on pooled data from all training users.
2. Foundation Models (Zero-Shot): Pre-trained models applied directly without task-specific training.
3. Personalised: Global models fine-tuned on individual user data.

Architectures Evaluated:

• Trained Deep Learning: MLP, LSTM, Temporal Convolutional Networks (TCN), Transformer, Mamba, and PatchTST.
• Foundation Models: TimesFM (decoder-only, pre-trained on 100B+ time points) and Reverso (hybrid convolution/recurrence).
• Statistical Baselines: Naive (Last), Moving Average, and ARIMA.

Experimental Setup:

• Features: Step counts, screen time, and sleep duration.
• Horizons: Forecasting evaluated across 1 to 8 days.
• Metrics: Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), symmetric MAPE (sMAPE), and Skill Score (relative to a Naive Last baseline).
• Personalisation Strategy: A two-stage transfer learning procedure where a global optimum is first established, followed by fine-tuning on individual user records with a reduced learning rate.

Key Results

1. No Single Dominant Architecture: Performance rankings are dataset-dependent. Among trained models, PatchTST generally leads on daily data (GLOBEM) for step counts and screen time. However, TCN, MLP, and Transformer show no meaningful performance differences in many scenarios. Mamba demonstrates robustness on hourly data (CAPTURE-24).
2. Foundation Model Efficacy: TimesFM matches or exceeds trained models in zero-shot settings, particularly in low-data regimes (e.g., the Wearables dataset with N=10). It excels on sleep duration but shows negative skill on hourly data, indicating a mismatch between its pre-training distribution and sub-daily health signals. Reverso generally underperformed.
3. Impact of Personalisation: Participant-level fine-tuning reduces per-feature RMSE by 16–60%. The benefit is highly feature-dependent:
   • Sleep: Benefits most (43–60% reduction) due to stable individual patterns.
   • Screen Time: Moderate benefit (19–36%).
   • Step Counts: Least benefit (16–46%).
   • Horizon Dependency: Personalisation gains increase with forecast horizon. At short horizons (h=1), global models already capture population patterns well; at longer horizons (h=8), personalisation is crucial to capture individual behavioural drift.
4. Data Scarcity Risks: Personalisation requires sufficient per-user data. On datasets with sparse observations (e.g., CAPTURE-24 with ~22 hourly points), fine-tuning led to overfitting and performance degradation of up to 40% at longer horizons.
5. Uncertainty Quantification: Empirical prediction intervals for PatchTST were significantly narrower (e.g., ±24 min for screen time) compared to Naive baselines (±90 min) while maintaining near-nominal coverage.

Significance and Claims
The authors claim this is the first study to jointly evaluate modern deep learning architectures, Foundation Models, and personalisation strategies for multi-horizon behavioural forecasting from wearables.

The study provides practical guidance for the mobile health community:

• Architecture Selection: No single model is universally superior; selection should depend on dataset size, temporal granularity, and sequence length. PatchTST is recommended for daily data with sufficient history, while Mamba shows promise for higher-frequency data.
• Foundation Model Applicability: TimesFM serves as a strong zero-shot default when per-user data is scarce, but its performance is contingent on the alignment between its pre-training domain and the target health signals.
• Personalisation Strategy: Fine-tuning is most valuable for features with high inter-individual regularity (like sleep) and for longer forecast horizons. However, it requires a minimum threshold of individual data to avoid overfitting.

The paper concludes that while deep learning offers significant improvements over statistical baselines, the choice of strategy (Global vs. Personalised vs. Zero-Shot) must be dictated by the specific data characteristics and the intended forecasting horizon.