Are different consumer sleep technologies measuring the same essential aspects of sleep?

This study of 74 older adults found that while aggregating consumer sleep technology data over 7–14 nights yields reliable measures, most metrics are device-specific rather than interchangeable with polysomnography, with sleep duration being the only essential aspect consistently transferable across different devices.

The Gist

The Big Question: Are All Sleep Trackers Telling the Same Story?

Imagine you and three of your friends are trying to describe a movie you all watched. You all use the same words like "the ending," "the action," and "the romance." But because you all have different eyes, different memories, and different ways of seeing things, your descriptions of the same movie end up being quite different.

This is exactly what this study investigated, but instead of a movie, the "movie" is your sleep, and the "friends" are different sleep trackers (like smartwatches, under-mattress sensors, and research-grade devices).

The researchers wanted to know: If two different devices say you slept 7 hours, are they actually measuring the exact same thing? Or are they just using the same label for two completely different experiences?

The Experiment: A Sleep Party with Four Guests

The researchers gathered 74 older adults (some with dementia, some without) and asked them to wear four different sleep trackers at the same time for up to two weeks:

1. A Research Watch (Axivity): The "gold standard" of wrist trackers, used in big scientific studies.
2. A Consumer Smartwatch (Withings Watch): The kind you might buy at a store.
3. A "Nearable" Sensor (Withings Sleep Analyzer): A pad that goes under your mattress.
4. A Sleep Diary: A notebook where people wrote down when they fell asleep and woke up.

They also brought everyone into a lab for one night to hook them up to a Polysomnography (PSG) machine. Think of the PSG as the "Truth Machine." It measures brain waves directly to see exactly what stage of sleep you are in.

The Findings: The "Translation" Problem

Here is what they discovered, broken down simply:

1. The "Duration" is the Only Common Language

If you ask all four trackers how many hours you slept (Duration), they generally agree with each other. It's like asking four people, "How long was the movie?" They might say 90 minutes, 95 minutes, or 92 minutes. They aren't perfect, but they are close enough to understand each other.

• The Takeaway: If you want to track how long you sleep, almost any device will do.

2. Everything Else is a Different Dialect

When it came to Quality (did you wake up a lot?), Timing (did you wake up too early?), or Deep Sleep (did you get that restorative sleep?), the devices started speaking different languages.

• The Analogy: Imagine one tracker says, "You had a restless night with 5 wake-ups." The other tracker looks at the same night and says, "You had a peaceful night with 0 wake-ups." They are both looking at the same person, but their internal "algorithms" (the rules they use to decide what is sleep and what is wakefulness) are so different that they can't agree.
• The Result: You cannot compare a study that used a Fitbit with a study that used an Oura Ring if they are looking at "sleep quality." They are measuring different things.

3. The "Blurry Photo" Problem (Reliability)

Sleep is messy. We toss, turn, and wake up briefly without remembering.

• The Analogy: Taking a picture of a sleeping person with a shaky hand (one night of data) results in a blurry photo. You can't tell if they are sleeping or just lying still.
• The Solution: The researchers found that if you take many photos over 7 to 14 nights and average them out, the picture becomes clear.
• The Catch: Some people (especially those with dementia) have very "shaky" sleep patterns naturally. For them, even 14 nights of photos might not be enough to get a clear picture of their "true" sleep habits.

4. The "Magic Filter" (PCA)

The researchers used a mathematical trick called Principal Component Analysis (PCA). Think of this as a noise-canceling filter.

• They took all the messy data from the trackers and filtered out the "static" (the unreliable measurements).
• What was left were two clear, stable "super-measures": Duration (how long) and Continuity (how broken up the sleep was).
• Even with this filter, Duration was the only thing that looked the same across all devices. The other measures remained specific to the device used.

5. Can We Predict Dementia?

Finally, they asked: "Can these sleep trackers help us spot people with dementia?"

• Yes, but... The models worked best when they used the "clean" data (the 7-14 night averages) and only the reliable measures.
• If they tried to use the "blurry" single-night data, the models got confused and made mistakes.
• The Lesson: To use sleep data as a medical tool, you need patience. You need to collect data for a long time to smooth out the noise.

The Bottom Line: What Should You Do?

1. Don't trust the labels blindly: Just because a smartwatch says "Deep Sleep" doesn't mean it's the same "Deep Sleep" as a hospital machine or a different brand of watch.
2. Patience is key: If you want to track your sleep habits for health reasons, don't look at one night's data. Look at the average over two weeks. That's when the data becomes reliable.
3. Duration is safe; Quality is tricky: You can trust your watch to tell you roughly how long you slept. Be very skeptical if it tells you exactly how many times you woke up or how much "Deep Sleep" you got, especially if you are comparing it to someone else's data from a different device.

In short: Consumer sleep trackers are great tools for seeing the "big picture" of your sleep duration over time, but they are currently terrible at agreeing on the "fine details" of sleep quality with each other. Treat them like a weather forecast: good for a general idea, but don't bet your life on the exact minute it will rain.

Technical Summary

1. Problem Statement

Consumer Sleep Technologies (CSTs) are increasingly used for longitudinal sleep monitoring in research and clinical settings, particularly for older adults and People Living with Dementia (PLWD). However, a critical gap exists in understanding whether identically labeled sleep metrics (e.g., Total Sleep Time, Wake After Sleep Onset) derived from different devices (wearables vs. nearables) measure the same underlying physiological or behavioral constructs.

• The Core Issue: CSTs rely on proprietary algorithms and peripheral signals (movement, heart rate) rather than direct brain activity (EEG). Consequently, their outputs are often treated as equivalent to Polysomnography (PSG) gold-standard measures, despite known biases and poor interchangeability.
• The Gap: There is a lack of device-agnostic frameworks to determine if CST-derived measures are reliable (stable over time) and interchangeable across different hardware, especially in populations with fragmented sleep like PLWD.

2. Methodology

The study employed a multi-modal, longitudinal data collection approach combined with advanced statistical modeling to assess reliability and interchangeability.

• Study Population: 74 older adults (mean age 70.5 years), including 20 PLWD and 54 cognitively intact controls.
• Data Collection:
  • Duration: Up to 14 nights of at-home monitoring followed by one in-lab PSG session.
  • Total Data: 752 nights of data.
  • Modalities: Four simultaneous tools were used:
    1. Research-grade Actigraphy: Axivity (AX3/6).
    2. Consumer Wearable: Withings Watch (WW).
    3. Consumer Nearable: Withings Sleep Analyzer (WSA) - under-mattress sensor.
    4. Subjective: Sleep Diary (SD).
  • Gold Standard: In-lab PSG (SomnoHD system) with consensus hypnograms.
• Statistical Framework:
  • Reliability Analysis: Used Intraclass Correlation Coefficients (ICC(1,1)) to assess single-night reliability and the Spearman-Brown prediction formula to estimate the number of nights ($k$) required to achieve a target reliability (ICC $\ge$ 0.7).
  • Associations: Repeated-measures correlations (rmcorr) to assess night-to-night changes within individuals across devices.
  • Dimensionality Reduction: Principal Component Analysis (PCA) with Varimax rotation to identify latent sleep aspects (e.g., Duration, Continuity, Timing). This was performed on both raw measures and "clean" measures (filtered for reliability).
  • Predictive Modeling: XGBoost binary classification models to distinguish PLWD from Controls, evaluated via AUC and Average Precision (AP). SHAP values were used to determine feature importance.

3. Key Contributions

• Reliability-Aware Framework: Proposes a shift from validating CSTs solely against PSG to assessing their longitudinal reliability (trait-like stability) as a prerequisite for biomarker discovery.
• Aggregation Thresholds: Quantifies the specific number of nights required to achieve reliable estimates for different sleep metrics, showing that a "one-size-fits-all" monitoring period is insufficient.
• Latent Sleep Aspects: Demonstrates that while raw metrics vary wildly, dimensionality reduction (PCA) can extract stable, interpretable latent constructs (e.g., Duration, Continuity) from noisy data.
• Device Interchangeability: Provides empirical evidence that most identically labeled metrics are not interchangeable across devices, with the notable exception of Duration.

4. Key Results

A. Agreement with PSG and Between-Device Correlations

• PSG Validation: CSTs showed only moderate correlations with PSG for duration/timing measures (e.g., TST $r \approx 0.4-0.5$). Sleep stage duration (REM, Deep Sleep) showed poor agreement, with significant overestimation of deep sleep by Withings devices.
• Between-Device: Single-night correlations between devices were generally weak ($r < 0.3$) for continuity measures (WASO, SEFF) and sleep stages. Moderate correlations ($0.3 \le r < 0.7$) were found primarily for Duration and Timing measures between objective devices (Axivity, WSA, WW).

B. Reliability and Aggregation

• Single-Night Reliability: Most measures had low single-night reliability (ICC < 0.7).
• Aggregation Effect:
  • 71% of measures reached acceptable reliability (ICC $\ge$ 0.7) after 7 nights of aggregation.
  • 79% reached this threshold after 14 nights.
  • Population Impact: PLWD required more nights to achieve stability than controls due to higher night-to-night variability.
  • Metric Specificity: Duration measures stabilized quickly (often within 2-7 nights), whereas Continuity and Nap measures often failed to reach acceptable reliability even after 14 nights.

C. Latent Sleep Aspects (PCA)

• PCA identified interpretable components: Duration, Continuity, Timing, and Naps.
• Reliability Filtering: When PCA was applied only to measures that passed the ICC $\ge$ 0.7 threshold ("clean" data), the resulting components were stable across all devices.
• Interchangeability:
  • Duration was the only latent aspect showing moderate, consistent associations across all objective devices.
  • Continuity and Timing aspects remained largely device-specific.

D. Predictive Utility (PLWD vs. Controls)

• Models using clean, reliability-filtered PCA components outperformed models using raw measures or unfiltered PCA components for Axivity and WSA (e.g., Axivity AUC improved from 0.59 to 0.71).
• Feature Importance: There was a strong positive correlation between a feature's reliability (ICC) and its importance in the predictive model (SHAP values), suggesting that stable measures are better biomarkers.

5. Significance and Conclusion

• Device Specificity: The study concludes that identical labels do not imply identical measurements. Except for Duration, CST measures are device-specific and should not be treated as interchangeable or equivalent to PSG measures.
• Study Design Implications: Longitudinal studies must tailor the monitoring duration to the specific metric of interest. While 7 nights may suffice for duration, continuity metrics may require longer periods or may be unsuitable for individual-level tracking in fragmented sleep populations.
• Biomarker Development: For CSTs to serve as valid digital biomarkers, researchers must:
  1. Filter measures based on reliability (ICC).
  2. Aggregate data over sufficient nights (7–14).
  3. Avoid assuming cross-device equivalence.
• Clinical Relevance: The findings urge caution in interpreting CST data in clinical trials and observational studies, particularly for populations like PLWD where sleep fragmentation is high and device algorithms may struggle.

Final Takeaway: Consumer sleep technologies can provide reliable, trait-like estimates of sleep duration and timing when aggregated over 7–14 nights, but they generally fail to provide interchangeable measures of sleep continuity or stages. A reliability-aware approach is essential for valid digital sleep phenotyping.