Performance of a Semi-Automated Hierarchical Rest Interval Detection Pipeline (actiSleep) for Wrist Actigraphy in Adolescents

The study demonstrates that the semi-automated actiSleep pipeline, which integrates event markers, diaries, light, and activity data, accurately estimates rest intervals in adolescents comparable to standard hand-scoring while outperforming purely activity-based algorithms, offering a viable alternative for large-scale research.

The Gist

Imagine you are trying to figure out exactly when your teenager went to sleep and when they woke up. You could ask them, but they might not remember or might be a little off. You could watch them all night, but that's exhausting and intrusive. Or, you could give them a special smartwatch that tracks their movement, light exposure, and button presses.

This is what actigraphy does. It's a popular tool for studying sleep, but there's a catch: interpreting the data is like trying to solve a puzzle with missing pieces.

The Problem: The "Human Bottleneck"

Currently, the "gold standard" for reading this data is hand-scoring. This means a human expert sits down, looks at the watch data, checks the teen's sleep diary, looks at light sensors, and manually decides, "Okay, they tried to sleep at 10:15 PM and got up at 7:00 AM."

Think of this like a master chef tasting a soup and adjusting the salt. It's the most accurate way, but it takes forever. If you have 50 teenagers, it takes a chef (or a researcher) about 17 hours to taste and adjust every single pot. If you have 5,000 people, you'd need a whole army of chefs, and they might all taste the soup slightly differently (lack of consistency).

On the other hand, there are fully automated algorithms (like the "Activity-Only" method). These are like instant soup mix. They just look at how much the person moved. If they were still, the computer assumes they were sleeping.

• The flaw: Sometimes people lie perfectly still while watching TV or reading. The computer thinks, "Ah, sleeping!" but they were actually awake. It's like the instant soup mix thinking a cold, still room is a hot kitchen.

The Solution: The "Semi-Automated Sous-Chef" (actiSleep)

The researchers in this paper built a new tool called actiSleep. Think of this as a smart sous-chef that does 90% of the work but knows when to ask the head chef for help.

Instead of just looking at movement, actiSleep acts like a detective that gathers clues from four different sources:

1. The Diary: What the teen said they did.
2. The Button Press: When they physically pressed a button on the watch to say, "I'm going to bed now."
3. The Light Sensor: When the lights in the room actually turned off.
4. The Movement: How much they were wiggling.

How it works (The Detective's Logic):
The algorithm follows a strict set of rules, like a hierarchy of trust.

• Scenario: The diary says "Lights out at 10:00," but the light sensor says "It's still bright until 10:30."
• The Human Chef would have to stare at the screen, think hard, and make a judgment call.
• The actiSleep Sous-Chef instantly applies the rule: "If the light is still on, the sleep attempt probably didn't start until the light went off." It makes the decision in a split second, following the exact same logic a human expert would use.

The Results: Speed vs. Accuracy

The researchers tested this new tool on 51 teenagers (some healthy, some with a family history of mood disorders). They compared three methods:

1. The Human Expert (Hand-scoring).
2. The Instant Soup Mix (Activity-Only).
3. The Smart Sous-Chef (actiSleep).

The Findings:

• The Instant Soup Mix was okay, but it often guessed wrong at the very beginning and end of the night (the "tails" of the sleep interval). It was like guessing the soup was ready 15 minutes too early.
• The Smart Sous-Chef (actiSleep) was almost identical to the Human Expert. It got the start and end times right, within just a few minutes of the human's judgment.
• The Speed: The Human Expert took 17 hours to process the whole group. The Smart Sous-Chef did it in 1 hour.

Why This Matters

Imagine you are running a massive study on sleep patterns for a whole city. You can't hire 100 human experts to stare at watches for weeks. But if you use a "Activity-Only" computer, you might get bad data for people who have trouble sleeping (like those with anxiety or depression), because they move less when they are awake.

actiSleep is the "best of both worlds." It brings the precision of a human expert to the speed of a computer. It allows researchers to study thousands of people accurately without needing a massive team of humans to do the math.

The Bottom Line

This paper introduces a new digital tool that teaches computers how to "think" like human sleep experts. It uses all the available clues (diaries, lights, buttons, and movement) to figure out sleep times much faster than a human can, but just as accurately. It's a game-changer for making sleep research faster, cheaper, and more reliable for everyone.

Technical Summary

1. Problem Statement

Wrist actigraphy is a standard tool for monitoring sleep in research and clinical settings due to its non-invasive nature and ability to track sleep over long periods. However, a critical bottleneck exists in rest interval detection (determining when a person attempts to sleep vs. when they wake up).

• Limitations of Automated Algorithms: Standard automated algorithms (e.g., Cole-Kripke, Oakley) rely solely on activity counts. They often misclassify immobile wakefulness as sleep and struggle to define the precise start and end of a rest interval, leading to biases in sleep latency and efficiency metrics.
• Limitations of Manual Hand-Scoring: Best-practice guidelines recommend using a hierarchical approach that integrates multiple data sources (activity, light sensors, sleep diaries, and event marker presses) to manually define rest intervals. While accurate, this process is time-intensive, prone to inter-rater variability, and impractical for large-scale cohort studies.
• The Gap: There is a lack of scalable, semi-automated solutions that replicate the precision of hierarchical hand-scoring by integrating multiple data streams without requiring manual review of every epoch.

2. Methodology

Study Design and Participants

• Sample: 51 adolescents (ages 14–19), comprising 29 Low-Risk (typically developing) and 22 High-Risk (family history of bipolar disorder) participants.
• Data Collection: Participants wore Philips Respironics Actiwatch Spectrum devices for 14 days, recording 60-second epochs of activity and light. They also completed daily sleep diaries and used device marker buttons to log "Good Night Time" (bedtime) and "Get Up Time."
• Reference Standard: An expert scorer performed Hand-Scoring using a standardized hierarchical algorithm (based on Patel et al.) that prioritized indicators (Marker > Diary > Light > Activity for onset; Activity > Light > Marker > Diary for offset) to establish the "ground truth."

The actiSleep Pipeline

The authors developed actiSleep, a semi-automated R-based pipeline designed to mimic the decision rules of human hand-scorers.

• Inputs: Activity data, light sensor data, sleep diary entries, and marker press events.
• Core Logic:
  1. Indicator Hierarchy: Applies the same priority rules as hand-scoring to determine the "winner" indicator for rest onset and offset.
  2. Concordance Rules: Checks for agreement between indicators within 15-minute and 30-minute windows.
  3. Special Cases: Includes logic to handle specific scenarios (e.g., shifting onset if a marker press occurs 120 minutes after a preliminary winner, or shifting offset if light remains on).
  4. Merging/Deletion: Merges closely adjacent rest intervals (≤1 hour gap) and deletes short naps (<15 min) unless corroborated by diary/marker.
  5. Flagging System: Automatically flags days with anomalies (e.g., extremely long/short intervals, missing data, or high indicator discrepancy) for limited manual review, rather than reviewing every day.

Comparators

The study compared actiSleep against two fully automated algorithms:

1. Activity-Only: Standard activity-based sleep estimation (Oakley algorithm) without external indicators.
2. Activity-Merged: Activity-based estimation with a rule to merge closely adjacent rest intervals (mimicking a basic hand-scoring step but lacking diary/light/marker integration).

Statistical Analysis

• Agreement: Bland-Altman plots were used to assess mean differences and limits of agreement for Rest Onset, Offset, and Duration.
• Epoch-Level Accuracy: Positive Predictive Value (PPV) and True Positive Rate (TPR) were calculated. Crucially, these were analyzed for the entire interval and specifically for the "tails" (the first and last 30 and 60 minutes of the rest interval), where misclassification is most common.

3. Key Contributions

• Development of actiSleep: A novel, open-source (GitHub) semi-automated pipeline that successfully translates complex, multi-indicator hierarchical hand-scoring rules into code.
• Hybrid Efficiency: The pipeline achieves near-hand-scoring accuracy while reducing processing time from ~17 hours (manual) to ~1 hour (semi-automated) for a 51-person cohort, requiring manual review only for flagged outliers.
• Validation in High-Risk Populations: Unlike many validation studies using healthy controls, this study validated the algorithm in adolescents with psychiatric risk factors and sleep disturbances, a group where automated algorithms typically fail.
• Focus on "Tail" Accuracy: The study highlights that while overall agreement is high, the critical differentiator between algorithms is performance at the boundaries (onset/offset) of the rest interval.

4. Key Results

• Mean Agreement:

  • actiSleep: Estimates were within 1–3 minutes of hand-scoring for onset, offset, and duration.
  • Activity-Merged: Within 2–4 minutes.
  • Activity-Only: Within 7–14 minutes.
  • Note: actiSleep showed significantly narrower margins of agreement (less scatter) in Bland-Altman plots compared to the other algorithms.

• Epoch-Level Accuracy (PPV and TPR):

  • For the entire rest interval, all algorithms performed similarly (PPV >97%, TPR >96%) because the middle of the sleep period is easy to detect.
  • For the tails (first/last 60 minutes):
    • actiSleep: Maintained high accuracy (TPR/PPV ~95%).
    • Activity-Merged/Only: Accuracy dropped significantly (TPR/PPV shifted to the low 90s/high 80s).
  • This confirms that actiSleep is superior at defining the precise start and end of sleep attempts.

• Subgroup Analysis:

  • Agreement was stronger for Low-Risk participants across all algorithms.
  • However, actiSleep maintained superior agreement with hand-scoring in the High-Risk group, whereas the performance of Activity-Merged and Activity-Only declined significantly in this population with more complex sleep patterns.

• Secondary Outcomes:

  • actiSleep slightly underestimated Wake After Sleep Onset (WASO) by ~1 minute and Sleep Offset by ~3 minutes.
  • Activity-based algorithms significantly underestimated Sleep Latency and overestimated Sleep Efficiency compared to hand-scoring.

5. Significance and Implications

• Scalability for Large Cohorts: actiSleep offers a viable solution for large-scale epidemiological studies (e.g., UK Biobank, ABCD study) where manual hand-scoring is impossible, but activity-only algorithms lack the necessary precision for clinical or nuanced research questions.
• Improved Validity in Clinical Populations: The study demonstrates that incorporating multiple data streams (diary, light, markers) is essential for accurately characterizing sleep in populations with psychiatric comorbidities or irregular sleep patterns.
• Reproducibility: By codifying hierarchical rules, actiSleep reduces inter-rater variability, a major source of noise in sleep research.
• Future Directions: The authors note that while actiSleep is a major step forward, future iterations could integrate machine learning to optimize indicator weighting and expand compatibility to other device types (e.g., GeneActiv, ActiGraph) and open-source pipelines like GGIR.

Conclusion: The actiSleep pipeline successfully bridges the gap between the labor-intensive precision of manual hand-scoring and the efficiency of automated processing, providing a robust, semi-automated tool for rest interval detection in adolescent populations, particularly those with sleep and psychiatric disturbances.