Freezing of Gait Prediction using Proactive Agent that Learns from Selected Experience and DDQN Algorithm

This study proposes a reinforcement learning framework utilizing a Double Deep Q-Network with Prioritized Experience Replay to predict Freezing of Gait in Parkinson's Disease patients up to 8.72 seconds in advance, enabling timely proactive interventions through wearable assistive devices.

The Gist

The Problem: The "Sudden Stop"

Imagine you are walking down a busy street, and suddenly, your feet feel like they are glued to the pavement. You want to move, but your brain and legs just won't cooperate. This is called Freezing of Gait (FOG), and it's a terrifying and dangerous symptom for people with Parkinson's disease. It often leads to falls.

The goal of this research is to build a "crystal ball" that can tell a person, "Hey, you're about to freeze in about 8 seconds. Start moving your feet differently right now!"

The Old Way vs. The New Way

The Old Way (The Fixed Window):
Imagine a security guard who checks the door every 5 seconds exactly. If someone looks suspicious at 4 seconds, the guard misses it. If they look suspicious at 6 seconds, the guard misses it too. Most previous computer models worked like this: they looked at a fixed chunk of time (like a 5-second window) and decided, "Is this a freeze?" They were rigid and often missed the early warning signs.

The New Way (The Proactive Agent):
The researchers built a smart, learning computer program (an "Agent") that acts more like a skilled chess player or a weather forecaster. Instead of checking the clock at fixed intervals, this agent constantly watches the walker's steps and asks itself: "Is it time to sound the alarm yet? Or should I wait a few more seconds to be sure?"

How the Agent Learns (The Video Game Analogy)

The team taught this agent using a method called Reinforcement Learning. Think of it like training a dog or playing a video game:

1. The Game: The agent watches data from a walker's sensors (like a smartwatch).
2. The Goal: It wants to press a "Prediction Button" (place a flag) just before the person freezes.
3. The Rewards & Penalties:
   • Gold Star (+150 points): It presses the button at the perfect time (e.g., 8 seconds before the freeze).
   • Time-Out (-40 points): It presses the button too early (false alarm).
   • Game Over (-60 points): It waits too long and the person has already frozen.
   • Wait (+0.1 points): It's okay to wait and gather more info, but not forever.

The Secret Sauce: "The Highlight Reel"

The agent didn't just learn from every single step it took. It used a special trick called Prioritized Experience Replay (PER).

Imagine you are studying for a big test. You don't just read the whole textbook once. You focus intensely on the questions you got wrong or the ones that were really hard.

• DDQN (The Brain): This is the part of the AI that makes decisions. It's like a double-check system to make sure the AI doesn't get overconfident or make silly mistakes.
• PER (The Highlight Reel): When the agent makes a mistake or has a "surprising" moment (like a sudden gait change), it saves that moment in a special "Highlight Reel." It then studies these difficult moments over and over again to learn faster. This helps it handle the weird, unpredictable ways different people walk.

The Results: Giving People a Head Start

The researchers tested this on real data from Parkinson's patients. Here is what they found:

• The Head Start: The agent could predict a freeze up to 8.72 seconds before it happened (in general tests) and 7.89 seconds (when tested on specific individuals).
• Why 8 seconds matters: In the world of Parkinson's, 8 seconds is an eternity. It gives the patient enough time to hear a cue (like a beep or a visual line on the floor), process it, and adjust their walking style to avoid the freeze entirely.
• Flexibility: Unlike the old "fixed window" models, this agent adapts. If a person's walking pattern is weird one day, the agent adjusts its strategy.

The Catch

It's not perfect yet.

• False Alarms: Sometimes, if a person has a lot of "freezing" episodes close together, the agent gets confused and sounds the alarm too early (like a smoke detector going off when you're just cooking toast).
• Data Gaps: The agent learned best on the data it saw most often. Some rare walking patterns were missed because the training data wasn't perfectly balanced.

The Bottom Line

This paper presents a smart, adaptive assistant for Parkinson's patients. Instead of a rigid computer program that checks the clock, it's a learning agent that watches, waits, and learns from its mistakes to give the user the maximum amount of time to react.

If you can get a warning 8 seconds before you trip, you can usually catch yourself. This technology aims to turn those terrifying "glued feet" moments into manageable, preventable events, potentially saving lives and preventing falls.

Technical Summary

1. Problem Statement

Freezing of Gait (FOG) is a debilitating motor symptom in Parkinson's Disease (PD) where patients suddenly lose the ability to walk, leading to falls and reduced mobility.

• Current Limitations: Most existing prediction methods rely on supervised learning with fixed-window approaches (e.g., CNN-LSTM). These methods often:
  • Lack flexibility in decision-making (cannot choose when to predict).
  • Depend on complete datasets and manual feature engineering.
  • Are constrained to short prediction horizons (typically 1–4 seconds), limiting the time available for assistive cueing systems to intervene.
  • Struggle with high inter-subject variability in pre-FOG patterns.
• Goal: To develop a proactive agent capable of dynamic decision-making that extends the prediction horizon, allowing for earlier and more personalized interventions.

2. Methodology

The study proposes a Reinforcement Learning (RL) framework where an agent learns to anticipate FOG by maximizing long-term rewards rather than fitting fixed labels.

A. Data and Feature Extraction

• Dataset: Daphnet Freezing of Gait Dataset (8 subjects).
• Signal Processing: Acceleration data is transformed into a Triple Index (TI) using Dynamic Mode Decomposition (DMD). A decrease in TI indicates the transition from stable to unstable gait.
• State Representation: The agent observes a 10-second window containing six specific parameters:
  1. Remaining time ($\tau_t$): Time until potential FOG.
  2. Mean ($\mu_t$): Average signal value.
  3. Standard Deviation ($\sigma_t$): Signal volatility.
  4. Trend/Slope ($\nabla_t$): Direction of gait deterioration.
  5. Spike Score ($\psi_t$): Detection of extreme fluctuations.
  6. Z-score ($\zeta_t$): Normalized deviation for faster convergence.

B. Agent Architecture

• Algorithm: Double Deep Q-Network (DDQN) combined with Prioritized Experience Replay (PER).
  • DDQN: Decouples action selection from value evaluation to mitigate Q-value overestimation bias.
  • PER: Samples transitions based on Temporal Difference (TD) error magnitude, allowing the agent to focus on "surprising" or high-impact experiences while correcting bias via Importance Sampling.
• Action Space: The agent chooses between two actions at each timestep:
  1. Wait: Continue observing.
  2. Place Flag: Predict an imminent FOG event.
• Reward Shaping: A time-based reward system guides the agent:
  • +150: Accurate prediction (within the target window).
  • -40: Early prediction (>15s before onset).
  • -60: Late prediction (<6s before onset).
  • -200: Failure (missed prediction).
  • +0.1: Reward for every "wait" action to encourage exploration without premature flagging.
• Training: 9,000 episodes with a target prediction horizon of 6 seconds.

C. Baseline Comparison

The proposed RL agent is compared against an internal CNN-LSTM baseline (both with and without the six extracted parameters) to evaluate the efficacy of the proactive approach versus fixed-window supervised learning.

3. Key Contributions

1. Paradigm Shift: Moves from fixed-window/threshold-based prediction to dynamic, proactive decision-making. The agent learns when to act to maximize safe cueing time.
2. Extended Prediction Horizon: Achieves significantly longer maximum prediction windows compared to state-of-the-art supervised methods.
3. Hybrid RL Architecture: Successfully applies the combination of DDQN and PER to non-stationary, highly personalized FOG signals, addressing the challenge of high inter-subject variability.
4. Personalization: Demonstrates robust performance in both Subject-Dependent (trained on same user) and Subject-Independent (Leave-One-Subject-Out) scenarios.

4. Experimental Results

• Prediction Horizons:
  • Subject-Independent (LOSO): Maximum prediction horizon of 8.72 seconds.
  • Subject-Dependent: Maximum prediction horizon of 7.89 seconds.
  • Comparison: While the mean prediction horizon (5.16s independent, 3.98s dependent) was comparable to or slightly lower than the best supervised baseline (Fu et al., 6.13s), the maximum achievable horizon was superior. This indicates the agent can exploit favorable signal conditions to provide earlier warnings than fixed-window models.
• Performance Metrics:
  • The agent achieved stable prediction behavior with mean values consistently exceeding standard deviations.
  • Misplaced Predictions: Some errors occurred in subjects with high FOG density (e.g., Subject 5), suggesting a correlation between FOG density and false alarms.
  • Undecided Points: In subject-dependent settings, the agent never failed to make a prediction (0 undecided points), indicating reliable recognition of individual gait patterns.
• Statistical Validation:
  • Shapiro-Wilk and Levene's tests confirmed normal distribution and equal variance, validating the use of t-tests.
  • No significant correlation was found between demographic factors (Age, Disease Duration, H&Y stage) and prediction horizon length.

5. Significance and Conclusion

• Clinical Impact: An extended prediction window of up to 8.72 seconds is critical for clinical applications. It provides sufficient time to account for device latency and patient neuromotor reaction times, enabling proactive cueing (e.g., auditory or visual stimuli) to prevent a full FOG episode and reduce fall risks.
• Adaptability: Unlike static models, this RL agent adapts to the evolving state of gait degradation, offering a personalized approach that can be integrated into wearable assistive devices.
• Future Work: The authors plan to integrate LSTM for better temporal encoding, implement signal smoothing to reduce false alarms, and validate the system through clinical trials to ensure real-world reliability and safety.

Summary: This paper presents a novel, proactive RL framework that outperforms traditional supervised learning in terms of maximum prediction horizon, offering a more flexible and timely solution for mitigating Freezing of Gait in Parkinson's patients.