A Graph Neural Network Model for Real-Time Gesture Recognition Based on sEMG Signals

Technical Summary: A Graph Neural Network Model for Real-Time Gesture Recognition Based on sEMG Signals

Problem Statement
Accurate and immediate hand gesture recognition is critical for seamless control in advanced hand prostheses, augmented reality (AR), and virtual reality (VR) systems. While surface electromyography (sEMG) signals from the forearm are commonly used for this purpose, existing real-time approaches face significant challenges. Traditional methods typically rely on binning sEMG signals and extracting temporal or frequency domain features for classification using algorithms like Support Vector Machines (SVMs) or Linear Discriminant Analysis (LDA). However, the computational complexity arising from continuous binning and the reliance on regular grid structures often fail to capture the complex spatial and temporal dependencies inherent in neuromuscular communication. Furthermore, many existing studies focus on pre-recorded data rather than true real-time prediction, limiting their applicability in dynamic human-computer interaction scenarios.

Methodology
The authors propose a novel framework that models sEMG signals as dynamic graphs to leverage Graph Neural Networks (GNNs) for real-time gesture recognition. The methodology consists of four primary stages:

1. Graph Generation:

   • Data Partitioning: Continuous sEMG time series from $M$ channels (electrodes) are partitioned into overlapping windows of length $L$.
   • Correlation Matrix: Within each window, the Pearson correlation coefficient is calculated between every pair of time series signals. This results in an $M \times M$ correlation matrix $P[n]$.
   • Graph Construction: The correlation matrix serves as the adjacency matrix for an undirected, fully connected graph $G[n]$. The nodes represent the electrodes, and the edge weights represent the calculated Pearson correlation coefficients, capturing both individual and mutual muscle activation patterns. Self-loops are pinned to zero.

2. Event Detection:

   • To identify meaningful gesture segments and avoid low signal-to-noise regions, the system employs an event detection measure based on the Frobenius norm of the difference between the current window's signal matrix and a previous window's matrix ($\Delta[n, \tau]$).
   • A subject-dependent threshold is established based on the variance of this metric. Only windows where the metric exceeds the threshold are processed for classification, ensuring the model focuses on significant gesture onsets.

3. Classification Architecture:

   • The generated graphs, paired with gesture labels, serve as training data.
   • A straightforward GNN architecture is utilized, consisting of two fully connected layers with ReLU activation functions.
   • The model is trained using Stochastic Gradient Descent (SGD) to classify hand gestures based on the graph structure, which propagates information between channels to understand contextual relationships.

4. Real-Time Pipeline:

   • The system operates in a loop: acquiring new data, waiting for a specific accumulation of samples ($t_0$), calculating the event detection metric, and, if a gesture is detected, generating a graph and feeding it into the trained GNN for prediction.

Key Contributions

• Graph-Based Representation: The paper introduces a method to transform raw sEMG time series into graph structures where nodes are electrodes and edges represent temporal correlations. This approach bridges the gap between raw sensor data and meaningful gesture representations, capturing spatial and temporal dynamics more effectively than grid-based methods.
• Real-Time Efficiency: The framework is designed specifically for low-latency operation. By filtering for events before graph construction and utilizing a lightweight GNN, the system minimizes computational overhead.
• Subject-Dependent Thresholding: The event detection mechanism utilizes a variance-based, subject-specific threshold, ensuring robustness across different individuals and signal conditions.

Experimental Results
The proposed method was evaluated using a dataset from 8 healthy subjects performing 5 distinct hand gestures (pointer, middle finger flexion, ring finger flexion, v-flexion, and hand closure) using a Myo band with 8 electrodes.

• Accuracy: The proposed method achieved an average classification accuracy of 99% (specifically 99.89% with 97.5% window overlap), significantly outperforming state-of-the-art techniques cited in the study, including muscle synergy-based approaches and Temporal Muscle Activation (TMA) image-based CNNs.
• Robustness: The model demonstrated strong generalization capabilities. Even with reduced training data (achieved by lowering the window overlap to 70%), the model maintained a high Area Under the Curve (AUC) score of 0.967.
• Latency: The average time for the entire process (graph construction and prediction) was 48 ms on an M1 Pro CPU (3 ms for graph generation, 45 ms for prediction). This duration is notably shorter than the time interval between successive events, confirming the system's suitability for real-time applications.
• Comparison: The results indicate that modeling the correlations between time series via graph networks yields superior performance compared to treating channels as independent entities or using standard CNNs on TMA images.

Significance and Claims
The authors claim that their graph-based approach offers a robust and efficient solution for real-time gesture recognition, surpassing current state-of-the-art methods in both accuracy and computational efficiency. By conceptualizing gestures as dynamic graphs, the method successfully unravels intricate patterns in neuromuscular communication that are difficult to capture with traditional binning or grid-based deep learning approaches. The paper posits that this technique is particularly effective for applications requiring seamless human-machine interaction, such as prosthetic control and AR/VR systems. The authors conclude that the method's ability to maintain high performance with limited training data and its low latency make it a significant step forward, with potential future applications in fluid gesture recognition for individuals with disabilities.