A Standardized Benchmark for Skeleton-Based Rehabilitation Assessment Using Deep Learning

Technical Summary: A Standardized Benchmark for Skeleton-Based Rehabilitation Assessment Using Deep Learning

Problem Statement
Automated assessment of human motion is critical for rehabilitation, offering objective evaluation of patient progress and exercise execution. However, rehabilitation motion assessment differs fundamentally from general human activity recognition. While activity recognition classifies what action is being performed (e.g., walking vs. running), rehabilitation assessment evaluates the quality of a specific action, requiring the detection of subtle deviations from an ideal motion.

Despite advances in deep learning and video-based skeleton extraction, the field lacks standardized benchmarks, consistent evaluation protocols, and reproducible methodologies. Existing research often relies on a limited set of datasets (notably KIMORE and UI-PRMD), uses inconsistent experimental protocols (varying data splits and metrics), and frequently fails to release source code. This absence of a unifying framework hinders the ability to objectively compare approaches, reproduce results, or assess generalizability across diverse populations and exercise types.

Methodology
To address these gaps, the authors propose a comprehensive benchmarking framework centered on three pillars:

1. Rehab-Pile Archive: The authors aggregated nine existing public repositories into a unified archive named "Rehab-Pile." This archive treats each specific exercise within a repository as an independent dataset, resulting in a total of 60 datasets:

   • 39 Classification Datasets: Covering binary and multi-class tasks (e.g., correct vs. incorrect execution, or specific error types).
   • 21 Extrinsic Regression Datasets: Covering continuous score prediction (e.g., severity levels or performance scores).
   • The data consists of skeleton sequences (joint coordinates) extracted from video streams (RGB or RGB-D), primarily captured via Kinect sensors or OpenPose.

2. Unified Evaluation Framework: The study evaluates nine deep learning architectures across all 60 datasets under a strictly standardized protocol:

   • Architectures: The models span four categories: Convolution-based (FCN, H-Inception, LITEMV, DisjointCNN), Recurrent-based (MotionGRU, ConvLSTM), Self-Attention-based (VanTran, ConvTran), and Graph-based (STGCN).
   • Data Preprocessing: All sequences are resampled to a common frame count per dataset, normalized using min-max scaling (computed on training data only), and converted into a multivariate time series format (flattening joint and dimension axes) for most models, except STGCN which uses raw skeleton sequences.
   • Experimental Protocol: A cross-subject split is enforced to prevent data leakage. The number of folds is determined by the number of subjects (5-fold for $\ge$10 subjects, leave-one-subject-out otherwise).
   • Ensembling: To stabilize results, each model is trained five times with different random initializations per fold, and predictions are averaged.
   • Metrics: Classification is evaluated using Accuracy and Balanced Accuracy. Regression uses Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). Efficiency is measured via training/inference time, parameter count, and FLOPs.

3. Statistical Analysis: Performance comparisons utilize Multi-Comparison Matrices (MCM) and the Wilcoxon Signed-Rank Test to determine statistical significance ($p < 0.05$) of performance differences between models.

Key Results
The extensive benchmarking yielded the following findings:

• Regression Performance: LITEMV (Light Inception with boosTing tEchniques MultiVariate) emerged as the statistically significant best-performing model for extrinsic regression, outperforming all competitors in both MAE and RMSE. The authors attribute this to LITEMV's use of Depthwise Separable Convolutions (DWSCs) and hand-crafted filters, which effectively capture both temporal and spatial correlations. ConvTran (a self-attention model with convolutional embedding) ranked second, suggesting long-term dependencies are crucial for regression tasks.
• Classification Performance: In classification tasks, LITEMV again achieved the highest average accuracy (75.49%) and balanced accuracy, though the margin over VanTran (Vanilla Transformer) was not statistically significant ($p > 0.05$). MotionGRU ranked second in balanced accuracy, performing well on unbalanced datasets. Notably, STGCN (Spatio-Temporal Graph Convolution Network), while performing reasonably in regression, ranked last in classification, suggesting GCN-based architectures may be less suitable for rehabilitation classification tasks under these conditions.
• Efficiency vs. Performance Trade-off: LITEMV demonstrated the best overall trade-off, offering top-tier performance with the lowest number of parameters and FLOPs. MotionGRU was the fastest for inference, while ConvTran was the fastest to train. VanTran was the most computationally expensive due to its quadratic FLOP scaling with sequence length and multiple attention layers.
• Model Characteristics: The results indicate that simple CNNs (FCN) and standard LSTMs (ConvLSTM) underperform compared to more specialized architectures. The success of LITEMV and ConvTran highlights the importance of capturing both local temporal patterns (via convolution) and global dependencies (via attention or recurrent memory).

Significance and Claims
The paper claims to establish a solid foundation for future research in automated rehabilitation assessment by:

1. Standardizing the Field: Providing the first unified archive (Rehab-Pile) with 60 diverse datasets and a reproducible evaluation protocol, enabling fair comparisons across studies.
2. Identifying State-of-the-Art: Demonstrating that LITEMV is currently the most effective and efficient architecture for skeleton-based rehabilitation assessment, particularly for regression tasks, while highlighting the potential of Transformer-based models for classification.
3. Promoting Reproducibility: Releasing all aggregated datasets, source code, and detailed experimental results to the community to foster transparency and accelerate the development of reliable, accessible, and personalized rehabilitation solutions.

The authors emphasize that while LITEMV currently leads, the classification task remains challenging (average accuracy ~75%), indicating a need for continued exploration of model architectures to further improve performance.