Joint angle based learning to refine kinematic human pose estimation

Technical Summary: Joint Angle Based Learning to Refine Kinematic Human Pose Estimation

Problem Statement
Marker-free Human Pose Estimation (HPE) is critical for applications in sports analysis, healthcare, and human-computer interaction. However, existing deep learning-based models face two primary challenges when analyzing kinematic human poses:

1. Incorrect Keypoint Recognition: Complex poses, occlusions, and distracting backgrounds lead to erroneous keypoint identification (outliers), which severely disrupts downstream kinematic analyses such as velocity, acceleration, and inverse dynamics calculations.
2. Spatiotemporal Jitters: Inconsistent localization of keypoints across consecutive frames results in fluctuating trajectories. This issue is exacerbated by the reliance on single-image datasets (e.g., COCO-WholeBody) where manual annotations often lack temporal consistency. While video datasets exist, they suffer from annotation inconsistencies, particularly in high-amplitude or fast-changing motions, leading models to prioritize Euclidean distance accuracy over trajectory continuity.

Current refinement methods, such as the state-of-the-art SmoothNet, often struggle with complex sports scenarios, resulting in position drift or insufficient smoothing of outliers.

Methodology
The authors propose a novel Joint Angle-based Refinement (JAR) framework designed to generate high-quality training data and refine single-image HPE outputs through a three-stage process:

1. Joint Angle-based Pose Description:

   • Instead of relying solely on absolute coordinates, the method converts 13 keypoint coordinates (extracted via a single-image HPE model like HRNet or ViTPose++) into 12 joint angles.
   • A stable base point (the "nose") is smoothed using a Savitzky-Golay filter.
   • Limb lengths are optimized using a trust-region algorithm based on two biomechanical constraints: proportional consistency between limbs and negligible inter-frame variation due to foreshortening. This approach makes the description robust to changes in perspective and shooting distance.

2. Fourier Series-based "Ground Truth" Generation:

   • Recognizing that joint motions often exhibit periodic characteristics, the authors approximate temporal joint angle variations using an 8th-order Fourier series.
   • Parameters for the series are estimated from open datasets to simulate diverse inter-individual differences.
   • A synthetic training dataset is constructed by sliding a window over these generated cycles. To simulate real-world noise, Gaussian white noise (small jitters) and large-amplitude outliers (simulating recognition errors) are injected into the "ground truth" sequences. This creates a high-quality dataset with spatiotemporal consistency that is difficult to obtain via manual annotation.

3. BiGRU-Attention Refinement Network:

   • A Bidirectional Gated Recurrent Unit (BiGRU) network with an attention mechanism is designed as a post-processing module.
   • This network takes the noisy joint angle sequences as input and outputs smoothed, refined angle sequences.
   • The refined angles, combined with the stabilized base point and optimized limb lengths, are used to reconstruct the final keypoint coordinates.
   • A sliding window approach with weighted averaging is employed during inference to ensure continuous smoothing across the entire video sequence.

Key Contributions

• Robust Pose Representation: The proposal of a joint angle-based model that resists perspective and distance interferences, significantly improving the accuracy and robustness of body recognition compared to coordinate-based methods.
• Synthetic High-Quality Dataset: A novel method for generating training data using high-order Fourier series to approximate physiological joint motion patterns, providing reliable "ground truth" for network training without relying on inconsistent manual annotations.
• Superior Refinement Architecture: The design of a BiGRU-Attention network that effectively eliminates outliers and jitters, outperforming existing refinement frameworks in challenging sports scenarios.
• Dataset Rectification Capability: Demonstration that the method can be applied to rectify existing video datasets (e.g., PoseTrack), correcting inter-frame inconsistencies caused by manual annotation errors.

Experimental Results
The JAR method was evaluated against the state-of-the-art SmoothNet across various athletic activities, including sprinting, standing triple jump, figure skating, and breaking (breakdancing).

• Outlier Correction: In a sprint case, JAR achieved a 100% outlier correction rate compared to SmoothNet's 45.45%. In a standing triple jump, JAR corrected 95.61% of errors versus 53.51% for SmoothNet. The overall correction rate for JAR was 97.77%, nearly double that of SmoothNet.
• Trajectory Smoothness: In complex scenarios like figure skating (flying spins) and breaking (handstands), JAR produced significantly smoother trajectories and more physiologically consistent velocity curves. SmoothNet exhibited noticeable coordinate drift and high-frequency fluctuations, particularly in the Y-axis components.
• Model Selection: Among ten tested sequence-to-sequence architectures (including TCN, LSTM, Mamba1/2, and their attention variants), the BiGRU-Attention model demonstrated the best balance of accuracy (lowest Mean Squared Error) and computational efficiency.

Significance and Claims
The paper claims that the limitations of current HPE refinement are largely due to the lack of high-quality, temporally consistent training datasets. By leveraging the physiological constraints of human motion (joint angles and limb lengths) and Fourier series approximation, the proposed JAR method:

• Enables deep learning models to learn robust spatiotemporal patterns, effectively correcting wrongly recognized joints and smoothing trajectories.
• Demonstrates superior generalization in challenging sports cases where current models fail.
• Offers a practical tool for rectifying existing video datasets, thereby reducing the adverse effects of inaccurate manual annotations and improving the reliability of HPE technology for motion analysis.

The authors conclude that their approach provides a reliable foundation for kinematic analysis, allowing for more accurate quantification of athletic performance and biomechanical assessment.