Integrating Skeleton Based Representations for Robust Yoga Pose Classification Using Deep Learning Models

Technical Summary: Integrating Skeleton Based Representations for Robust Yoga Pose Classification Using Deep Learning Models

Problem Statement
While automated yoga pose classification offers the potential to democratize access to yoga training and provide personalized feedback, developing robust systems remains challenging. Existing approaches often rely on raw image inputs or single pose extraction models, facing limitations such as noisy data, low-resolution images, class imbalance, and inadequate diversity in benchmark datasets. Furthermore, the nuanced variations in yoga postures—often defined by subtle differences in limb angles and body alignment—require sophisticated methodologies to capture intricate details accurately. There is a lack of systematic benchmarking comparing skeleton-based representations against raw image inputs specifically for yoga pose recognition.

Methodology
The study introduces a structured workflow involving dataset curation, pose representation generation, and deep learning classification:

1. Dataset Creation (Yoga-16): The authors curated a new dataset, "Yoga-16," comprising 16 distinct yoga poses. This dataset was constructed by filtering and merging images from the Yoga-82 dataset and the Yoga Poses Dataset. The curation process addressed common issues such as multiple subjects per image, low quality, non-human subjects, and zoomed-out views. The final dataset ensures balanced class distribution (approximately 80 images per class) and includes diverse movement patterns (standing, reclining, sitting) and complexity levels.
2. Input Modalities: The study systematically evaluated three input modalities:
   • Direct Images: Raw input images.
   • MediaPipe Pose Skeletons: Skeleton images generated using the MediaPipe Pose (BlazePose) model, which detects 33 keypoints.
   • YOLOv8 Pose Skeletons: Skeleton images generated using the YOLOv8-Pose model.
3. Deep Learning Architectures: Three Convolutional Neural Network (CNN) architectures were employed and trained from scratch on the Yoga-16 dataset to learn task-specific representations:
   • VGG16: Utilizing sequential small convolutional filters.
   • ResNet50: Leveraging residual blocks to train deeper networks.
   • Xception: Built on depthwise separable convolutions for efficient feature learning.
4. Training and Evaluation: Models were trained using the Adam optimizer with categorical cross-entropy loss. The study utilized 5-fold cross-validation to assess generalization and employed Grad-CAM for interpretability analysis to visualize the features driving classification decisions.

Key Contributions

1. Yoga-16 Dataset: Introduction of a standardized, balanced benchmark dataset of 16 yoga poses designed to address limitations in existing datasets, such as class imbalance and low resolution.
2. Systematic Modality Comparison: A comprehensive evaluation comparing direct image inputs against two distinct skeleton-based representations (MediaPipe and YOLOv8-Pose).
3. Architectural Benchmarking: Evaluation of VGG16, ResNet50, and Xception across the different input modalities to determine optimal configurations for yoga pose classification.
4. Interpretability Analysis: Application of Grad-CAM to provide insights into model decision-making, specifically highlighting how models attend to skeletal features like limb alignments and joint angles.
5. Generalization Validation: Demonstration of model robustness through cross-validation and testing on a custom dataset derived from real-world YouTube videos to simulate diverse environmental conditions.

Experimental Results
The experiments demonstrated that skeleton-based representations consistently outperformed raw image inputs across all tested architectures.

• Best Performance: The VGG16 model combined with MediaPipe Pose skeleton inputs achieved the highest accuracy of 96.09%, with a precision of 96.27% and an F1-score of 96.10%.
• Modality Comparison: MediaPipe skeletons generally yielded superior results compared to YOLOv8-Pose skeletons. For instance, VGG16 achieved 96.09% with MediaPipe versus 91.41% with YOLOv8-Pose. The authors attribute this to MediaPipe's higher stability and anatomical consistency in landmark detection, particularly for symmetrical poses and fine-grained limb orientations.
• Architecture Comparison: Among the three models, VGG16 consistently outperformed ResNet50 and Xception across all input types. ResNet50 achieved the lowest performance (66.41% on direct images, 88.28% on MediaPipe skeletons), while Xception showed intermediate performance.
• Generalization: When tested on a custom dataset simulating real-world scenarios (20 images per class from YouTube), the VGG16-MediaPipe model maintained strong performance with an accuracy of 93.75%, demonstrating its ability to generalize beyond the controlled training environment.
• Error Analysis: Misclassifications primarily occurred between poses with overlapping skeletal features (e.g., Dolphin Plank vs. Wide Angle Seated Forward Bend), indicating that subtle joint angle ambiguities remain a challenge.

Significance and Claims
The paper claims that the integration of skeleton-based representations, specifically using MediaPipe Pose, significantly enhances the precision and efficiency of automated yoga pose classification by isolating pose-specific features and removing background noise. The study establishes that VGG16 is a highly reliable architecture for this task when paired with skeletal inputs, achieving state-of-the-art performance on the proposed Yoga-16 benchmark.

The authors emphasize that their approach comprehensively addresses key challenges in the field—low-resolution datasets, class imbalance, and overlapping class features—by combining a curated dataset with a dual-strategy evaluation of skeletonized and non-skeletonized representations. The work suggests that these findings provide a foundation for developing robust, real-time digital fitness applications and rehabilitation systems that can offer accurate, personalized feedback without reliance on expert practitioners. The study concludes that while current models show high accuracy, future improvements require larger, more diverse datasets and refined architectures to better distinguish poses with subtle or overlapping skeletal structures.