Technical Summary: An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories

Problem Statement

Deep generative models, particularly Variational Autoencoders (VAEs), have proven effective for structured data like images and text. However, when applied to sequences of human skeletons, standard VAEs often struggle to isolate the intrinsic geometry of shapes and their motion. Instead, they allocate significant model capacity to "nuisance factors"—variations in camera orientation, subject scale, viewpoint, and execution speed (temporal rate variability). These factors are unavoidable in naturalistic recording setups but are irrelevant for tasks such as activity classification or clinical mobility assessment. Traditional shape analysis methods remove these transformations before statistical analysis, but integrating these geometric invariances into powerful deep learning frameworks, which typically operate on Euclidean variables, remains a challenge.

Methodology

The authors propose the Elastic Shape Variational Autoencoder (ES-VAE), a geometry-aware generative model designed specifically for skeletal trajectories. The core innovation lies in representing skeleton data on Kendall's shape manifold and utilizing the Transported Square-Root Velocity Field (TSRVF) framework.

1. Geometric Representation

• Kendall Shape Space: Each skeleton is treated as a configuration of $k$ landmarks in $\mathbb{R}^m$. The model removes translation (by centering), scale (by normalizing to unit Frobenius norm), and rotation (by quotienting out the special orthogonal group $SO(m)$). This results in a representation invariant to rigid transformations and global scaling.
• TSRVF for Temporal Alignment: To address rate variability (e.g., different walking speeds), the method employs the TSRVF framework. This involves computing the covariant derivative of the trajectory, parallel transporting tangent vectors to a common reference point (the Fréchet mean), and normalizing them. This allows for optimal temporal alignment via Dynamic Programming, effectively removing speed profiles while preserving shape dynamics.

2. Riemannian VAE Architecture

The ES-VAE operates directly on the manifold rather than approximating it in Euclidean space:

• Encoder: Takes the tangent vector representation of a trajectory (computed via the Riemannian logarithm map from the Fréchet mean) and maps it to a low-dimensional latent space $z \in \mathbb{R}^L$.
• Decoder: Maps the latent code $z$ back to the tangent space via a neural network, then projects it onto the shape manifold using the Riemannian exponential map to reconstruct the trajectory.
• Loss Function: The model is trained by maximizing the Riemannian Evidence Lower Bound (ELBO). The reconstruction term utilizes the squared geodesic distance on the shape manifold, and the regularization term is the Kullback-Leibler (KL) divergence against a standard normal prior in the latent space.

Key Contributions

1. Elastic Shape VAE: Introduction of a Riemannian VAE that embeds skeleton trajectories (represented by transported SRVFs on Kendall shape space) into a low-dimensional latent space. This design ensures complete invariance to translation, rotation, scaling, and activity speed profiles by construction.
2. Manifold Learning Superiority: Using synthetic data on a unit sphere ($S^2$), the authors demonstrate that ES-VAE approximates non-geodesic, one-dimensional submanifolds more accurately than Euclidean PCA, Euclidean VAE, and Tangent-space PCA. This highlights the importance of preserving nonlinear geometric structures that Euclidean approximations lose.
3. Clinical Interpretability and Performance: On a clinical dataset of 155 participants (healthy vs. post-stroke), the learned latent dimensions were shown to:
   • Map to interpretable gait features (e.g., stride length, limb stiffness, arm variability).
   • Correlate highly with clinical mobility scores (POMA).
   • Outperform classical and deep learning baselines in predicting stroke severity and classifying stroke laterality.
4. Action Recognition Performance: On a curated subset of the NTU RGB+D dataset, the ES-VAE latent representations outperformed various baselines (including TCNs, Transformers, and Graph Convolutional Networks) in classifying difficult action categories, demonstrating the generalizability of the shape-based representation.

Experimental Results

The paper evaluates ES-VAE against a comprehensive set of baselines, including raw skeleton methods (PCA, VAE, TCN, LSTM, Transformer, ST-GCN), joint angle methods, and tangent vector methods.

• Stroke Gait Dataset (Regression): For predicting POMA scores, ES-VAE achieved an $R^2$ of 0.74 and RMSE of 2.82, significantly outperforming the best raw-skeleton deep learning model (LSTM, $R^2=0.64$) and Tangent PCA ($R^2=0.70$).
• Stroke Gait Dataset (Classification): In three-class classification (Healthy, Left Stroke, Right Stroke), ES-VAE achieved a macro F1 of 0.83, surpassing Tangent PCA (0.79) and all raw-skeleton deep learning models.
• NTU-60 Action Recognition: On a difficult 10-class subset, ES-VAE achieved a macro F1 of 0.56, outperforming the best tangent-vector baseline (Tangent PCA at 0.50) and raw-skeleton baselines. Notably, it improved upon a Vanilla VAE (0.27) by 0.29, demonstrating the value of the geometric preprocessing.
• Ablation Studies: Results confirmed that geometric alignment is critical. Removing translation alone was insufficient; the combination of Kendall shape space (removing scale/rotation) and TSRVF (removing rate variability) yielded the best performance. The geodesic reconstruction loss also provided a consistent advantage over Euclidean tangent-space MSE.

Significance and Claims

The paper claims that the primary advantage of ES-VAE is structural: by geometrically removing nuisance variability (translation, rotation, scale, speed) before modeling, the deep network can allocate its capacity entirely to learning meaningful shape dynamics.

• Interpretability: Unlike "black box" deep learning approaches, the latent dimensions of ES-VAE correspond to specific, clinically relevant gait features (e.g., stride reduction, limb stiffness), offering a principled framework for analyzing longitudinal data on pose shape manifolds.
• Generalizability: The approach is shown to be effective across both clinical gait analysis and general action recognition, suggesting that modeling directly on the manifold preserves essential geometric structure that is lost in Euclidean approximations.
• Limitations: The authors note that the study is limited by dataset size (155 participants for stroke, 40 for NTU subset) and the use of kinematic data only. They suggest future work could involve larger multi-center datasets, multimodal inputs (e.g., EMG), and more expressive architectures, though potentially at the cost of interpretability.

In conclusion, ES-VAE provides a robust, interpretable, and high-performing framework for skeleton pose trajectory analysis, demonstrating that incorporating geometric invariances into the generative model architecture yields superior results compared to standard deep learning approaches that attempt to learn these invariances from raw data.