Technical Summary: Zero-Shot Human Pose Estimation Using Diffusion-Based Inverse Solvers

Problem Statement

Human pose estimation aims to reconstruct a full-body 3D posture (head, torso, limbs) from sparse sensor data. While recent advances using conditional diffusion models have shown promise in predicting full-body poses from just three sensors (typically head and two wrists), these methods suffer from a critical limitation: poor generalization across users with different body shapes.

Existing approaches, such as those by Castillo et al. (2023), condition diffusion models on both joint locations and rotations. However, joint locations are inherently scale-dependent; they vary significantly based on a user's bone lengths and body proportions. Consequently, a model trained on a specific user's data fails to generalize to users with different body shapes without extensive fine-tuning. Jointly training on multiple body shapes (as attempted by Aliakbarian et al., 2022) increases model complexity and offers no guarantee of covering all possible body variations. There is a need for an algorithm that can generalize to body shape outliers without requiring personalized training data.

Methodology: InPose

The authors propose InPose, a diffusion-based method that formulates pose estimation as an inverse problem to achieve zero-shot generalization. The core insight is decomposing the human pose into two components:

1. Scale-free pose: Defined by joint rotations (angles), which are independent of body size.
2. Scale-dependent component: Defined by joint locations in 3D space, which depend on bone lengths and body shape.

1. Inverse Problem Formulation

Instead of conditioning the diffusion model on both location and rotation (which couples the model to specific body shapes), InPose conditions the generative prior only on rotational measurements.

• Prior: A Conditional Flow Matching (CFG) diffusion model is trained to predict the distribution of scale-free poses given noisy rotational measurements ($r_m$) from the sensors.
• Likelihood (Guidance): The measured joint locations ($l_m$) are used not as a conditioning input for training, but as a likelihood term to guide the inference process. This term evaluates how well a generated pose explains the observed locations given the user's specific bone lengths.

2. Mathematical Framework

The goal is to sample from the posterior $p(r_M | y_m)$, where $r_M$ represents the 22 global joint rotations and $y_m$ are the sensor measurements. Using Bayes' rule, the score function is decomposed:
$$\nabla_{r_M^t} \log p_t(r_M^t | \{l_m, r_m\}) = \nabla_{r_M^t} \log p_t(r_M^t | r_m) + \nabla_{r_M^t} \log p_t(l_m | r_M^t)$$

• The Prior Term ($\nabla \log p(r_M^t | r_m)$): Learned by a CFG-based score model conditioned solely on rotations. This provides a scale-free distribution of plausible poses.
• The Likelihood Term ($\nabla \log p(l_m | r_M^t)$): This term acts as an inverse kinematics guidance. It requires mapping the estimated rotations to locations via a non-linear operator $D(\cdot)$ (converting 6D rotation vectors to matrices) and a linear operator $A$ (forward kinematics).

3. Handling Non-Linearity via Pseudoinverse Guidance

A key challenge is that the likelihood term involves propagating a Gaussian random variable through the non-linear function $D(\cdot)$. Standard Pseudoinverse-Guidance for Diffusion Models (ΠGDM) assumes linear operators.

• Theorem 1: The authors prove that if the score model ensures specific unit-norm and orthogonality constraints on the 6D rotation vectors, the distribution of the transformed rotations $D(r_M^0)$ given the noisy input $r_M^t$ can be approximated as a Gaussian distribution.
• This approximation allows the authors to compute the likelihood score analytically, enabling the use of ΠGDM to guide the denoising process without requiring the operator to be linear.

4. Pipeline

The inference process (Algorithm 1) iteratively refines the pose:

1. Generate a conditionally denoised estimate $\hat{r}_M^t$ using the CFG prior (conditioned on rotations).
2. Compute the likelihood score using the ΠGDM framework, incorporating the user's specific bone lengths to map rotations to expected locations.
3. Combine the prior and likelihood scores using a modified DDIM sampler to update the pose estimate for the next timestep.

Key Contributions

1. Zero-Shot Generalization: By decoupling the generative prior (rotations) from the scale-dependent measurements (locations), InPose can estimate poses for users with arbitrary body shapes without retraining or fine-tuning.
2. Inverse Solver Formulation: The paper reframes pose estimation as an inverse problem where a pre-trained diffusion prior is guided by a likelihood term derived from sparse location measurements.
3. Gaussian Approximation for Non-Linear Operators: The authors provide a theoretical proof (Theorem 1) allowing the propagation of Gaussian variables through the non-linear 6D-to-matrix rotation mapping, making ΠGDM applicable to this specific inverse problem.
4. Robustness: The method demonstrates inherent robustness to location measurement noise, as the generative prior plays a dominant role in the final estimate, unlike methods that rely heavily on noisy location inputs.

Experimental Results

Experiments were conducted on the AMASS dataset using protocols similar to the BoDiffusion baseline.

• Body Shape Scaling: When tested on users with scaled body shapes (e.g., taller or shorter), baseline models (AvatarJLM, BoDiffusion) showed significant degradation in both position error (MPJPE) and rotation error (MPJRE). In contrast, InPose maintained consistent performance across all scaling factors, demonstrating its zero-shot capability.
• Varying Relative Body Parts: In scenarios where specific body parts (e.g., arms vs. torso) were scaled differently, InPose outperformed baselines in MPJPE and Upper Body Position Error (UPE), as the inverse guidance correctly accounted for the changed bone lengths.
• Noise Robustness: Under increasing Gaussian noise in location measurements, InPose remained stable, while baselines degraded significantly.
• Trade-offs:
  • InPose slightly underperforms baselines on the default body shape (the shape used for training), likely because the baselines can learn a direct non-linear mapping for that specific shape, whereas InPose relies on a general approximation.
  • Lower Body Estimation: Without head translation as a CFG input, InPose struggles with lower body pose estimation (high LPE) because the root translation is not directly observed. However, adding head position as a CFG input (InPose (head)) significantly improves lower body estimation, outperforming all baselines across various body shapes.

Significance and Claims

The paper claims that InPose offers a pathway to zero-shot pose prediction by leveraging the decomposition of pose into scale-free and scale-dependent components. The significance lies in eliminating the need for personalized data collection and retraining for every new user, a major bottleneck in practical wearable sensor applications.

The authors modestly acknowledge limitations:

• The method does not outperform baselines on the specific body shape used for training.
• Lower body estimation relies heavily on the prior or the inclusion of head position, as root translation is not directly incorporated.
• The algorithm requires accurate knowledge of the user's bone lengths during inference; errors in bone length estimation can degrade performance.

The work suggests that future research should focus on improving lower-body modeling through physics constraints and developing methods to blindly estimate bone lengths without explicit calibration.