MuPPet: Multi-person 2D-to-3D Pose Lifting

1. Problem Statement

The paper addresses the challenge of multi-person 2D-to-3D pose lifting, specifically in social interaction scenarios. While 2D pose estimation has advanced significantly, lifting these 2D detections to 3D absolute coordinates in a group setting remains difficult due to:

• Inter-person Relationships: Existing methods often treat individuals independently, neglecting the "coherence," "synchrony," and "mimicry" inherent in social groups.
• Variable Group Sizes: Many state-of-the-art (SOTA) multi-person methods (e.g., POTR-3D) are trained on fixed group sizes, requiring zero-padding for smaller groups, which is impractical for real-world dynamic scenes.
• Occlusion: In crowded social settings, individuals frequently occlude one another. Single-person models struggle to infer occluded poses without context from surrounding people.
• Absolute vs. Relative Space: Social interaction analysis requires absolute 3D poses (positions in the real world) rather than relative poses, as this allows for the analysis of spatial relationships and group dynamics (e.g., F-formations).

2. Methodology: MuPPet

MuPPet is a diffusion-based framework designed to lift sequences of detected 2D poses from multiple people into absolute 3D poses. The architecture consists of three core innovations:

A. Architecture Overview

• Input: A sequence of 2D pose joints ($T \times P \times J \times 2$) where $T$ is time, $P$ is the number of people (dynamic), and $J$ is the number of joints.
• Backbone: A Transformer-based denoiser integrated into a Denoising Diffusion Probabilistic Model (DDPM).
• Process: The model takes a noisy 3D pose and iteratively denoises it over $N$ steps, conditioned on the input 2D poses, to predict the final absolute 3D pose.

B. Key Components

1. Dynamic Multi-Person Spatial Attention:

   • Extends standard self-attention (usually applied within a single person) to the entire set of joints across all people in the scene.
   • By reshaping features to $(P \cdot J) \times 512$, the model captures relationships between joints of different individuals.
   • The attention weights are independent of the number of people $P$, allowing the model to handle dynamic group sizes without retraining or padding.

2. Person Encoding (PE):

   • Since the attention mechanism treats all tokens similarly, it cannot inherently distinguish which joint belongs to which person.
   • MuPPet introduces learnable Person Encodings ($E \in \mathbb{R}^{P \times 512}$) added to the features of all joints belonging to a specific person.
   • This explicitly informs the model of individual identities, enabling it to learn inter-person correlations effectively.

3. Permutation Learning (Data Augmentation):

   • Standard augmentations (rotation/translation) break inter-person relationships. MuPPet introduces a novel augmentation strategy:
     • Superset Permutation: Randomly reordering the sequence of people ($P$) in the input.
     • Subset Permutation: Randomly selecting a subset of people (e.g., $P-1$ or $P-2$) and permuting them.
   • Because of the Person Encoding, the model can still distinguish individuals even after their order is shuffled. This forces the model to learn robust relationships regardless of input ordering or group size, significantly increasing training diversity.

4. Diffusion Process & Loss:

   • Uses a diffusion process to generate diverse hypotheses for occluded joints, aggregating them via averaging to refine the final prediction.
   • Loss Function: Separates the loss into Absolute Root (global position) and Relative Joints (body shape). A weighted sum ($\lambda \mathcal{L}_{abs} + (1-\lambda)\mathcal{L}_{rel}$) balances the difficulty of predicting global location versus relative pose.

3. Key Contributions

1. Dynamic Group Handling: A method capable of lifting poses for a dynamic number of people in a group interaction, overcoming the fixed-size limitations of previous works.
2. Explicit Inter-person Modeling: The combination of Person Encoding and Dynamic Spatial Attention allows the model to explicitly learn and leverage correlations between individuals.
3. Novel Augmentation: The introduction of Permutation Learning (superset and subset) to enhance training diversity without disrupting social relationships.
4. Superior Occlusion Handling: The framework significantly outperforms single-person baselines in occluded scenarios by inferring missing poses based on the spatial context of the group.

4. Experimental Results

The authors evaluated MuPPet on the Haggling and CMU Panoptic datasets.

• vs. Single-Person Baselines (D3DP):
  • MuPPet outperformed the SOTA single-person diffusion method (D3DP) by 17.2% in absolute joint error and 6.4% in relative joint error on the Haggling dataset.
  • The improvement was most significant in absolute root error, proving the benefit of modeling group dynamics for global positioning.
• vs. Multi-Person SOTA (VirtualPose, POTR-3D):
  • On the CMU Panoptic dataset, MuPPet achieved the lowest Mean Per Joint Positional Error (MPJPE) among all compared methods (55.8 mm average), outperforming both direct 3D estimation and other 2D-to-3D lifting methods.
• Occlusion Study:
  • As the number of occluded joints increased, the performance gap between MuPPet and single-person models widened. MuPPet maintained high accuracy even with significant occlusion, whereas single-person models degraded rapidly.
• Ablation Studies:
  • Removing Person Encoding caused a significant drop in absolute pose accuracy.
  • Removing Permutation Learning (both superset and subset) degraded performance, confirming that these augmentations are critical for generalization to varying group sizes.
  • The Diffusion Process was proven necessary, reducing absolute error from 167.5 mm (without diffusion) to 119.5 mm.

5. Significance

MuPPet represents a significant step forward in socially-aware 3D pose estimation. By explicitly modeling the inter-person relationships that define social interactions, it moves beyond treating people as isolated entities.

• Practical Application: The ability to handle dynamic group sizes and occlusions makes it suitable for real-world applications like humanoid robot interaction, crowd analysis, and social behavior studies.
• Absolute Space: The focus on absolute 3D coordinates enables the analysis of physical distances and group formations (e.g., F-formations), which are crucial for understanding social dynamics but were previously difficult to estimate accurately in multi-person settings.
• Future Direction: The paper suggests that incorporating multi-modal data (e.g., audio) could further enhance pose inference in group settings.