Expanding mmWave Datasets for Human Pose Estimation with Unlabeled Data and LiDAR Datasets

1. Problem Statement

Human Pose Estimation (HPE) using millimeter-wave (mmWave) radar is gaining traction due to its privacy-preserving nature, robustness to lighting, and 3D capabilities. However, the field faces two critical bottlenecks:

1. Data Scarcity: Labeled mmWave datasets are extremely limited in volume.
2. Lack of Diversity: Existing datasets suffer from homogeneity in two areas:
   • Point Cloud (PC) Attributes: Lack of device heterogeneity and environmental variety leads to uniform detection noise, point density, and motion sensitivity.
   • Human Poses: Subjects often perform simple, uniform postures facing the radar, failing to cover complex or unseen scenarios.

These limitations severely degrade the generalization ability of trained models when deployed in real-world or unseen environments. While unlabeled mmWave data (easy to collect) and diverse LiDAR datasets (abundant and pose-rich) are available, they cannot be directly used because:

• Unlabeled mmWave data lacks ground-truth skeletons.
• LiDAR point clouds have fundamentally different physical attributes (sensing principles) compared to mmWave, causing domain shift.

2. Methodology: EMDUL

The authors propose EMDUL, a novel pipeline designed to expand existing mmWave datasets by integrating unlabeled mmWave data and annotated LiDAR datasets. The framework consists of two independent modules:

A. Pseudo-Label Estimator for Unlabeled mmWave Data

To utilize unlabeled mmWave data, the authors train a Pseudo-label Estimator ($\theta_{pl}$) to generate skeleton annotations.

• Training Strategy: The estimator is trained on a small set of labeled data ($D_{lab}$) using Mean Squared Error (MSE) loss and on unlabeled data ($D_{unlab}$) using a novel Unsupervised Temporal Consistency Loss (UTCL).
• Unsupervised Temporal Consistency Loss (UTCL): This loss leverages the physical principle of mmWave radar: Doppler effect. Moving body parts generate stronger signals than static ones.
  • Dynamic Consistency Loss (DCL): Penalizes joints embedded within the point cloud (likely moving) if their predicted flow (velocity) is near zero.
  • Static Consistency Loss (SCL): Penalizes joints far from detected points (likely static) if they show significant flow.
  • This enforces temporal consistency and physical plausibility, improving the reliability of pseudo-labels.

B. Closed-Form PC Converter for LiDAR Data

To bridge the gap between LiDAR and mmWave, the authors propose a Closed-Form Converter that translates LiDAR point clouds into synthetic mmWave counterparts ($D_{conv}$). This process simulates mmWave-specific attributes through a sequential augmentation pipeline:

1. Noisy Point Addition (NPA): Adds noise points to simulate environmental clutter.
2. Flow-based Point Filtering (FPF): The core innovation. It interpolates flow vectors based on skeleton motion and filters out points with low flow magnitude. This mimics the mmWave radar's tendency to detect moving parts while missing static ones.
3. Random Sampling (RS): Reduces point density to match the sparsity of mmWave data.
4. Noise Injection (NI): Adds coordinate noise to simulate lower spatial resolution.

C. The EMDUL Pipeline

1. Conversion: LiDAR data is converted to $D_{conv}$ (mmWave-like).
2. Labeling: The pseudo-label estimator is trained on $D_{lab} \cup D_{conv}$ and then used to generate pseudo-labels for $D_{unlab}$, creating $D_{pl}$.
3. Expansion: The final expanded dataset is $D_{exp} = D_{lab} \cup D_{conv} \cup D_{pl}$.
4. Training: The final HPE inference model is trained from scratch on $D_{exp}$.

3. Key Contributions

1. Novel Pseudo-Labeling Strategy: Introduction of UTCL, which incorporates the physical motion-detection mechanism of mmWave radar (Doppler effect) to enforce temporal consistency in pseudo-labels, significantly improving their quality without ground truth.
2. Closed-Form LiDAR-to-mmWave Converter: A domain adaptation method that does not require paired mmWave-LiDAR data. It uses Flow-based Point Filtering (FPF) to realistically simulate the motion-dependent detection characteristics of mmWave sensors.
3. Comprehensive Dataset Expansion: A unified pipeline that simultaneously increases data volume (via unlabeled data) and diversity (via LiDAR conversion), addressing both scarcity and homogeneity issues.

4. Experimental Results

The authors evaluated EMDUL on standard mmWave datasets (MM-Fi and mmBody) and LiDAR datasets (LiDARHuman26M and HmPEAR).

• Performance Gains:
  • In-Domain (MM-Fi $\to$ MM-Fi): Reduced MPJPE (Mean Per Joint Position Error) by 15.1%.
  • Out-of-Domain (mmBody $\to$ MM-Fi): Reduced MPJPE by 18.9%, demonstrating superior generalization.
• Comparison with Baselines: EMDUL consistently outperformed state-of-the-art methods (including P4T, SPiKE, and Mean-Teacher based semi-supervised approaches) across all settings.
• Ablation Studies:
  • FPF Importance: Removing Flow-based Point Filtering from the converter significantly degraded performance, proving the necessity of simulating motion detection.
  • UTCL Components: Combining Dynamic and Static consistency losses yielded the best results; using them individually led to bias toward overly dynamic or static predictions.
  • Data Scarcity: EMDUL showed the most significant improvement when labeled data was extremely scarce (1% of the dataset), reducing error by a large margin compared to baselines.

5. Significance

This work addresses a fundamental barrier in mmWave HPE: the lack of diverse, large-scale labeled data. By effectively leveraging unlabeled data and cross-modal LiDAR data without requiring expensive paired annotations, EMDUL provides a scalable solution to improve model robustness.

• Practical Impact: It enables the training of HPE models that generalize well to unseen environments and complex poses, which is crucial for real-world deployment in robotics, human-computer interaction, and surveillance.
• Methodological Insight: The paper highlights that incorporating physical priors (like Doppler-based motion detection) into deep learning loss functions and domain adaptation pipelines is essential for sensor-specific tasks.

The code is publicly available, facilitating further research in mmWave sensing and cross-modal data augmentation.