💻 computer science

LiCamPose: Combining Multi-View LiDAR and RGB Cameras for Robust Single-timestamp 3D Human Pose Estimation

This paper introduces LiCamPose, a robust single-timestamp 3D human pose estimation pipeline that fuses multi-view RGB and sparse LiDAR data using a volumetric architecture, leveraging a synthetic dataset generator and unsupervised domain adaptation to achieve strong generalization across diverse scenarios without manual 3D annotations.

Original authors: Zhiyu Pan, Zhicheng Zhong, Wenxuan Guo, Yifan Chen, Jianjiang Feng, Jie Zhou

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhiyu Pan, Zhicheng Zhong, Wenxuan Guo, Yifan Chen, Jianjiang Feng, Jie Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out exactly how a basketball player is moving their body in 3D space. If you only have one camera, it's like trying to guess the shape of a cloud by looking at a single shadow; you might get it wrong if the player is hiding behind someone or if the angle is tricky. If you have multiple cameras, it's better, but if the player is wearing dark clothes or the lighting is bad, the cameras might still get confused.

This paper introduces LiCamPose, a new "super-sense" system that combines two different types of eyes to solve this problem: RGB Cameras (which see colors and textures, like our eyes) and LiDAR Sensors (which shoot out invisible laser beams to measure distance, like a bat using echolocation).

Here is a breakdown of how it works, using simple analogies:

1. The Problem: The "Blind Spot" Dilemma

Existing methods often rely only on cameras. But in a crowded, fast-paced game like basketball, players block each other (occlusion), and cameras can't see depth well. Other methods use LiDAR, but LiDAR data is often "sparse"—think of it like a low-resolution 3D sketch made of just a few dots. It's great for knowing where something is, but hard to tell exactly how a joint is bent just from a few dots.

2. The Solution: The "Volumetric Soup"

LiCamPose doesn't just look at the camera image or the laser dots separately. It mixes them together into a 3D grid, which the authors call a "volumetric space."

The Analogy: Imagine a giant, invisible cube of Jell-O floating in the air around the player.
- The Cameras project their 2D photos into this Jell-O, painting the "skin" and "clothes" onto the inside of the cube.
- The LiDAR shoots its laser dots into the Jell-O, marking the exact physical "bones" and edges.
- The system then looks at the whole cube at once. Even if the camera can't see the player's elbow because it's hidden, the LiDAR dots might still be there. Even if the LiDAR dots are too sparse to see the curve of the arm, the camera image fills in the texture. By combining them in this 3D "soup," the system gets a much clearer picture of the pose.

3. The Training Challenge: Learning Without a Teacher

Usually, to teach a computer to recognize poses, you need a human teacher to draw lines on thousands of videos saying, "This is a knee, this is an elbow." Doing this for 3D data in a real basketball game is incredibly hard, expensive, and slow.

LiCamPose's Trick:
Instead of using a human teacher for the real game, they used a Video Game Simulator (called SyncHuman).

The Analogy: Think of it like a pilot training in a flight simulator. The simulator creates thousands of fake basketball games with perfect, computer-generated 3D skeletons. The system learns the rules of "how humans move" in this fake world first.
The Bridge (Domain Adaptation): Once the system is good at the fake game, they move it to the real basketball game. But the real game looks different (different lighting, different people). To bridge this gap without a human teacher, the system uses Self-Correction:
1. The "Confidence Meter" (Entropy): The system guesses the pose. If it's very unsure (high "entropy" or confusion), it ignores that guess. If it's very confident, it treats that guess as a "pseudo-label" (a temporary truth) to teach itself.
2. The "Anatomy Check" (Human Prior): The system has a built-in rulebook of human anatomy. It knows, for example, that your legs can't bend backward like a spider, and your left and right arms should be roughly the same length. If the system predicts a pose that breaks these rules, it gets a "penalty" and learns to fix it.

4. The Results

The researchers tested this on four different datasets, including a real, challenging basketball game they filmed themselves (the BasketBall dataset).

The Outcome: By mixing the laser dots and the camera photos, LiCamPose was able to track players much better than systems using just cameras or just lasers.
The "No-Label" Win: Even without human teachers labeling the real basketball game, the system learned effectively by using its "confidence meter" and "anatomy check" to clean up its own mistakes.

Summary

LiCamPose is like giving a computer a pair of glasses that can see both the "painting" (the camera image) and the "wireframe" (the laser dots) at the same time. It learns by practicing in a video game first, then moves to the real world where it acts like a self-correcting student: it only trusts its own guesses when it feels confident, and it constantly checks its work against the basic rules of human anatomy to make sure it doesn't do anything impossible.

Technical Summary: LiCamPose

Problem Statement

While multi-view image-based methods have achieved satisfactory performance in 3D human pose estimation under simple conditions, there is a significant gap in approaches that extract 3D human skeletons from multimodal inputs, specifically combining RGB images and sparse LiDAR point cloud data. Practical scenarios present challenges such as diverse human motions, severe occlusions, and large scene scales that existing public datasets often fail to capture. Furthermore, manually annotating 3D poses for multiple individuals in large, complex scenes is time-consuming and difficult, hindering the training of robust models without relying on extensive labeled data.

Methodology

The authors propose LiCamPose, a pipeline designed to estimate robust 3D human poses from single-timestamp, multi-view LiDAR and RGB inputs. The methodology is structured around three core components:

1. Volumetric Multi-Modal Fusion

LiCamPose adopts a top-down approach. First, it uses PointPillars to detect 3D bounding boxes for individuals. For each detected person, the system defines a volumetric space centered on the point cloud's centroid.

Point Cloud Processing: The sparse point cloud is discretized into a voxel grid ( $X \times Y \times Z$ ), where voxels containing points are assigned a value of 1. A V2V-Net extracts point cloud-related features ( $f_p$ ).
RGB Processing: 2D pose heatmaps are generated from RGB images of multiple views using off-the-shelf detectors. These heatmaps are back-projected into the same volumetric space, and V2V-Net extracts RGB-related features ( $f_c$ ).
Fusion: The features from both modalities are concatenated in the 3D feature space and processed by a Fusion-Net to generate a final 3D human pose heatmap. Joint coordinates are derived using Soft-argmax to minimize quantization errors.

2. Synthetic Data Generation (SyncHuman)

To address the scarcity of annotated multi-view LiDAR-RGB datasets, the authors developed SyncHuman, a synthetic dataset generator based on the Unity Engine.

Sensor Simulation: It simulates RGB cameras and LiDAR scanners (specifically mimicking the Livox Mid-40 and Mid-100 scanning patterns) to generate colored point clouds and depth maps.
Avatar Animation: Avatars are driven by action files or keypoint annotations from public motion capture datasets (e.g., AMASS), ensuring diverse and complex human actions.
Ground Truth: The system provides accurate 3D pose ground truth, 2D pose labels, and semantic segmentation for point clouds.

3. Unsupervised Domain Adaptation

To transfer the model from synthetic data to real-world scenarios without manual 3D annotations, the authors propose an unsupervised domain adaptation strategy utilizing a composite loss function:

Pseudo 2D Supervision: Off-the-shelf 2D pose estimators generate pseudo 2D labels, and an $L_{2D}$ loss aligns the projected 3D poses with these labels.
Entropy-Guided Pseudo 3D Supervision: The system calculates the information entropy of the predicted 3D joint heatmaps. High entropy indicates uncertainty. The authors filter predictions, selecting only those with entropy below a threshold ( $\lambda$ ) to serve as reliable pseudo 3D labels ( $L_{3D}$ ) for the next training epoch.
Human Prior Loss: To ensure anatomical plausibility, a prior loss ( $L_{prior}$ ) is introduced, penalizing implausible bone lengths, asymmetric limb lengths, and unreasonable joint angles (specifically nose-neck-midhip and hip-knee-ankle angles).

Key Contributions

LiCamPose Pipeline: A novel framework that fuses multi-view sparse point clouds and RGB information within a volumetric space to estimate 3D poses of multiple individuals from a single timestamp.
SyncHuman Generator: A synthetic data generation tool capable of producing diverse multi-modal data with accurate 3D pose ground truth under various sensor configurations, alongside the creation of the BasketBall dataset (a real-world, four-view LiDAR-Camera dataset of a basketball game).
Unsupervised Training Strategy: A method that avoids manual 3D annotations by pretraining on synthetic data and adapting to real-world data using entropy-guided pseudo-labels and human prior constraints.

Experimental Results

The method was evaluated on four datasets: CMU Panoptic Studio, MVOR, the self-collected BasketBall dataset, and the synthetic BasketBallSync.

Performance: LiCamPose outperformed existing supervised methods (such as VoxelPose and PlaneSweepPose) in terms of Mean Per-Joint Position Error (MPJPE) and Procrustes Aligned MPJPE (PA-MPJPE), particularly in large-scene settings like BasketBallSync.
Unsupervised Capability: The unsupervised version of LiCamPose demonstrated strong generalization, even outperforming some supervised multi-view RGB-only methods on the Panoptic dataset.
Modality Analysis: Experiments confirmed that while sparse point clouds alone are insufficient for accurate skeleton extraction, fusing them with RGB data significantly enhances robustness against occlusions and 2D pose estimation errors.
Ablation Studies: The study validated that aligning synthetic sensor settings with the target dataset improves transfer performance. Furthermore, the ablation of loss functions confirmed that both entropy-guided 3D supervision and human prior losses are critical for performance.
Entropy Analysis: The authors observed a correlation between entropy values and pose rationality, demonstrating that entropy can effectively serve as a confidence metric for filtering unreliable predictions.

Significance and Claims

The paper claims that LiCamPose exhibits "great generalization performance and significant application potential" for robust 3D human pose estimation in complex, real-world scenarios. By integrating LiDAR and RGB data, the method leverages the complementary nature of precise long-range depth measurements and high-resolution texture, mitigating the impact of occlusions that plague single-view or RGB-only methods.

The authors emphasize that their approach reduces the dependency on costly manual 3D annotations through the use of synthetic pretraining and unsupervised domain adaptation. They highlight the practical utility of their system in scenarios like sports analytics (demonstrated via the BasketBall dataset) and surveillance, where minimal sensor setups are advantageous. The paper concludes that the volumetric architecture effectively preserves the geometric characteristics of both modalities, facilitating accurate single-timestamp 3D pose estimation.