Direct Reward Fine-Tuning on Poses for Single Image to 3D Human in the Wild

This paper introduces DrPose, a direct reward fine-tuning algorithm that leverages a novel dataset of pose-image pairs to optimize multi-view diffusion models for generating 3D humans with more natural and diverse poses from single images, eliminating the need for expensive 3D assets.

The Gist

🎨 The Big Problem: The "Clumsy" 3D Artist

Imagine you have a magical artist who can look at a single photo of a person and instantly build a perfect 3D statue of them. This artist is great at making people stand still or walk normally.

However, if you show this artist a photo of someone doing a backflip, a breakdance move, or a gymnastic split, the artist gets confused. Because they haven't seen enough photos of people doing these crazy moves, they try to guess what the back looks like, but they often get the pose wrong. The resulting 3D statue might have legs twisted in impossible directions or arms floating in the air.

Why? The artist was trained on a library of photos that mostly showed people standing or walking. They lack a "muscle memory" for extreme, dynamic poses.

🚀 The Solution: DrPose (The "Pose Coach")

The authors created a new method called DrPose (Direct Reward Fine-tuning on Poses). Think of DrPose as a specialized coach that trains the magical artist specifically on how to handle difficult, acrobatic poses.

Here is how the coach works, broken down into three simple steps:

1. The Training Camp: DrPose15K

You can't just show the artist a 3D statue of a backflip because those are hard and expensive to make. Instead, the researchers built a new training camp called DrPose15K.

• The Analogy: Imagine you want to teach a chef how to cook a complex dish, but you don't have the ingredients yet. So, you use a robot to simulate the ingredients perfectly based on a recipe.
• What they did: They took a massive database of human motion data (like a library of dance moves) and used a video generator to create fake single photos of people doing those moves.
• The Result: They now have 15,000 pairs of "Crazy Pose + Photo" that the artist can study. This library is much more diverse than any previous library, covering everything from yoga to parkour.

2. The Grading System: PoseScore

How does the coach know if the artist is getting better? They need a grading system.

• The Analogy: Imagine the artist draws a picture of a person jumping. The coach doesn't just look at the picture; they use a special X-ray machine (called PoseScore) to see the "skeleton" inside the drawing.
• How it works: The coach compares the skeleton in the artist's drawing against the perfect skeleton from the original motion data.
  • If the knees are bent the right way? Good grade.
  • If the legs are twisted like a pretzel? Bad grade.
• The Magic: This "X-ray machine" is differentiable, meaning it can give the artist specific feedback on exactly how to fix the drawing to get a better score.

3. The "Don't Forget" Rule: KL Divergence

There is a risk in training. If you push the artist too hard to get a perfect score on the skeleton, they might start drawing weird, ugly monsters just to trick the grading system (this is called "reward hacking").

• The Analogy: It's like a student who memorizes the answers to a test but forgets how to write in cursive. They pass the test but can't write a letter.
• The Fix: The coach adds a rule called KL Divergence. This is a "memory anchor." It reminds the artist: "Hey, while you're fixing the pose, don't forget to keep the face, the clothes, and the overall look looking natural and realistic." It ensures the 3D human still looks like a human, not a glitchy mess.

🏆 The Results: From "Wobbly" to "Awesome"

The researchers tested their new method on three types of challenges:

1. Standard Tests: Normal people standing and walking.
2. Wild Photos: Real internet photos of people in random poses.
3. MixamoRP (The Boss Level): A new test they created with extreme poses like swinging a bat or doing a handstand.

The Outcome:
The DrPose-trained artist was significantly better than all previous methods.

• Geometric Accuracy: The 3D models had fewer twisted limbs and more accurate body shapes.
• Visual Quality: The textures and details looked sharper.
• Dynamic Poses: Most importantly, when the input was a crazy acrobatic pose, the 3D result actually looked like a person doing that move, rather than a broken mannequin.

🧠 Summary in One Sentence

The paper introduces a way to teach AI how to build 3D humans in crazy poses by creating a massive library of "fake" training photos and using a smart "skeleton-checking" coach to fine-tune the AI, ensuring the results are both acrobatically accurate and visually realistic.

Technical Summary

1. Problem Statement

Single-view 3D human reconstruction has advanced significantly through Image-to-Multi-View (I2MV) diffusion models. These models generate multiple views from a single input image, which are then lifted into 3D space. However, a critical bottleneck remains: reconstructed 3D humans often exhibit unnatural or distorted poses, particularly in dynamic, acrobatic, or extreme scenarios.

The authors attribute this failure to the limited scale and diversity of existing 3D human training datasets. Creating large-scale 3D datasets with diverse poses is expensive (requiring multi-camera setups) and privacy-sensitive. Consequently, current models struggle with out-of-distribution poses (e.g., breakdancing, extreme athletic movements) because they lack sufficient training data covering such variations.

2. Methodology: DrPose

The authors propose DrPose (Direct Reward Fine-tuning on Poses), a post-training algorithm designed to align I2MV diffusion models with natural human poses without requiring expensive 3D assets. The methodology consists of three core components:

A. The Training Dataset: DrPose15K

To overcome data scarcity, the authors constructed DrPose15K, a novel dataset containing 15,000 samples of human poses paired with single-view images.

• Source: Derived from the Motion-X dataset (specifically the AIST subset), which offers broad pose coverage.
• Generation: Instead of capturing real 3D scans, they used a pose-conditioned video generative model (MIMO) to animate 3D human models and generate corresponding single-view images for each pose.
• Diversity: DrPose15K exhibits a significantly higher standard deviation in joint locations (1.73× larger than THuman2.1), ensuring coverage of dynamic and challenging poses.

B. The Algorithm: Direct Reward Fine-Tuning

DrPose employs a Direct Reward Fine-tuning strategy (building on DRTune) to optimize the diffusion model directly against a pose-consistency metric, avoiding the instability of Reinforcement Learning (RL).

1. PoseScore (Differentiable Reward):
   • The core innovation is a differentiable reward function, $r(x_0, \theta)$, called PoseScore.
   • It measures the consistency between the generated multi-view latent image ($x_0$) and the ground-truth pose ($\theta$).
   • Mechanism: A pre-trained skeletal predictor ($g_{skel}$) converts the generated latent image into a skeletal image ($\hat{I}_{skel}$). The ground-truth pose $\theta$ is projected into a skeletal image ($I_{skel}$). The reward is the negative distance between these two: $r = -E(||\hat{I}_{skel} - I_{skel}||)$.
2. KL Divergence Regularization:
   • To prevent "reward hacking" (where the model degrades image quality to maximize the reward score), a KL divergence loss ($L_{KL}$) is added.
   • This term penalizes deviations from the initial pre-trained model's noise predictions ($\hat{\epsilon}_0$) at intermediate denoising steps, ensuring the generated images remain realistic while improving pose accuracy.
3. Optimization Objective:
   The model minimizes: $L_{total} = L_{reward} + w_{KL} \cdot L_{KL}$.
   • Training is conducted efficiently by sampling only a subset of denoising steps and using gradient stopping.

C. 3D Reconstruction Pipeline

The post-trained I2MV model is integrated into a standard single-view reconstruction pipeline:

1. Generate multi-view RGB and Normal maps from the input image.
2. Use explicit carving (SMPL-X initialization, differentiable remeshing, and appearance fusion) to reconstruct the final 3D mesh.

3. Key Contributions

1. DrPose Algorithm: A novel post-training method that fine-tunes I2MV diffusion models using direct reward optimization on poses, eliminating the need for 3D ground truth during the fine-tuning phase.
2. DrPose15K Dataset: A large-scale dataset of 15K diverse human poses paired with generated single-view images, constructed to address the lack of dynamic pose data in existing 3D benchmarks.
3. MixamoRP Benchmark: A new evaluation benchmark specifically designed to test reconstruction performance on extreme and dynamic poses (e.g., acrobatics), filling a gap in existing benchmarks like THuman2.1 and CustomHumans.
4. PoseScore: A differentiable reward function that quantifies pose consistency by projecting latent images and ground-truth poses into a shared skeletal representation.

4. Experimental Results

The authors evaluated their method on three benchmarks: THuman2.1-test, CustomHumans-test, and the new MixamoRP.

• Quantitative Improvements:
  • Geometry: On MixamoRP (the most challenging benchmark), DrPose improved the Chamfer Distance (CD) from 150.01 (Era3D*) to 126.06, and the F-Score from 7.35 to 8.31. Similar improvements were seen on standard benchmarks.
  • Appearance: PSNR and SSIM metrics also showed consistent gains, with PSNR increasing from 17.53 to 17.51 (Era3D*) to 17.66 (Ours) on MixamoRP, and LPIPS decreasing (better similarity).
• Qualitative Improvements:
  • Visualizations (Figures 7-9) demonstrate that DrPose significantly reduces unnatural limb twisting and pose distortions in dynamic scenarios (e.g., jumping, swinging bats) compared to baselines like SiTH, H3D, and standard fine-tuned Era3D.
• Ablation:
  • The method works effectively on different base models (Era3D and PSHuman).
  • The skeletal predictor ($g_{skel}$) was validated to be reliable for measuring pose consistency.

5. Significance and Impact

• Bridging the Data Gap: DrPose offers a scalable solution to the data bottleneck in 3D human reconstruction. By leveraging 2D pose data and generative video models, it creates high-quality training signals without the cost of multi-view 3D capture.
• Robustness in the Wild: The method significantly enhances the applicability of 3D reconstruction in real-world scenarios ("in the wild") where subjects often perform dynamic or complex movements that current state-of-the-art models fail to capture accurately.
• Efficiency: By using direct reward fine-tuning rather than RL, the approach achieves faster convergence and better stability.
• Future Direction: The introduction of MixamoRP sets a new standard for evaluating 3D human reconstruction, shifting focus from static or mildly dynamic poses to truly challenging, acrobatic configurations.

Limitations: The pipeline still requires segmented input images (artifacts appear with imperfect segmentation), and the training process is computationally intensive due to the need to generate multiple high-resolution views for reward calculation.