Technical Summary: TGRHuman

Problem Statement

Realistic 3D human generation is critical for immersive VR/AR, gaming, and film, yet current methods struggle to simultaneously achieve high-quality geometry, high-resolution textures, 3D consistency, and inference efficiency. Existing approaches generally fall into two categories, both with significant limitations:

1. Native 3D Generative Models: These train directly on 3D representations (e.g., tri-planes, SDFs). While they possess strong 3D awareness, the high dimensionality of 3D data makes it difficult to leverage pre-trained 2D diffusion models, often necessitating training from scratch with high computational costs. Furthermore, the scarcity of real 3D human datasets hampers generalization.
2. Two-Stage Generation Methods: These rely on pre-trained 2D diffusion models and Score Distillation Sampling (SDS) to optimize 3D representations like NeRF or DMTet. While they leverage 2D priors, SDS-based optimization is extremely time-consuming (often hours per instance) and frequently results in poor view consistency, multi-face artifacts, and over-smoothed results. Additionally, many NeRF-based methods fail to decouple high-quality geometry from texture, and SMPL-based displacement methods often cannot support loose clothing due to fixed topology.

Methodology

TGRHuman addresses these challenges by decoupling geometry and texture generation through explicit 2D observation generation and dimension elevation, avoiding the tight coupling and slow optimization of SDS-based methods. The framework consists of two primary stages:

1. Geometry Generation Stage

• Multi-view Normal Generation: The method employs a high-resolution generative module to produce consistent four-view (front, back, left, right) normal maps at 1024 resolution. This module utilizes a pre-trained Latent Diffusion Model (LDM) conditioned on SMPL latents (for pose and shape guidance), text embeddings (via CLIP), and camera parameters. To ensure view consistency, the model employs cross-view interaction during the diffusion process, integrating global details from all views into the attention mechanism.
• Geometry Carving: Instead of relying on sparse views or super-resolution post-processing, TGRHuman uses a geometry-carving strategy. It deforms an initial SMPL mesh by comparing the generated high-resolution normal maps with renderings from a differentiable rasterizer. The optimization minimizes pixel-aligned errors between the observed normals and the rendered mesh. Crucially, the topology is dynamically adjusted (via remeshing) during optimization, allowing the method to support humans wearing loose clothing without requiring a refinement process.

2. Texture Generation Stage

• Texture Prior Acquisition: Recognizing that sparse views cannot cover self-occluded regions, the method first constructs a robust texture prior based on the SMPL UV space.
  • A shape-guided diffusion model generates a coarse front-view appearance aligned with the geometry.
  • This front view is unwrapped into SMPL UV space.
  • A UV inpainting model (fine-tuned Stable Diffusion) completes the missing regions in the UV map, creating a complete SMPL UV texture map.
• Diffusion Renderer for Free-View Rendering: To generate high-resolution, consistent textures for arbitrary viewpoints, TGRHuman introduces a diffusion renderer.
  • Architecture: The renderer consists of a main RenderNet and a ReferenceNet. RenderNet takes camera parameters, initial RGB, and normal maps as input. ReferenceNet processes the initial front-view observation and normal map to inject identity-consistent features into RenderNet via summed self-attention inputs.
  • Process: The renderer synthesizes dense, 360-degree RGB observations guided by the texture prior.
  • Optimization: A differentiable rasterizer optimizes the final texture map by minimizing the L1, SSIM, and total variation losses between the rendered images and the diffusion-generated observations, ensuring structural similarity and smoothness.

Key Contributions

1. Decoupled Generation Strategy: A novel approach that separates geometry and texture generation via explicit 2D observation generation and dimension elevation. This avoids the slow, implicit supervision of SDS-based methods while preserving diversity.
2. High-Resolution Geometry Module: A high-resolution generative module for multi-view normals combined with a geometry-carving strategy that maintains view consistency and effectively supports loose clothing without post-refinement.
3. Texture Prior and Diffusion Renderer: A strategy for acquiring a robust texture prior via SMPL UV inpainting and a specialized diffusion renderer that ensures view consistency and enables dense, surround-view RGB observation for high-resolution texture synthesis.

Experimental Results

• Quantitative Performance: TGRHuman outperforms existing state-of-the-art methods (including HumanNorm, En3D, TADA, and Chupa) in both geometry and texture quality. It achieves the lowest FID scores for normals (29.48) and RGB (25.36) and the highest CLIP scores for text-to-3D consistency.
• Efficiency: The method demonstrates a significant speed advantage over SDS-based approaches. While competitors often require 1–2+ hours per instance, TGRHuman completes the full pipeline (geometry, prior, and texture) in approximately 5 minutes on an A800 GPU.
• Novel View Synthesis: In novel-view synthesis tasks, the diffusion renderer significantly outperforms baselines like MagicMan, Wonder3D, and SV3D in PSNR (28.3), SSIM (0.951), and LPIPS (0.043).
• Qualitative Quality: Visual comparisons show that TGRHuman generates more realistic and consistent 3D humans, particularly in handling loose clothing and complex textures, without the multi-face artifacts or over-smoothing common in other methods.

Significance and Claims

The paper claims that TGRHuman successfully balances performance and efficiency in text-driven 3D human generation. By decoupling geometry and texture and utilizing explicit multi-view observations rather than implicit SDS optimization, the method achieves high-quality, diverse, and consistent 3D humans efficiently. The authors emphasize that their approach supports loose clothing—a limitation in many SMPL-based displacement methods—and outputs explicit meshes and texture maps, which facilitates downstream graphics applications such as animation and texture editing. The study concludes that while the current focus is on text-to-3D, the framework is inherently flexible and can be extended to image-conditioned settings.