💻 computer science

CrowdGaussian: Reconstructing High-Fidelity 3D Gaussians for Human Crowd from a Single Image

CrowdGaussian is a unified framework that reconstructs high-fidelity, photorealistic 3D human crowds from single images by employing a self-supervised adaptation pipeline to handle extensive occlusions and a Self-Calibrated Learning strategy to refine multi-person 3D Gaussian Splatting representations.

Original authors: Yizheng Song, Yiyu Zhuang, Qipeng Xu, Haixiang Wang, Jiahe Zhu, Jing Tian, Siyu Zhu, Hao Zhu

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yizheng Song, Yiyu Zhuang, Qipeng Xu, Haixiang Wang, Jiahe Zhu, Jing Tian, Siyu Zhu, Hao Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you take a single photo of a busy street corner filled with people. Some are partially hidden behind others, some are far away and blurry, and the lighting might be tricky. Now, imagine you want to turn that flat, 2D photo into a fully explorable 3D world where you can walk around the crowd and see everyone from any angle, even the parts that were hidden in the original photo.

That is exactly what CrowdGaussian does, but it's a massive technical challenge. Here is how it works, explained simply.

The Problem: The "Puzzle with Missing Pieces"

Most 3D reconstruction tools are like puzzle solvers that only work well if you give them a clear, close-up picture of a single person. But real life is messy. In a crowd:

Occlusion: People block each other. You can't see the back of the person in front, or the legs of the person behind.
Blur: People far away look like tiny, blurry blobs.
Complexity: There are too many people to process one by one without it taking forever.

If you try to use old methods on a crowd photo, the result is usually a ghostly mess with holes, transparency, or weirdly distorted faces.

The Solution: A Two-Stage "Magic Trick"

The researchers built a system called CrowdGaussian that acts like a two-step magic trick to fix these problems.

Stage 1: The "Imaginative Architect" (LORM)

First, the system looks at the messy photo and identifies everyone. It knows where their bodies should be based on standard human shapes (like a skeleton), even if parts are missing.

The Analogy: Imagine a child drawing a picture of a person but accidentally smudging the arm off the page. A normal artist might leave a blank spot. But our "Architect" (called LORM) is like a master painter who knows exactly what an arm looks like. It doesn't just guess; it uses a massive library of human knowledge to "hallucinate" (reconstruct) the missing arm perfectly, even though it was never in the photo.
How it learns: Instead of needing a teacher to show it the missing parts (which is impossible in real life), it teaches itself. It takes a perfect photo, hides parts of it, and tries to rebuild the missing pieces to match the original. It gets really good at filling in the blanks without needing 3D blueprints.

Stage 2: The "Detail Polisher" (CrowdRefiner)

Now, the system has a 3D crowd, but it might look a bit like a smooth, plastic mannequin. The clothes look flat, and the hair is blurry because the original photo was low-resolution.

The Analogy: Think of this stage like a high-end photo editor or a sculptor adding the final touches. It takes the "plastic" 3D model and uses a powerful AI (a diffusion model) to add realistic wrinkles, fabric textures, and sharp hair strands.
The "Self-Calibrated" Secret: Usually, when AI tries to add details, it gets too excited and starts inventing weird things (like giving a person three eyes or a weird shirt pattern). To stop this, the researchers used a strategy called Self-Calibrated Learning.
- Imagine a teacher telling a student: "If you get the face right, don't touch it. Only fix the blurry parts."
- The AI learns to be smart: it sharpens the blurry areas but leaves the good parts alone. This prevents it from "over-correcting" and ruining the image.

The Result: A 3D Crowd You Can Walk Through

Once the details are polished, the system turns everything back into a 3D format called 3D Gaussians.

What are 3D Gaussians? Imagine the 3D world isn't made of solid triangles (like a video game character), but of millions of tiny, glowing, fuzzy clouds (like glitter). These clouds are arranged so precisely that when you look at them, they look like a solid, high-definition photo.
The Payoff: Because of this method, you can take that single, messy crowd photo and turn it into a 3D scene where you can:
- Walk around the people.
- See the back of the person who was hidden.
- Zoom in and see sharp details on clothes, even if the original photo was blurry.

Why This Matters

Before this, trying to 3D-model a crowd from one photo was like trying to build a house out of wet sand—it would just collapse. CrowdGaussian is the tool that turns that wet sand into a sturdy, detailed castle, making it possible to create immersive virtual worlds, better video games, and realistic digital avatars for the metaverse, all from a single snapshot.

1. Problem Statement

The paper addresses the challenge of single-view 3D reconstruction of human crowds. While recent advancements have enabled high-quality 3D reconstruction of single individuals from monocular images, extending this to multi-person scenes remains an unexplored and difficult territory. The authors identify three primary challenges in this domain:

Extensive Occlusions: In crowded scenes, individuals are frequently occluded by others or objects, leading to missing body parts in the input image. Standard 2D inpainting often fails to produce geometrically consistent or semantically plausible completions.
Low Resolution & Degradation: Individuals in crowd scenes often appear small or in low-resolution crops, causing existing large human models to lose high-frequency details (e.g., facial features, clothing textures), resulting in blurred outputs.
Efficiency: Reconstructing dozens of individuals individually using separate models is computationally prohibitive. A unified, efficient framework is required.

2. Methodology: CrowdGaussian Framework

The proposed CrowdGaussian is a unified two-stage framework designed to reconstruct multi-person 3D Gaussian Splatting (3DGS) scenes from a single image.

Stage 1: Coarse Crowd Completion (LORM)

The first stage focuses on recovering complete 3D geometry from heavily occluded inputs.

Pipeline: The system first estimates per-person 3D poses and locations using a multi-person Human Mesh Recovery (HMR) model (PromptHMR) and segments individuals using SAM (Segment Anything Model).
LORM (Large Occluded Human Reconstruction Model):
- Base: Built upon a pre-trained large human reconstruction model (LHM).
- Challenge: Pre-trained models fail on occluded inputs because they lack occlusion-aware training.
- Solution: A Self-Supervised Adaptation Framework is introduced.
  - Teacher-Student Distillation: A "Teacher" model (frozen pre-trained LHM) processes complete images to generate clean 3D Gaussians and renderings. A "Student" model (LORM) processes occluded versions of the same images (synthesized via masking).
  - Training Objective: The student is trained to hallucinate complete geometries by minimizing the rendering difference between its output and the teacher's clean renderings.
  - Parameter Efficiency: Only the transformer backbone (MBHT) is adapted using LoRA (Low-Rank Adaptation), while the visual encoder and Gaussian decoder remain frozen.
- Outcome: LORM produces a coarse, geometrically complete multi-person 3DGS scene, even when input crops are severely occluded.

Stage 2: Diffusion-based Crowd Refinement (CrowdRefiner)

The second stage enhances the coarse 3DGS representation to recover high-frequency details and sharp textures.

CrowdRefiner: A single-step diffusion model designed to refine coarse 2D renderings of the crowd.
Inputs: It takes the coarse RGB rendering and a corresponding SMPL normal map (as a geometric prior) to guide the refinement.
Self-Calibrated Learning (SCL):
- Problem: Standard training on degradation-to-clean pairs often leads to "over-refinement," causing facial distortions and artifacts.
- Solution: SCL mixes two types of training samples:
  1. Degradation-to-Clean: (Coarse input $\to$ Ground Truth).
  2. Identity-Preserving: (Ground Truth $\to$ Ground Truth).
- Mechanism: This mixed supervision forces the model to adaptively refine under-recovered regions while strictly preserving well-structured areas, preventing hallucinations.
Distillation: The high-fidelity 2D outputs from CrowdRefiner serve as pseudo-ground truths. These are distilled back into the 3D Gaussians via differentiable rendering optimization, iteratively updating the 3DGS parameters to match the refined 2D details.

3. Key Contributions

Unified Framework for Crowds: The first method to reconstruct high-fidelity, multi-person 3D Gaussian scenes from a single in-the-wild image, addressing the specific challenges of occlusion and low resolution.
LORM (Self-Supervised Adaptation): A novel strategy to adapt large human reconstruction models to handle occlusions without requiring 3D annotations, using a teacher-student distillation approach.
CrowdRefiner & SCL: A single-step diffusion refiner trained with Self-Calibrated Learning. This strategy balances detail enhancement with structural preservation, effectively solving the over-refinement problem common in diffusion-based 3D refinement.
Robustness: The method demonstrates exceptional robustness to severe occlusions and significant input resolution degradation (downsampling).

4. Experimental Results

Qualitative Performance:
- Outperforms state-of-the-art baselines (LHM, IDOL, PSHuman, SyncHuman) in complex crowd scenarios.
- Successfully recovers complete geometry and sharp textures (faces, hands, clothing) where baselines produce transparency artifacts, blurred details, or geometric inconsistencies.
- Demonstrates superior performance on low-resolution inputs (2x and 4x downsampling).
Quantitative Metrics:
- On the THuman2.1 dataset with simulated occlusions, CrowdGaussian achieves the highest PSNR (18.619), SSIM (0.931), and lowest LPIPS (0.914) compared to all baselines.
- Maintains stable performance even as occlusion ratios increase from 20% to 60%, whereas baselines degrade significantly.
Ablation Studies:
- SCL: Removing SCL leads to facial distortions and a drop in metrics, confirming its role in preventing over-refinement.
- Geometric Conditioning: Removing the SMPL normal map input causes structural collapse (e.g., distorted hands), proving the necessity of geometric priors.

5. Significance

CrowdGaussian represents a significant leap forward in 3D computer vision by bridging the gap between single-person and multi-person reconstruction. Its significance lies in:

Practical Applicability: It enables the creation of immersive 3D scenes from casual, single-camera crowd photos, which are ubiquitous in real-world data but previously difficult to process.
Handling Real-World Noise: By explicitly addressing occlusion and resolution degradation, the method is robust enough for "in-the-wild" scenarios, moving beyond controlled studio environments.
Efficiency: The use of a unified framework and single-step diffusion allows for efficient reconstruction of large crowds, avoiding the computational cost of processing individuals sequentially.
Future Applications: The high-fidelity, animatable 3D crowds generated by this method have immediate applications in virtual reality, teleconferencing, digital content creation, and crowd simulation.