💻 computer science

AHOY! Animatable Humans under Occlusion from YouTube Videos with Gaussian Splatting and Video Diffusion Priors

AHOY is a novel method that reconstructs complete, animatable 3D Gaussian avatars from heavily occluded, in-the-wild monocular videos by leveraging identity-finetuned diffusion models to hallucinate unobserved body regions and employing a specialized architecture to resolve multi-view inconsistencies.

Original authors: Aymen Mir, Riza Alp Guler, Xiangjun Tang, Peter Wonka, Gerard Pons-Moll

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aymen Mir, Riza Alp Guler, Xiangjun Tang, Peter Wonka, Gerard Pons-Moll

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a digital twin of a person for a video game or a movie. Usually, you need a special studio with dozens of cameras, bright lights, and the person standing perfectly still so nothing blocks the view.

AHOY is a new method that says, "No problem! We can do this with just a regular YouTube video, even if the person is hiding behind a table, a tree, or another person."

Here is how it works, broken down into simple steps with some creative analogies:

The Big Problem: The "Blind Spot"

Most 3D reconstruction tools are like a photographer who only takes pictures when the subject is standing in an empty room. If the person is sitting on a couch, the photographer can't see their legs. If they are holding a coffee cup, the photographer can't see their hand.

In the real world (like YouTube videos), people are constantly blocked by things. Traditional tools get confused and leave big holes in the 3D model, or they just give up.

The Solution: AHOY's "Magic Imagination" Pipeline

AHOY solves this by using a mix of 3D scanning and AI imagination (specifically, a Video Diffusion model). Think of it as a four-step cooking recipe:

Step 1: The Rough Draft (The "Coarse Avatar")

First, AHOY looks at the video and builds a very rough, blocky 3D model of the person using only the parts it can see.

Analogy: Imagine a sculptor trying to make a statue of a person sitting on a chair. They can only carve the visible parts (the head, the shoulders). The legs and the back are just empty space because the chair is in the way. This is the "Coarse Avatar."

Step 2: The "Hallucination" (The AI Artist)

This is the secret sauce. AHOY takes that rough, blocky statue and asks a super-smart AI video generator (trained on the specific person's face and clothes) to "imagine" what the missing parts look like.

Analogy: The sculptor hands the rough statue to a magical painter. The painter looks at the empty space where the legs should be and says, "Based on the style of the shirt and the person's identity, I'm going to paint the legs, the back, and the hidden hands."
The Trick: The AI doesn't just guess randomly; it uses "Diffusion Inversion." It takes the rough 3D model, turns it into a video, and asks the AI to "re-imagine" that video but fill in the missing details with high-quality, realistic textures. It's like asking an artist to redraw a sketch, but this time, they fill in all the blank spots with perfect detail.

Step 3: The "Two-Pose" Dance (Fixing the Glitches)

Here is the tricky part. The AI's "imagination" isn't perfect. If the AI generates a video of the person turning around, the left side might look slightly different from the right side because the AI generated them separately. It's like a team of painters working on different parts of a mural without talking to each other—the seams might not match.

AHOY solves this with a clever "Two-Pose" system:

The Map Pose: This is the "blueprint." It stays the same for the whole movement. It tells the system, "This is what the person looks like when their arms are up."
The LBS Pose: This is the "movement." It changes every single frame to match the video.

Analogy: Think of a puppet show. The Map Pose is the puppet's body shape (which stays consistent). The LBS Pose is the puppeteer's hands moving the strings. If the AI's video looks a little wobbly or inconsistent, the "puppeteer" (the LBS pose) adjusts the strings frame-by-frame to smooth out the glitches, while the "body shape" (the Map) stays true to the person's identity.

Step 4: The Face Guard (Keeping Identity)

AI video generators are great at bodies but terrible at faces—they often change the person's identity (making them look like a different person).

Analogy: AHOY puts a "Face Guard" on the process. It separates the head from the body. It uses a special, stricter AI just for the face to make sure the digital twin looks exactly like the real person, while letting the "imagination" AI handle the body.

The Result

At the end of the process, you have a complete, 3D, animatable human.

You can take this digital person and make them dance, sit, or jump.
You can put them into a 3D scene (like a living room) that was also filmed with a phone.
Even though the original video had the person hiding behind a sofa, the final 3D model has the sofa's shape filled in perfectly, and the person can be moved anywhere.

Why This Matters

Before AHOY, if you wanted a 3D model of a person, you needed a Hollywood studio. Now, you can take a random video from YouTube where someone is cooking, dancing, or playing with their dog (and getting blocked by furniture), and turn it into a high-quality, moving 3D character. It unlocks the entire internet as a source for 3D characters!

1. Problem Statement

Current state-of-the-art methods for reconstructing photorealistic, animatable 3D human avatars (using NeRFs or 3D Gaussian Splatting) rely heavily on unoccluded, multi-view, or canonical-pose inputs. They fail in "in-the-wild" scenarios, such as standard YouTube videos, where:

Heavy Occlusion: Subjects are frequently blocked by furniture, objects, or other people.
Partial Observability: Large body regions (e.g., the back of the body, inner arms) may never be observed in the video.
Lack of Multi-view Supervision: Single-camera inputs provide no geometric consistency for unseen regions.
Pose Dependency: Existing methods struggle to learn how clothing and appearance change with body configuration when multi-view data for specific poses is missing.

The goal of AHOY is to reconstruct a complete, animatable 3D Gaussian avatar from a single monocular YouTube video despite these severe occlusions and lack of multi-view data.

2. Methodology

AHOY employs a four-stage pipeline that bridges the gap between sparse, occluded observations and dense, realistic supervision using identity-finetuned video diffusion models.

Stage 1: Coarse Avatar Construction (Canonical Representation)

Input Processing: The system estimates SMPL parameters and visibility masks from the input video.
Texture Mapping: Using DensePose, visible pixels are mapped to a canonical UV atlas.
Inpainting: A pretrained FLUX inpainting model fills in missing regions of the canonical texture atlas (regions never observed).
Multi-view Generation: A multi-view diffusion model generates 4 canonical-pose images (front, back, left, right) from the inpainted atlas.
Coarse 3DGS: A StyleUNet predicts a canonical Gaussian map (time-invariant) based on these 4 views. This coarse avatar is trained using the generated views and the original occluded video frames, resulting in a rough 3D model with artifacts in unobserved areas.

Stage 2: Hallucinated Supervision Generation

To address the lack of data for unobserved regions and novel poses, AHOY generates "hallucinated" supervision videos:

Identity Finetuning: A video diffusion model (Wan 2.2) is finetuned using LoRA on the input video to capture the specific subject's identity and clothing, treating frames as an unordered set to focus on appearance rather than motion dynamics.
Structured Motion Sequences: The system curates motion sequences (e.g., full 360° turns paired with specific actions like "arms raised," or static poses like "sitting" from multiple angles).
Coarse Rendering & RF-Inversion: The coarse avatar is animated with these motions and rendered. These renderings contain artifacts. The system then uses Rectified Flow (RF) Inversion to encode these renderings into the diffusion latent space and decode them back using the identity-finetuned model.
Result: This process "hallucinates" plausible, identity-consistent textures for previously unobserved body parts, generating dense, multi-view supervision videos.

Stage 3: Full Avatar Learning (Pose-Dependent & Inconsistency Absorption)

The hallucinated videos are used to train the final high-fidelity avatar, addressing two key challenges:

Canonical-to-Pose-Dependent Transition: Unlike the coarse stage, the final model learns pose-dependent Gaussian maps. The hallucinated sequences provide the necessary multi-view coverage for specific poses (e.g., a 360° turn provides 360° views of the "arms raised" pose).
Map-Pose / LBS-Pose Decoupling: Since diffusion-generated videos are not geometrically consistent across frames, AHOY decouples the pose into two components:
1. Map Pose ( $\theta_{map}$ ): Shared across frames of a sequence. It defines the input to the StyleUNet, ensuring consistent Gaussian attributes for a specific body configuration.
2. LBS Pose ( $\theta_{lbs}$ ): Optimized per-frame. It deforms the Gaussians to absorb geometric inconsistencies and multi-view noise inherent in the diffusion output.
Head/Body Split: To preserve facial identity (which diffusion models often distort), the head is separated from the body using SMPL-to-FLAME correspondence. The body is supervised by the hallucinated videos, while the head is supervised by a dedicated multi-view face diffusion model (CAP4D) that ensures identity consistency.

Stage 4: Animation and Scene Composition

The resulting 3DGS avatar can be driven by arbitrary SMPL poses. At inference, poses are projected via PCA to stay within the training distribution. The avatar can be composited into 3DGS scenes reconstructed from casual cell-phone videos for photorealistic rendering.

3. Key Contributions

Occlusion-Robust Reconstruction: The first method to reconstruct complete, animatable 3D avatars from heavily occluded, single-view YouTube videos.
Hallucination-as-Supervision: A novel pipeline using identity-finetuned video diffusion and RF-Inversion to generate dense, multi-view supervision for unobserved body regions.
Two-Stage Architecture: A progression from a canonical representation to a pose-dependent representation, enabled by structured motion sequences in the hallucination stage.
Inconsistency Absorption: A Map-Pose/LBS-Pose decoupling mechanism that allows the model to learn high-fidelity pose-dependent deformations from geometrically inconsistent diffusion data.
Identity Preservation: A head/body split strategy that prevents facial identity degradation during the hallucination process.

4. Results

Datasets: Evaluated on YouTube videos (occluded, real-world) and the BEHAVE dataset (multi-view captures with significant occlusion).
Metrics: Outperforms state-of-the-art baselines (LHM, IDOL, PSHuman, SyncHuman) in PSNR, SSIM, and LPIPS for both static reconstruction and novel pose animation.
- Example: On BEHAVE novel-view tasks, AHOY achieved 24.12 PSNR vs. ~19.34 for the next best method.
Qualitative: Demonstrates the ability to reconstruct complete bodies where baselines fail (producing holes or artifacts) and successfully animates avatars into novel poses and 3D scenes.
Ablation Studies: Confirmed that removing any component (hallucination, RF-inversion, pose decoupling, or head/body split) significantly degrades reconstruction quality, particularly in unobserved regions and facial identity.

5. Significance

AHOY represents a paradigm shift in 3D human reconstruction by treating occlusion not as a failure condition but as a solvable problem through generative priors.

Scalability: It unlocks the vast, diverse library of YouTube videos as a source for 3D human assets, removing the need for controlled capture rigs or subject cooperation.
Generative 3D: It effectively bridges 2D video generation (diffusion) and 3D reconstruction (Gaussian Splatting), showing how generative models can provide the "missing data" required for 3D learning.
Application: Enables the creation of photorealistic, animatable digital humans for gaming, VR, and film directly from casual footage, democratizing access to high-quality 3D assets.

Limitations: The method is computationally intensive (multi-stage optimization and diffusion inference) and inherits the "hallucination" risk of diffusion models (potentially generating plausible but incorrect details for regions never observed). Future work aims to integrate stronger geometric constraints.