Human Video Generation from a Single Image with 3D Pose and View Control

This paper presents HVG, a latent video diffusion model that generates high-quality, multi-view, and spatiotemporally coherent human videos from a single image by leveraging articulated pose modulation, view-temporal alignment, and progressive spatio-temporal sampling to achieve precise 3D pose and view control.

The Gist

Imagine you have a single, perfect photograph of a friend standing still. Now, imagine you want to turn that photo into a full movie where your friend dances, spins, and walks around, all while the camera flies around them to show every angle.

That is the magic trick this paper, HVG (Human Video Generation in 4D), is trying to perform.

Here is the story of how they did it, explained without the heavy technical jargon.

The Problem: The "Flat" vs. The "Rigid"

Before HVG, other AI methods tried to do this, but they had two main flaws:

1. The "Stick Figure" Problem: Some methods used 2D skeletons (like a stick figure drawing) to tell the AI how to move. This works okay if the person just turns their head. But if they spin around, the AI gets confused. It might twist an arm backward like a pretzel or make a leg disappear because it doesn't understand that arms have volume and can't pass through bodies. It's like trying to direct a play using only a shadow puppet; the AI doesn't know where the "real" body is in 3D space.
2. The "Mannequin" Problem: Other methods used 3D digital mannequins (called SMPL) that have a fixed shape. The problem is, real people wear clothes! If your friend is wearing a big, fluffy coat, a rigid mannequin can't show the coat flapping in the wind. It treats the clothes like part of the skin, leading to weird "shape leaking" where the coat looks like it's melting into the legs.

The Solution: HVG's Three Secret Weapons

The authors built a new system called HVG that solves these problems using three clever tricks.

1. The "3D Bone Map" (The Invisible Skeleton)

Instead of using a flat stick figure or a rigid mannequin, HVG creates a 3D "Bone Map."

• The Analogy: Imagine your friend's skeleton isn't just thin lines, but is made of soft, 3D sausages (ellipsoids) connecting the joints.
• Why it works: These "sausages" have thickness. When the AI sees the arm cross in front of the body, the "sausage" knows it's blocking the view. It knows exactly how much space the arm takes up. This prevents the AI from making impossible moves (like a hip dislocating) and keeps the clothes looking real, because the AI knows the clothes are draped over these 3D shapes, not stuck to a flat surface.

2. The "Centering Trick" (Keeping the Camera Calm)

When you watch a video of someone spinning, if the camera stays fixed, the person moves from the left side of the screen to the right. This confuses the AI because it has to constantly relearn where the person is.

• The Analogy: Imagine a stagehand who constantly moves the actor so they are always standing in the exact center of the stage, no matter which way they turn.
• Why it works: HVG uses a "View Alignment" strategy. It mathematically shifts the person so they stay centered in the "AI's mind" for every camera angle. This makes it much easier for the AI to learn that "this is the same person" from every angle, resulting in a video that doesn't flicker or glitch when the camera moves.

3. The "Puzzle Piece" Strategy (Building the Movie)

Making a long video with many camera angles is like trying to solve a giant puzzle all at once. It's too heavy for the computer's brain, and the edges often don't match up.

• The Analogy: Instead of trying to paint the whole mural in one go, HVG paints it in small, overlapping tiles. It paints a few seconds of time, then a few camera angles, then overlaps them slightly to blend the edges perfectly.
• Why it works: This "Progressive Spatio-Temporal Sampling" allows the AI to generate long, smooth videos without running out of memory or creating choppy transitions. It ensures the video flows like butter, even when the camera is spinning wildly.

The Result: A Digital Twin That Breathes

When you put these three tricks together, HVG can take a single photo and generate a 4D video (3D space + time).

• Clothes look real: You can see wrinkles in a shirt as the person twists.
• No weird glitches: Limbs don't twist backward, and clothes don't melt into skin.
• Smooth camera moves: You can watch the person dance from the front, the side, and the back, and it looks like a real camera crew filmed it.

The One Flaw

The paper admits one small weakness: The Face.
Because the AI is so focused on getting the big body movements and the clothes right, the face sometimes gets a little blurry or distorted (like a nose looking slightly off). It's a trade-off: the AI is great at the "big picture" but sometimes misses the tiny details of the face. The authors suggest that in the future, they might use a special "face-only" AI to fix this, like adding a high-definition filter just for the head.

In a Nutshell

HVG is like a super-smart digital puppeteer. Instead of using flat strings (2D skeletons) or stiff mannequins, it uses soft, 3D "sausage bones" to guide the movement, keeps the actor centered so the camera doesn't get dizzy, and builds the movie piece-by-piece to ensure it looks smooth and realistic. It's a huge step toward creating virtual humans that look and move just like the real thing.

Technical Summary

1. Problem Statement

The paper addresses the significant challenges in 4D human video generation (generating multi-view, multi-frame videos from a single static image). While recent diffusion models have advanced image-to-video synthesis, they struggle with:

• View-Consistency: Generating coherent videos when the camera angle changes (novel views).
• Motion Realism: Inferring anatomically correct movements and clothing dynamics (e.g., wrinkles) from a single image.
• Limitations of Existing Driving Signals:
  • 2D Skeletons: Lack anatomical depth and joint dependencies, leading to implausible motions (e.g., dislocated limbs) and self-occlusion errors when viewed from new angles.
  • SMPL Meshes: Oversimplify character geometry, failing to represent loose clothing or accessories, which causes "shape leakage" (distorted garment edges) and inconsistent proportions in multi-view synthesis.
  • Computational Cost: Existing 4D methods (like Human4DiT) often rely on computationally expensive 3D view attention mechanisms that limit scalability.

2. Methodology: HVG Framework

The authors propose HVG (Human Video Generation in 4D), a latent video diffusion model built upon the Stable Video Diffusion (SVD) architecture. It introduces three core innovations to solve the aforementioned problems:

A. Articulated Pose Modulation (Dual-Dimensional Bone Maps)

Instead of using 2D skeletons or raw SMPL meshes, HVG uses a novel Dual-Dimensional Bone Map as the driving signal.

• Construction: It extracts 3D skeletal joints from SMPL-X and models the volume of each bone segment using 3D ellipsoids. These ellipsoids are projected onto 2D planes to create two complementary maps:
  1. Depth Map: Encodes $z$-ordering (distance to camera) to resolve occlusions and maintain depth relationships.
  2. Normal Map: Encodes surface orientation to preserve body shape and clothing details.
• Advantage: This approach captures volumetric cues (preventing limb intersections) and decouples shape from pose (preserving clothing/accessories), avoiding the rigidity of SMPL and the fragility of 2D skeletons.
• Modulation: A Pose Modulator processes these maps via separate Depth and Normal Guider networks, fusing them into the DenoisingNet via cross-attention.

B. View and Temporal Alignment

To ensure consistency across different camera angles and time steps without excessive computational cost:

• Human-Centric View Alignment: Instead of using expensive 3D view attention, HVG aligns the human subject to a consistent position (centered) across all views by shifting the pelvis joint. This allows the model to use efficient 2D attention to learn cross-view correlations, significantly reducing computational overhead while maintaining spatial consistency.
• Temporal Alignment: The reference image is aligned with the first frame of the pose sequence to minimize flickering and pose discontinuities, ensuring frame-to-frame stability.

C. Progressive Spatio-Temporal Sampling

To generate long, multi-view videos efficiently, HVG employs a Progressive Spatio-Temporal Sampling strategy:

• The video generation is decomposed into overlapping segments along both the temporal dimension (frames) and the view dimension (camera angles).
• The model denoises these segments independently and then fuses them using a learned weighting strategy.
• This approach balances computational efficiency with the need for long-range temporal and multi-view coherence, preventing artifacts like texture shifting or logo disappearance.

3. Key Contributions

1. Novel Driving Signal: Introduction of Dual-Dimensional Bone Maps (Depth + Normal) derived from 3D ellipsoids, which effectively resolves self-occlusions and shape leakage issues common in previous methods.
2. Efficient Architecture: A lightweight View Alignment strategy that replaces heavy 3D attention with 2D attention on aligned subjects, enabling scalable multi-view generation.
3. Sampling Strategy: A Progressive Spatio-Temporal Sampling technique that enables the generation of long, consistent multi-view videos by overlapping and fusing temporal and view segments.
4. Comprehensive Performance: The model successfully bridges pose-guided animation, multi-view synthesis, and long-sequence generation.

4. Experimental Results

The authors evaluated HVG on novel view and novel pose synthesis tasks using datasets like THuman2.0/2.1, CustomHuman, and MVHumanNet.

• Quantitative Metrics: HVG outperforms state-of-the-art methods (including AnimateAnyone, MagicAnimate, Champ, MimicMotion, and 4D reconstruction models like LHM) across all metrics:
  • FID: 59.35 (vs. 81.60 for LHM) – indicating higher image quality.
  • SSIM: 0.923 – indicating superior structural similarity.
  • PSNR: 22.13 – indicating better pixel-level accuracy.
  • FVD (Video Fidelity): 152.1 (vs. 248.4 for LHM) – indicating better temporal coherence.
• Qualitative Results:
  • Occlusion Handling: HVG correctly reconstructs occluded body parts (e.g., arms crossing the torso) where 2D-skeleton methods fail.
  • Clothing Detail: It preserves fine-grained clothing textures and dynamics (e.g., coat stretching) without the "smoothing" artifacts seen in other methods.
  • Multi-View Consistency: It maintains consistent geometry and identity across 360-degree rotations, avoiding the "shape leakage" (distorted limbs/clothing) seen in SMPL-based methods.

5. Significance and Future Work

• Significance: HVG represents a major step forward in 4D content creation, enabling the generation of high-fidelity, controllable human videos from a single image. It solves the critical trade-off between anatomical accuracy, clothing fidelity, and computational efficiency in multi-view synthesis.
• Limitations: The paper notes that while global structure is preserved, facial details (nose, lips) can sometimes suffer from distortion due to the trade-off between global motion coherence and local high-frequency details.
• Future Direction: The authors suggest a modular approach where the head region is cropped and processed by a specialized facial generation network to achieve higher fidelity in facial features.

In conclusion, HVG provides a robust framework for generating realistic, controllable 4D human videos, overcoming the geometric and computational bottlenecks of previous diffusion-based approaches.