3D-Aware Implicit Motion Control for View-Adaptive Human Video Generation

This paper introduces 3DiMo, a novel framework that achieves flexible, view-adaptive human video generation by training a motion encoder to distill driving frames into view-agnostic tokens, thereby overcoming the limitations of rigid 2D poses and inaccurate explicit 3D models while leveraging the intrinsic 3D priors of large-scale video generators.

The Gist

Imagine you want to make a movie where a specific person (let's call him "Bob") performs a dance, but you don't have a camera crew or a dance studio. Instead, you just have a video of a stranger dancing in a small room, and a single photo of Bob standing still.

Your goal is to make Bob dance exactly like the stranger, but with a twist: you want to be able to move the camera around Bob however you like (zooming in, circling him, looking from above) just by typing a sentence like, "The camera circles Bob from the left."

This is the problem the paper 3DiMo solves. Here is how they did it, explained simply:

The Problem: The "Flat Shadow" Trap

Previous methods tried to solve this by looking at the stranger's video and trying to map their movements onto a flat, 2D shadow.

• The 2D Approach: Imagine looking at a shadow puppet on a wall. If the puppet moves its arm, the shadow moves. But if you try to walk around the wall to see the puppet from the side, the shadow doesn't change because it's stuck to the wall. Old AI models were like this: they could make Bob dance, but if you asked the camera to move, the video would glitch or look flat because the AI didn't understand that Bob is a 3D person, not a 2D drawing.
• The 3D Skeleton Approach: Other methods tried to build a digital skeleton (like a stick figure) of the dancer first. But these skeletons are often clumsy and inaccurate. It's like trying to direct a movie by forcing the actors to wear stiff, ill-fitting mannequin suits. The AI gets confused by the bad data and stops being creative.

The Solution: 3DiMo (The "Intuitive Ghost")

The authors of 3DiMo took a different approach. Instead of forcing the AI to use a rigid skeleton or a flat shadow, they taught it to understand the "spirit" of the movement.

Think of 3DiMo as a master choreographer who doesn't care about the specific camera angle of the original video.

1. The "Ghost" Encoder: They built a special tool (a "Motion Encoder") that watches the driving video and extracts the pure essence of the movement. It ignores the background, the lighting, and the specific camera angle. It's like listening to a song and understanding the melody without caring about the volume or the speaker quality.
2. The "Invisible" Connection: Instead of forcing the AI to draw a skeleton, they feed this "essence" directly into the video generator using a technique called Cross-Attention. Imagine the video generator is a painter, and the motion encoder is a whispering ghost telling the painter, "Move the arm like this, but don't worry about where the camera is."
3. The Camera Control: Because the AI understands the movement as a 3D concept (not a 2D picture), it can naturally handle camera instructions. If you say, "Zoom out," the AI knows to pull the camera back while keeping Bob's dance consistent, because it understands Bob exists in 3D space.

The Secret Sauce: Training with "Many Eyes"

How did they teach the AI to understand 3D space without using those clumsy skeletons?

• The "View-Rich" Gym: They trained the AI on a massive dataset of videos. Some were filmed from one angle, some from many angles at once (like a security camera room), and some with the camera moving around the subject.
• The "Teacher's Cheat Sheet" (Annealing): At the very beginning of training, they gave the AI a "cheat sheet" (a rough 3D skeleton estimate) to help it get started. But as the AI got smarter, they slowly took the cheat sheet away. This forced the AI to stop relying on the cheat sheet and start learning the true 3D nature of movement from the video data itself.

The Result

The result is a system that can take a video of a stranger dancing and make a photo of Bob dance along perfectly.

• No more flat videos: You can spin the camera around Bob, and he will still look like a real 3D person.
• No more glitches: The AI understands that if Bob's hand touches his hip, it stays there even if the camera moves to the side.
• Better than the alternatives: In tests, 3DiMo produced videos that looked more natural and physically realistic than any previous method.

In short: 3DiMo teaches AI to "feel" the movement in 3D space rather than just "seeing" it on a 2D screen, allowing for magical, camera-free control over generated human videos.

Technical Summary

1. Problem Statement

Current methods for human motion control in video generation face a fundamental trade-off between motion fidelity and viewpoint flexibility:

• 2D Pose-Based Methods: Rely on 2D pose images (e.g., OpenPose, DensePose) as control signals. While effective for static camera views, they rigidly bind motion to the driving video's viewpoint. This prevents novel-view synthesis and causes generated videos to collapse into 2D projections, lacking true 3D spatial reasoning.
• Explicit 3D Parametric Methods: Use models like SMPL/SMPL-X to reconstruct 3D geometry from driving videos. While they offer 3D structure, they suffer from inherent inaccuracies (depth ambiguity, incorrect limb contacts, distorted Z-axis motion). When used as strong constraints, these imperfect reconstructions override the intrinsic 3D spatial priors of large-scale video generators, limiting the model's ability to produce physically plausible motion.

The Goal: To develop a motion control framework that can faithfully reproduce the underlying 3D motion from 2D driving videos while supporting flexible, text-guided camera control (e.g., rotating the camera around the subject) without relying on error-prone external 3D reconstructions.

2. Methodology: 3DiMo Framework

The authors propose 3DiMo, an end-to-end framework that learns implicit, view-agnostic motion representations aligned with a pretrained video generator's spatial priors.

A. Core Architecture

• Backbone: A pretrained DiT (Diffusion Transformer) based video generation model (Latent Diffusion Model) with strong 3D spatial and motion awareness.
• Implicit Motion Encoder:
  • A Transformer-based 1D tokenizer that processes driving video frames.
  • View-Agnostic Design: It compresses 2D frames into compact 1D motion tokens, intentionally discarding spatial layout and appearance details to force the extraction of semantic motion dynamics.
  • Dual-Scale Encoding: Uses two separate encoders: one for global body motion ($E_b$) and one for fine-grained hand gestures ($E_h$).
  • Augmentation: Applies random perspective transformations and appearance jittering to decouple motion from specific viewpoints and prevent identity leakage.
• Conditioning Mechanism: Instead of rigid pixel-aligned projection, motion tokens are injected into the DiT generator via cross-attention. This allows for flexible semantic interaction between motion, text, and visual tokens.

B. Training Strategy: View-Rich Supervision

To force the model to learn genuine 3D understanding rather than just 2D patterns, the authors employ a multi-stage training strategy using a large-scale, view-rich dataset:

1. Dataset Composition: Combines internet videos (single-view), synthetic UE5 renders (multi-view/camera motion), and real-world multi-view captures.
2. Training Objectives:
   • Same-View Reconstruction: Learns expressive motion dynamics.
   • Cross-View Reproduction: The model must reproduce the same 3D motion from a different viewpoint or camera trajectory (guided by text prompts). This forces the model to reason about 3D space invariant to the camera.
3. Progressive Stages:
   • Stage 1: Single-view reconstruction to initialize motion learning.
   • Stage 2: Mixed reconstruction and cross-view reproduction.
   • Stage 3: Exclusive focus on multi-view and camera-motion data to refine 3D awareness.

C. Auxiliary Geometric Supervision (Annealing)

To address slow convergence and instability in early training:

• Initialization: A lightweight MLP decoder predicts SMPL/MANO parameters from the motion tokens, supervised by off-the-shelf 3D estimators.
• Annealing: This geometric loss is applied only in early stages and gradually annealed to zero. This allows the model to start with a reliable 3D prior but eventually transition to learning "genuine" 3D understanding directly from the data and the generator's intrinsic priors, avoiding the biases of external estimators.

3. Key Contributions

1. 3D-Aware Motion Control Paradigm: Reframes motion control as a 3D reasoning task, recovering underlying 3D motion from 2D frames without relying on explicit parametric reconstruction (SMPL) as a hard constraint.
2. End-to-End Implicit Framework (3DiMo): Proposes a joint training approach for a motion encoder and a video generator. The encoder learns to distill view-agnostic motion tokens that align with the generator's spatial priors, enabling flexible text-driven camera control.
3. View-Rich Supervision: Introduces a comprehensive dataset and training strategy (single-view, multi-view, moving-camera) that compels the model to learn 3D spatial consistency, moving beyond 2D projection patterns.
4. Annealed Geometric Guidance: A novel training schedule that uses imperfect external 3D estimates only for initialization, allowing the model to eventually rely on its own learned 3D understanding.

4. Results

• Quantitative Performance: 3DiMo outperforms state-of-the-art methods (AnimateAnyone, MimicMotion, Uni3C, MTVCrafter) in visual fidelity metrics (FID, FVD, LPIPS) and user study scores for naturalness and 3D plausibility.
• Qualitative Improvements:
  • Depth Accuracy: Correctly handles limb depth ordering and contact points (e.g., hand-on-hip) from novel angles, where 2D methods fail and SMPL-based methods show depth ambiguity.
  • Camera Control: Successfully generates videos where the camera rotates, pans, or zooms around the subject while maintaining consistent 3D human motion, a capability lacking in baseline methods.
• Ablation Studies: Confirm that removing view-rich supervision, the hand encoder, or the cross-attention mechanism significantly degrades performance. Removing the auxiliary geometric supervision leads to unstable training.

5. Significance

This work represents a significant shift in controllable video generation. By moving away from explicit 3D reconstruction (which is often inaccurate and limits the generator) toward implicit 3D reasoning (leveraging the generator's own priors), 3DiMo achieves:

• Higher Fidelity: More physically plausible and visually consistent human motion.
• True View Adaptability: The ability to generate novel views and dynamic camera movements that were previously impossible with 2D-conditioned methods.
• Scalability: The approach demonstrates that large-scale video models can be guided to understand 3D space effectively if the conditioning mechanism and training data are designed to respect 3D invariance, rather than forcing rigid geometric constraints.

The paper suggests that future motion control should focus on aligning with the generator's intrinsic spatial understanding rather than imposing external, potentially flawed, geometric constraints.