🤖 AI

Beyond Skeletons: Learning Animation Directly from Driving Videos with Same2X Training Strategy

The paper introduces DirectAnimator, a novel framework that bypasses error-prone pose estimators by directly learning from raw driving videos through a Driving Cue Triplet and a Same2X training strategy to achieve state-of-the-art, robust human image animation with improved identity preservation and efficiency.

Original authors: Yuan Zeng, Yujia Shi, Yuhao Yang, Dongxia Liu, Zongqing Lu, Wenming Yang, Qingmin Liao

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuan Zeng, Yujia Shi, Yuhao Yang, Dongxia Liu, Zongqing Lu, Wenming Yang, Qingmin Liao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a digital character (a static photo) how to dance. In the past, the "teacher" (the AI) had to first look at a video of a real dancer, try to draw a stick-figure skeleton over them, and then tell the digital character, "Move your left arm here, bend your knee there."

The problem with this old way is that drawing stick figures is hard. If the dancer's arm crosses their body, the stick figure gets confused. If the dancer waves their hand quickly, the stick figure might lose the hand entirely. When the teacher gives bad instructions, the digital dancer ends up with extra limbs, twisted bodies, or a blank, emotionless face.

DirectAnimator is a new approach that skips the stick figures entirely. Instead of trying to translate the video into a simplified drawing, it lets the AI learn directly from the raw video pixels, just like a human learns by watching a demonstration rather than reading a manual.

Here is how it works, broken down into simple concepts:

1. The "Driving Cue Triplet" (The Three-Legged Stool)

Instead of one messy stick figure, the authors created a three-part "instruction set" called the Driving Cue Triplet. Think of this as giving the AI three specific lenses to look at the video through:

The Pose Cue (The Motion Lens): Imagine taking the video of the dancer and blurring out all the details like their shirt pattern or hair texture. You are left with a "ghostly" shape that shows only how they are moving. This helps the AI focus on the dance moves without getting distracted by the clothes.
The Face Cue (The Expression Lens): The authors realized that stick figures are terrible at showing facial expressions. So, they simply crop out the dancer's face from the video and feed that directly to the AI. This ensures the digital character can smile, frown, or look surprised just like the real person.
The Location Cue (The Map Lens): Sometimes the dancer in the video is standing far away, while the photo you want to animate is a close-up. The "Location Cue" acts like a map, telling the AI exactly where the body and face should be placed so the dance fits the photo perfectly without stretching or squishing it.

2. The "CueFusion DiT" (The Smart Mixer)

Once the AI has these three lenses, it needs to mix them together. The paper introduces a special "mixer" block (called CueFusion DiT).

Think of the Reference Photo (the person you want to animate) as the Base Cake.
Think of the Driving Cues (the dance moves and expressions) as the Frosting and Decorations.
This mixer ensures the frosting (the dance) is applied perfectly to the cake (the person) without changing the flavor of the cake (the person's identity). It makes sure the dancer looks like you, but moves like the video.

3. The "Same2X" Training Strategy (The Practice Drill)

Training an AI to make a stranger dance like someone else is incredibly difficult. It's like asking a student to copy a dance they've never seen, while wearing a mask that hides their own face. The AI gets confused and learns slowly.

The authors solved this with a clever two-step training method called Same2X:

Step 1 (The Easy Drill): First, they teach the AI using videos where the dancer and the photo are the same person. This is easy; the AI learns, "Okay, when the arm goes up, the arm goes up."
Step 2 (The Hard Drill): Next, they try to teach it with different people. But instead of starting from scratch, they use a "ghost" of the first lesson. They tell the AI, "Remember how you learned to move the arm in Step 1? Do that same movement pattern here, even though the person is different."
The Result: This acts like a safety net. It stops the AI from getting confused and allows it to learn the difficult "different person" task 6.7 times faster than before.

Why This Matters (According to the Paper)

The paper claims that by skipping the "stick figure" step and using these three smart lenses plus the two-step training, their system:

Fixes the "Ghost Hand" problem: It doesn't accidentally add extra limbs when a person crosses their arms.
Captures Emotions: The faces look natural and expressive, not like frozen masks.
Saves Time: It trains much faster and uses fewer computer resources than previous methods.
Handles Chaos: It works well even when the video has fast movements, blurry motion, or people blocking each other.

In short, DirectAnimator stops trying to translate a video into a bad drawing and instead teaches the AI to "watch and learn" directly from the video, using a special training drill to make sure it gets it right the first time.

Technical Summary: Beyond Skeletons: Learning Animation Directly from Driving Videos with Same2X Training Strategy

1. Problem Statement

Human Image Animation (HIA) aims to generate photorealistic videos of a static reference identity performing poses and expressions derived from a driving video. Existing state-of-the-art methods typically rely on intermediate representations—such as 2D skeleton maps, DensePose, or SMPL parameters—as driving signals. The authors identify two critical limitations in this paradigm:

Noise and Unreliability: Pose estimators often fail under occlusion, complex articulation, or motion blur, producing erroneous or incomplete skeletons. These errors propagate directly into the generated video, causing anatomical distortions (e.g., front-back confusion, missing limbs).
Limited Expressiveness: Sparse keypoints and skeleton maps struggle to capture nuanced facial expressions and complex body dynamics, often resulting in unnatural or expressionless animations.

Furthermore, training models for cross-identity scenarios (where the reference and driving videos feature different people) is inherently difficult. The model must simultaneously learn to transfer motion/expression while preserving the reference identity's appearance, leading to noisy gradients and slow convergence.

2. Methodology

The authors propose DirectAnimator, a framework that bypasses explicit pose estimation and learns animation directly from raw driving video pixels. The methodology consists of three core components:

A. Driving Cue Triplet

Instead of relying on estimated skeletons, DirectAnimator extracts a structured "Driving Cue Triplet" from the raw driving video to serve as the conditioning signal:

Pose Cue: Obtained by segmenting the foreground (using models like Grounded SAM) and applying frequency-domain low-pass filtering. This suppresses high-frequency appearance details (clothing textures) while preserving temporally coherent pose dynamics.
Face Cue: Involves cropping, resizing, and centering face regions from the driving video to provide rich semantic information for expressive synthesis, addressing the limitations of 68-point landmarks.
Location Cue: Comprises grid-based, softened body and face masks. These provide spatial priors to align the driving video's scale and position with the reference image, preventing identity leakage and structural distortion.

B. CueFusion DiT (CF-DiT) Block

The framework utilizes a Diffusion Transformer (DiT) backbone. To inject the driving cues effectively, the authors introduce the CueFusion DiT Block:

The reference image is encoded into vision embeddings that flow through the main denoising path.
The three driving cues are encoded into separate control embeddings.
These control embeddings modulate the DiT blocks via Adaptive Layer Normalization (AdaLN) and gated residual connections. This design ensures that the identity ("who") is carried by the main path, while motion and expression ("how") are injected through conditional modulation without disrupting the denoising process.

C. Same2X Training Strategy

To address the optimization challenges of cross-identity training, the authors propose a two-stage Same2X strategy:

Same-ID Pretraining: The model is first trained on pairs where the reference and driving identities are the same to establish robust motion and expression transfer capabilities.
Cross-ID Adaptation: The model is then adapted to cross-identity scenarios. Crucially, instead of relying solely on noisy cross-ID data, the authors generate pseudo driving cues from same-ID data to simulate cross-ID conditions.
Alignment Loss: A novel Same2X Alignment Loss ( $L_{S2X}$ ) is introduced. This loss minimizes the cosine distance between the feature embeddings of the cross-ID model and a pre-trained same-ID model at specific DiT blocks. This regularizes the optimization, aligning the cross-ID feature space with the well-learned same-ID space, thereby accelerating convergence and stabilizing training.

3. Key Contributions

Direct Video Conditioning: The paper reformulates HIA to use raw driving video pixels (processed into a structured cue triplet) as the driving signal, eliminating the accumulation of errors from intermediate pose estimators.
Structured Cue Triplet & CF-DiT: The introduction of the Pose, Face, and Location cues, fused via a specialized CueFusion DiT block, provides a more robust and semantically rich representation for controlling motion and expression.
Same2X Training Strategy: A generalizable framework that leverages feature alignment between same-ID and cross-ID settings. This strategy significantly accelerates convergence in cross-ID scenarios and reduces reliance on large-scale real-world cross-ID datasets by utilizing pseudo data.

4. Experimental Results

The authors evaluated DirectAnimator on the TikTok and Unseen datasets, comparing it against SD-based (AnimateAnyone, MagicAnimate), SVD-based (StableAnimator, MimicMotion), and DiT-based (UniAnimate-DiT, DynamiCtrl) baselines.

Quantitative Performance: DirectAnimator achieved state-of-the-art results in FID, SSIM, L1 error, and Face Identity Similarity (FIS) on both datasets. Notably, on the challenging "Unseen" dataset (featuring complex dance motions), it consistently outperformed all baselines across all metrics.
Pose and Expression Transfer: The method demonstrated superior performance in Pose Landmark Consistency (PLC) and Facial Landmark Consistency (FLC), confirming more accurate body pose following and faithful expression transfer.
Qualitative Improvements: Visual comparisons showed that DirectAnimator avoids common artifacts seen in skeleton-based methods, such as extra limbs, front-back ambiguity, and missing keypoints. It maintained high identity preservation and background consistency.
Efficiency: The Same2X strategy was shown to reach the same loss level 6.7× faster than training without it in the cross-ID stage. The method also achieved these results with fewer computational resources (4 H20 GPUs) compared to some baselines requiring 8 A100s.

5. Significance and Claims

The paper claims that DirectAnimator represents a shift from relying on noisy, intermediate pose abstractions to learning directly from holistic visual demonstrations. By bypassing pose estimators, the method achieves higher robustness to occlusions and complex articulations.

The authors emphasize that the Same2X training strategy is a significant contribution to the training of Diffusion Transformers for HIA, as it effectively transfers supervision from easier same-ID settings to challenging cross-ID scenarios. This approach not only improves generation quality but also offers a pathway to reduce reliance on expensive, curated real-world cross-identity datasets by leveraging pseudo data and feature alignment.

The work concludes that while limitations regarding pseudo cue quality remain, DirectAnimator paves the way for more intuitive, human-level animation learning that mirrors how humans perceive motion—through holistic visual cues rather than simplified skeletal representations.