💻 computer science

MotionWeaver: Holistic 4D-Anchored Framework for Multi-Humanoid Image Animation

MotionWeaver is an end-to-end framework that overcomes the limitations of existing single-human image animation methods by introducing unified motion representations and a holistic 4D-anchored paradigm to achieve state-of-the-art results in complex multi-humanoid scenarios involving diverse forms, interactions, and occlusions.

Original authors: Xirui Hu, Yanbo Ding, Jiahao Wang, Tingting Shi, Yali Wang, Guo Zhi Zhi, Weizhan Zhang

Published 2026-02-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xirui Hu, Yanbo Ding, Jiahao Wang, Tingting Shi, Yali Wang, Guo Zhi Zhi, Weizhan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director trying to film a movie with a cast of characters. Some are humans, some are robots, and some are cartoon animals. You have a photo of each character and a video of how you want them to move. Your goal is to make the photos come alive and dance, fight, or hug exactly like the video, even when they bump into each other or hide behind one another.

For a long time, computers were great at animating one person. But as soon as you added a second person, or a robot, or an alien, the computer got confused. It would mix up who was who, or it would make them walk through each other like ghosts.

Enter MotionWeaver. Think of it as a master puppeteer who can handle an entire troupe of different characters at once without getting tangled up. Here is how it works, broken down into simple concepts:

1. The Problem: The "Identity Crisis"

Imagine you have two dancers, Alice and Bob. In old computer programs, if you told the computer "Alice, do Bob's dance," the computer might get confused because it was looking at the shape of the body to decide who was who.

The Old Way: It was like trying to recognize a friend only by their height. If a tall person and a short person swapped places, the computer got lost.
The New Way (MotionWeaver): It separates the dance steps from the dancer. It realizes, "Oh, these are the steps, and that is Alice's face. I will give Alice's face these steps." It doesn't care if Alice is a human, a robot, or a dragon; it just knows how to move the specific character.

2. The Secret Sauce: The "4D Map"

Most video generators work in 2D (flat images) and time. They don't really understand depth.

The Analogy: Imagine a 2D drawing of two people hugging. From the front, you can't tell who is in front and who is in back. A 2D computer might draw them melting into a blob.
MotionWeaver's Trick: It builds a 4D Map (3D space + time). It understands that "Person A is behind Person B."
- It uses a special "Depth Sensor" (called the Hyper-Scene Integrator) that acts like a pair of 3D glasses for the computer.
- When two characters cross paths, the computer knows exactly who should be visible and who should be hidden, just like in real life.

3. The Training: "The Dance School"

To teach this computer to be so good, the authors didn't just use YouTube videos of people dancing. They built a massive, custom dance school called MultiHuman46.

The Dataset: They collected 46 hours of videos showing people (and AI-generated characters) interacting, fighting, and dancing together.
The Benchmark: They created a test called DualDynamics with 300 videos of two characters interacting. It's like a final exam where the computer has to prove it can handle complex moves without the characters glitching or swapping identities.

4. The Learning Process: "The Two-Stage Lesson"

The computer learns in two distinct phases, similar to how a human learns to draw:

The Big Picture (High Noise): At the start of learning, the computer focuses on the big shapes and the order of things. "Who is standing where? Who is blocking whom?" It learns the occlusion (who is in front) first.
The Details (Low Noise): Later, it focuses on the fine details of the movement. "How does the arm bend? How does the foot land?"

Why This Matters

Before MotionWeaver, if you wanted to animate a scene with a robot fighting a knight, you had to do it frame-by-frame by hand, which takes forever.

MotionWeaver allows you to upload a photo of a robot and a photo of a knight, show it a video of a fight, and it instantly generates a video where the robot and knight fight realistically, respecting who is behind whom, without the robot turning into a human or the knight disappearing.

In Summary:
MotionWeaver is like a universal translator for movement. It takes the "language" of motion (how things move in 3D space) and teaches it to any character, whether it's a human, a robot, or a cartoon monster, ensuring they never lose their identity and always know who is standing in front of whom. It turns the chaotic mess of multi-character animation into a smooth, coordinated performance.

1. Problem Statement

Character image animation aims to synthesize videos of a reference character driven by pose sequences. While existing methods have achieved high fidelity in single-human scenarios, they fail significantly in multi-humanoid settings. The paper identifies three primary limitations in current state-of-the-art (SOTA) approaches:

Deficient Motion Representations: Existing methods often entangle motion with morphology (body shape/size) and identity. Control signals like skeleton maps or SMPL renderings carry appearance biases, hindering generalization to diverse humanoid forms (e.g., robots, anthropomorphic animals). Furthermore, compositing multiple skeletons into a single map causes identity confusion when characters swap positions.
Lack of Explicit 4D Modeling: Prior approaches fuse video latents with control signals using naive cross-attention without explicitly modeling spatiotemporal relationships or depth. This leads to failures in handling occlusions and distinguishing between a character being physically smaller versus being further away in the scene.
Suboptimal Training Strategy: Current frameworks rely on 2D pixel-wise MSE loss, which provides no explicit supervision for 4D motion dynamics. This causes models to overfit to human-specific visual features rather than learning true motion dynamics, acting as "mindless renderers" rather than motion-guided generators.

2. Methodology: MotionWeaver

MotionWeaver is an end-to-end framework built on a holistic 4D-anchored paradigm. It consists of three core innovations:

A. Unified-Choreography Core (UCC)

The UCC is designed to extract identity-agnostic motion representations ( $z_{uni}$ ) that generalize across diverse humanoid forms.

Motion Normalization: Instead of using raw SMPL parameters which carry body-proportion biases, the system extracts joint coordinates and maps them to a standardized skeleton with fixed Euclidean distances between joints. This decouples motion from specific body shapes.
Identity Binding: To prevent identity confusion in multi-character scenarios, the UCC uses a pre-trained Image-to-Video (I2V) model to extract identity tokens ( $z_{id}$ ) from the reference frame. A Group Attention Module then binds the identity-agnostic motion tokens ( $z_{mo}$ ) to the corresponding identity tokens. This ensures that motion signals are explicitly associated with the correct character, even during complex interactions or position swaps.

B. Hyper-Scene Integrator (HSI)

The HSI constructs a shared 4D space to fuse motion representations with video latents, addressing the challenge of occlusions and dense interactions.

Depth-Aware Attention (for Z-axis): Since video latents lack explicit depth, the HSI introduces a depth token derived from the SMPL translation parameters ( $z$ -coordinate). It employs a depth-aware attention mechanism supervised by an occlusion loss. This loss enforces the model to learn correct depth ordering, ensuring that characters in the foreground correctly occlude those in the background.
Dynamic C-RoPE (for T, X, Y axes): To model temporal and spatial relationships, the framework uses Dynamic Cross-Attention Rotary Positional Encoding (C-RoPE). Unlike static RoPE, this dynamically selects rotary matrices based on the global 4D position ( $t, x, y, z$ ) of each motion unit. This allows the model to rapidly capture complex spatiotemporal dependencies between characters and the scene.

C. Hierarchical-4D Supervision (H4S)

H4S provides a training strategy that aligns with the denoising characteristics of Diffusion Transformers (DiTs).

High-Noise Steps ( $t \geq \alpha T$ ): The model is supervised using standard MSE loss combined with the Occlusion Loss ( $L_{OCC}$ ). This forces the model to establish correct global layout and depth relationships early in the generation process.
Low-Noise Steps ( $t < \alpha T$ ): The occlusion loss is replaced with a Motion-Level Loss ( $L_{MO}$ ). The denoised latents are decoded and passed through a 4D pose detector to extract motion features, which are then aligned with ground truth. This transfers strong 4D motion priors into the model, reducing overfitting to appearance and improving motion fidelity.

3. Key Contributions

MotionWeaver Framework: A novel end-to-end framework for multi-humanoid image animation that effectively handles diverse forms, rich interactions, and frequent occlusions.
Unified Motion Representations: The introduction of UCC, which separates motion from morphology and explicitly binds motion to identity, enabling robust generalization.
Holistic 4D Paradigm: The development of HSI and H4S, which create a shared 4D space for fusion and provide hierarchical supervision to resolve occlusions and learn motion dynamics.
New Dataset and Benchmark:
- MultiHuman46: A curated dataset of 46 hours of multi-human videos with diverse interaction patterns (boxing, fencing, dancing).
- DualDynamics: A benchmark of 300 videos featuring pairs of diverse humanoid characters (robots, animals, avatars) in interaction-rich scenarios, specifically designed to evaluate multi-humanoid animation.

4. Experimental Results

Quantitative Performance: On the DualDynamics benchmark, MotionWeaver outperforms all SOTA baselines (including RealisDance-DiT, UniAnimate-DiT, HumanVid, and StableAnimator) across all metrics, including FVD, FID, PSNR, SSIM, and CLIP scores.
Qualitative Performance: Visual comparisons demonstrate that MotionWeaver is the only method capable of correctly handling dense inter-character interactions and occlusions without identity confusion. It successfully animates diverse forms (e.g., a robot interacting with a human) where other methods fail or produce artifacts.
Generalization: The model generalizes effectively to scenarios with more than two characters and maintains performance in real-world capture sequences.
Ablation Studies: Removing any core component (UCC, HSI, or H4S) leads to significant performance degradation, confirming the necessity of the unified 4D-anchored design.

5. Significance

MotionWeaver represents a paradigm shift in character animation by moving from 2D, appearance-entangled control signals to a 4D-anchored, identity-agnostic approach.

Scientific Impact: It solves the long-standing challenge of generalizing animation to non-humanoid forms and complex multi-character interactions, which previous 2D-centric methods could not address.
Practical Application: The framework enables high-fidelity animation for diverse use cases, including immersive gaming, virtual reality, digital avatars, and professional animation, where characters often have non-human shapes and interact dynamically.
Community Resource: By releasing the MultiHuman46 dataset and DualDynamics benchmark, the authors provide the community with the first dedicated resources for training and evaluating multi-humanoid animation, filling a critical gap in the field.

In summary, MotionWeaver establishes a robust foundation for the holistic modeling of multi-character dynamics in video synthesis, proving that explicit 4D modeling and unified motion representations are essential for the next generation of generative video AI.