MultiAnimate: Pose-Guided Image Animation Made Extensible

This paper presents MultiAnimate, an extensible framework leveraging Diffusion Transformers and novel Identifier components to achieve state-of-the-art, identity-preserving pose-guided video animation for multiple characters, including scenarios with more participants than seen during training.

The Gist

Imagine you are a director trying to film a movie scene. You have a photo of an actor (let's call him "Bob") and a sequence of stick-figure drawings showing how Bob should move. Your goal is to turn that photo into a video where Bob dances exactly like the stick figures.

This is what Pose-Guided Image Animation does. It's like a magic puppeteer that makes a static photo come to life based on a dance script.

The Problem: The "Identity Crisis" in Group Scenes
For a long time, these magic puppeteers could only handle one actor at a time. If you tried to make two people dance together, the AI got confused. It would mix up their faces, swap their clothes, or make them walk through each other like ghosts.

Why? Imagine two dancers spinning around each other. At a certain point, they look identical from the camera's perspective. The AI asks, "Wait, which one is Bob, and which one is Alice? Did they just swap places, or did they both keep spinning?" Without extra help, the AI guesses wrong, leading to a chaotic mess where identities get lost.

The Solution: MultiAnimate
The researchers behind this paper, MultiAnimate, built a new system that acts like a super-organized stage manager. They solved the confusion with two main tricks:

1. The "Name Tag" System (Identifier Assigner & Adapter)

Think of the AI as a chaotic classroom. In the old methods, the teacher just shouted, "Everyone move!" and the kids (the characters) got mixed up.

MultiAnimate gives every single person in the video a unique, invisible Name Tag (an "Identifier").

• The Assigner: This is the teacher who looks at the video and says, "Okay, the person on the left is wearing the 'Red' tag, and the person on the right is wearing the 'Blue' tag."
• The Adapter: This is the system that ensures the AI only moves the "Red" person when the Red tag is active, and the "Blue" person when the Blue tag is active.

Even if the two dancers spin around and swap positions, the "Red" tag stays with the Red person, and the "Blue" tag stays with the Blue person. The AI never loses track of who is who.

2. The "Universal Translator" (Scalable Training)

Here is the really cool part. Usually, if you want an AI to learn how to handle a group of 5 people, you have to show it thousands of videos of 5 people dancing. That's expensive and hard to find.

MultiAnimate is like a student who learns math by practicing with just two numbers but somehow learns how to handle seven numbers without ever seeing them before.

• The researchers trained the AI only on videos of two people dancing.
• They used a special training trick where they randomly shuffled the "Name Tags" during practice.
• Because the AI learned to associate the movement with the tag (not a fixed position), it realized: "Oh, I can handle a 'Red' tag and a 'Blue' tag. I can also handle a 'Green' tag and a 'Yellow' tag!"

So, when they tested it on a video with three or even seven people, the AI just assigned new Name Tags to the new people and handled them perfectly, even though it had never seen a group that big before.

The Result

The paper shows that this new system:

• Keeps identities perfect: Bob stays Bob, and Alice stays Alice, even when they are hugging, fighting, or dancing in a circle.
• Handles occlusions: If one person walks in front of another, the AI knows who is hiding whom, instead of making them melt into a blob.
• Works for solo acts too: Even though it was trained for groups, it's just as good at animating a single person as the old methods.

In a Nutshell:
MultiAnimate is like giving the AI a set of invisible, unbreakable name tags and a rulebook that says, "No matter how many people are on stage, just follow the tags." This allows it to create complex, multi-person dance videos that look realistic and consistent, all while learning from a much smaller dataset than anyone thought was possible.

Technical Summary

1. Problem Statement

Pose-guided human image animation aims to synthesize realistic videos of a reference character driven by a sequence of poses. While diffusion-based methods have achieved success in single-character animation, extending these methods to multi-character scenarios presents significant challenges:

• Identity Confusion: Naively extending single-character models often leads to characters swapping identities or blending together.
• Implausible Occlusions: Existing models struggle to model complex spatial interactions (e.g., one person blocking another) correctly.
• Scalability & Generalization: Most current models are trained on a fixed number of participants (e.g., exactly two people). They fail to generalize to scenarios with unseen numbers of characters (e.g., three or more) without costly retraining on new datasets.
• Motion Ambiguity: In multi-character settings, identical pose sequences can lead to multiple plausible motion trajectories (e.g., two people rotating and exchanging positions), making the mapping between poses and specific identities underdetermined without explicit spatial cues.

2. Methodology: MultiAnimate Framework

The authors propose MultiAnimate, an extensible framework built upon modern Diffusion Transformers (DiTs) (specifically leveraging the Wan 2.1 architecture). The core innovation lies in a mask-driven scheme that explicitly models per-person positional cues and inter-person spatial relationships.

Key Components

1. Pipeline Architecture:

   • Reference Stream: Encodes the reference image and its initial pose to capture appearance information.
   • Motion Stream: Encodes the driving pose sequence and per-person tracking masks to model motion and spatial conditions.
   • Fusion: The two streams are fused via element-wise addition of latent tokens within the DiT backbone.

2. Identifier Assigner:

   • Takes a set of per-person tracking masks $\{M_i\}$ and unifies them into a single structured label map $L$.
   • It assigns unique, distinct non-zero identifiers (e.g., $a, b$) to each character's mask while labeling the background as 0.
   • This map is converted into a one-hot encoded binary tensor, preserving the spatial occupancy and relationships (proximity, occlusion) of each character.

3. Identifier Adapter:

   • A module consisting of stacked 3D convolutional layers.
   • It takes the one-hot label map from the Identifier Assigner as input.
   • It embeds these identity cues into the feature space of the DiT backbone, allowing the model to explicitly track which pixels belong to which character throughout the video generation process.

Scalable Training Strategy

To address the generalization problem (training on 2 people, testing on 3+), the authors introduce a novel training strategy:

• Identifier Weight Bank: A learnable embedding space containing $n$ identifier channels (where $n$ is the maximum desired number of characters, e.g., 7).
• Random Sampling: During training on a two-character dataset, the model randomly selects two distinct identifiers from the $n$ available channels for the two characters in each batch.
• Symmetry Breaking: By randomizing which channel corresponds to which character, the model learns to associate identities with the spatial mask rather than a fixed channel index.
• Result: By the time training converges, all $n$ channels in the Weight Bank are trained to be mutually distinguishable. Consequently, during inference, the model can handle up to $n$ characters even if it was never trained on that specific number.

3. Key Contributions

1. First Extensible DiT-based Framework: MultiAnimate is the first framework built on modern Diffusion Transformers specifically designed for extensible multi-character image animation.
2. Novel Modules: Introduction of the Identifier Assigner and Identifier Adapter to jointly and explicitly model per-person spatial features and inter-person interactions, solving the motion ambiguity problem.
3. Scalable Training Strategy: A method that allows a model trained on limited data (e.g., two-person videos) to generalize to scenarios with more participants (e.g., three to seven people) without retraining on new data.
4. High-Quality Dataset Curation: Creation of a new high-quality dataset (Gen-dataset) using Wan 2.2 to supplement training data and improve temporal consistency and scene dynamics.

4. Experimental Results

The authors evaluated MultiAnimate on the Swing Dance dataset, a generated Gen-dataset, and unseen TikTok dance videos (involving 3-7 characters).

• Quantitative Performance:

  • MultiAnimate achieved State-of-the-Art (SOTA) performance across all metrics (PSNR, SSIM, L1, LPIPS, FVD, FID-VID) on both two-character and unseen multi-character scenarios.
  • On the Swing Dance test set, it significantly outperformed baselines like UniAnimate-DiT, VACE, and DisPose (e.g., FVD of 648.84 vs. 891.89 for UniAnimate-DiT).
  • On unseen 3-7 person videos, the "Extended model" achieved an FVD of 358.74, far surpassing the next best baseline (624.45).

• Qualitative Performance:

  • Identity Consistency: Unlike baselines that suffer from identity swapping or blurring during complex interactions, MultiAnimate maintains consistent identities throughout the video.
  • Spatial Relationships: The model correctly handles occlusions and spatial arrangements (e.g., one person standing behind another).
  • Backward Compatibility: Despite the added complexity for multi-person tasks, the model remains highly competitive in single-character animation tasks, matching or exceeding SOTA single-person baselines.

5. Significance

• Data Efficiency: The framework demonstrates that high-quality multi-character animation can be achieved by training solely on two-character data, eliminating the need for expensive, large-scale multi-person datasets.
• Real-World Applicability: The ability to generalize to unseen numbers of participants makes the technology viable for complex applications like film production, digital avatars, and social media content creation where the number of actors varies dynamically.
• Theoretical Insight: The work provides a solution to the "underdetermined association" problem in pose-guided animation by explicitly decoupling identity from fixed channels and binding it to spatial masks, a concept that could influence future controllable video generation research.