ProFashion: Prototype-guided Fashion Video Generation with Multiple Reference Images

ProFashion is a novel framework for fashion video generation that leverages multiple reference images and introduces a Pose-aware Prototype Aggregator and Flow-enhanced Prototype Instantiator to overcome the limitations of single-image inputs and improve spatiotemporal consistency in generated videos.

The Gist

Imagine you are a fashion designer trying to show off a new jacket to a customer. You have a single photo of the jacket from the front. If you try to make a video of a model wearing it and turning around, a standard AI might get confused. It might try to guess what the back looks like, but since it only has the front photo, it might "hallucinate" (make things up), turning the back of the jacket into a weird, blurry mess or copying the front pattern onto the back where it doesn't belong.

This is the problem the paper ProFashion solves. It's a new AI system designed to create fashion videos that look real and consistent, even when the model spins, turns, or moves around.

Here is how it works, explained with simple analogies:

1. The Problem: The "One-Photo" Blind Spot

Think of existing AI video generators like a painter who is only allowed to look at one single photograph of a shirt.

• The Issue: If the shirt has a cool pattern on the front but a plain color on the back, the painter has no idea what the back looks like. When the model turns around in the video, the AI panics and paints something random.
• The Result: The video looks glitchy and fake.

2. The Solution: The "Photo Album" Approach

ProFashion changes the rules. Instead of giving the AI just one photo, it gives it a small photo album (multiple reference images) showing the outfit from the front, the back, and the side.

• The Goal: Now the AI has all the information it needs to know exactly what the outfit looks like from every angle.

3. The Secret Sauce: How ProFashion Uses the Album

Just throwing all the photos at the AI isn't enough; it needs a smart way to pick the right details at the right time. ProFashion uses two special tools to do this:

A. The "Pose-Aware Librarian" (Pose-aware Prototype Aggregator)

Imagine you are reading a book, and you need to find a specific fact. You don't read the whole book every time; you use the index to jump to the right page.

• How it works: As the model in the video starts to turn, ProFashion looks at the model's pose (is it facing left? right? back?).
• The Magic: It acts like a librarian who instantly grabs the exact reference photo that matches that pose. If the model turns to the back, the librarian grabs the "back view" photo and hands it to the AI. If the model faces front, it grabs the "front view."
• Why it's cool: It does this so quickly that it doesn't slow down the computer, even though it's looking at multiple photos.

B. The "Flow-Enhanced Motion Guide" (Flow-enhanced Prototype Instantiator)

Imagine you are animating a stick figure. If you just draw the arm in a new spot for the next frame, it might look like the arm is teleporting or jittering.

• How it works: ProFashion looks at the "motion flow"—the invisible path the body parts are traveling. It knows that if a hand moves from left to right, the pixels should follow a smooth curve, not jump.
• The Magic: It uses this motion map to "warp" the image smoothly. It ensures that the fabric of the shirt stretches and folds naturally as the model moves, keeping the video fluid and realistic.

4. The Result: A Perfect Fashion Show

When you put these tools together, ProFashion can generate a video where:

• No Hallucinations: The back of the jacket looks exactly like the back of the jacket in the reference photo, not a made-up mess.
• Smooth Motion: The model turns and walks without the clothes glitching or warping strangely.
• High Detail: You can see the texture and patterns of the fabric clearly, just like in a real high-end fashion show.

Why Does This Matter?

Currently, online stores use static photos. If you want to see how a dress moves, you have to imagine it. ProFashion allows stores to turn a few photos of a model into a full, 360-degree video instantly. This helps customers "try on" clothes virtually with much higher accuracy, potentially saving them from buying clothes that don't look right.

In short: ProFashion is like giving an AI a complete photo album and a smart guidebook, so it can paint a perfect, moving fashion show without ever guessing or making things up.

Technical Summary

1. Problem Statement

Fashion video generation aims to synthesize temporally consistent videos of a character wearing specific garments based on reference images. While diffusion-based methods have advanced significantly, existing approaches suffer from two critical limitations:

• Single-Reference Limitation: Current methods typically accept only a single reference image. This fails to capture view-dependent patterns (e.g., different designs on the front and back of a garment), leading to severe hallucinations and poor view consistency when the character rotates or moves.
• Insufficient Motion Modeling: Widely adopted motion modules often propagate information only along the temporal dimension at the same spatial position. This is insufficient for handling large human body movements (e.g., turning around), resulting in sub-optimal spatiotemporal consistency and jittery motion.

2. Methodology: ProFashion Framework

ProFashion is a prototype-guided framework built upon a Latent Diffusion Model (LDM). It accepts multiple reference images ($I_r$) and a driving pose sequence ($p_f$) to generate a coherent video. The architecture consists of three core components:

A. Reference Encoder

• Function: Extracts multi-scale, fine-grained features from multiple reference images.
• Mechanism: Based on a U-Net structure, it processes encoded reference latents. It incorporates Spatial Self-Attention to enrich spatial details and Semantic Cross-Attention to inject global visual information from CLIP image encoders into the latent representations.

B. Pose-aware Prototype Aggregator (PPA)

• Goal: To effectively integrate features from multiple reference images into a single guidance signal without incurring high computational costs.
• Mechanism:
  • Pose-Aware Selector: Calculates the similarity between the driving pose of the current frame and the poses of all reference images.
  • Aggregation: Based on pose similarity, it generates aggregation maps to select and weight features from the reference images.
  • Prototype Formation: It aggregates both fine-grained features (from the Reference Encoder) and global features (from CLIP) into "prototypes."
  • Efficiency: The resulting prototypes share the same shape as a single reference feature, allowing the model to leverage multi-view information with the same computational cost as a single-reference setup.

C. Flow-enhanced Prototype Instantiator (FPI)

• Goal: To ensure smooth motion and detail consistency across frames, particularly during large body movements.
• Mechanism:
  • Prototype-guided Spatial Attention: Uses the aggregated fine-grained prototypes to guide the spatial attention layers in the denoiser, ensuring garment details are preserved.
  • Flow-enhanced Temporal Attention (FTA): Introduces an additional spatiotemporal attention layer. It leverages human keypoint motion flow (derived from the driving pose sequence) to supervise a warping process. This allows features of the same body part to propagate across adjacent frames even when spatial positions shift significantly, enhancing motion smoothness.

D. Training Strategy

The model is trained in two stages:

1. Stage 1: Trained on single frames with multiple references (excluding FTA layers) to learn view consistency.
2. Stage 2: Trained on video clips to optimize motion consistency, updating only the FTA layers.

3. Key Contributions

1. Multi-Reference Extension: The first framework to extend fashion video generation to multiple reference images, solving the ill-posed nature of generating view-consistent videos from a single image.
2. Pose-aware Prototype Aggregator (PPA): A novel module that dynamically selects and aggregates reference features based on pose similarity, creating frame-wise prototypes that guide denoising efficiently.
3. Flow-enhanced Prototype Instantiator (FPI): A mechanism that integrates human keypoint motion flow into spatiotemporal attention, significantly improving motion smoothness and handling large body movements.
4. MRFashion-7K Dataset: The authors collected a new dataset of 7,335 fashion videos featuring diverse garment details from multiple perspectives and significant body movements to facilitate research in this domain.

4. Experimental Results

The authors evaluated ProFashion on the MRFashion-7K dataset and the UBC Fashion dataset.

• Quantitative Performance (MRFashion-7K):
  • ProFashion significantly outperformed single-reference baselines (e.g., Animate Anyone, Champ) and ablation variants.
  • Metrics: Achieved the best SSIM (0.885), PSNR (23.57), LPIPS (0.086), and FVD (126.92).
  • Ablation: Adding PPA improved SSIM from 0.838 to 0.867; adding FPI reduced FVD by 35.56% (from 196.95 to 126.92), demonstrating the efficacy of flow-guided motion modeling.
• Human Evaluation:
  • In a user study (13 volunteers), ProFashion scored highest in Character Authenticity, Clothing Detail, and Motion Fluency.
  • It notably outperformed methods using simple averaging or concatenation of references.
• UBC Fashion Dataset:
  • ProFashion surpassed state-of-the-art methods (including DreamPose and Animate Anyone) across all metrics, achieving a 31.4% improvement in FVD over the previous best method.
• Qualitative Results:
  • Visualizations show ProFashion successfully generates view-consistent videos (e.g., showing both front and back patterns) without the hallucinations seen in single-reference methods.

5. Significance and Impact

• Retail Application: ProFashion offers a low-cost solution for online retail by generating accurate, detailed, and view-consistent fashion videos from static images, enhancing the customer shopping experience.
• Technical Advancement: It addresses the fundamental limitations of current diffusion models regarding multi-view consistency and complex motion modeling in human video generation.
• Future Work: While effective for patterns, the authors note limitations in preserving fine textual details on clothing, which remains an area for future improvement.

In summary, ProFashion represents a significant leap in fashion video generation by effectively leveraging multiple reference views and human motion flow to produce high-fidelity, temporally consistent videos.