The devil is in the details: Enhancing Video Virtual Try-On via Keyframe-Driven Details Injection

This paper introduces KeyTailor, a novel framework featuring a keyframe-driven details injection strategy to enhance garment dynamics and background integrity in video virtual try-on without increasing architectural complexity, accompanied by the large-scale ViT-HD dataset to address data limitations.

The Gist

Imagine you want to try on a new outfit, but instead of going to a fitting room, you want to see yourself wearing it in a video of you dancing, walking, or jumping. This is the dream of Video Virtual Try-On (VVT).

However, current technology is a bit like a clumsy tailor. It can put the shirt on you, but often the fabric looks flat, the wrinkles don't move when you raise your arm, and the background (like the wall behind you) starts to blur or warp. It's like wearing a cardboard cutout of a shirt instead of real cloth.

This paper introduces a new system called KeyTailor (along with a massive new library of training videos called ViT-HD) that acts like a master tailor who pays attention to the "devil in the details."

Here is how it works, broken down into simple concepts:

1. The Problem: The "Blurry Background" and "Flat Shirt"

Existing AI models try to swap clothes in a video, but they struggle with two main things:

• The Flat Shirt: When you move, real clothes wrinkle, stretch, and catch the light. Current AI often makes the clothes look like a smooth, static sticker that doesn't react to your movement.
• The Wobbly Background: To make room for the new clothes, the AI often "paints over" the old clothes. In doing so, it accidentally blurs the floor, the wall, or your hair. It's like trying to edit a photo and accidentally smudging the background while fixing the foreground.
• The Heavy Cost: To fix these issues, other methods try to add massive, complex extra layers to the AI brain, making it slow and expensive to run.

2. The Solution: The "Keyframe" Strategy

The authors realized that you don't need to analyze every single frame of a video to understand how a shirt moves. You just need the Keyframes.

Think of a flipbook animation. You don't need to draw every single millisecond of movement to understand the motion; you just need the key poses (e.g., "arm up," "arm down," "turning around").

KeyTailor uses a smart "Instruction-Guided" system to find these key moments:

• The Instruction: You tell the AI what you want to see (e.g., "Show the back of the shirt" or "Show the sleeves when the arm is raised").
• The Selection: The AI scans the video and picks the specific frames that best show those angles and movements. These are the Keyframes.

3. The Magic Trick: "Details Injection"

Once the AI has these perfect Keyframes, it doesn't just paste them into the video. Instead, it uses them as a reference guide to teach the main AI how to behave.

• Garment Dynamics (The Shirt): The AI looks at the Keyframes to see exactly how the fabric wrinkles when the arm is raised. It then "injects" this knowledge into the video generation process. It's like the tailor saying, "Remember, when the arm goes up, the fabric pulls tight here."
• Background Integrity (The Room): The AI also looks at the Keyframes to see what the background should look like without the clothes. It uses this to ensure the floor and walls stay sharp and don't get blurry or warped. It's like having a high-resolution photo of the room to make sure the AI doesn't accidentally paint over the wallpaper.

4. The Result: A Lightweight, High-Quality Fit

The best part is that KeyTailor doesn't need to rebuild the entire AI brain. Instead of adding a giant, heavy engine to the car, they just added a high-tech GPS navigation system (the Keyframe modules) to the existing engine.

• Efficiency: It runs much faster and uses fewer computer resources than previous methods.
• Quality: The clothes look real (with wrinkles and movement), and the background stays perfectly clear.
• The Dataset (ViT-HD): To teach this system, the authors collected over 15,000 high-definition videos of models wearing different clothes. Think of this as a massive "fashion library" that the AI studied to learn how real clothes behave in the real world.

The Bottom Line

KeyTailor is like upgrading a virtual fitting room from a blurry, cardboard-mannequin experience to a high-definition, realistic simulation. By focusing on the most important moments (Keyframes) and using them to guide the details, it creates videos where the clothes move naturally, and the world around them stays perfectly intact.

Technical Summary

1. Problem Statement

Video Virtual Try-On (VVT) aims to replace a person's clothing in a video with a target garment while maintaining spatiotemporal consistency (motion, background, and garment dynamics). Despite progress with Diffusion Transformers (DiTs), existing methods face three critical challenges:

• Insufficient Garment Dynamic Details: Current models fail to capture fine-grained dynamics such as backside textures, wrinkles caused by body motion (e.g., raising an arm), and subtle lighting variations. This leads to over-smoothed appearances and incorrect garment positioning.
• Background Inconsistency: Methods relying solely on garment-agnostic videos (inpainting results) often lose fine background textures, suffer from temporal flickering, and exhibit environmental incoherence (e.g., blurred floor textures or misaligned wall frames).
• High Complexity and Data Scarcity: State-of-the-art (SOTA) methods often introduce heavy interaction modules into DiT backbones, drastically increasing computational costs and parameter counts. Furthermore, existing public datasets (e.g., VVT, ViViD) are limited in scale, resolution, and garment diversity, hindering model generalization.

2. Methodology: KeyTailor

The authors propose KeyTailor, a novel DiT-based framework that avoids modifying the core DiT architecture. Instead, it employs a Keyframe-Driven Details Injection strategy to enrich the generation process with fine-grained information.

A. Instruction-Guided Keyframe Sampling (IKS)

To capture multi-view garment dynamics and background consistency, the system does not use all frames. Instead, it uses a Large Visual-Language Model (e.g., Qwen) to parse user instructions (e.g., "show front/back," "raise hand").

• Process: The system generates anchor poses corresponding to the requested views and actions. It then scores input video frames based on:
  1. Motion Difference: How much the frame's skeleton differs from the anchors.
  2. Garment Area Ratio: The visibility of the garment.
• Selection: A dual-selection strategy filters frames to ensure temporal uniformity and reduce redundancy, resulting in a set of informative Keyframes ($F_{key}$).

B. Two Lightweight Injection Modules

The core innovation lies in two modules that distill details from the selected keyframes and inject them into the DiT without adding heavy interaction layers:

1. Garment Dynamic Details Enhancement (GDDE):

   • Goal: Enrich garment latents with dynamic details (wrinkles, back textures).
   • Mechanism: It takes the initial try-on frame and the garment regions from the keyframes. It encodes these regions into latent representations and uses a lightweight distillation component ($D$) to inject multi-view garment variations into the primary garment latent ($L_g$).

2. Collaborative Background Details Optimization (CBDO):

   • Goal: Preserve background integrity and semantic consistency.
   • Mechanism: It addresses the loss of detail in garment-agnostic videos. It extracts the background regions from the keyframe with the highest background completeness score and encodes them into a latent ($L_{bg}^{key}$). This is fused with the standard background latent ($L_{bg}$) via a weighted merging strategy to create an optimized background latent ($\bar{L}_{bg}$).

C. Video Generation

The enriched latents ($\bar{L}_g$ and $\bar{L}_{bg}$) are fused with pose, mask, and noise latents. These are injected into standard DiT blocks. Crucially, the model uses LoRA (Low-Rank Adaptation) to fine-tune the DiT attention mechanisms, avoiding the need to retrain the entire backbone or add massive interaction layers.

3. Key Contributions

1. KeyTailor Framework: A novel DiT-based VVT framework that enhances fidelity and background integrity via a keyframe-driven injection strategy. It achieves this without modifying the DiT architecture or introducing heavy interaction modules.
2. ViT-HD Dataset: The authors curated a new, large-scale, high-definition dataset containing 15,070 video samples at 810×1080 resolution. It covers diverse garment styles and scenarios, significantly surpassing existing datasets in quality and scale.
3. Efficiency: By using LoRA and avoiding architectural changes, KeyTailor significantly reduces computational overhead compared to SOTA methods (e.g., MagicTryOn, ViViD).
4. Instruction-Guided Sampling: A unique approach to selecting keyframes based on semantic instructions, ensuring the model focuses on the most informative frames for dynamic details.

4. Results

• Quantitative Performance: On the ViT-HD, VVT, and ViViD datasets, KeyTailor outperforms SOTA baselines (including MagicTryOn, CatV2TON, and ViViD) across all metrics:
  • Garment Fidelity: Lower VFID (Video Fréchet Inception Distance) and LPIPS scores.
  • Background Integrity: Higher SSIM scores, indicating better preservation of background structures.
  • Generalization: Performs well on image-based try-on tasks (VITON-HD, DressCode), demonstrating strong transferability.
• Qualitative Improvements: Visual comparisons show KeyTailor produces videos with realistic wrinkles, correct belt positions, and consistent background textures (e.g., floor patterns, wall frames) that other methods blur or distort.
• Efficiency Analysis:
  • Parameters: KeyTailor adds only 2.10% parameters to the base model, compared to 15.11% (MagicTryOn) and 157.10% (ViViD).
  • Training: Trains on only 0.2B parameters (vs. 16.4B for MagicTryOn).
  • Inference: Maintains inference speed comparable to the base Wan 2.1 model.
• User Study: In a blind preference test, KeyTailor was preferred over SOTA methods in ~55-65% of cases across visual quality, semantic consistency, and overall quality.

5. Significance

This work addresses the "devil in the details" problem in video generation, proving that fine-grained consistency does not require massive architectural overhauls.

• Paradigm Shift: It demonstrates that keyframe-driven injection is a more efficient and effective way to handle temporal dynamics and background consistency than adding complex interaction modules to DiTs.
• Resource Contribution: The release of ViT-HD fills a critical gap in high-resolution, diverse video try-on data, enabling future research in high-fidelity video generation.
• Practicality: The method's low computational cost and high fidelity make it highly suitable for real-world applications in e-commerce and short-video platforms.