Technical Summary: iTryOn – Mastering Interactive Video Virtual Try-On with Spatial-Semantic Guidance

1. Problem Definition and Motivation

Video Virtual Try-On (VVT) aims to replace a garment on a person in a video with a new one while maintaining temporal consistency. While existing methods have advanced significantly in handling passive motion (e.g., walking or posing), they are confined to non-interactive scenarios. Current benchmarks and models fail to address Interactive Video Virtual Try-On (Interactive VVT), where subjects actively engage with their clothing (e.g., pulling a hem, unzipping a jacket, or adjusting a collar).

The transition to Interactive VVT introduces two primary challenges not present in traditional VVT:

1. Semantic Ambiguity of Interactions: Standard conditioning signals like 2D pose keypoints lack 3D orientation and shape. A 2D projection cannot distinguish between a hand pressing flat against a chest (non-interactive) and a hand pulling a shirt (interactive), leading to ambiguous generation.
2. Learning from Sparse Events: Interactive moments involving complex, physics-driven deformations are often brief and infrequent compared to static or passive segments. This data imbalance creates a sparse supervisory signal, causing models to converge on simpler, non-interactive dynamics rather than learning complex interactions.
3. Data and Evaluation Scarcity: Existing datasets consist almost entirely of non-interactive sequences, and standard metrics fail to verify the semantic success of human-garment interactions.

2. Methodology: The iTryOn Framework

To address these challenges, the authors propose iTryOn, a novel framework built upon a large-scale video diffusion Transformer (specifically the Wan2.1-VACE backbone). The framework integrates a multi-level interaction injection mechanism and a targeted constraint loss.

2.1. Multi-Level Interaction Injection

The framework resolves ambiguity by providing guidance at both spatial and semantic levels:

• Spatial Guidance (The "How"): To resolve spatial ambiguity, iTryOn introduces a garment-agnostic 3D hand prior. Unlike depth-based priors which suffer from information leakage (retaining source garment geometry), this approach uses the HaMeR model to extract clean 3D hand meshes. These are processed by a lightweight Interaction Guider module to provide fine-grained cues on hand shape, orientation, and proximity, ensuring physically plausible hand-garment contact.
• Semantic Guidance (The "What" and "When"): To resolve semantic ambiguity, the system leverages two types of text inputs:
  • Global Captions: Provide overall context for the video.
  • Time-Stamped Action Captions: Provide localized, categorical descriptions of specific interactions (e.g., "Rolling sleeves").
  • Action-aware Rotational Position Embedding (A-RoPE): A novel embedding strategy designed to synchronize these action captions with specific video segments. By applying scaled 1D-RoPE to the keys of interactive segments only, A-RoPE prevents semantic "bleeding" into non-interactive frames, ensuring precise temporal alignment.

2.2. Action-Aware Constraint Loss (AC Loss)

To address the challenge of learning from sparse interactive events, the authors introduce an Action-Aware Constraint Loss. This loss function strategically re-weights the standard diffusion loss. By applying a binary mask derived from action captions, the loss amplifies the supervisory signal specifically on frames containing interactions. This prevents the model from being overwhelmed by gradients from simpler, non-interactive frames, forcing it to prioritize and converge on complex dynamics.

2.3. Dataset: VVT-Interact

The authors constructed VVT-Interact, the first large-scale dataset for this task, containing 5,292 high-quality video-garment pairs. The dataset was curated from e-commerce live streams and social media, filtered for quality, and annotated using a VLM-based pipeline (Qwen-VL) to generate global captions, time-stamped action captions, and binary interaction labels.

3. Key Contributions

1. Task Formalization: The paper formally defines Interactive Video Virtual Try-On (Interactive VVT), establishing a new frontier for dynamic apparel presentation.
2. The iTryOn Framework: A novel architecture featuring a multi-level interaction injection mechanism (3D hand priors + A-RoPE synchronized captions) and an action-aware constraint loss to stabilize training on sparse data.
3. Dataset and Metrics: The introduction of VVT-Interact and the Interaction Success Rate (ISR) metric. ISR utilizes a Vision-Language Model to semantically verify if the generated interaction (e.g., "zipping up") is recognizable and coherent, moving beyond purely visual fidelity metrics.

4. Experimental Results

Extensive experiments were conducted on both the new VVT-Interact dataset and the traditional non-interactive ViViD benchmark.

• Interactive Performance: On VVT-Interact, iTryOn establishes a commanding lead over state-of-the-art baselines (ViViD, CatV2TON, MagicTryOn).
  • Visual Fidelity: It achieves superior scores in SSIM, LPIPS, and VFID, indicating high-quality texture preservation and reduced flickering.
  • Interaction Fidelity: Crucially, iTryOn achieves an Interaction Success Rate (ISR) of 61.00%, significantly outperforming baselines which score below 49%. This confirms the model's ability to execute specific physical interactions rather than just generating plausible-looking motion.
• Non-Interactive Performance: Despite its focus on interaction, iTryOn also achieves state-of-the-art performance on the traditional ViViD benchmark, demonstrating the versatility of the underlying backbone and training strategies.

5. Significance and Claims

The paper claims that iTryOn marks a significant step towards more dynamic and controllable virtual try-on experiences. By bridging the gap between passive display and active human-garment interaction, the work addresses a critical limitation in current e-commerce and digital content creation tools.

The authors emphasize that their success is not merely due to additional data but stems from the synergistic combination of spatial guidance (3D hands), semantic precision (A-RoPE), and targeted supervision (AC loss). They acknowledge limitations, such as the model's inability to reason about garment semantics (e.g., attempting to unzip a T-shirt without a zipper), noting that future work should incorporate explicit garment-aware action validation. However, the current framework successfully generates physically plausible deformations for a wide range of complex interactions, setting a new standard for the field.