💻 computer science

PROMO: Promptable Outfitting for Efficient High-Fidelity Virtual Try-On

PROMO is a promptable virtual try-on framework built on a Flow Matching DiT backbone with latent multi-modal conditioning and self-reference mechanisms that achieves high-fidelity, efficient image synthesis by treating VTON as a structured editing problem, thereby surpassing existing methods in both visual quality and inference speed.

Original authors: Haohua Chen, Tianze Zhou, Wei Zhu, Runqi Wang, Yandong Guan, Dejia Song, Yibo Chen, Xu Tang, Yao Hu, Lu Sheng, Zhiyong Wu

Published 2026-03-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haohua Chen, Tianze Zhou, Wei Zhu, Runqi Wang, Yandong Guan, Dejia Song, Yibo Chen, Xu Tang, Yao Hu, Lu Sheng, Zhiyong Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're shopping online for a new outfit. You find a beautiful sweater and a pair of jeans, but you can't be sure how they'll look on you. You'd love to see a photo of yourself wearing them, but you don't want to wait for a tailor or deal with the hassle of returning things that don't fit.

This is the problem PROMO solves. It's a new AI tool that acts like a magic digital dressing room. You upload a photo of yourself and a photo of the clothes you want, and it instantly generates a realistic photo of you wearing those exact items.

Here is how PROMO works, explained through simple analogies:

1. The "Smart Tailor" vs. The "Old Sewing Machine"

The Old Way: Previous AI tools were like old sewing machines. They tried to physically "warp" or stretch the image of the clothes to fit your body. This often resulted in weird distortions, like a sweater that looked melted or a shirt that didn't match your arm length.
The PROMO Way: PROMO is like a master digital tailor who understands the fabric, the fit, and the style perfectly. Instead of stretching the image, it "paints" the new clothes onto you, pixel by pixel, ensuring the texture (like the knit of a sweater) and the lighting look completely real.

2. The "Recipe Book" (Promptable Style)

Sometimes, you don't just want to wear a shirt; you want to wear it specifically. Maybe you want the sleeves rolled up, or the shirt tucked in, or the collar popped.

The Innovation: PROMO has a special "Style Prompt System." Think of this as a smart assistant who looks at the clothes and writes a detailed recipe for how they should look.
How it helps: You can tell the AI, "Make the sleeves short" or "Make it look like a casual fit." The AI reads this instruction and adjusts the digital outfit accordingly, giving you control over the final look without needing to take a new photo.

3. The "Efficient Chef" (Speed and Memory)

Usually, creating these high-quality photos takes a long time and requires a supercomputer because the AI has to juggle too many ingredients (your body, the clothes, the background, the pose).

The Trick: PROMO uses a clever trick called "Temporal Self-Reference." Imagine a chef who, instead of re-reading the entire recipe book for every single step of cooking, remembers the key ingredients from the first step and reuses them for the rest of the meal.
The Result: This allows PROMO to cook up a high-quality photo much faster than its competitors, saving time and computer power without losing quality.

4. The "Smart Mask" (Not Getting Confused)

When you try on a new shirt, the AI needs to know exactly where your old shirt ends and the new one begins. If it gets confused, it might accidentally paint the new shirt over your face or your background.

The Solution: PROMO uses a "Region-Aware Loss" strategy. Think of this as a smart highlighter. It highlights your body and the clothes you are changing, telling the AI: "Focus all your energy here! Ignore the background and the rest of the room." This ensures the clothes look crisp and the background stays exactly as it was.

5. The "Outfit Coordinator" (Multi-Garment)

Most old tools could only try on one item at a time (like just a shirt). If you wanted to see a full outfit (shirt + pants + shoes), you had to do it in three separate, clunky steps.

The Upgrade: PROMO is a full outfit coordinator. You can upload a picture of a dress, a pair of shoes, and a handbag all at once. It figures out how they all fit together on your body simultaneously, creating a complete, cohesive look in one go.

Why Does This Matter?

For Shoppers: You can try on hundreds of outfits in seconds, seeing exactly how they fit and look before you buy. This means fewer returns and more confidence.
For Stores: They can show you how their clothes look on your body type, not just on a generic model, making online shopping feel personal and real.

In short, PROMO takes the complex math of AI and turns it into a simple, fast, and magical way to see yourself in any outfit, anytime, anywhere. It's the difference between guessing how a shirt fits and actually seeing it on you.

1. Problem Statement

Virtual Try-On (VTON) is a critical technology for e-commerce, aiming to render target garments onto a person image with high fidelity. Despite recent advances, existing methods face three persistent challenges:

Fidelity vs. Efficiency Trade-off: Diffusion-based methods (e.g., using UNets or Transformers with auxiliary reference networks) achieve photorealism but suffer from complex architectures, slow sampling speeds, and high inference overhead.
Limited Controllability: Many models lack fine-grained control over "dressing styles" (e.g., tucking in a shirt, sleeve length) or struggle to handle multi-garment scenarios (e.g., trying on a top and pants simultaneously).
Real-World Robustness: Existing approaches often fail in "in-the-wild" scenarios involving loose clothing, occlusions, or incomplete inputs (e.g., missing garment images or model images).

2. Methodology

The authors propose PROMO, a framework built on a Flow Matching Diffusion Transformer (DiT) backbone. It treats VTON as a structured image editing problem requiring subject preservation, faithful texture transfer, and seamless harmonization.

Core Architectural Components:

Latent Multi-Modal Conditional Concatenation: Instead of using separate reference networks (which double parameters), PROMO concatenates conditions (person, garments, masks, pose) directly into the latent space.
Efficient Spatial Conditioning:
- Agnostic Mask Generation: Uses a combination of human body parsing and DWPose to automatically generate masks, preventing information leakage from original clothing.
- Hierarchical Compression: To reduce token counts, pose and mask conditions are downsampled (2x) and merged. This reduces the token count to 12.5% of the original dual-condition representation, significantly boosting training and inference efficiency.
- Region-Aware Loss Weighting: A weighted loss function is applied where body regions receive higher weights ( $1+\lambda$ ) and backgrounds receive lower weights ( $1-\lambda$ ), forcing the model to focus on garment details.
Temporal Self-Reference (TSR):
- A parameter-free mechanism using 3D-RoPE (Rotary Positional Embedding) to distinguish between different condition groups (e.g., multiple garments).
- Caching Mechanism: Key-Value (KV) pairs for spatial conditions are cached at the first timestep and reused for subsequent timesteps. This avoids the need for a separate reference network, drastically reducing inference time while maintaining quality.
Promptable Dressing Style System:
- A Style Prompt System based on a fine-tuned Qwen2.5-VL-7B model.
- It converts images of models wearing garments into structured JSON descriptions (using Pydantic schemas) and then into natural language prompts.
- This allows users to control specific attributes (e.g., "fitted," "waist-level," "sleeveless") via text, overcoming the limitations of previous closed-source or single-garment extraction methods.

System for Real-World Scenarios:

PROMO includes auxiliary modules to handle incomplete inputs:

TryOff Model: Extracts a garment from a model image if the isolated garment image is missing.
Style Prompt System: Generates style descriptions from garment catalogs if the model image is missing.

3. Key Contributions

PROMO Framework: A promptable, multi-garment VTON framework using a Flow Matching DiT backbone with latent multi-modal concatenation. It achieves a superior balance between visual fidelity and inference speed.
Efficiency Innovations: Introduction of Temporal Self-Reference and Spatial Token Merging, which eliminate the need for auxiliary reference networks and reduce token counts by up to 87.5%, enabling fast inference on large-parameter models.
Style Prompt System: A novel pipeline using a distilled Qwen2.5-VL-7B model to generate reliable, multi-garment style descriptions, enabling prompt-based control over dressing styles without relying on closed-source APIs.
Robustness: Demonstrated capability to handle "in-the-wild" scenarios, including multi-garment try-ons and scenarios with missing inputs (model or garment images).

4. Experimental Results

The authors evaluated PROMO on VITON-HD, DressCode, and a self-collected In-The-Wild dataset.

Quantitative Performance:
- PROMO outperformed state-of-the-art VTON models (e.g., OOTDiffusion, CatVTON, Any2AnyTryon) and general image editing models (e.g., Seedream, Qwen Image Edit) on standard metrics (SSIM, LPIPS, FID, KID).
- On the DressCode dataset (Paired), PROMO achieved an FID of 3.3103 and LPIPS of 0.0887, surpassing competitors.
- It significantly outperformed general image editing models, which often suffer from color inconsistencies and artifacts in VTON tasks.
Qualitative Performance:
- Visualizations show PROMO preserves intricate textures, text patterns, and natural human geometry better than baselines.
- It successfully handles complex scenarios like loose-fitting garments and multi-garment combinations.
User Study:
- On an in-the-wild dataset, PROMO achieved the highest Overall Excellence Rate (84.42%) compared to commercial services (Huiwa, Kling, Douyin), particularly in garment texture and style consistency.
Efficiency:
- Temporal Self-Reference reduced inference time from 22s to **9.2s** (on H800 GPU) while maintaining quality.
- Spatial Token Merging further optimized speed with negligible quality loss.

5. Significance

Paradigm Shift: PROMO demonstrates that Flow Matching Transformers combined with latent multi-modal conditioning and self-reference acceleration can replace complex, multi-network architectures for high-quality VTON.
Practical Applicability: By solving the "missing input" problem (via TryOff/Style Prompt modules) and enabling prompt-based style control, PROMO bridges the gap between academic research and real-world e-commerce applications.
Training Efficiency: The framework is generic and transfers well to broader image editing tasks, suggesting that VTON paired data can serve as a rich supervisory resource for training general-purpose editors.

In conclusion, PROMO sets a new state-of-the-art by delivering high-fidelity, multi-garment virtual try-on with prompt-based controllability and significantly reduced computational costs.