Inverse Virtual Try-On: Generating Multi-Category Product-Style Images from Clothed Individuals

This paper introduces TEMU-VTOFF, a novel text-enhanced framework that leverages a dual DiT backbone and multimodal attention to overcome the limitations of existing methods in generating high-quality, multi-category product-style garment images from clothed individuals, thereby achieving state-of-the-art performance in the inverse virtual try-on task.

The Gist

Imagine you walk into a clothing store, try on a beautiful dress, and take a selfie. Now, imagine you want to buy that exact dress online, but the store only has a photo of you wearing it, not a photo of the dress hanging neatly on a rack. Usually, you'd have to wait for a professional photographer to take a "flat lay" photo of the item to list it for sale.

TEMU-VTOFF is like a magical, AI-powered "reverse magic trick" that solves this problem instantly. It takes a photo of a person wearing clothes and magically "undresses" them to reveal the pristine, catalog-ready version of the garment underneath.

Here is a simple breakdown of how it works, using everyday analogies:

1. The Problem: The "Messy Room" vs. The "Showroom"

• Virtual Try-On (VTON): This is the old way. You take a picture of a shirt and a picture of a person, and the AI tries to paste the shirt onto the person. It's like trying to fit a puzzle piece into a moving, wiggling puzzle. It's hard, and the result often looks weird or distorted.
• Virtual Try-Off (VTOFF): This is what this paper does. It's the opposite. You give the AI a photo of a person in a messy room (wearing the clothes, maybe sitting down, maybe with arms crossed), and it cleans up the room to show you the furniture (the clothes) exactly as it would look in a showroom.
• The Challenge: Previous AI attempts at this were like trying to guess what a car looks like just by seeing a blurry photo of someone driving it. They often got the color right but messed up the shape, or they lost the fine details like buttons and patterns.

2. The Solution: The "Dual-Brain" System (TEMU-VTOFF)

The authors built a new AI system called TEMU-VTOFF. Think of it as a team of two specialized detectives working together:

• Detective A (The Feature Extractor): This detective's job is to look at the person in the photo and figure out exactly what the clothes look like underneath all the wrinkles, folds, and body shapes. It ignores the person's face and pose and focuses purely on the fabric.
• Detective B (The Generator): This detective takes the clues from Detective A and paints a brand new, perfect picture of the clothes hanging flat on a wall.

The Secret Sauce:

• Text Clues: Sometimes, just looking at the photo isn't enough. Is that a "sleeveless summer dress" or a "long-sleeve winter coat"? The AI also reads a short text description (like a caption) to help it understand the style. It's like having a shopping list while you look at the clothes.
• The "Mask" (The Cookie Cutter): The AI uses a digital outline (a mask) to know exactly where the clothes end and the person begins. It's like using a cookie cutter to cut the dress shape out of the photo of the person.

3. The "Garment Aligner": The Quality Control Inspector

Even with two detectives, the AI sometimes makes small mistakes, like blurring a logo or making a pattern look wavy.

To fix this, the team added a Garment Aligner. Think of this as a strict art teacher or a quality control inspector.

• During training, the AI tries to draw the clothes.
• The "Inspector" (a pre-trained expert AI called DINOv2) looks at the drawing and compares it to a perfect reference image.
• If the AI draws a button in the wrong spot or makes the texture too smooth, the Inspector says, "No, look closer!" and forces the AI to correct its work.
• Crucially: This inspector only helps during the learning phase. Once the AI is smart enough, the inspector is fired, so the final result is generated super fast without slowing down.

4. Why This Matters

This technology is a game-changer for the fashion industry:

• For Online Stores: They can take a photo of a model wearing a shirt and instantly generate the "flat lay" photo needed for the website, saving thousands of dollars on photoshoots.
• For You: It means better search results. If you see a cool jacket on a celebrity, this tech could help find that exact jacket for sale, even if the store only has photos of people wearing it.
• For AI: It helps train better AI models by creating huge libraries of clean, perfect clothing images from messy real-world photos.

In a Nutshell

TEMU-VTOFF is an AI that looks at a photo of you wearing an outfit and says, "I know exactly what that shirt looks like when it's not being worn." It uses a team of specialized AI brains, text descriptions, and a strict quality-checker to turn a messy, real-world photo into a perfect, store-ready product image. It's like having a magic wand that turns a "lived-in" photo into a "catalog" photo instantly.

Technical Summary

1. Problem Definition

The paper addresses Virtual Try-Off (VTOFF), the inverse task of the widely studied Virtual Try-On (VTON).

• Goal: To reconstruct a standardized, "in-shop" (flat-lay) product image of a garment directly from a photo of a person wearing that garment.
• Challenges:
  • Visual Ambiguity: Unlike VTON, which has diverse valid outputs (different poses), VTOFF has a single ground truth (the flat garment). However, inferring the flat shape from a warped, occluded, and pose-distorted human image is highly ambiguous.
  • Detail Loss: Existing methods often fail to preserve fine-grained textures, logos, and structural details (e.g., necklines, seams) due to the complexity of disentangling the garment from the human body.
  • Multi-Category Handling: Most existing solutions are limited to single categories (e.g., only upper-body) or struggle to generalize across diverse garment types (dresses, pants, tops) within a unified framework.
  • Architectural Mismatch: Current approaches often simply reverse VTON pipelines, which are not optimized for the specific constraints of extracting clean garment features from complex human inputs.

2. Methodology: TEMU-VTOFF

The authors propose TEMU-VTOFF (Text-Enhanced MUlti-category Virtual Try-OFF), a novel architecture based on a Dual Diffusion Transformer (DiT) framework.

A. Dual-DiT Architecture

The system utilizes two distinct DiT components based on Stable Diffusion 3 (SD3):

1. Feature Extractor ($F_E$):
   • Role: Encodes the input "clothed person" image ($x_{model}$) to extract meaningful intermediate features.
   • Input: It takes the person image, a binary mask, and the latent noise. Crucially, it operates at timestep $t=0$ (clean data) to extract noise-free, high-fidelity features of the garment as it appears on the person.
   • Output: It produces intermediate keys ($K_{extractor}$) and values ($V_{extractor}$) from its layers, which serve as rich conditioning signals for the generator.
2. Garment Generator ($F_D$):
   • Role: Generates the final clean, flat-lay garment image ($x_g$).
   • Mechanism: It leverages the features extracted by $F_E$ via a Multimodal Hybrid Attention (MHA) mechanism.

B. Multimodal Hybrid Attention (MHA)

To resolve ambiguities, the MHA module fuses three information sources within the attention mechanism:

• Latent Features ($z_t$): The noisy latent representation of the target garment.
• Text Embeddings: Concatenated CLIP and T5 embeddings derived from garment descriptions.
• Extractor Features: The $K_{extractor}$ and $V_{extractor}$ from the feature extractor.
• Function: This allows the model to attend to the text for semantic guidance (e.g., "long sleeves"), the mask for spatial boundaries, and the person image features for texture and color transfer, effectively grounding the generation in the input visual data.

C. Text and Mask Conditioning

• Text: The model uses LLMs (Qwen2.5-VL) to generate structural descriptions (e.g., "fitted waist," "button-down") while explicitly excluding color/texture to avoid redundancy with visual features.
• Mask: A binary mask acts as a "hard discriminator" to define the garment's spatial extent, while text acts as a "soft discriminator" for semantic attributes.

D. Garment Aligner Module

To mitigate the loss of high-frequency details (textures, patterns) common in diffusion models:

• Mechanism: A lightweight alignment module enforces feature-level consistency between the 8th Transformer block of the generator ($F_D$) and a frozen DINOv2 vision encoder.
• Loss: A cosine similarity loss ($L_{align}$) is added during training to ensure the generated garment's internal representations match the structural and textural fidelity of the clean ground-truth garment.
• Inference: This module is discarded during inference, adding no computational overhead.

3. Key Contributions

1. Unified Multi-Category Framework: TEMU-VTOFF is the first dedicated architecture to handle upper-body, lower-body, and full-body garments (dresses) simultaneously without category-specific pipelines.
2. Dual-DiT Design with Asynchronous Conditioning: The use of a separate feature extractor operating at $t=0$ to provide clean conditioning to a denoising generator operating at $t>0$ significantly improves feature extraction quality compared to synchronous approaches.
3. Multimodal Hybrid Attention: A novel attention mechanism that jointly processes text, masks, and visual features to resolve the ambiguity of reconstructing flat garments from 3D poses.
4. Garment Aligner: A training-time component that explicitly aligns generated features with a pre-trained vision encoder (DINOv2) to preserve fine-grained textures and structural integrity.

4. Experimental Results

The method was evaluated on VITON-HD (upper-body only) and Dress Code (multi-category) datasets.

• Quantitative Performance:
  • TEMU-VTOFF achieved State-of-the-Art (SOTA) performance on both datasets.
  • On Dress Code, it significantly outperformed competitors (TryOffDiff, MGT, Any2AnyTryon) in distributional metrics (FID, KID) and perceptual similarity (DISTS, LPIPS).
  • It showed robust generalization in cross-dataset experiments (e.g., trained on Dress Code, tested on VITON-HD).
• Qualitative Results:
  • The model successfully preserved complex patterns, logos, and structural details (necklines, hems) that competitors often distorted or lost.
  • It handled occlusions and complex poses better than prior art.
• Downstream Utility:
  • When used as a data augmentation tool to generate synthetic "in-shop" images for training VTON models (CatVTON), it improved the performance of the downstream VTON task, proving the high fidelity of the generated data.
• User Study:
  • In a pairwise comparison with 42 participants, TEMU-VTOFF was preferred over MGT (75.77%) and Any2AnyTryon (77.74%) for realism and texture preservation.

5. Significance

• E-commerce Impact: The ability to automatically generate standardized catalog images from customer/model photos reduces the cost and time of product photography, enabling scalable dataset curation.
• Foundation Model Training: High-quality, standardized garment images are crucial for training better foundation models for fashion AI.
• Architectural Insight: The paper demonstrates that simply reversing VTON pipelines is insufficient; dedicated architectures that separate feature extraction from generation and leverage multimodal conditioning are necessary for high-fidelity inverse tasks.
• Ethical Consideration: The authors acknowledge potential copyright issues regarding reconstructing third-party designs but emphasize the tool's value for research and responsible data augmentation.

In summary, TEMU-VTOFF represents a significant leap forward in the inverse virtual try-on task, solving the critical issues of detail preservation and multi-category generalization through a sophisticated dual-DiT architecture enhanced by text, masks, and feature alignment.