💻 computer science

Dress-ED: Instruction-Guided Editing for Virtual Try-On and Try-Off

This paper introduces Dress-ED, the first large-scale dataset unifying Virtual Try-On, Virtual Try-Off, and text-guided garment editing with over 146k instruction-driven samples, alongside a unified multimodal diffusion framework to enable controllable and interactive fashion generation.

Original authors: Fulvio Sanguigni, Davide Lobba, Bin Ren, Marcella Cornia, Nicu Sebe, Rita Cucchiara

Published 2026-03-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fulvio Sanguigni, Davide Lobba, Bin Ren, Marcella Cornia, Nicu Sebe, Rita Cucchiara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical wardrobe in your living room. You pick out a dress, put it on, and instantly, you can change its color, turn it into leather, add a pocket, or even shorten the sleeves—all just by speaking to it. That's the dream of "Virtual Try-On" (VTON). But until now, this magic was a bit clumsy. You could try on a dress, but you couldn't easily edit it with specific instructions like "make the sleeves long" or "add a camouflage pattern."

This paper introduces Dress-ED, a massive new project that acts like a "training school" for AI to master this kind of fashion magic. Here is the breakdown in simple terms:

1. The Problem: The "Static Mannequin"

Think of previous fashion AI datasets as a library of mannequins. You could swap a mannequin's shirt for another one, but you couldn't tell the mannequin, "Hey, turn that blue shirt into a red leather jacket with a zipper." The AI didn't have a big enough library of examples showing how to make those specific changes while keeping the person looking like themselves.

2. The Solution: The "Fashion Chef" (Dress-ED)

The authors built Dress-ED, a giant cookbook with over 146,000 recipes.

The Ingredients: Every "recipe" (or data sample) has four parts:
1. The original photo of a person wearing a garment.
2. The original photo of the garment alone (like a catalog picture).
3. The edited version of the garment (e.g., now it's yellow leather).
4. The edited version of the person wearing that new garment.
5. The Instruction: A text note saying exactly what changed (e.g., "Change the color to yellow").
The Magic Trick (How they made it): They didn't ask humans to draw 146,000 pictures (that would take forever). Instead, they built an automated assembly line:
1. The Analyst (Qwen3-VL): A super-smart AI looks at a photo and writes a detailed description of the clothes (e.g., "Blue cotton shirt, short sleeves").
2. The Chef (FLUX.2): Another AI takes that description and follows a command like "Make it yellow" to create a new, edited photo of the shirt.
3. The Tailor (FitDiT): A third AI takes the new yellow shirt and "sews" it onto the person in the photo.
4. The Inspector (GPT-5 & InternVL): A final AI acts as a strict quality control manager. It checks: "Did the shirt actually turn yellow? Did the person's face stay the same? Is the zipper in the right place?" If the answer is no, the sample is thrown in the trash. If yes, it goes into the dataset.

3. The Two Main Games

With this new dataset, they set up two challenges for AI models to solve:

Virtual Try-On (The "Fitting Room"): You give the AI a person and a command ("Make the dress red"). The AI must show you the person wearing the red dress.
Virtual Try-Off (The "Catalog"): You give the AI a person wearing a dress and a command ("Remove the belt"). The AI must generate a clean photo of just the dress without the belt, as if it were a product photo for a store.

4. The New Star: Dress-EM

The authors didn't just build the dataset; they also built a new AI model called Dress-EM to play these games.

Think of Dress-EM as a bilingual fashion designer. It can read your English instructions ("Add a pocket") and look at the visual clues of the clothes at the same time.
It uses a "connector" to bridge the gap between language and images, ensuring that when you ask for a change, it happens exactly where you want it, without messing up the rest of the outfit.

5. Why This Matters

Before this, if you wanted to edit clothes in a photo, you had to use Photoshop or hope a generic AI guessed what you meant.

Dress-ED gives AI a massive, structured library of "before and after" examples with clear instructions.
Dress-EM proves that AI can now understand complex fashion commands, like changing the material from cotton to leather or adding a specific pattern, while keeping the person's face and pose perfectly natural.

In a nutshell: The authors built a massive, automated "fashion school" where AI learns to listen to your voice and instantly redesign clothes on people. They created the curriculum (the dataset) and the top student (the model) to show that the future of online shopping might soon let you say, "I want this dress in green velvet with long sleeves," and see it happen instantly.

1. Problem Statement

While recent advances in Virtual Try-On (VTON) and Virtual Try-Off (VTOFF) have improved photo-realistic fashion synthesis, existing approaches face significant limitations:

Static Datasets: Current datasets (e.g., VITON-HD, Dress Code) contain fixed person-garment pairs without support for instruction-level control or semantic editing.
Lack of Unified Framework: Existing solutions are fragmented. VTON datasets lack editability; general image editing datasets lack consistent garment-person correspondences; and fashion editing methods often address narrow sub-tasks without unified benchmarks.
Control Gap: There is a lack of large-scale, instruction-driven benchmarks that allow for fine-grained, controllable modifications (both appearance and structural) to garments while maintaining realistic human-garment interaction.

2. Methodology

The paper introduces Dress-ED, a large-scale dataset and a unified multimodal framework (Dress-EM) to address these gaps.

A. The Dress-ED Dataset

Dress-ED is the first benchmark unifying VTON, VTOFF, and text-guided garment editing. It contains 146,460 verified quadruplets spanning three garment categories (dresses, upper-body, lower-body) and seven editing types.

Data Structure: Each sample consists of:
1. $I_{garment}$ : In-shop garment image.
2. $I_{person}$ : Person wearing the garment.
3. $I_{garment}^{edit}$ : Edited in-shop garment.
4. $I_{person}^{edit}$ : Person wearing the edited garment.
5. $T_{inst}$ : Natural language instruction describing the modification.
Automated Generation Pipeline:
1. Attribute Extraction: Uses Qwen3-VL to extract structured garment attributes (color, pattern, material, structure) from the original pair.
2. Instruction Synthesis: Generates natural language instructions using rule-based templates based on the extracted attributes.
3. Garment Editing: Employs FLUX.2 Klein (a diffusion model) to synthesize the edited in-shop garment ( $I_{garment}^{edit}$ ) based on the instruction.
4. Try-On Generation: Uses FitDiT to render the edited garment onto the person, generating $I_{person}^{edit}$ .
5. Quality Verification: A multi-stage filtering process ensures semantic consistency and visual fidelity:
  - GPT-5 scores the edit adherence and quality (0–100).
  - InternVL-3.5 is fine-tuned on a subset of GPT-5 annotated data to act as a scalable verifier.
  - Samples with scores below a threshold (80) are filtered out.
Editing Taxonomy: The dataset covers 7 edit types grouped into two categories:
- Appearance: Change Color, Change Pattern, Change Material, Fine-Grained.
- Structure: Add Detail, Remove Element, Modify Structure.

B. The Proposed Model: Dress-EM

The authors propose Dress-EM (Dress Editing Model), a unified multimodal diffusion framework for instruction-driven editing in both VTON and VTOFF scenarios.

Architecture:
- MLLM Encoder: Uses InternVL-3.5 to jointly encode the visual context (garment/person) and the text instruction, producing fused token embeddings.
- Connector: A lightweight Transformer that bridges the MLLM feature space and the Diffusion Transformer (DiT). It splits the output into:
  - Global Guidance ( $g$ ): A high-level semantic vector (2048-dim) summarizing the edit intent.
  - Token Refinement ( $z_c$ ): Fine-grained token embeddings preserving specific details.
- Diffusion Transformer (DiT): Based on Stable Diffusion 3. It takes the VAE latent features of the input (person + garment for VTON; garment for VTOFF) and conditions the denoising process on both $g$ and $z_c$ .
Training Strategy: The VAE and MLLM are frozen. The Connector and DiT are trained end-to-end using a mixed supervision strategy (original and edited images as targets) to learn robust alignment with real-world textures while mastering structural modifications.

3. Key Contributions

Dress-ED Dataset: The first large-scale, semantically consistent benchmark unifying VTON, VTOFF, and instruction-driven editing with over 140k verified samples.
Automated Pipeline: A novel pipeline integrating MLLM-based understanding, diffusion-based synthesis, and LLM-guided verification to generate high-quality, instruction-grounded data without manual annotation.
Unified Baseline (Dress-EM): A state-of-the-art multimodal diffusion model that jointly reasons over linguistic instructions and visual cues, establishing a strong baseline for controllable fashion generation.
Comprehensive Benchmarks: Defined evaluation protocols for both paired/unpaired VTON and VTOFF, utilizing metrics for visual fidelity (FID, LPIPS, SSIM) and edit correctness (DINO-I).

4. Experimental Results

The model was evaluated on the Dress-ED test split (14,259 samples) against task-specific baselines (CatVTON, Any2AnyTryon) and general-purpose editors (FLUX.2, Qwen-Image-Edit).

Quantitative Performance:
- Dress-EM consistently outperformed all baselines across both Paired and Unpaired VTON settings, achieving the lowest FID (3.79), KID (1.13), and DISTS (0.0733) scores.
- In VTOFF, Dress-EM achieved the best results in reconstructing edited garments (FID 5.06, DINO-I 0.8071).
- Per-Edit Analysis: The model showed superior performance in both appearance edits (color/pattern) and complex structural edits (sleeve length, adding/removing elements), maintaining high DINO-I similarity scores (>0.94 in VTON).
Qualitative Results: Visual comparisons demonstrated that Dress-EM preserves person identity, pose, and background better than competitors while accurately following complex instructions (e.g., "change sleeves to long," "add a belt").

5. Significance

Bridging the Gap: Dress-ED fills a critical void in fashion AI by providing a unified framework for both synthesizing garments on humans (VTON) and reconstructing garments from humans (VTOFF) under instruction.
Controllable Generation: It shifts the paradigm from static synthesis to interactive, controllable fashion generation, enabling applications like virtual shopping with customization and digital fashion design.
Scalability: The fully automated pipeline demonstrates that high-quality, instruction-aligned datasets can be generated at scale using MLLMs and diffusion models, reducing reliance on expensive manual annotation.
Foundation for Future Research: By releasing the dataset and code, the authors provide a standardized benchmark to accelerate research in multimodal understanding, controllable diffusion models, and digital fashion.