💻 computer science

Garments2Look: A Multi-Reference Dataset for High-Fidelity Outfit-Level Virtual Try-On with Clothing and Accessories

This paper introduces Garments2Look, the first large-scale multimodal dataset comprising 80K outfit-level virtual try-on pairs with multiple garments and accessories, alongside a synthesis pipeline and baseline evaluations that reveal current methods struggle with the complexities of seamless multi-item styling and layering.

Original authors: Junyao Hu, Zhongwei Cheng, Waikeung Wong, Xingxing Zou

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junyao Hu, Zhongwei Cheng, Waikeung Wong, Xingxing Zou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "One-Item" Wardrobe

Imagine you go to a virtual dressing room app. You pick out a shirt, and the app puts it on you. Great! But then you want to add a jacket, a scarf, a belt, and a pair of cool sunglasses. Suddenly, the app gets confused. It might make the jacket disappear, turn the scarf into a weird blob, or forget that the jacket goes over the shirt.

Existing virtual try-on tools are like single-item chefs. They are great at cooking one perfect steak (a single shirt), but they struggle to cook a full, complex banquet (a whole outfit with layers and accessories). They don't understand how clothes interact, how to layer them correctly, or how to style them (like tucking a shirt in or rolling up sleeves).

The Solution: Garments2Look

The researchers created Garments2Look, which is like building a massive, super-detailed fashion library to teach computers how to dress a person properly.

Think of it this way:

Old Datasets: Like a library that only has books about "Shirts."
Garments2Look: A massive library with 80,000 "Lookbooks." Each entry doesn't just show a shirt; it shows a complete outfit (3 to 12 items) including shoes, bags, and jewelry.

How They Built It: The "Fashion Architect" Pipeline

They didn't just take photos from the internet; they built a sophisticated factory to create this data. Here is the process:

The "Style Guide" (The Knowledge Base):
Imagine hiring 100 fashion experts to write a rulebook for every style imaginable (from "Y2K Punk" to "Minimalist Office"). They defined what goes with what, what colors work, and what rules to break.
The "AI Stylist" (The Synthesis):
They used a smart AI (like a creative writer) to imagine a scenario: "A woman going to a business casual dinner in autumn." The AI then picks a shirt, pants, shoes, and a bag that fit that specific story.
The "Virtual Mannequin" (The Try-On):
Instead of just pasting the clothes onto a person (which looks fake), they used advanced AI to generate a brand-new photo of a model wearing that exact combination.
The "Fashion Police" (The Filtering):
This is crucial. They had real human fashion experts review the AI's work. If the AI made the jacket look like it was melting into the shirt, or if the shoes didn't match the pants, they threw it away. Only the best 40% of the AI's creations made the cut.

What Makes It Special? (The Secret Sauce)

Most datasets just show a picture of a shirt and a picture of a model. Garments2Look adds three superpowers:

The "Layering Map":
It tells the computer exactly what goes on top of what.
- Analogy: It's like giving a builder a blueprint that says, "Put the brick wall inside the frame, not outside." It knows a thin sweater goes under a coat, not over it.
The "Styling Instructions":
It includes text instructions like "tuck the shirt in," "roll up the sleeves," or "drape the scarf over the shoulder."
- Analogy: It's the difference between a robot that just puts a hat on your head and a stylist who knows to tilt the hat slightly for a cool look.
The "Accessory Party":
It includes bags, belts, glasses, and jewelry, teaching the AI that these items are part of the outfit, not just background noise.

The Reality Check: Testing the Computers

The researchers took the best AI models in the world and tried to make them use this new library. The results were a bit of a wake-up call:

The Struggle: Even the smartest AIs struggled. When asked to put on 5 or 6 items at once, they often dropped an item, messed up the layers, or made the clothes look like they were melting together.
The Insight: The study found that text is key. When they gave the AI the outfit and the written instructions (e.g., "tuck the shirt"), the results got much better. It proved that computers need to "read" the fashion rules, not just "see" the pictures.

Why Should You Care?

This paper is a huge step forward for the future of online shopping and fashion.

For Shoppers: Imagine an app where you can upload a photo of your entire closet, pick a "vibe" (like "Date Night"), and see a realistic photo of yourself wearing a perfectly coordinated outfit with the right layers and accessories.
For Designers: It helps computers understand the complex logic of fashion, leading to better design tools and more realistic virtual fashion shows.

In a nutshell: Garments2Look is the first massive training manual that teaches computers that fashion isn't just about putting a shirt on a body; it's about the art of layering, styling, and putting together a complete look. It's the difference between a robot that can tie a shoe and a robot that can be a personal stylist.

1. Problem Statement

Virtual Try-On (VTON) has advanced significantly for single-garment visualization. However, real-world fashion involves complex outfits comprising multiple garments, accessories, fine-grained categories, layering, and specific styling techniques (e.g., tucking in, draping).

Limitations of Existing Data: Current datasets (e.g., VITON-HD, DressCode) are primarily designed for single-item or limited multi-item try-on. They lack:
- Outfit-level diversity: They rarely support 3+ items simultaneously.
- Accessories: They often ignore shoes, bags, and jewelry.
- Structured Annotations: They lack textual descriptions of layering order (inner vs. outer) and styling techniques (e.g., "rolled sleeves," "tucked-in").
- Complexity: Existing methods struggle with occlusion, correct layering logic, and maintaining consistency across multiple items.

2. Methodology: Garments2Look Dataset Construction

The authors propose Garments2Look, the first large-scale multimodal dataset for outfit-level VTON. The construction process involves four key stages:

A. Data Collection & Categorization

The dataset aggregates data from four streams:

Gold Standard: Real-world paired images (garments + model wearing them).
Paired Outfits: Garment lists without the final model image.
Unpaired Garments: Raw garment images.
Look Images: Model images without reference garments.

Composition: The final dataset contains 80,041 outfit pairs (approx. 50% real, 50% synthetic) covering 40 major categories and 300+ fine-grained subcategories. Each pair includes 3–12 reference images (avg. 4.48).

B. Data Synthesis Pipeline

To overcome data scarcity and ensure diversity, the authors developed a synthesis pipeline:

Outfit Synthesis (Retrieval-Augmented Generation):
- A Style Knowledge Base (65 styles, e.g., Y2K, Minimalist) is built using LLMs and refined by fashion experts.
- An LLM generates a user context (demographics, occasion) and a specific outfit list (3–9 items) based on the style.
- Re-weighted Sampling: To avoid bias toward popular items, a retrieval mechanism selects items based on an inverse frequency strategy (rare items get higher selection probability).
Look Synthesis:
- Instead of feeding individual images directly, the system arranges the outfit list into an OOTD (Outfit of the Day) grid image.
- Advanced image generation models (e.g., Nano Banana/Gemini-2.5-Flash-Image) synthesize the final "Look" image using the OOTD grid as a unified reference.
- Prompt Engineering: Prompts explicitly specify layering order and styling techniques (e.g., "tucked-in," "draped over shoulders") to guide the generation.

C. Data Filtering & Quality Control

Automated Filtering: Uses vision models (Gemini-2.5-Flash, DWPose) to filter for full garment visibility and frontal model views.
Human Validation: 13 fashion experts review the data. They validate:
- Item Consistency: Does the label match the image?
- Outfit Plausibility: Are there logical errors (e.g., two dresses worn simultaneously)?
- Look Quality: Are the layering and styling accurate?
Result: Only ~40% of synthetic images pass the rigorous expert review.

D. Data Annotation

The dataset includes rich textual annotations generated by Vision-Language Models (VLMs) and verified by experts:

Item-level: Descriptions of material, texture, and style.
Outfit-level: Overall description, color palette, and theme.
Structural: Explicit Layering Order (Inner $\to$ Outer) and Styling Techniques (e.g., "unbuttoned," "cinched").

3. Key Contributions

First Large-Scale Outfit-Level Dataset: Garments2Look is the first dataset to support high-fidelity try-on for full outfits with 3–12 items, including accessories, across 40 categories.
Structured Multimodal Annotations: It introduces a new task format that leverages structured text (layering logic, styling rules) to guide generation, bridging the gap between visual inputs and complex outfit semantics.
Comprehensive Benchmark: The paper establishes baselines by testing SOTA VTON models (e.g., OmniTry, FastFit) and general-purpose image editing models (e.g., GPT-4o, Nano Banana) on this challenging task.

4. Experimental Results & Analysis

The authors evaluated existing models on Garments2Look to reveal current limitations:

Performance Gap: Current SOTA VTON models perform poorly on outfit-level tasks. They struggle with:
- Item Count: Models often drop items or fail to render more than 2–3 layers.
- Consistency: Significant shape distortion, texture loss, and color deviation occur as the number of reference items increases.
- Layering & Styling: Models fail to correctly infer inner-outer relationships or apply specific styling (e.g., keeping a shirt untucked).
Editing Models vs. VTON Models: Surprisingly, general-purpose image editing models (like Nano Banana) often outperform specialized VTON models on this dataset.
- Reason: Editing models are more flexible with variable-length inputs and OOTD grids, whereas specialized VTON models are often constrained to fixed label sets or single-layer logic.
Impact of Text: An ablation study (Table 4) proves that providing structured textual annotations (layering order, styling techniques) significantly improves generation quality (lower FID, higher SSIM). Text acts as essential semantic guidance for complex outfit synthesis.
Pose Control: Adding skeleton images as references did not significantly improve editing models, suggesting they struggle to integrate different reference types (pose vs. object) simultaneously without specific training.

5. Significance & Future Impact

Bridging the Gap: Garments2Look addresses the critical disconnect between academic VTON research (single-item focus) and real-world e-commerce needs (full outfit visualization).
New Research Direction: It highlights that future VTON systems must move beyond purely visual frameworks to multimodal approaches that integrate structured text (layering, styling) for precise control.
Open Source: The dataset and code are open-sourced, providing a standardized benchmark to drive progress in high-fidelity, multi-reference virtual try-on.

In conclusion, Garments2Look demonstrates that while current AI struggles with the complexity of full-outfit try-on, the combination of high-quality data, structured textual annotations, and advanced editing models offers a viable path forward for realistic fashion visualization.