Virtual Try-On for Cultural Clothing: A Benchmarking Study

This paper introduces BD-VITON, a new benchmark dataset featuring culturally diverse Bangladeshi garments with complex draping and layering challenges, and evaluates the performance of state-of-the-art virtual try-on models on this dataset to demonstrate significant improvements over zero-shot inference.

The Gist

Imagine you have a magical mirror. You hold up a picture of a beautiful dress, and the mirror instantly shows you wearing it, looking perfect. This is the dream of Virtual Try-On (VTO) technology, a tool that lets online shoppers see how clothes look on them without ever stepping into a fitting room.

However, until now, this magic mirror has had a very specific blind spot. It was trained almost exclusively on Western fashion (like t-shirts, jeans, and fitted dresses) and mostly on female models. If you tried to use it to see how a traditional South Asian outfit looked, the mirror would get confused, often producing weird, distorted results.

This paper introduces a solution: BD-VITON. Think of it as a "cultural upgrade" for the magic mirror.

Here is the breakdown of their work in simple terms:

1. The Problem: The "Western" Blind Spot

The existing AI models are like chefs who only know how to cook burgers and fries. They are experts at handling simple, structured clothes (like a button-down shirt). But when you ask them to "cook" a Saree (a long, unstitched cloth draped in complex folds), a Panjabi (a loose tunic), or a Salwar Kameez, they struggle.

• The Analogy: Imagine trying to fold a fitted sheet. It's hard enough. Now imagine trying to fold a giant, flowing river of fabric that wraps around your body in intricate layers. That is the challenge of traditional Bangladeshi clothing. The old AI models tried to force these complex fabrics into the shape of a simple t-shirt, resulting in a mess.

2. The Solution: Building a New Library (BD-VITON)

The researchers realized they couldn't just blame the AI; they needed to teach it new things. They built a brand new dataset called BD-VITON.

• What is it? A collection of 1,013 photos featuring real Bangladeshi people wearing traditional clothes (Sarees, Panjabis, and Kameez).
• Why is it special? It includes both men and women (unlike previous datasets that mostly had women) and features full-body shots, not just upper-body shots.
• The "Training" Process: They didn't just dump photos in. They used smart computer programs to "label" every part of the image—identifying where the arm is, where the fabric starts, and how the folds look. This is like giving the AI a detailed map before sending it on a journey.

3. The Experiment: Teaching the Old Dogs New Tricks

The team took three of the most advanced AI models currently available (VITON-HD, HR-VITON, and StableVITON) and gave them a crash course using their new BD-VITON dataset.

• Before Training (Zero-Shot): They asked the AI to try on a Saree without any prior learning.
  • Result: The AI looked confused. It tried to wrap the Saree like a tight Western dress, ignoring the folds and the way the fabric actually hangs.
• After Training: They let the AI study the BD-VITON photos.
  • Result: The AI learned the "rules" of drape and fold. It started understanding that a Saree flows differently than a shirt. The results became much more realistic and respectful of the clothing's structure.

4. The Results: A Clear Win

The study showed that when you train an AI on culturally specific data, it gets significantly better at handling that specific culture's clothes.

• The Metaphor: It's like teaching a driver who only knows how to drive on smooth, straight highways (Western clothes) how to drive on winding, hilly mountain roads (South Asian clothes). Once they practice on the mountain roads, they don't just get better at the mountains; they become a better, more adaptable driver overall.
• The Outcome: The models trained on BD-VITON produced images that looked much more natural. The fabrics draped correctly, the folds looked real, and the clothes fit the body's shape properly.

5. Why This Matters

This isn't just about making pretty pictures. It's about inclusion.

• There are billions of people in South Asia who wear these traditional clothes.
• Currently, if they want to shop online, the "virtual fitting room" doesn't work for them.
• By proving that these AI models can be taught to handle complex, cultural clothing, the researchers are opening the door for a more inclusive digital fashion world.

The Bottom Line

The paper argues that you can't expect a tool built for one culture to work perfectly for another without retraining it. By creating a specialized dataset (BD-VITON) and teaching the AI the unique "language" of Bangladeshi fashion, they successfully upgraded the technology to serve a much wider, more diverse audience.

In short: They taught the AI that not all clothes are like T-shirts. Some are like flowing rivers, and now the AI knows how to handle the flow.

Technical Summary

Here is a detailed technical summary of the paper "Virtual Try-On for Cultural Clothing: A Benchmarking Study".

1. Problem Statement

Current Virtual Try-On (VTON) systems, driven by diffusion models and generative AI, have achieved significant photorealism. However, they suffer from a critical cultural and structural bias:

• Dataset Limitation: Existing benchmarks (e.g., VITON, VITON-HD, DressCode) are dominated by Western-style clothing (shirts, tops) and female models.
• Generalization Failure: Models trained on these datasets fail to generalize to culturally diverse garments with complex topologies, such as draping, asymmetric layering, and high deformation.
• Specific Gap: Traditional South Asian attire (e.g., Saree, Panjabi, Salwar Kameez) involves intricate folding and continuous draping that standard VTON architectures cannot handle effectively, rendering existing systems ineffective for the 2.4 billion people in South Asia.

2. Methodology

A. Dataset Construction: BD-VITON

The authors introduced BD-VITON, a new benchmark dataset specifically for Bangladeshi traditional attire.

• Composition: Contains 1,013 paired images (810 for training, 203 for testing) covering three garment types:
  • Saree: Unstitched continuous drape (Female).
  • Panjabi: Collarless tunic (Male).
  • Kameez: Long tailored garment (Female).
• Demographics: Includes both male and female subjects, addressing the gender imbalance in previous datasets.
• Resolution: High-resolution images (median 1080×1440), suitable for detailed texture synthesis.
• Annotation Pipeline: The authors developed a fully automated pipeline to generate necessary inputs for VTON models:
  1. Semantic Human Parsing: Using SCHP and FASHN parsers to create dense segmentation maps.
  2. Parse-Agnostic Representation: Removing garment and arm regions to prevent identity copying while preserving pose.
  3. Cloth Mask & Extraction: Isolating garment-only images from the original photos.
  4. Agnostic Mask Generation: Using morphological operations (convex hull, dilation) to create smooth silhouettes that remove fabric texture but preserve pose.
  5. Pose Estimation: Extracting 2D keypoints (OpenPose) and DensePose maps (24-part body surface correspondence) to guide garment warping.

B. Benchmarking Framework

The study retrained and evaluated three state-of-the-art VTON architectures on BD-VITON:

1. VITON-HD: A GAN-based model using Thin-Plate Spline (TPS) for geometric alignment and an ALIAS generator for synthesis.
   • Adaptation: Custom training script implemented using transfer learning from official checkpoints.
2. HR-VITON: A two-stage model (Condition Generator + Image Generator) that uses target human parsing maps to guide warping.
   • Adaptation: Addressed a "copy-paste" failure mode where the model simply pasted the cloth image onto the person. The authors introduced Gaussian noise on cloth edges and rotational augmentation to force the model to learn genuine warping rather than memorization.
3. StableVITON: A diffusion-based model using latent feature representations and cross-attention to warp cloth.
   • Adaptation: Required no fundamental architectural changes, only fine-tuning on the new data.

3. Key Contributions

1. BD-VITON Dataset: The first large-scale dataset focused on South Asian traditional clothing, introducing complex structural challenges (folds, drapes) absent in Western-centric datasets.
2. Empirical Evidence of Distributional Mismatch: Demonstrated that even with unaltered model architectures, in-distribution training on culturally specific data significantly outperforms zero-shot inference.
3. Benchmarking Baselines: Established new performance baselines for VITON-HD, HR-VITON, and StableVITON on complex, non-Western garments.
4. Technical Adaptations: Proposed specific data augmentation strategies (noise injection, rotation) to prevent overfitting and copy-paste artifacts in HR-VITON when dealing with pseudo-cloth inputs.

4. Results

Quantitative Analysis

The study evaluated models using SSIM (Structural Similarity), LPIPS (Perceptual Similarity), and FID (Fréchet Inception Distance).

• Training vs. Zero-Shot: All three models showed consistent improvements across all metrics after fine-tuning on BD-VITON compared to zero-shot inference.
  • Example: HR-VITON's FID improved from 259.25 (Zero-Shot) to 42.92 (Trained).
  • Example: VITON-HD's FID improved from 100.07 to 49.89.
• Model Performance:
  • VITON-HD and HR-VITON achieved the best results after training, showing similar performance levels.
  • StableVITON showed the lowest performance among the trained models. The authors attribute this to limited training epochs (only 5 due to resource constraints) and the attention layers requiring longer convergence times.

Qualitative Analysis

• StableVITON: In zero-shot, it tended to alter garment structures to look "Western." After training, results were more coherent but retained some color inconsistencies.
• HR-VITON: Struggled with deformation in zero-shot (copy-paste behavior). Training mitigated this, but misalignment issues persisted compared to VITON-HD.
• VITON-HD: Produced the most stable outputs due to its reliance on TPS-based warping, avoiding the severe deformation failures seen in HR-VITON, though it lacked the fine detail of diffusion-based models.

5. Significance and Future Work

• Significance: This work proves that the "Western bias" in VTON datasets is a primary bottleneck for global applicability. It establishes that culturally specific training data is essential for models to handle complex garment topologies, even without changing the underlying model architecture.
• Future Directions:
  • Diversity Expansion: Plan to include other complex cultural garments (e.g., Japanese Kimono, Chinese Hanfu).
  • Scale Increase: The current dataset size (1,013 samples) is too small for training generative models from scratch; the authors aim to expand the dataset to prevent overfitting and enable full model training.

In conclusion, BD-VITON serves as a critical benchmark for evaluating the generalization capabilities of VTON systems, highlighting the necessity of diverse, culturally representative data to achieve robust performance in real-world global markets.