💻 computer science

VTONQA: A Multi-Dimensional Quality Assessment Dataset for Virtual Try-on

This paper introduces VTONQA, the first multi-dimensional quality assessment dataset for virtual try-on containing 8,132 images and 24,396 human ratings across three dimensions, which is used to benchmark existing VTON models and image quality metrics to reveal their limitations and guide future improvements.

Original authors: Xinyi Wei, Sijing Wu, Zitong Xu, Yunhao Li, Huiyu Duan, Jia Wang, Ning Liu

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinyi Wei, Sijing Wu, Zitong Xu, Yunhao Li, Huiyu Duan, Jia Wang, Ning Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the bustling world of online shopping, a familiar frustration persists: the gap between a garment on a screen and how it might look on a real person. To bridge this gap, technology has developed virtual try-on systems, which act as digital fitting rooms. These tools take a photograph of a person and a separate image of a piece of clothing, then attempt to merge them into a single, realistic picture showing the person wearing that outfit. While these systems have become common in e-commerce and digital fashion, they are not perfect. Often, the results look strange; clothes might appear stretched or torn, body shapes can warp unnaturally, or the fabric might not drape correctly. For these tools to become truly useful, developers need a way to measure exactly how good or bad a result is, moving beyond simple pixel comparisons to understand what a human eye actually perceives as a successful try-on.

A team of researchers at Shanghai Jiao Tong University has addressed this need by creating a new resource called VTONQA, the first large-scale dataset designed specifically to judge the quality of these virtual try-on images. Instead of relying on a single overall score, the researchers broke down the evaluation into three distinct dimensions: how well the clothing fits the body, how naturally the body shape and pose remain consistent, and the overall aesthetic quality of the final image. To build this, they gathered 8,132 images generated by 11 different virtual try-on models, ranging from classic computer vision techniques to modern artificial intelligence systems. These images were created by pairing 184 different people—representing a diverse mix of ages, genders, ethnicities, and body types, including children, the elderly, and pregnant individuals—with 80 different garments across eight categories, from t-shirts and sweaters to dresses and trousers.

Once the images were generated, the researchers did not leave the judging to machines. They recruited 40 human volunteers to act as the ultimate arbiters of quality. These volunteers, who had backgrounds in undergraduate and graduate education, were trained to look at each image and assign a score from one to five for each of the three dimensions. They evaluated whether the shirt looked like it was actually being worn, if the person's arms and legs still looked like human limbs, and if the final picture felt pleasing to look at. This process resulted in nearly 24,400 individual scores, creating a rich, detailed map of how humans perceive success and failure in virtual try-on. The researchers then used this human data to test how well existing computer programs could predict these scores on their own.

The findings revealed a significant gap between what computers currently measure and what humans actually see. Traditional methods used to judge image quality, which often focus on how closely pixels match between two images or how much noise is present, failed to align with human opinion. These standard tools could not detect the subtle distortions that make a virtual outfit look fake, such as a sleeve twisting unnaturally or a waistline disappearing. Even more advanced artificial intelligence models, which had been trained on general images, struggled when faced with the specific challenges of virtual try-on. The study showed that while some modern systems performed better than others, none were perfect. The closed-source commercial systems tested generally outperformed the open-source research models, but even the best open-source models showed inconsistency, often doing well with upper-body clothing like shirts but struggling significantly with lower-body items like trousers and skirts.

One of the most revealing aspects of the study was how different types of clothing and different body types affected the results. The data suggested that current technology handles the contours of the upper body and full-body dresses more reliably than it handles the complex shapes of the lower body. Similarly, while the systems worked reasonably well across various body shapes and ethnicities, the performance varied enough to show that no single model is yet a universal solution. The researchers found that the overall quality of an image was often more dependent on whether the body looked natural than on whether the clothing fit perfectly, suggesting that preserving the human form is the most critical factor for a convincing result. By providing this detailed, human-verified dataset, the researchers have given the field a new standard for measurement. This resource allows developers to see exactly where their models fail, moving the technology from a trial-and-error phase toward a future where digital fitting rooms can offer a truly realistic and reliable experience for shoppers.

Technical Summary: VTONQA

Problem Statement

Image-based Virtual Try-on (VTON) has become a critical technology for e-commerce and digital fashion, yet current models frequently produce low-quality outputs characterized by artifacts such as garment distortion, blurred faces, and inconsistent body structures. While existing evaluation methods rely heavily on objective metrics like Fréchet Inception Distance (FID), Learned Perceptual Image Patch Similarity (LPIPS), and pixel-level criteria (SSIM, PSNR), these measures often exhibit weak correlation with human perception. Furthermore, existing Image Quality Assessment (IQA) datasets are designed for natural images or generic AI-generated content, failing to address the specific distortion types and reference information formats unique to VTON. Additionally, current evaluation protocols typically provide a single holistic score, which is insufficient to diagnose the diverse failure modes of VTON systems, such as the distinction between poor garment transfer and body incompatibility.

Methodology

To address these gaps, the authors constructed VTONQA, the first multi-dimensional quality assessment dataset specifically for VTON. The methodology encompasses three primary components:

1. Dataset Construction

Data Generation: The dataset comprises 8,132 synthesized images generated by 11 representative VTON models. These models span three categories: classical warp-based methods (e.g., VITON-HD, TPD), diffusion-based approaches (e.g., StableVITON, OOTDiffusion), and closed-source commercial systems (e.g., Kling, LinkFox).
Input Diversity: The synthesis process utilizes 184 reference person images covering nine demographic categories (Black, Caucasian, Asian, children, young, elderly, pregnant, men, women) and various poses. These are paired with 80 garment images across eight categories (T-shirt, shirt, sweater, shorts, trousers, maxiskirt, skirt, dress).
Annotation Protocol: A subjective experiment was conducted with 40 independent volunteers. Participants rated each image on a 1–5 scale (precision 0.1) across three distinct dimensions:
1. Clothing Fit: Accuracy and completeness of the garment transfer.
2. Body Compatibility: Physical consistency of the human body shape and pose post-try-on.
3. Overall Quality: Alignment with human aesthetic perception.
Data Processing: To ensure reliability, outlier detection was performed (removing scores deviating >2 $\sigma$ ), followed by within-subject Z-score normalization and linear mapping to a [0, 100] range. The final Mean Opinion Scores (MOS) were computed by averaging normalized scores.

2. Benchmarking Framework

The authors established a comprehensive benchmark to evaluate:

VTON Models: 11 representative algorithms were tested in an inference-only setting (no fine-tuning on VTONQA).
IQA Metrics: 17 existing IQA metrics were evaluated, including traditional full-reference (FR) and no-reference (NR) methods, as well as deep learning-based approaches. Both zero-shot evaluation and fine-tuned scenarios (where models were trained on VTONQA scores) were tested.

3. Evaluation Metrics

Consistency between predicted scores and ground-truth MOS was measured using Spearman's Rank Correlation Coefficient (SRCC), Kendall's Rank Correlation Coefficient (KRCC), and Pearson Linear Correlation Coefficient (PLCC).

Key Results

Dataset Characteristics

The dataset contains 24,396 MOS annotations across the three dimensions.
Inter-annotator agreement analysis demonstrated high consistency across all dimensions (SRCC/PLCC ranging from ~0.73 to ~0.79).
Dimensional Trends: Analysis revealed that current models generally perform better in body compatibility (preserving pose/structure) than in clothing fit (accurate alignment). Lower-body garments (trousers, shorts) present significantly more challenges than upper-body or full-body garments.

Model Performance

Closed-source systems (Kling, LinkFox) achieved the highest and most stable performance across all categories.
Among open-source methods, diffusion-based models (e.g., StableVITON) generally outperformed classical warp-based methods (e.g., FS-VTON) and showed greater robustness across diverse scenarios.
DS-VTON emerged as the top performer among classical warp-based methods.

IQA Metric Performance

Traditional Metrics: Standard pixel-level metrics (MSE, PSNR, SSIM) and traditional NR-IQA methods showed poor correlation with human judgments, particularly for the "Clothing Fit" dimension.
Deep Learning Metrics: Deep learning-based NR-IQA methods, particularly MANIQA, achieved the strongest correlations with human scores.
Fine-tuning Impact: Models fine-tuned on VTONQA demonstrated significantly improved correlation with human judgments compared to their zero-shot counterparts, validating the utility of the dataset for training perceptual evaluators.

Significance and Contributions

The paper positions VTONQA as a foundational resource for the VTON community, offering three main contributions:

First Comprehensive Subjective Study: It provides the first large-scale analysis of human perceptual preferences across diverse VTON scenarios, highlighting specific failure modes in garment transfer and body interaction.
Multi-Dimensional Dataset: By moving beyond a single holistic score, VTONQA enables fine-grained evaluation through its three distinct dimensions (Clothing Fit, Body Compatibility, Overall Quality), facilitating more targeted model improvements.
Benchmarking and Insights: The extensive benchmarking of 11 VTON models and 17 IQA metrics reveals the limitations of current objective metrics and underscores the necessity of subjective supervision for developing perceptually aligned evaluation methods.

The authors conclude that VTONQA will serve as a solid foundation for advancing both quality assessment methodologies and the development of more robust, realistic VTON models. The dataset is publicly available to support further research.