MV-Fashion: Towards Enabling Virtual Try-On and Size Estimation with Multi-View Paired Data

This paper introduces MV-Fashion, a large-scale multi-view video dataset featuring 3,273 sequences with pixel-level annotations, ground-truth material properties, and paired flat/worn garment images, designed to bridge the realism and annotation gaps in existing datasets for virtual try-on and size estimation tasks.

The Gist

Imagine you're trying to buy a coat online. You see a picture of a model wearing it, but you have no idea how it will look on you. Will the sleeves be too long? Will the shoulders feel tight? Will it look good if you roll up the sleeves or leave the jacket open?

Currently, the internet is like a giant library where some books only show the cover (the flat photo of the coat), and others only show the inside pages (3D models of people moving), but no single book has both. This makes it incredibly hard for computers to learn how to "try on" clothes virtually or tell you your perfect size.

Enter MV-Fashion, a massive new project by researchers that is like building the ultimate "fashion simulator" to fix this problem.

Here is a breakdown of what they did, using some everyday analogies:

1. The Problem: The "Missing Link"

Think of existing fashion data as two separate worlds:

• World A (The Catalog): Flat, pretty photos of clothes on a hanger. Great for seeing the design, but useless for seeing how the fabric moves.
• World B (The Runway): Videos of models walking and dancing in 3D. Great for seeing movement, but they don't have the "flat" photo of the specific item to compare against.

Because these worlds don't talk to each other, computers can't learn the rules of how fabric stretches, folds, or fits different body types.

2. The Solution: The "Fashion Time Machine"

The researchers built MV-Fashion, a giant dataset that acts as a bridge between these two worlds.

• The Setup: Imagine a room with 68 cameras (like a security system on steroids) arranged in a circle around a person.
• The Actors: They filmed 80 different people wearing 474 different outfits.
• The Magic: For every single outfit, they did two things:
  1. They took a flat, perfect photo of the clothes (like a catalog).
  2. They filmed the person wearing those exact clothes moving, posing, and even layering them (like wearing a t-shirt under a jacket).

It's like having a "before and after" photo for every single piece of clothing, but the "after" is a 3D movie of a real human wearing it.

3. What Makes It Special? (The "Secret Sauce")

Most datasets are boring or fake. MV-Fashion is special because it captures the messy reality of fashion:

• Layering: It shows what happens when you wear a sweater under a coat.
• Styling: It captures the difference between a shirt tucked in vs. hanging loose, or sleeves rolled up vs. down.
• The "Elasticity" Score: They didn't just film the clothes; they measured them. They noted if a fabric is stiff (like denim) or stretchy (like a yoga pant), giving the computer a "physics textbook" for how that specific fabric should behave.

4. What Can We Do With It?

The researchers used this dataset to teach computers three new tricks:

• Virtual Try-On (The "Magic Mirror"):
  Imagine uploading a photo of your body and a photo of a dress you like. The computer uses MV-Fashion to learn how that dress would drape over your specific shoulders and hips, not just the model's. It's like a magic mirror that shows you the truth before you buy.
• Size Estimation (The "Crystal Ball"):
  Instead of guessing if you are a "Medium" or "Large," the computer can look at a photo of you in a shirt and calculate the exact measurements (in centimeters) of the garment. It's like having a tailor who can measure you through a webcam.
• New View Synthesis (The "360-Degree Spin"):
  If you only have a photo of a person from the front, the computer can use this data to generate a realistic photo of them from the back or side, even if the camera never actually took that picture. It's like filling in the missing pieces of a puzzle.

5. Why Does This Matter?

• For You: Fewer returns! If you know exactly how a shirt fits and looks on your body before buying, you won't waste money on clothes that don't fit.
• For the Planet: Returns create a huge amount of waste (clothes often get thrown away). Better fitting clothes mean fewer returns and less trash.
• For Designers: They can simulate how a new design will look on thousands of different body types without needing to sew a single physical prototype.

The Bottom Line

MV-Fashion is like giving computers a "fashion degree." By feeding them millions of frames of real people wearing real clothes in every possible configuration, the researchers have built the foundation for a future where online shopping feels as real as walking into a store. It's not just about looking cool; it's about making the digital world fit the real world perfectly.

Technical Summary

Here is a detailed technical summary of the paper "MV-Fashion: Towards Enabling Virtual Try-On and Size Estimation with Multi-View Paired Data."

1. Problem Statement

The field of fashion-centric computer vision faces a critical data gap. Existing datasets are fragmented and insufficient for advanced tasks like Virtual Try-On (VTON), clothing size estimation, and dynamic garment simulation:

• 2D VTON Datasets: While they provide paired "catalogue-to-worn" images (essential for VTON), they lack 3D geometric data, multi-view synchronization, and dynamic motion, limiting them to static, single-view synthesis.
• 3D/4D Human Datasets: These capture realistic geometry and motion (e.g., Human3.6M, MVHumanNet++) but often lack the specific "paired" data (flat garment images) required for VTON and miss fine-grained fashion annotations like material properties, elasticity, and detailed sizing charts.
• Synthetic Datasets: While offering perfect ground truth, they suffer from a "reality gap," failing to capture complex real-world garment dynamics such as multi-layering, draping, and fabric elasticity.

Consequently, there is no single dataset that bridges the gap between multi-view 4D capture and domain-specific fashion analysis (paired catalogue data, material properties, and complex styling).

2. Methodology: The MV-Fashion Dataset

The authors introduce MV-Fashion, a large-scale, multi-view video dataset engineered to address these limitations.

A. Data Acquisition Setup

• Hardware: A custom hybrid capture rig consisting of 60 Raspberry Pi global shutter cameras (1.6 MP, RGB) and 8 Orbbec Femto Bolt cameras (4K RGB + Depth).
• Synchronization: All cameras are synchronized via an external electronic trigger (PWM signal) achieving sub-2ms alignment.
• Lighting: 40 high-power LED panels ensure consistent, diffused illumination to minimize motion blur.
• Protocol: The dataset includes three sequence types for each subject:
  1. Body Sequence: Minimal clothing for body shape estimation.
  2. Template Sequence: Static poses to capture garment geometry.
  3. Motion Sequence: Dynamic poses (5 random fashion poses) to model garment deformation.
• Scale: 80 diverse subjects, 3,273 sequences, 474 unique outfit sets (754 garment pieces), totaling 72.5 million frames.

B. Key Data Features & Annotations

MV-Fashion distinguishes itself through rich, multi-modal annotations:

• Paired Data: Every multi-view video sequence is paired with flat/catalogue images of the garments (front and back views), enabling VTON tasks.
• Complex Styling: Includes multi-layered outfits (up to 3 layers) and diverse styling variations (e.g., rolled sleeves, tucked shirts, open/closed jackets).
• Pixel-Level Segmentation: Foreground masks and layered garment segmentation (separating inner/outer layers) generated via a pipeline using YOLOv8, Qwen3-VL, and SAM2.
• 3D Geometry: 3D point clouds recovered from depth sensors and SMPL-X body parameters (pose and shape).
• Garment Attributes:
  • Sizing Charts: Detailed physical measurements (19 distinct parts) for 6 garment groups.
  • Material Properties: Manually annotated fabric types and elasticity (scale 1–5).
  • Draping Styles: Categorical labels for fit (slim, regular, loose) and styling (tucked, rolled, etc.).
  • Textual Descriptions: Generated by Large Language Models (Qwen3-VL) based on garment images.

3. Key Contributions

1. MV-Fashion Dataset: The first large-scale, multi-view video dataset specifically designed for fashion, featuring paired catalogue-to-worn data, 474 outfit sets, and 72.5M frames.
2. Complex Capture Scenarios: Unique inclusion of multi-layered outfits and diverse styling configurations (e.g., rolled sleeves, layered jackets) to test garment dynamics.
3. Rich Multi-Modal Annotations: Provides ground truth for material elasticity, detailed sizing charts, pixel-level segmentation, and 3D point clouds, which are rare in existing fashion datasets.
4. Paired Flat-to-Worn Data: Explicitly links static catalogue images with dynamic multi-view video, unlocking new avenues for VTON and size estimation.
5. Baseline Benchmarks: Established state-of-the-art baselines for Virtual Try-On, Size Estimation, and Novel View Synthesis (NVS).

4. Experimental Results & Baselines

The authors evaluated three core tasks using the dataset:

A. Virtual Try-On (VTON)

• Models: Tested IDM-VTON (diffusion-based) and InsertAnything (flow-matching based).
• Single-View: Both models achieved high-fidelity results (SSIM ~0.88–0.92, FID ~8–10) on the frontal-paired subset, proving compatibility with existing pipelines.
• Semantic Controllability: By augmenting prompts with styling annotations (e.g., "jacket fully open"), the model demonstrated the ability to follow fine-grained instructions, though subtle artifacts remain.
• Multi-View Geometric Analysis: A "View-Adaptive Try-On" experiment showed that using both frontal and rear catalogue images reduces geometric inconsistencies compared to using only frontal images.

B. Size Estimation

• Approach: Adapted SPnet to predict garment sewing parameters (and thus measurements) from posed images.
• Method: A two-stage pipeline: (1) A normal predictor ($\Psi$) disentangles pose from garment geometry to generate a canonical normal map; (2) A size regressor ($\Phi$) predicts measurements from the canonical map.
• Results: The Multi-Task + SwinV2 model (conditioned on target pose) achieved the best performance with a Mean Absolute Error (MAE) of 4.279 cm. This validates that real-world data with MV-Fashion's annotations is sufficient for accurate size estimation without synthetic data.

C. Novel View Synthesis (NVS)

• Models: Benchmarked Instant-NGP, Nerfacto, and 3D Gaussian Splatting (Splatfacto).
• Results: All methods achieved high-fidelity reconstructions. 3D Gaussian Splatting performed best, preserving fine textures and wrinkles.
• Ablation: Performance saturated at 56 views, suggesting the 60-camera setup is optimal for balancing fidelity and complexity.

5. Significance and Impact

• Bridging the Gap: MV-Fashion is the first dataset to simultaneously provide the geometric richness of 4D capture and the paired data structure required for commercial VTON applications.
• Sustainability & E-commerce: By enabling accurate size estimation and virtual try-on, the dataset supports technologies that can reduce return rates and physical sampling, addressing environmental concerns in the fashion industry.
• Research Foundation: The dataset and baselines provide a robust testbed for future research in garment physics, dynamic simulation, and 3D fashion generation.
• Accessibility: The dataset and code are publicly released to foster community-driven progress in digital human modeling.

In conclusion, MV-Fashion represents a significant leap forward in fashion AI, moving beyond static 2D analysis to a comprehensive, multi-view, and physically grounded understanding of clothing dynamics and fit.