Mobile-VTON: High-Fidelity On-Device Virtual Try-On

Mobile-VTON is a high-fidelity, privacy-preserving framework that enables fully offline virtual try-on on commodity mobile devices by utilizing a modular TGT architecture with feature-guided adversarial distillation and trajectory-consistency training to match server-based performance without requiring cloud computing.

The Gist

Imagine you are shopping for a new shirt online. Usually, you have to guess if it will look good on you, or you have to upload your photo to a massive, cloud-based computer server to see a digital simulation. This raises two big problems: privacy (you don't want your photo stored on a stranger's server) and speed (waiting for the server to process the image takes time).

The paper "MOBILE-VTON" introduces a solution that solves both problems. It's like having a personal fashion stylist living inside your phone that works entirely offline, without ever sending your data to the internet.

Here is a breakdown of how they did it, using simple analogies:

1. The Big Problem: The "Heavy" vs. The "Light"

Most high-quality virtual try-on systems are like giant, heavy trucks. They are incredibly powerful and can do amazing things, but they are too big to fit in a small car (your mobile phone). They require massive cloud servers to run.

The researchers wanted to build a sleek, high-speed sports car that fits in your pocket but can still drive just as well as the truck.

2. The Solution: The "Teacher-Student" System

To shrink the giant truck down to a sports car, they used a clever Teacher-Student approach (called the TGT architecture):

• The Teacher (TeacherNet): Imagine a world-famous, master chef who has cooked every dish in the world. This chef is huge, has a massive kitchen, and knows exactly how to make a perfect virtual try-on. However, this chef is too big to fit in your phone.
• The Student (Light-UNets): This is a young, talented apprentice chef. They are small and fast, perfect for a tiny kitchen (your phone), but they don't have the master's experience yet.

The Magic Trick (Distillation): Instead of the student trying to learn everything from scratch, the Master Chef stands right next to the student and whispers instructions. The student doesn't just copy the final dish; they learn how the Master thinks and moves. This allows the small student to produce results that look almost as good as the Master, but using a tiny fraction of the energy.

3. The Three Special Tools

To make this work perfectly on a phone, they added three specific "tools" to the student's kit:

A. The "Steady Hand" (Trajectory-Consistent GarmentNet)

The Problem: When you try to put a shirt on a digital body, sometimes the shirt's pattern gets blurry or shifts as the computer processes the image step-by-step. It's like trying to draw a straight line while your hand is shaking.
The Fix: They trained the "GarmentNet" to be a steady hand. They taught it to look at the shirt at every single step of the drawing process and ensure the pattern (like stripes or logos) stays in the exact same place. This prevents the shirt from looking like a melted mess.

B. The "Double-Check" (Adversarial Learning)

The Problem: Sometimes, a computer-generated image looks "too smooth" or fake, like a plastic mannequin.
The Fix: They introduced a skeptical art critic (a discriminator). The student tries to create a realistic image, and the critic tries to spot the fake one. If the critic says, "That fabric looks plastic!" the student tries again. They keep playing this game until the student creates an image so real that even the critic can't tell the difference.

C. The "Direct Line" (Latent Concatenation)

The Problem: Usually, computers need to be pre-trained on millions of images before they can understand how clothes fit. The researchers wanted to skip this huge, expensive training step.
The Fix: Instead of guessing, they simply glued the person's photo and the shirt's photo together side-by-side and fed them directly into the system. It's like showing the student, "Here is the body, here is the shirt, now figure out how they fit." This direct connection helps the phone understand the alignment perfectly without needing a massive database of prior knowledge.

4. The Result: Why It Matters

The final result is a system called MOBILE-VTON.

• Privacy: It runs 100% on your phone. Your photo never leaves your device.
• Speed: No waiting for a server. It's instant.
• Quality: Even though it's small, it looks just as good as the giant server-based systems.

In summary: The researchers took a massive, cloud-based AI, shrunk it down using a "master chef" teaching method, added special tools to keep the clothes steady and realistic, and packed it all into a tiny app that runs on your phone. It's the difference between ordering a pizza from a factory and having a master pizzaiolo come to your kitchen to make it fresh, right in front of you.

Technical Summary

1. Problem Statement

Virtual Try-On (VTON) has achieved high visual fidelity using large-scale diffusion models, but current state-of-the-art systems face three critical barriers to practical, real-world deployment:

• Privacy and Latency: Existing systems rely on cloud-based GPUs, requiring users to upload personal photos. This raises privacy concerns and introduces network latency.
• Hardware Constraints: Diffusion models typically have massive parameter sizes (often >1B parameters), exceeding the memory and compute capabilities of mobile NPUs and GPUs.
• Semantic Instability & Pretraining Dependence:
  • Garment representations often drift across diffusion timesteps, causing texture distortion and loss of identity.
  • Most models rely on extensive pretraining on large-scale image datasets before fine-tuning for VTON. This limits the ability to train lightweight, standalone models directly on the VTON task without upstream data.

2. Methodology: The TGT Architecture

The authors propose MOBILE-VTON, a unified framework optimized for commodity mobile devices. It utilizes a TeacherNet–GarmentNet–TryonNet (TGT) architecture and runs entirely offline using only a single person image and a single garment image.

A. Core Components

1. TeacherNet (Frozen):

   • Based on Stable Diffusion 3.5 Large.
   • Acts as a high-capacity "oracle" providing supervisory signals (score estimates) but remains frozen during training.
   • It does not perform pixel-level regression but guides the student networks via gradient-based supervision.

2. Light-UNets (Student Networks):

   • GarmentNet: A lightweight network trained to extract consistent garment features.
   • TryonNet: A lightweight network responsible for synthesizing the final try-on image by aligning person and garment representations.
   • Both are adapted from the SnapGen architecture for mobile efficiency, utilizing mobile-optimized attention and dual text encoders (CLIP-G/L).

3. Light-Adapter:

   • A unified image-to-text conditioning module replacing large CLIP visual encoders with DINOv2-base for efficiency.
   • Extracts visual features from the garment and projects them into Key/Value tensors for cross-attention mechanisms.

B. Key Strategies & Loss Functions

1. Feature-Guided Adversarial (FGA) Distillation
To bridge the gap between the heavy teacher and lightweight students:

• Feature-Level Distillation: Instead of regressing pixels, the student minimizes the $\ell_2$ distance between its predicted score function ($s_{fake}$) and the teacher's score function ($s_{true}$) at each diffusion timestep. This aligns the generative distribution in latent space.
• Adversarial Realism: A lightweight discriminator is trained to distinguish real images from generated ones. TryonNet is optimized to fool the discriminator, enhancing photorealism and texture sharpness.

2. Trajectory-Consistent GarmentNet (TCG)
To prevent semantic drift (garment texture/shape changing over time):

• GarmentNet is trained with a trajectory-consistency loss.
• Instead of adding noise during training, the diffusion process is applied deterministically. The model must reconstruct the original garment image consistently across all timesteps $t \in [1, T]$.
• This enforces temporal stability, ensuring garment semantics remain invariant during the diffusion trajectory.

3. Garment-Aware TryonNet with Latent Concatenation (LC)
To enable training from scratch without large-scale pretraining:

• Latent Concatenation: The person image ($X_p$) and garment image ($X_g$) are spatially concatenated and encoded into a unified latent space. A reference input is also concatenated to guide identity and appearance preservation.
• Feature Fusion: TryonNet integrates multi-scale garment features from GarmentNet and visual Key/Value pairs from the Light-Adapter into its self-attention layers.
• Dual-Branch Cross-Attention: Combines textual guidance and visual garment semantics to ensure precise alignment.

3. Key Contributions

1. First On-Device Diffusion VTON: MOBILE-VTON is the first diffusion-based VTON system capable of running entirely on commodity mobile devices (415M parameters) without requiring external pretraining or user data transmission.
2. Novel TGT Framework: Introduces a modular architecture combining:
   • FGA Distillation: For efficient, realistic generation under mobile constraints.
   • TCG GarmentNet: For semantic stability across diffusion steps.
   • Garment-Aware TryonNet: Uses latent concatenation and cross-modal fusion to learn alignment directly from task-specific data.
3. Mask-Free Operation: Unlike many baselines that require segmentation masks, MOBILE-VTON is fully mask-free, synthesizing the entire image (body, clothing, background) from scratch.
4. Privacy-Preserving: The system operates 100% offline, eliminating privacy risks associated with cloud processing.

4. Experimental Results

Experiments were conducted on VITON-HD, DressCode, and VITON-HD In-the-Wild at a resolution of 1024 × 768.

• Performance vs. Server Baselines:

  • MOBILE-VTON matches or outperforms strong server-based baselines (e.g., IDM-VTON, StableVITON, CatVTON) in perceptual metrics (LPIPS, SSIM, CLIP-I).
  • On the DressCode Upper-body dataset, it achieved the best SSIM (0.893) and lowest LPIPS (0.088) among all compared methods.
  • On VITON-HD In-the-Wild, it achieved competitive results (LPIPS: 0.133) compared to server models, despite being mask-free.

• Efficiency:

  • Memory Usage: Requires only 2.84 GB of memory, compared to 5–18 GB for server-side baselines.
  • Parameters: 415M parameters (significantly smaller than 2–7x larger server models).

• Ablation Studies:

  • Removing TCG caused a drop in CLIP-I (0.805 → 0.798) and increased LPIPS, confirming its role in semantic stability.
  • Removing Latent Concatenation (LC) significantly degraded performance (LPIPS rose to 0.111), proving its necessity for learning alignment without pretraining.

5. Significance

MOBILE-VTON demonstrates that high-fidelity, privacy-preserving virtual try-on is feasible on mobile devices. By successfully distilling knowledge from a large teacher model into a compact student architecture and solving the issues of semantic drift and pretraining dependence, the paper provides a blueprint for deploying complex generative AI models on edge devices. This enables secure, responsive, and accessible fashion applications without relying on cloud infrastructure.