CORAL: Correspondence Alignment for Improved Virtual Try-On

This paper introduces CORAL, a DiT-based virtual try-on framework that improves garment detail preservation by explicitly aligning person-garment query-key matching through correspondence distillation and entropy minimization losses, addressing the limitations of existing unpaired methods.

The Gist

Imagine you are trying on a beautiful, intricate sweater in a virtual dressing room. You upload a photo of yourself and a photo of the sweater. The goal is for the computer to "paint" the sweater onto your body perfectly, keeping your pose, your face, and your hands exactly as they are, while making the sweater look like it was actually worn by you.

For a long time, these virtual dressing rooms have been a bit clumsy. They often get the details wrong: the sweater might look stretched, the logo might be blurry, or the computer might accidentally paint the sweater over your face or hands.

Enter CORAL.

Think of CORAL not just as a new computer program, but as a super-precise GPS system for fabric.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Lost in Translation" Garment

In the old methods, the computer was like a painter who had a photo of a sweater and a photo of a person, but they didn't really understand how the two connected.

• They knew the sweater had a collar, but they weren't sure which part of your neck it should sit on.
• They knew the sweater had a hem, but they didn't know exactly where your waist was.
• The Result: The computer would guess. Sometimes it guessed right, but often it would duplicate parts of the sweater (like two hems), stretch the fabric weirdly, or lose the small details like a brand logo.

2. The Insight: The "Secret Handshake"

The researchers behind CORAL discovered something fascinating about the new generation of AI (called Diffusion Transformers). These AIs work by looking at every single pixel of the person and the sweater and asking, "Which pixel of the sweater belongs to which pixel of the person?"

They realized that for the AI to do a good job, it needs a "Secret Handshake" between the person and the garment.

• Imagine the AI is holding a map. The "Query" is the person's body, and the "Key" is the sweater.
• If the AI's map is blurry, it might think the sweater's sleeve belongs to your leg.
• CORAL's job is to sharpen that map so the "handshake" is perfect.

3. The Solution: The "Trusty Guide" (DINOv3)

How does CORAL fix the map? It uses a Trusty Guide (a pre-trained AI model called DINOv3).

• Think of DINOv3 as an expert tailor who has seen millions of clothes and knows exactly how a sleeve aligns with an arm, or how a collar sits on a neck.
• CORAL asks this expert tailor: "Hey, look at this person and this sweater. Which parts should match?"
• The tailor points and says, "This patch of the sweater goes here on the arm, and this logo goes right here on the chest."
• CORAL then forces the main AI to listen to this expert and adjust its "Secret Handshake" to match the tailor's advice.

4. The Two Magic Tools

To make this work, CORAL uses two specific techniques, which we can think of as The Magnet and The Laser.

• The Magnet (Correspondence Distillation): This pulls the AI's attention toward the correct spots. If the AI was trying to match a sweater sleeve to a person's knee, the Magnet pulls it back to the person's arm, guided by the expert tailor's map.
• The Laser (Entropy Minimization): Sometimes, even when the AI knows the right spot, it gets "wishy-washy." It might think, "Maybe the sleeve goes here, or maybe a little bit there?" This makes the image blurry. The Laser forces the AI to be confident. It says, "No, the sleeve goes exactly here, and nowhere else." This creates sharp, crisp edges and clear logos.

5. The Result: A Perfect Fit

When you combine the GPS (CORAL), the Expert Tailor (DINOv3), the Magnet, and the Laser, you get a virtual try-on that feels magical.

• No more double hems: The bottom of the shirt stops exactly where it should.
• No more blurry logos: The brand name on the shirt is readable and sharp.
• No more weird stretching: The fabric looks like it naturally drapes over your specific body shape.

Why Does This Matter?

Before CORAL, virtual try-ons were like wearing a costume that didn't quite fit. With CORAL, it's like having a tailor who can instantly sew a perfect outfit onto your digital self, no matter how you are posing or what the background looks like. It bridges the gap between a flat picture of a shirt and a real, 3D experience of wearing it.

In short: CORAL teaches the computer to stop guessing and start knowing exactly where every piece of fabric belongs.

Technical Summary

1. Problem Statement

Virtual Try-On (VTON) aims to synthesize an image of a person wearing a specific garment while preserving the person's identity and pose. Despite recent advances, existing methods—particularly those based on Diffusion Transformers (DiTs)—struggle with accurate person-garment correspondence.

• The Core Issue: In unpaired settings (where the person and garment images are not pre-aligned), current models often fail to map specific garment details (e.g., logos, patterns, hemlines) to the correct locations on the person's body. This leads to artifacts like duplicated hems, distorted textures, or "logo drift."
• The Gap: While DiTs offer powerful full 3D attention mechanisms, the paper reveals that these models do not explicitly enforce precise query-key matching between person tokens and garment tokens. Consequently, the attention maps often become dispersed, leading to poor localization of garment features.

2. Methodology: CORAL

The authors propose CORAL (Correspondence Alignment), a DiT-based framework that explicitly enhances person-garment correspondence by aligning the internal attention mechanisms of the diffusion model with robust external correspondences.

A. Key Insight: Query-Key Correspondence

The authors first analyzed the behavior of full 3D attention in DiTs. They demonstrated a strong linear correlation between the accuracy of person-to-garment query-key matching and the final VTON quality.

• Finding: High-quality VTON results correspond to sharp, localized attention maps where person queries correctly attend to specific garment keys.
• Hypothesis: Improving the precision of this matching process is critical for preserving fine details and global shape.

B. Network Architecture

• Diptych Formulation: The model uses a two-panel layout where the noisy garment latent ($z_g$) and person latent ($z_p$) are horizontally concatenated. This allows the DiT to process them as a single sequence, enabling direct token-level interaction via multimodal attention.
• Pose Injection: Instead of concatenating pose conditions along the channel dimension (which entangles appearance and pose), CORAL concatenates pose tokens along the token dimension. It utilizes Rotary Position Embeddings (RoPE) to ensure the person and pose tokens share spatial indices, enforcing strict spatial alignment.

C. The CORAL Loss Function

To enforce accurate correspondence, CORAL introduces two complementary loss terms applied to the person-to-garment attention maps ($A_{P \to G}$):

1. Correspondence Distillation Loss ($\mathcal{L}_{corr}$):

   • Goal: Align the model's internal query-key matching with reliable external correspondences.
   • Mechanism: Uses DINOv3 (a vision foundation model) to generate pseudo ground-truth dense correspondences between the person and garment images.
   • Process: A "soft argmax" is used to estimate the correspondence from the attention map. The model is trained to minimize the $L_2$ distance between this estimated correspondence and the DINOv3 pseudo-ground-truth.
   • Refinement: A cycle-consistency check filters out unreliable matches in the DINOv3 data to ensure the supervision signal is robust.

2. Entropy Minimization Loss ($\mathcal{L}_{ent}$):

   • Goal: Sharpen the attention distribution to prevent dispersed or ambiguous matching.
   • Mechanism: Minimizes the Shannon entropy of the attention weights for each query.
   • Effect: This forces the model to make confident, localized decisions (attending to a specific key) rather than spreading attention across multiple incorrect keys.

Total Objective: The training objective combines the standard rectified flow velocity loss with the CORAL loss:
$$\mathcal{L}_{total} = \mathcal{L}_{velocity} + \lambda_{corr}\mathcal{L}_{corr} + \lambda_{ent}\mathcal{L}_{ent}$$

3. Key Contributions

1. Theoretical Insight: The paper is the first to explicitly link the quality of VTON to the precision of query-key correspondence within the full 3D attention of DiTs, proving that accurate RGB-space alignment depends on precise token matching.
2. Novel Framework (CORAL): A DiT-based architecture that integrates Correspondence Distillation (aligning with DINOv3) and Entropy Minimization to explicitly guide attention toward correct garment regions.
3. Robust Evaluation Protocol:
   • Introduced a new VLM-based evaluation (using GPT-4/5) to assess Garment Transfer Consistency (GTC), Textual Attribute Consistency (TAC), and Fit Pose Coherence (FPC), addressing the limitations of traditional metrics (FID, SSIM) in capturing human preference.
   • Evaluated on a challenging in-the-wild dataset (PPR10K) involving person-to-person garment transfer, a scenario where most existing methods fail.

4. Experimental Results

CORAL was evaluated on standard benchmarks (VITON-HD, DressCode) and the challenging PPR10K dataset.

• Quantitative Performance:
  • VITON-HD & DressCode: CORAL achieved State-of-the-Art (SOTA) results across all metrics (SSIM, LPIPS, FID, KID) in both paired and unpaired settings. For example, on VITON-HD unpaired, it improved FID from 9.076 (CatVTON) to 8.763 and KID from 1.184 to 0.880.
  • PPR10K: In the difficult in-the-wild setting, CORAL significantly outperformed baselines, demonstrating strong generalization.
• Qualitative Improvements:
  • Detail Preservation: CORAL successfully preserves fine details like logos, text, and repetitive patterns that baseline models often blur or distort.
  • Artifact Reduction: It eliminates common artifacts such as duplicated garment hems or misaligned textures.
  • Pose Consistency: The model maintains better adherence to the input pose, avoiding "pose hallucination" (e.g., distorted hands).
• Ablation Studies:
  • Removing $\mathcal{L}_{corr}$ leads to misaligned geometry (wrong silhouette).
  • Removing $\mathcal{L}_{ent}$ leads to dispersed attention and loss of fine details.
  • Combining both yields the best performance, confirming their complementary nature.
• VLM & Human Evaluation: CORAL scored highest in human preference studies and VLM-based assessments, particularly in GTC and FPC, validating that the method aligns better with human visual perception.

5. Significance

• Paradigm Shift: The paper shifts the focus of VTON from merely improving inference sampling or adding auxiliary encoders to fundamentally improving the attention mechanism within the diffusion backbone.
• Scalability: By leveraging foundation models (DINOv3) for pseudo-supervision, CORAL provides a scalable way to train high-fidelity VTON models without requiring expensive, manually annotated dense correspondence datasets.
• Real-World Applicability: The success on the PPR10K dataset and in person-to-person transfer scenarios suggests that CORAL is a viable solution for real-world applications where input images are uncurated, occluded, or taken in the wild.

In summary, CORAL solves the critical bottleneck of "where to put the garment details" in Diffusion Transformers by explicitly aligning the model's internal attention with robust external semantic correspondences, resulting in significantly higher fidelity and realism in virtual try-on generation.