Technical Summary: Oxygen-TryOn

Problem Statement

Virtual try-on (VTON) has emerged as a critical application for generative AI in e-commerce, yet current solutions face significant limitations in fidelity, flexibility, and scope. Existing methods generally fall into two categories:

1. Specialized Try-On Models: These often rely on constrained inpainting or garment warping techniques. They are typically limited to single garment categories (e.g., tops or bottoms), require clean studio settings with flat-lay product shots, and struggle with accessories, footwear, or free multi-item composition. They frequently fail to preserve fine-grained textures, logos, or subject identity when faced with complex poses or "in-the-wild" reference photos.
2. General-Purpose Generative Models: While powerful proprietary and open-source models (e.g., GPT-Image-2, FLUX.2) exhibit strong semantic understanding, they are not optimized for the strict consistency requirements of fashion. When prompted for try-on, they tend to hallucinate garment details, drift on subject identity, or fail to maintain structural plausibility under occlusion and layering.

There is a lack of a unified, open-source foundation model capable of handling any-item (garments, accessories, shoes, bags) try-on across multi-reference scenarios while faithfully preserving both the reference item's appearance and the target subject's identity.

Methodology

1. Architecture

Oxygen-TryOn is built upon the JoyAI-Image-Edit foundation, inheriting a strong multimodal prior. The architecture integrates three core components:

• Multimodal Large Language Model (MLLM): Based on Qwen3-VL-8B-Instruct, this serves as the cognitive core. It parses heterogeneous inputs (subject image, multiple reference item images, and textual instructions) to resolve item-to-body-region mappings and understand layering constraints.
• Variational Autoencoder (VAE): Utilizing Wan-2.1-VAE, it compresses images into latent tokens while preserving high-frequency details essential for reconstructing fabric textures and small logos.
• Multimodal Diffusion Transformer (MMDiT): A 16B-parameter dual-stream model that fuses semantic priors from the MLLM with image latents from the VAE. It performs high-fidelity synthesis conditioned on the target subject and reference items.

Key Design Choice: Instead of using explicit segmentation masks or region-type embeddings, Oxygen-TryOn employs a multi-reference conditioning strategy. The subject and all reference items are encoded into a shared token sequence. The model distinguishes between the target (noise-to-image) and references (clean conditioning) solely via the temporal axis of the multimodal RoPE (MRoPE) position encoding. Item identity and placement are resolved through natural language instructions (e.g., "Picture 1," "Picture 2") rather than dedicated tokens, allowing for variable numbers of references.

2. Data Engine

To overcome the scarcity of aligned try-on triplets, the authors built a dedicated Data Engine comprising three stages:

• Collection & Filtering: Aggregates over 50M raw images from e-commerce, open-domain fashion, and model-generated sources. A multi-stage filtering pipeline removes low-quality, unsafe, or mismatched samples using VLMs, OCR, and aesthetic scoring models.
• Annotation & Captioning: Generates structured metadata for items (category, material, texture) and subjects (pose, body region, demographics). This ensures consistency between product attributes and target body regions.
• Try-On Pairing: Constructs item–subject–result triplets through four complementary pipelines:
  1. Real product + Real dressed person (for authentic textures).
  2. Real product + Synthetic subject (for identity/pose diversity).
  3. Synthetic product + Real subject (for long-tail categories).
  4. Fully synthetic compositions (for open-ended scenarios).
     Multi-item triplets are constructed with constraints on body regions and physical coexistence to teach plausible layering.

3. Training Recipe

The model undergoes a three-stage training process:

1. Continued Pre-Training (CPT): Adapts the general-purpose foundation to the try-on domain using a 1:1 mixture of general data and try-on data, preserving broad multimodal capabilities while injecting domain knowledge.
2. Supervised Fine-Tuning (SFT): Specializes the model on high-quality paired data from the data engine. The model learns to map reference items to target subjects under diverse conditions (single/multi-item, full/half-body) using a conditional flow-matching objective.
3. Reinforcement Learning (RL): Optimizes the model using a hybrid reward mechanism to improve consistency and realism beyond pixel-level losses.
   • In-house Reward Model: A 8B VLM trained on 100k human preference pairs, evaluating item consistency, identity consistency, and visual quality with uncertainty-aware ranking.
   • Rubric-Guided Judge: Uses Gemini 3.1 Pro to score results against explicit criteria (instruction adherence, identity, item fidelity, physical plausibility).
   • Aggregation: The final reward is a weighted average of both sources, penalizing imbalance to prevent "reward hacking."

Key Contributions

• A Unified Any-Item Try-On System: Oxygen-TryOn is presented as the first system to deliver high-fidelity, multi-reference try-on for arbitrary combinations of garments, accessories, shoes, and bags. It supports both clean product shots and "in-the-wild" worn photos as references.
• Fashion-Native Foundation Model: Unlike repurposed general editors, Oxygen-TryOn is purpose-built for fashion, reformulating try-on as an understanding-driven generation task rather than a mask-based inpainting problem.
• Transparent Training Pipeline: The authors release the model weights alongside a detailed recipe (CPT–SFT–RL) and a dedicated data engine design, aiming to make high-fidelity try-on reproducible and extensible for the open-source community.
• Built-in General Editing: The model retains general instruction-following capabilities (e.g., pose changes, background edits) within the same generation pass, eliminating the need for task switching.

Results

Oxygen-TryOn was evaluated across public benchmarks (DressCode, VITON-HD, TStars-VTON) and a proprietary Oxygen-TryOn Bench (1,000 real-world samples).

• Quantitative Performance:
  • Single-Item Try-On: Outperforms leading proprietary systems (Nano Banana Pro, GPT-Image-2, Seedream5 Lite) and open-source models (FLUX.2) on TStars-VTON, achieving the highest scores in Identity Consistency, Item Fidelity, and Overall quality.
  • Multi-Item Try-On: Leads in Overall score, Identity Consistency, Background Preservation, and Physical Logic on TStars-VTON, demonstrating superior ability to handle complex layering and occlusion.
  • Real-World Usability: On the Oxygen-TryOn Bench, it achieves a Usability Rate (results requiring no manual rework) of 86.79% for Cloth-to-Model and 85.43% for Model-to-Model, significantly surpassing the best proprietary competitor (GPT-Image-2 at 80%) and open-source baselines (67%).
• Qualitative Performance: Visual comparisons show Oxygen-TryOn better preserves fine-grained details (logos, textures), maintains subject identity (face, body shape), and handles complex physical interactions (draping, occlusion) compared to baselines, which often suffer from identity drift, distorted proportions, or missing item details.

Significance

The paper claims that Oxygen-TryOn represents a shift from constrained, single-category prototypes toward truly universal virtual try-on systems. By demonstrating that a unified foundation model can handle diverse items and complex compositions with state-of-the-art fidelity, it challenges the notion that specialized warping pipelines are necessary for high-quality results. Furthermore, by releasing the model and its training infrastructure, the authors aim to close the gap between proprietary capabilities and open-source reproducibility, enabling the community to audit, reproduce, and extend high-fidelity fashion generation.