TrendGen: An Outfit Recommendation and Display System

1. Problem Statement

The fashion industry faces significant challenges in deploying robust AI systems for online shopping due to the complexity of real-world data. While recent advances in Computer Vision (CV) and Deep Learning (DL) have improved image understanding, existing solutions often fail in production environments because they rely on:

• Curated datasets: Most research uses controlled, high-quality images, ignoring real-world issues like inconsistent lighting, non-ideal angles, complex backgrounds, and occlusions (e.g., hair or limbs covering garments).
• Suboptimal Inputs: Traditional outfit recommendation systems often rely on raw human-worn images and unstructured text descriptions. Raw images contain distracting elements, and text descriptions often include non-essential details that hinder accurate stylistic matching.
• Lack of Diversity: Existing recommendation engines often prioritize aesthetic compatibility but fail to account for the frequency of item selection, leading to repetitive outfit suggestions that do not cater to diverse consumer preferences.

2. Methodology

TrendGen is a comprehensive AI system deployed on a major e-commerce platform. It addresses the above challenges through a three-stage pipeline: Product Attribution, Virtual Try-Off (Image Generation), and Outfit Recommendation.

A. Product Attribution Engine

• Goal: Transform raw product titles and images into structured, meaningful attributes (e.g., category, color, hem type, sleeve length).
• Architecture:
  • Feature Extraction: Utilizes Fashion-CLIP, a domain-adapted version of the CLIP model, to generate joint embeddings from both the garment image and the product title. This leverages contrastive language-image pre-training to capture fine-grained visual-textual relationships.
  • Classification: A custom classifier (fully connected layers with PReLU activation and Dropout) is trained on top of the Fashion-CLIP embeddings to predict specific attributes.
• Advantage: This multi-modal approach outperforms traditional separate CNN (image) and NLP (text) models by integrating complementary information.

B. Virtual Try-Off (Image Generation)

• Goal: Convert "human-worn" garment images (Figure 1b) into clean, high-resolution "lay-down" (flat-lay) images (Figure 1a) to serve as standardized inputs for the recommendation engine and improve product display.
• Architecture:
  • Base Model: A Latent Diffusion Model (LDM) based on the inpainting variant of Stable Diffusion v1.5.
  • Input Conditioning: Unlike standard text-to-image models, this system uses a binary cloth mask (derived via segmentation) and the masked human-worn image as conditioning inputs, eliminating the need for text prompts.
  • Training Strategy: The U-Net component is fine-tuned using pairs of human-worn images and corresponding flat-lay images (from the VITON-HD dataset and proprietary data). The authors found that fine-tuning only the transformer blocks of the U-Net yielded the best balance between performance and computational efficiency.
• Output: High-fidelity, distraction-free garment images that ensure consistent input quality for downstream tasks.

C. Outfit Recommendation Engine

• Goal: Generate cohesive, diverse, and trend-aligned outfit combinations.
• Embedding Generation:
  • Combines the lay-down images (from the Virtual Try-Off module) and structured attributes (from the Attribution Engine) using Fashion-CLIP to create 1024-dimensional multi-modal embeddings (512 visual + 512 textual).
• Learning Framework:
  • TripleNet: A Siamese network trained with Triplet Loss. It learns to pull "compatible" items (Anchor + Positive) closer in the embedding space while pushing "incompatible" items (Anchor + Negative) further apart.
  • Negative Mining: Uses domain expert rules (e.g., avoiding multiple multicolor garments) to construct semantically meaningful negative samples, ensuring the model learns strict styling constraints.
• Diversity Algorithm (STYLERANK):
  • To prevent repetition, the system employs a ranking algorithm that balances aesthetic compatibility (based on TripletNet distances) with historical appearance frequency.
  • It calculates a final score: $S(p) = R_D(p) + \lambda \cdot R_A(p)$, where $R_D$ is the compatibility rank and $R_A$ is the appearance rank. This ensures the system suggests fresh combinations rather than the same popular items repeatedly.

3. Key Contributions

1. Precise Product Attribution: A multi-modal system using Fashion-CLIP that achieves high accuracy in classifying complex fashion attributes, surpassing traditional separate image/text models.
2. State-of-the-Art Virtual Try-Off: A novel diffusion-based model that converts human-worn images to flat-lay views using mask-guided inpainting, optimized for production environments without requiring heavy computational resources.
3. Diverse Outfit Recommendation: A recommendation engine that integrates visual and textual features and introduces the STYLERANK algorithm to balance aesthetic compatibility with item diversity, solving the "repetition" problem in existing systems.
4. End-to-End Production Evaluation: A comprehensive evaluation of the entire pipeline on proprietary production data, demonstrating real-world viability and business impact.

4. Results

The system was evaluated on a mix of public datasets (VITON-HD) and proprietary data from a major retailer.

• Attribution Performance: The proposed Fashion-CLIP + Classifier system achieved 92.3% overall accuracy, significantly outperforming the baseline Fashion-CLIP alone (73.8%) and a ResNet+NLP hybrid (89.1%). It reached 99.6% accuracy in critical category classification (e.g., Tops vs. Bottoms).
• Virtual Try-Off Performance:
  • Outperformed TryOffDiff and MGT on key perceptual metrics.
  • Achieved a DISTS score of 21.01 (lower is better), compared to 23.0 for TryOffDiff and 24.7 for MGT, indicating superior visual fidelity and structural similarity.
  • Demonstrated better texture and pattern accuracy in qualitative comparisons.
• Recommendation Accuracy:
  • The full system (Lay-down images + Attributes) achieved 91.3% accuracy in human expert evaluations.
  • This significantly outperformed the OutfitTransformer baseline (75.2%) and the system using raw images/text (79.8%).
  • The STYLERANK algorithm was crucial, as removing it dropped accuracy to 55.7%.
• Business Impact (A/B Testing):
  • Revenue Per Visit (RPV): +6.60%
  • Net Margin Per Visit (NMPV): +6.62%
  • Conversion Rate (CVR): +5.61%
  • Click-Through Rate (CTR): +8.2%

5. Significance

TrendGen represents a significant advancement in applying AI to the fashion retail sector by bridging the gap between academic research and production reality.

• Robustness: It successfully handles the "messy" nature of real-world e-commerce data (occlusions, poor lighting) by preprocessing images into standardized flat-lay views.
• Holistic Approach: Unlike isolated research focusing only on generation or only on recommendation, TrendGen integrates data cleaning, attribute extraction, image generation, and recommendation into a single, cohesive pipeline.
• Commercial Viability: The system has been proven to drive measurable business growth (increased sales, margins, and engagement) on a major platform, validating that AI-driven personalization and display improvements directly translate to revenue.
• Scalability: The use of efficient fine-tuning strategies and optimized diffusion models makes the system deployable on standard consumer-grade hardware, unlike many resource-heavy academic models.