KiseKloset for Fashion Retrieval and Recommendation

1. Problem Statement

The fashion e-commerce industry faces significant challenges in transitioning from physical to digital retail. Key pain points include:

• Inefficient Search: Users spend excessive time searching for items without the guidance of in-store staff.
• Lack of Fit Visualization: The inability to try on clothes virtually leads to high return rates and damaged items.
• Fragmented Solutions: Existing systems often handle Outfit Retrieval/Recommendation (ORR) and Virtual Try-On (VTON) separately. Furthermore, current ORR methods often struggle with generalized complementary item recommendations (recommending multiple missing items rather than just one) and suffer from high computational latency in large-scale datasets.

The paper proposes KiseKloset, a comprehensive end-to-end system designed to bridge the gap between physical and digital fashion retail by integrating outfit retrieval, recommendation, and real-time virtual try-on.

2. Methodology

KiseKloset is built upon three core technical pillars:

A. Outfit Retrieval (Intra-Category)

The system employs two distinct retrieval strategies using the CLIP (Contrastive Language-Image Pre-training) model for feature extraction:

1. Similar Item Retrieval: Extracts visual embeddings from a reference image and retrieves visually similar items.
2. Text Feedback-Guided Retrieval: Utilizes the CLIP4Cir architecture. It combines embeddings from a reference image and natural language feedback (e.g., "make it longer," "green") via a combiner network trained with symmetric contrastive loss.
   • Diversity Augmentation: To prevent result uniformity, the system introduces variation in the output: the top results are accurate, while subsequent results are randomized from broader search ranges (top-100 to top-1000) to expose users to diverse styles.

B. Outfit Recommendation (Inter-Category)

The authors propose a novel Transformer-based architecture to solve the "fill-in-the-blank" problem for multiple missing items in an outfit.

• Formulation: An outfit is treated as a sequence of items (e.g., bags, tops, shoes). Items are categorized as Input (reference item), Output (target complementary items), or Unavailable (missing categories).
• Embedding Strategy:
  • Input items use visual embeddings.
  • Output and Unavailable items use shared, learnable special tokens (<OUT> and <UN>).
  • Positional embeddings denote item categories.
• Training: The model uses a Noise Contrastive Loss to maximize the similarity between the predicted output embeddings and the ground-truth visual embeddings of the complementary items.
• Inference: The system generates embeddings for missing categories and uses Approximate Nearest Neighbor (ANN) search to retrieve the best matching items from the database.
• Modes: It supports "Tone sur tone" (similar colors/textures) and "Mix and match" (diverse styles) by manipulating the search result ranking.

C. Virtual Try-On (VTON)

To address the computational complexity of existing VTON methods, the system employs a lightweight Teacher-Student network based on DM-VTON:

• Teacher Network: Built on FS-VTON, uses parser-based training to ensure high-fidelity output.
• Student Network: Built on MobileNetV2 with inverted residual blocks. It learns from the teacher via knowledge distillation, enabling real-time, resource-efficient inference without relying on complex human parsing during the generation phase.

D. Search Optimization

To handle large-scale datasets, the system integrates Approximate Search Algorithms (IVF, ANNOY, HNSW) to replace exhaustive K-Nearest Neighbor (KNN) searches, significantly reducing query time while maintaining high recall.

3. Key Contributions

1. Integrated System: KiseKloset is the first system to support both ORR (similar and text-guided) and real-time VTON in a single application for real-world use.
2. Generalized Recommendation: Unlike previous works that recommend a single missing item, the proposed Transformer network generalizes to recommend multiple complementary items of varying categories simultaneously.
3. Efficient Search Pipeline: The integration of approximate search algorithms (specifically HNSW) significantly accelerates retrieval speed without sacrificing accuracy.
4. Lightweight VTON: The deployment of a teacher-student distillation framework allows for real-time, memory-efficient virtual try-on suitable for web/mobile deployment.
5. User-Centric Validation: The system was deployed for live user testing, providing empirical data on satisfaction and usability.

4. Results

Technical Performance (Search Algorithms)

Experiments on large-scale fashion datasets (PolyvoreOutfits and DeepFashion) compared KNN against approximate methods:

• HNSW emerged as the optimal algorithm:
  • Query Time: 1.80 ms (vs. 35.24 ms for KNN).
  • Recall: 98.11% (vs. 100% for KNN).
  • Memory: 0.84 GB (comparable to KNN's 0.76 GB).
• This demonstrates that approximate search offers a scalable solution for real-time fashion retrieval.

User Study

The system was deployed for online users to test the end-to-end experience. Key findings:

• Usefulness: 84.4% of participants found the system highly useful for enhancing their online shopping experience.
• Advocacy: 65.6% of users expressed willingness to recommend the system to friends and family.
• Satisfaction Factors: Users rated Response Time and Ease of Use highest, followed by VTON quality and ORR quality. The seamless visual search and fast processing were highlighted as major strengths.

5. Significance

The KiseKloset system represents a significant step forward in fashion e-commerce technology by unifying retrieval, recommendation, and visualization into a cohesive, scalable platform.

• For Retailers: It offers a tool to reduce return rates (via better fit visualization) and increase conversion rates through personalized, multi-item outfit recommendations.
• For Consumers: It bridges the "try-on" gap in digital shopping, saving time and providing a more immersive, personalized experience.
• Scientific Impact: The paper advances the state-of-the-art in generalized outfit completion (handling variable missing items) and demonstrates the practical viability of lightweight, distillation-based VTON models for real-time applications.

Note: The authors acknowledge that the current system focuses exclusively on upper-body garments and plan to extend the framework to lower-body and full-body outfits in future work.