LookBench: A Live and Holistic Open Benchmark for Fashion Image Retrieval

This paper introduces LookBench, a live and holistic open benchmark for fashion image retrieval that features time-stamped real-world and AI-generated images, a fine-grained attribute taxonomy, and periodic updates to provide a durable, contamination-aware evaluation of retrieval models in real e-commerce settings.

The Gist

Imagine you are shopping for a very specific outfit. You see a photo of a stranger on the street wearing a perfect "high-neck, cream-colored, oversized knit sweater" paired with "wide-leg denim jeans." You want to find that exact sweater, or at least one that looks and feels exactly the same, to buy online.

This is the daily challenge of Fashion Image Retrieval. But for a long time, the "tests" used to see if computer programs are good at this job have been outdated, like using a map from 1990 to navigate a city that has completely changed since then.

Here is a simple breakdown of the paper "LookBench" and what the authors achieved, using some everyday analogies.

1. The Problem: The "Old Map" vs. The "Live City"

For years, researchers tested fashion search engines using static datasets (like DeepFashion or Fashion200K).

• The Analogy: Imagine training a taxi driver using a map of a city from 2010. The driver might memorize the streets perfectly, but if you take them to the city today, they will get lost because new buildings, one-way streets, and traffic patterns have changed.
• The Reality: Modern AI models (like CLIP or DINO) are trained on massive amounts of internet data. Because the old fashion test datasets are so old, these AI models have likely already "seen" the test pictures during their training. It's like giving a student a practice exam that they already have the answers to. They get a perfect score, but they aren't actually smart; they just memorized the test.

2. The Solution: LookBench (The "Live" Test)

The authors created LookBench, a new, "live" benchmark.

• The Analogy: Instead of a static map, LookBench is like a live traffic camera feed. It constantly updates with new photos taken today from real e-commerce websites and even AI-generated fashion images.
• Key Feature: Every photo has a "timestamp." If a computer model was trained on data from 2023, but LookBench uses photos from 2025, the model can't cheat by memorizing the answers. It has to actually understand the clothes.

3. The Four Levels of Difficulty

LookBench isn't just one test; it's a video game with four levels of difficulty, ranging from "Easy" to "Nightmare Mode."

• Level 1: RealStudioFlat (Easy)
  • What it is: Clean, white-background photos of a single shirt or dress, just like you see on a store website.
  • The Goal: Find the exact same shirt.
• Level 2: AIGen-Studio (Medium)
  • What it is: AI-generated photos of clothes in a studio setting.
  • The Goal: Find the item, but the lighting and texture are slightly different because they were made by a computer.
• Level 3: AIGen-StreetLook (Hard)
  • What it is: AI-generated photos of people wearing full outfits on busy streets.
  • The Goal: Find the specific jacket or shoes in a messy, complex scene.
• Level 4: RealStreetLook (Nightmare Mode)
  • What it is: Real photos of people on the street. The clothes are wrinkled, partially hidden by bags, the lighting is weird, and the person is walking.
  • The Goal: This is the hardest test. The computer has to ignore the background, the pose, and the wrinkles to find the exact item.

4. The Secret Sauce: "Attribute" Training

Most AI just looks at the "big picture." LookBench forces the AI to look at the details.

• The Analogy: Imagine a detective. A bad detective says, "It's a red car." A good detective says, "It's a red 2024 sedan with a scratch on the left door and a specific license plate."
• How they did it: The authors taught their AI to recognize over 100 specific details (attributes) like "V-neck," "sleeveless," "linen," or "pleated." They used a super-smart AI (Qwen) to label these details on thousands of photos. Now, when the computer searches, it doesn't just look for "red"; it looks for "red, V-neck, linen."

5. The Results: Who Won the Race?

The authors tested many famous AI models against LookBench.

• The Generic Models (CLIP, DINO): These are like general-purpose detectives. They are good at finding "a car," but they failed miserably at finding "a red 2024 sedan with a scratch." On the hardest test (RealStreetLook), many got less than 40% right.
• The Fashion Specialists (Marqo): These are better, like detectives who specialize in cars. They got around 60-66% right.
• The Winners (GR-Pro & GR-Lite): The authors built their own models specifically trained on their "attribute" system.
  • GR-Pro: The "Pro" version is a super-advanced model (kept secret for business reasons) that got the highest scores.
  • GR-Lite: The "Lite" version is open-source (free for everyone to use). It performed almost as well as the Pro version and crushed all other public models.

6. Why This Matters

This paper changes the game in two ways:

1. It stops the cheating: By using fresh, timestamped data, we can finally tell which AI models are actually smart and which ones just memorized old test questions.
2. It raises the bar: It shows that to truly understand fashion, AI needs to pay attention to tiny details (like the type of fabric or the cut of a sleeve), not just the general shape.

In summary: LookBench is a new, constantly updating "driving test" for fashion AI. It proves that to find the perfect outfit in a crowded digital world, you need a detective that notices the tiny details, not just a robot that guesses the general color. The authors have released their "best detective" (GR-Lite) to the public so everyone can build better fashion search engines.

Technical Summary

1. Problem Statement

Fashion image retrieval is a critical capability for e-commerce, powering visual search and recommendations. However, the field faces three major challenges:

• Stagnant Evaluation: Existing benchmarks (e.g., DeepFashion, Fashion200K) are static, product-centric, and often suffer from data contamination. Modern Vision-Language Models (VLMs) like CLIP and DINO are trained on web-crawled data that likely overlaps with these legacy test sets, leading to inflated performance metrics due to overfitting rather than genuine understanding.
• Lack of Realism: Current benchmarks primarily focus on single-item retrieval in controlled studio settings. They fail to capture complex, real-world user intents, such as retrieving specific items from cluttered street-style photos (outfit-level retrieval) or handling AI-generated fashion trends.
• Insufficient Granularity: Most metrics rely on category-level matching (e.g., "shirt"), ignoring fine-grained attributes (e.g., "v-neck," "silk," "short sleeve") which are crucial for accurate fashion search.

2. Methodology: LookBench

The authors introduce LookBench, a live, holistic, and contamination-aware benchmark designed to reflect real e-commerce scenarios.

A. Benchmark Construction & Design Principles

• Live & Time-Stamped: Samples are collected via time-constrained web queries from recent retail sites. Each sample is time-stamped, allowing for "contamination-aware" evaluation where test data is filtered to ensure it post-dates the training cutoff of any model being evaluated.
• Four Evaluation Subsets:
  1. RealStudioFlat: Clean, flat-lay studio product photos (Easy, Single-item).
  2. AIGen-Studio: AI-generated product images in lifestyle contexts (Medium, Single-item).
  3. RealStreetLook: Real-world street-style photos of full outfits (Hard, Multi-item).
  4. AIGen-StreetLook: AI-generated street-style outfits (Hard, Multi-item).
• Attribute Taxonomy: The benchmark utilizes a fine-grained, category-specific taxonomy with over 100 visually grounded attributes (e.g., 10–25 attributes per category like "v-neck," "pleated," "wool").
• Pre-annotation Pipeline: All query and gallery images are processed using YOLOv11 for object detection and Qwen2.5-VL-72B for attribute extraction. An LLM-as-a-judge (GPT-5.1) audit verified ~93% attribute correctness, providing reliable weak supervision.
• Corpus Construction: Retrieval corpora are augmented with "soft negatives" from Fashion200K. Distractors are selected based on cosine similarity bands to ensure they are visually plausible but semantically distinct, avoiding near-duplicates.

B. Proposed Models: GensmoRetro (GR)

To establish strong baselines, the authors trained a family of text-free visual encoders under attribute supervision:

• GR-Pro: A high-capacity (0.3B parameters) proprietary vision encoder trained on 6.5M in-house fashion images. It uses a DINOv3 backbone and is optimized via ArcFace loss for discriminative embedding.
• GR-Lite: An open-source counterpart with a similar architecture (DINOv3 ViT-L/16 backbone + projection head) trained on 1.8M images (1.3M open-source + 0.5M in-house). It serves as the strongest publicly available reference.

C. Evaluation Metrics

The paper introduces Fine Recall@1 as the primary metric. A retrieval is only considered correct if the top-ranked item matches both the garment category and all annotated attributes. This is stricter than standard Recall@1, which often only checks category.

3. Key Contributions

1. LookBench Benchmark: The first live, contamination-aware benchmark for fashion retrieval that includes both real and AI-generated data across studio and street domains. It supports single-item and multi-item (outfit) retrieval.
2. Attribute-Supervised Training: Demonstrates that training visual encoders with fine-grained attribute supervision significantly outperforms generic VLM backbones (CLIP, SigLIP, DINO) in fashion-specific tasks.
3. Open Baselines: Release of GR-Lite, a state-of-the-art open-source model that outperforms existing fashion-specific models (e.g., Marqo-FashionCLIP) and generic encoders.
4. Comprehensive Analysis: Provides deep insights into model scaling, data scaling, and the specific difficulties of outfit-level retrieval vs. single-item retrieval.

4. Experimental Results

• Performance on LookBench:
  • GR-Pro achieved the highest Fine Recall@1 across all subsets (Overall: 67.38%), significantly outperforming the best public baseline (Marqo-FashionCLIP at 63.24%).
  • GR-Lite (Open Source) ranked second overall (65.71%), trailing GR-Pro by only 1.67 points but beating all other public models.
  • Generic Models: Strong generic backbones (CLIP-L/14, DINOv3) performed poorly on LookBench (30–45% Fine Recall@1), highlighting the domain gap and the need for fashion-specific fine-tuning.
  • RealStreetLook Challenge: This subset proved the most difficult, with all models dropping in performance compared to studio settings, yet GR-Pro maintained a clear lead.
• Outfit-Level Retrieval: On the RealStreetLook subset, requiring all items in an outfit to be correctly retrieved, GR-Pro achieved 53.2% Fine Recall@1, while generic models struggled below 40%.
• Generalization: Despite being trained on LookBench's specific protocol, GR-Pro and GR-Lite achieved new state-of-the-art results on the legacy Fashion200K benchmark (88.7% and 88.3% Recall@1, respectively), proving their representations transfer well to standard tasks.
• Scaling Laws:
  • Data Scaling: Performance improved sharply from 0.59M to 6.5M images, plateauing thereafter.
  • Model Scaling: Increasing parameters from 21M to 0.3B yielded significant gains, but scaling to 0.8B offered diminishing returns relative to the increased computational cost.

5. Significance and Future Impact

• Contamination Awareness: LookBench addresses the critical issue of data contamination in modern AI evaluation, providing a protocol to fairly assess models against "future" data.
• Holistic Evaluation: By moving beyond single-item matching to outfit-level and attribute-level retrieval, it better aligns with actual user shopping behaviors.
• Reproducibility: The authors publicly release the dataset splits, evaluation code, trained models (GR-Lite), and attribute annotations, fostering transparent research.
• Future Directions: The paper suggests future work in multimodal intent-conditioned retrieval (combining text and image), temporal trend analysis, and aesthetic-aware ranking.

In conclusion, LookBench sets a new standard for fashion image retrieval evaluation, demonstrating that while generic foundation models are powerful, they require domain-specific, attribute-supervised fine-tuning to handle the nuances of real-world fashion search.