💻 computer science

Image Score: Learning and Evaluating Human Preferences for Mercari Search

This paper presents a cost-effective, LLM-driven approach using Chain-of-Thought prompting to generate image quality labels that correlate with human behavior on Mercari's e-commerce platform, ultimately leading to significant sales growth through improved search relevance and explainability.

Original authors: Chingis Oinar, Miao Cao, Shanshan Fu

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chingis Oinar, Miao Cao, Shanshan Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Mercari as a massive, bustling digital flea market where millions of people buy and sell everything from vintage clothes to used electronics. In this market, the most important thing for a seller is their "shop window"—the photo they take of their item. If the photo is blurry, dark, or messy, buyers scroll right past it, even if the price is perfect. If the photo is bright, clear, and looks professional, buyers stop and click.

The team at Mercari wanted to solve a specific problem: How do we automatically tell which photos are "good" and which are "bad" without hiring an army of human judges?

Here is the story of how they did it, explained simply:

1. The Problem: The "Silent" Buyer

In a normal store, you might ask a friend, "Do you like this shirt?" In an online market, buyers don't say anything. They just click or they don't. This is called "implicit feedback."

The Challenge: If a buyer clicks on an item, is it because the item is exactly what they wanted (relevance), or is it simply because the photo looked amazing?
The Goal: The team wanted to separate the "good photo" factor from the "right product" factor. They wanted to know: If two items are the same price and the same type, does the one with the better photo get more clicks?

2. The Solution: The "AI Art Critic"

Usually, to teach a computer what a "good photo" looks like, you need thousands of humans to look at pictures and give them grades. That is slow and expensive.
Instead, the Mercari team used a Large Language Model (LLM)—a super-smart AI that can "see" images. Think of this AI as a virtual art critic.

The Process: They took groups of similar items (like five different listings for the same type of jeans) and asked the AI: "Look at these photos. Compare them to each other. Which one looks the best? Why?"
The Secret Sauce (Chain-of-Thought): They didn't just ask for a score. They asked the AI to "think out loud" (Chain-of-Thought). The AI had to explain why a photo was good: "The lighting is bright," "The colors match," "The background is clean." This helped the AI give much more accurate scores that matched what real humans would think.

3. Training the "Smart Sorter"

The AI critic is great, but it's too slow and expensive to use on every single item in the market every time someone searches. It's like hiring a famous art critic to grade every single photo in a library; it would take forever.

So, they used the AI critic's grades to train a lightweight, fast computer model (a "Smart Sorter").

The Analogy: Imagine the AI critic is a master chef tasting a dish and writing a detailed review. The "Smart Sorter" is a junior chef who watches the master chef taste many dishes, learns the patterns, and then can quickly guess the quality of new dishes without needing the master chef's help every time.
The Result: This new model learned to predict which photos would get clicks based on the "grades" the AI critic gave them.

4. The Big Test: Putting it to Work

They tested this system in the real world on the Mercari website. They split users into two groups:

Group A (The Control): Saw the search results as usual.
Group B (The Test): Saw search results where items with "high-quality photos" were bumped up to the top, even if they were just slightly less relevant to the search term.

The Outcome:

On the Web: The test group bought almost 7% more than the control group. The "Smart Sorter" successfully put the prettiest photos in front of the buyers, and the buyers responded by clicking and buying more.
On Mobile: Interestingly, this didn't work as well on mobile phones. The authors suggest this might be because phone screens are smaller, so users might care less about tiny visual details compared to people browsing on big computer monitors.

5. The Catch: The "Fake" Photos

The paper also notes a funny side effect. Because the AI loves pretty pictures, some sellers started posting AI-generated images (computer-made art) instead of real photos of their items. These fake images got perfect scores and shot to the top of the list, even though they weren't real products.

The Lesson: The system needs a "truth detector" to make sure the pretty pictures are actually real items, not just digital art.

Summary

The Mercari team built a system that uses a smart AI to grade product photos, teaches a faster model to mimic that grading, and uses those grades to show the best-looking items first. The result? On the website, people bought more because the items looked better. It's a bit like rearranging a grocery store so the freshest, most colorful vegetables are always at eye level—naturally, people buy more of them.

1. Problem Statement

Mercari, Japan's largest C2C e-commerce marketplace, faces a challenge in optimizing search results where image quality is a primary differentiator. Unlike traditional e-commerce where product titles and descriptions dominate, Mercari users often make decisions based on images and prices before reading titles (as seen in the app's grid layout).

The Core Issue: While implicit feedback (clicks) is abundant, it is heavily confounded by relevance and position bias. It is difficult to isolate the specific impact of image quality on user behavior from these other factors.
Data Scarcity: Traditional image quality assessment relies on expensive, time-consuming human annotation, which is not scalable for a marketplace with millions of listings.
Goal: Develop a cost-effective method to assess image quality, decouple it from relevance, and use it to improve search ranking and user engagement (clicks and sales).

2. Methodology

The authors propose a pipeline that leverages Large Language Models (LLMs) for weak supervision to generate training labels, which are then used to train a lightweight machine learning model for online deployment.

A. Data Collection & Label Generation

Hypothesis: If two items are similarly relevant and priced, the one with better image quality will receive more clicks.
Data Filtering: The team extracted search logs and applied strict filtering to create "similar item pairs":
- Price Filtering: Items within a specific price range of the first clicked item.
- Position Windowing: Retaining items immediately before and after the clicked item (to minimize position bias).
- Result: A dataset of 150k training SERPs (Search Engine Results Pages) and 36k validation SERPs.
LLM Annotation (Weak Supervision):
- Instead of human annotators, they used GPT-4 via a Vision API.
- Chain-of-Thought (CoT) Prompting: The LLM was prompted to analyze aesthetic elements (color harmony, brightness, composition, etc.) and assign a score (0–10) to images in a batch, comparing them against each other rather than in isolation.
- Output: JSON-formatted aesthetic scores and descriptive adjectives.

B. Model Architecture

Backbone: The authors utilized CLIP (Contrastive Language-Image Pre-training) embeddings (specifically ViT-B/16 pre-trained on translated CC12M). They found that frozen CLIP embeddings already capture quality attributes effectively.
Fine-Tuning: A Multi-Layer Perceptron (MLP) was added on top of the frozen CLIP encoder.
Training Objective:
- Formulated as a Pairwise Learning-to-Rank (LTR) problem.
- Loss Function: Focal Loss was used instead of standard Cross-Entropy. This was chosen to handle noisy labels (from LLMs) and prevent the model from being overwhelmed by "easy" negative examples.
- Data Construction: For a query, if the LLM ranks image $x_i$ higher than $x_j$ , the model is trained to predict a higher score for $x_i$ .

C. Deployment & Integration

Inference: The model is hosted on NVIDIA Triton Inference Server.
Pipeline: Integrated into the Google Cloud Dataflow indexing pipeline. When an item is updated, the image score is predicted synchronously and indexed into Elasticsearch.
Ranking Strategy: The image score is integrated into the Elasticsearch ranking formula using two approaches tested in A/B tests:
1. Multiplicative: final_score = existing_relevance_score * image_score
2. Additive: final_score = existing_relevance_score + image_score

3. Key Contributions

Decoupling Relevance from Implicit Feedback: A novel data generation approach that isolates image quality impact by controlling for price and relevance, allowing clicks to be used as a valid metric for image quality evaluation.
LLM-Driven Weak Supervision: Demonstrated that LLMs (GPT-4) with CoT prompting can generate high-quality, interpretable image aesthetic labels that correlate strongly with human behavior, offering a cost-effective alternative to human annotation.
Cost-Efficient Model: Successfully trained a lightweight model (CLIP + MLP) that outperforms baselines, proving that expensive LLM inference is only needed for label generation, not real-time ranking.
Business Impact: Validated the approach through online A/B testing, showing significant revenue growth.

4. Experimental Results

Offline Evaluation

Metrics: Ordered Pair Accuracy (OPA) and Click Accuracy (CA).
Baselines: Random Guess, Historical CTR, and CLIP Score (zero-shot).
Performance:
- Image Score (Focal Loss): Achieved 67.75% OPA and 64.42% CA.
- Comparison: Significantly outperformed Random Guess (50%) and Historical CTR (which performed worse than random guessing, ~42-44%).
- Insight: Historical CTR failed because it is confounded by relevance; the Image Score successfully isolated quality.

Online A/B Testing (Web Platform)

Setup: Three groups: Control (Baseline), Additive Boost, Multiplicative Boost.
Key Metric: Average Transactions Per User (ATPU).
Results:
- Variant 1 (Additive): ATPU increased by 6.99% (statistically significant, t-value 2.22).
- Variant 2 (Multiplicative): ATPU increased by 6.66%.
- Observation: The improvements were significant on the Web platform but showed negative/neutral impacts on Mobile. The authors hypothesize web users are more sensitive to visual quality due to higher screen resolutions.

5. Significance and Future Work

Significance: This work bridges the gap between Deep Learning image quality assessment and practical e-commerce optimization. It proves that LLMs can serve as scalable "human proxies" for training ranking models, drastically reducing the cost of data labeling while improving explainability.
Limitations:
- AI-Generated Images: The system currently struggles to distinguish between high-quality real photos and AI-generated images, which can exploit the scoring system.
- Platform Variance: The model's effectiveness varies by device (Web vs. Mobile), suggesting the need for platform-specific weighting.
Future Directions:
- Integrating AI-generated content detection to prevent gaming the system.
- Developing continuous feedback loops for real-time monitoring.
- Refining the model to account for platform-specific user preferences.

In conclusion, the paper presents a robust, scalable framework for enhancing search relevance in C2C marketplaces by quantifying image quality, directly leading to a ~7% increase in sales transactions on the web platform.