Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments

This paper addresses the scarcity of expert textual relevance labels in large-scale app store search by leveraging a specialized, fine-tuned LLM to generate millions of high-quality labels, which, when used to augment the production ranker, significantly improves both offline metrics and real-world conversion rates, particularly for tail queries lacking reliable behavioral data.

The Gist

Imagine the App Store is a massive, bustling library with millions of books (apps). Your job is to be the librarian who helps visitors find exactly what they want.

To do this job well, you need to answer two questions for every book on the shelf:

1. The "Click" Test (Behavioral Relevance): Do people actually pick this book up and buy it when they see it? (This is easy to track; you just count the sales.)
2. The "Description" Test (Textual Relevance): Does the book's title and summary actually match what the visitor asked for? (This is hard to track; you need a human expert to read the book and the request and decide if they match.)

The Problem: The "Expert" Bottleneck

The library is so huge that you have millions of "Click" records, but you only have a tiny handful of "Description" records because hiring human experts to read and judge every book is too slow and expensive.

Because of this, your librarian system is great at showing popular books, but it sometimes fails at showing the right book for specific, weird, or rare requests (like "an app for knitting cats in space"). Without enough expert judges, the system gets lost on these rare requests.

The Solution: The "Super-Intern" (The LLM)

The authors of this paper decided to hire a Super-Intern (a Large Language Model, or LLM) to help the human experts.

Instead of asking the intern to just guess, they trained it by showing it thousands of examples of how the real human experts judged books. They asked the intern: "Here is a request and a book. Based on what you learned from the experts, does this book match the request?"

The Big Discovery:
They tried two types of interns:

1. A Giant Intern (a huge, pre-trained AI model) who knows everything about the world but hasn't been trained on your specific library rules.
2. A Specialized Intern (a smaller model that was specifically trained on your library's rules).

The Surprise: The Specialized Intern was far better than the Giant Intern. Even though the Giant Intern was ten times bigger, the Specialized Intern, having studied your specific rules, gave much more accurate answers. It was like a junior employee who knows your company's specific culture beating a genius who just walked in off the street.

The Result: A "Force Multiplier"

Once they found the best Specialized Intern, they let it work overnight. In a single night, it generated millions of "Description" labels, effectively doing the work of thousands of human experts.

They then fed all this new data back into the librarian system.

The Payoff: The "Pareto Frontier" Shift

In the paper, they talk about something called the "Pareto Frontier." Let's translate that:
Imagine a graph where the X-axis is "How much people click" and the Y-axis is "How accurate the description is." Usually, if you try to improve one, the other gets worse. You have to make a trade-off.

What happened here?
By adding the millions of new labels from the Super-Intern, the librarian system didn't just move up or to the right; it moved diagonally outward.

• It got better at matching descriptions (Textual Relevance).
• AND it also got better at getting people to click (Behavioral Relevance).

It's as if the librarian suddenly became smarter at reading, which made them so much more helpful that people started buying more books, too.

The Real-World Win: Saving the "Tail" Queries

The most exciting part happened with the "Tail Queries."

• Head Queries: "iPhone 15 case." (Everyone searches this; we have tons of data on what people click).
• Tail Queries: "App to translate ancient Sumerian poetry." (Very few people search this; we have almost no data on what people click).

For the rare, weird searches, the old system was guessing in the dark. The new system, powered by the Super-Intern's millions of "Description" labels, suddenly knew exactly what to show.

The Bottom Line:
In a real-world test across the whole world, this new system increased the number of people downloading apps by 0.24%.

• Is 0.24% small? For a single person, maybe.
• For the App Store? That's millions of extra downloads and happy users.

In short: They used a smart, specialized AI to act as a "force multiplier" for human experts, generating millions of new clues to help the App Store find the perfect app for even the weirdest searches, making the whole system better for everyone.

Technical Summary

Here is a detailed technical summary of the paper "Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments."

1. Problem Statement

Large-scale commercial search systems, such as the Apple App Store, rely on optimizing two complementary objectives to ensure user satisfaction:

1. Behavioral Relevance: Based on user actions (clicks, downloads). This data is abundant but noisy and biased toward popular items.
2. Textual Relevance: Based on the semantic fit between a query and an app's metadata, judged by human experts. This data is high-quality but scarce and expensive to produce.

The Core Challenge: The scarcity of high-quality textual relevance labels creates a bottleneck in multi-objective training. Without sufficient textual labels, the ranking model cannot effectively learn to surface semantically relevant results, particularly for "tail queries" (low-frequency searches) where behavioral data is non-existent or unreliable.

2. Methodology

The authors propose a two-stage approach to overcome label scarcity by using Large Language Models (LLMs) as scalable offline annotators ("LLM-as-a-Judge").

A. LLM Relevance Label Generation

Instead of using LLMs as real-time rerankers, the team uses them offline to generate millions of pointwise textual relevance labels.

• Model Selection: They evaluated three configurations:
  1. Pretrained 3B parameter model.
  2. Pretrained 30B parameter model.
  3. Fine-tuned 3B model: A smaller model fine-tuned on existing human judge data.
• Prompting Strategy: The system uses few-shot prompting with specific guidelines on how to evaluate query-app pairs based on metadata.
• Key Finding: Contrary to the intuition that larger models are always better, the fine-tuned 3B model significantly outperformed the much larger 30B pretrained model. The fine-tuned model learned the specific internal rubric of the human judges more effectively.
• Output: The optimal model (FT-3B) generated millions of high-quality textual labels across multiple languages and storefronts.

B. Multi-Objective Ranker Training

The production ranker is trained using a Multi-Objective Optimization (MOO) framework that jointly optimizes for behavioral and textual relevance.

• Data Mixing: The training dataset is constructed by mixing two distinct sources:
  1. Behavioral labels (clicks/downloads) from search logs.
  2. Textual labels (Human + LLM-generated).
• Scalarization: The authors use a weighted mix of these data sources as a tunable hyperparameter. This allows them to control the influence of each objective and explore different points on the Pareto frontier (the set of optimal trade-offs between objectives).
• Integration: The LLM-generated labels are treated as ground truth for the textual objective, effectively expanding the training data volume without requiring additional human annotation.

3. Key Contributions

1. Systematic LLM Evaluation: Demonstrated that a specialized, fine-tuned smaller model is superior to massive pretrained models for generating domain-specific relevance labels, offering a cost-effective production path.
2. Scale: Generated millions of new textual relevance labels across multiple languages, vastly expanding the training corpus.
3. Pareto Frontier Shift: Proved that augmenting training data with LLM labels does not just improve one metric at the expense of another; it achieves a strict Pareto improvement, simultaneously boosting both textual and behavioral relevance.
4. Industrial Validation: Validated the approach through rigorous offline metrics and a worldwide online A/B test on the App Store.

4. Experimental Results

Offline Evaluation

• Label Quality: The Fine-tuned 3B model achieved an F1 score of 0.800, significantly outperforming the Pretrained 30B model (F1 0.382).
• Ranker Performance: The "LLM-augmented" model showed improvements in NDCG@k (Normalized Discounted Cumulative Gain) for both objectives compared to the baseline production model:
  • Textual NDCG@1: Increased from 0.867 to 0.868.
  • Behavioral NDCG@1: Increased from 0.646 to 0.652.
  • This confirmed a shift outward of the Pareto frontier, meaning the new model strictly dominates the old one.

Online A/B Test

• Conversion Rate: The LLM-augmented model achieved a statistically significant +0.24% increase in conversion rate (defined as search sessions resulting in at least one app download) across worldwide traffic.
• Tail Query Impact: The most significant gains were observed in tail queries (low-frequency searches).
  • Reasoning: For tail queries, behavioral signals (clicks/downloads) are sparse or absent. The LLM-generated textual labels provided a robust signal where behavioral data failed, effectively bridging the relevance gap for long-tail searches.
  • The improvement was observed in 89% of App Store storefronts.

5. Significance

This paper provides a practical blueprint for large-scale search systems to overcome the "data scarcity" bottleneck in relevance labeling.

• Cost Efficiency: It proves that fine-tuning smaller, efficient models is more effective and cheaper than relying on massive pretrained models for specific annotation tasks.
• Scalability: It transforms the "LLM-as-a-Judge" paradigm from an evaluation tool into a force multiplier for training data generation.
• Real-World Impact: The approach successfully improved a mature, high-stakes production system (Apple App Store), demonstrating that synthetic labels can be trusted to drive real business metrics, particularly in scenarios where human data is insufficient.