🤖 AI

AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Search

The paper proposes AutoRelAnnotator, a cost-efficient system that combines domain-specific fine-tuning, a cascading architecture, and per-class isotonic calibration to generate high-quality relevance annotations at scale for sponsored search, successfully processing over 150 million queries while significantly reducing human labeling costs and compute expenses.

Original authors: Md Omar Faruk Rokon, Shasvat Desai, Hong Yao, Kuang-chih Lee

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Omar Faruk Rokon, Shasvat Desai, Hong Yao, Kuang-chih Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a massive online store, like Walmart. Every time a customer types a search query (like "running shoes"), your computer has to decide which products are the most relevant to show them. To make this decision, the computer needs to learn from examples. But who teaches the computer? Humans.

The problem is that hiring humans to label millions of search results is slow (taking days) and expensive (costing hundreds of thousands of dollars). On the other hand, using fancy, expensive AI models (like GPT-4) to do the labeling is faster, but they aren't very good at understanding your specific store's products, leading to mistakes.

The authors of this paper, from Walmart Global Tech, built a smart system called AutoRelAnnotator to solve this. Think of it as a highly efficient "Triage System" for labeling search results.

Here is how it works, broken down into simple concepts:

1. The Problem with "One-Size-Fits-All" AI

Imagine you have a team of experts. If you ask a world-famous professor (a huge AI model) to do a simple task like "Is this a red apple?", they might overthink it, take a long time, and still get it wrong because they aren't a fruit specialist.
The paper found that standard, off-the-shelf AI models were only about 68–70% accurate at judging relevance for their specific store. They needed 89% accuracy to be useful.

2. The Solution: A Three-Layer "Filter" (The Cascade)

Instead of asking one giant, expensive AI to do everything, the authors built a relay race with three runners, each getting slightly more expensive but also smarter.

Runner 1 (The Sprinter): A small, fast, cheap model (Cross-Encoder). It looks at the search term and the product title. It's very fast (5 milliseconds).
Runner 2 (The Middle Distance): A medium-sized model (Gemma-2B). It takes about 50 milliseconds.
Runner 3 (The Marathoner): A large, powerful model (LLaMA-8B). It takes 200 milliseconds.

How the race works:
The system asks Runner 1 to make a guess.

If Runner 1 is very confident (e.g., "I'm 95% sure this is a perfect match!"), the system accepts the answer and stops. No need to call the expensive runners.
If Runner 1 is unsure (e.g., "I think it's a match, but I'm only 60% sure"), it passes the baton to Runner 2.
If Runner 2 is also unsure, it passes to Runner 3.
If even Runner 3 is confused, the three models vote together to make a final decision.

The Magic: This "cascade" approach means the system only uses the expensive, slow models for the tricky questions. For the easy questions (which are most of them), it uses the cheap, fast model. This cuts the computing cost in half without losing accuracy.

3. The Secret Sauce: "Calibration" (The Confidence Coach)

There's a catch: AI models often lie about how sure they are. They might say, "I'm 90% sure," when they are actually only 60% sure. If the system trusts this lie, it stops too early and makes a mistake.

The authors added a Calibration Coach (specifically, "per-class isotonic calibration").

Think of this coach as a referee who watches the runners and says, "Hey, when you say you are 90% sure on 'Class A', you are actually only 80% sure. Let's adjust your confidence score."
The paper found that fixing these confidence scores for each specific type of answer (e.g., "Perfect Match" vs. "Bad Match") separately helped the system route queries more accurately. It added a small but statistically significant boost to the final accuracy.

4. The Result: Training the Models First

Before the relay race even starts, the authors did something crucial: they fine-tuned all three models.

Instead of using generic AI models that know everything about the world but nothing about your store, they trained these models specifically on 1.2 million examples of their own search data.
The Analogy: It's like taking a general doctor and training them specifically on "Walmart's inventory." Suddenly, they become a specialist.
This step alone boosted accuracy from ~68% to ~89%.

The Bottom Line

By combining these three ingredients, the team achieved a "sweet spot":

Fine-tuning made the models accurate (the "Brain").
Cascading made the process cheap and fast (the "Economy").
Calibration made sure the models knew when to stop and when to ask for help (the "Trust").

Real-World Impact:

Speed: What used to take human teams 5 days to label now takes 1 to 3 hours.
Volume: They processed over 150 million annotations.
Cost: They cut the computing cost in half compared to just using the big, powerful model for every single query.

In short, they built a smart, self-correcting assembly line that labels search results as well as a team of experts, but at a fraction of the cost and time.

Technical Summary: AutoRelAnnotator

Problem Statement
Relevance annotations are critical for search ranking systems, serving as the foundation for training data preparation, NDCG evaluation, and root cause analysis. However, generating high-quality annotations at scale faces two primary bottlenecks:

Human Annotation: While accurate, it is slow (3–5 day turnaround for batches >10k) and expensive ($0.50/annotation), creating a bottleneck for rapid experimentation and model iteration.
Off-the-Shelf LLMs: Generic Large Language Models (e.g., GPT-4, Claude-3) suffer from low accuracy on domain-specific tasks (68–70% on a 5-class relevance task) compared to human inter-annotator agreement (~89%). Furthermore, existing model cascade methods that route queries between off-the-shelf LLMs cannot exceed the accuracy ceiling of their constituent models.

Methodology: Calibrated Model Cascades
The authors propose AutoRelAnnotator, a systematic approach for offline relevance annotation that decouples accuracy improvement from cost reduction. The system employs a three-stage cascade of progressively larger, domain-specific fine-tuned classifiers, enhanced by per-class isotonic calibration.

System Architecture:
- Stage 1: A lightweight Cross-Encoder (DeBERTa-v3-base, 184M parameters) processes query-title pairs in ~5ms.
- Stage 2: A Gemma-2B model fine-tuned with LoRA ( $r=16, \alpha=16$ ), inference time ~50ms.
- Stage 3: A LLaMA-8B model (Meta LLaMA-3-8B) with similar LoRA fine-tuning, inference time ~200ms.
- Fallback: If no model meets its confidence threshold, an ensemble fallback using majority voting (with a tie-breaker biasing toward higher relevance) is applied.
Domain-Specific Fine-Tuning:
Unlike generative approaches requiring regex parsing, the models use classification heads to produce well-defined probability distributions. All models are fine-tuned on 1.2M query-product pairs from sponsored search data. This step is the primary driver of accuracy, raising performance from ~68% (off-the-shelf) to ~89%.
Per-Class Isotonic Calibration:
The authors observe that raw confidence scores are miscalibrated, with models being overconfident on extreme classes (0, 4) and underconfident on middle classes (1, 2, 3). To address this, they apply per-class isotonic regression (following Zadrozny and Elkan) rather than global calibration.
- Separate calibration functions $f_c$ are learned for each predicted class $c$ .
- This ensures that routing decisions are based on reliable, class-conditional confidence scores, allowing the system to route high-confidence predictions to cheaper models while deferring ambiguous cases to larger models.
Cascade Logic:
Queries are routed through the cascade (Cross-Encoder $\to$ Gemma-2B $\to$ LLaMA-8B). At each stage, if the calibrated probability $p^*_m$ exceeds a threshold $\tau_m$ , the prediction is returned. Otherwise, the query proceeds to the next, more expensive model.

Key Contributions
The paper makes three specific contributions:

Orthogonal Optimization: The authors demonstrate that accuracy and cost are orthogonal. Fine-tuning drives the majority of accuracy gains (~20 points), while cascading reduces compute costs by ~50% without sacrificing accuracy.
Per-Class Calibration: They introduce per-class isotonic calibration as a critical component of the cascade. This adds a statistically significant gain (+0.6 points over the strongest global calibration baseline, $p<0.05$ ) by better handling class-conditional miscalibration.
Production Validation: The system was deployed across six offline use cases (including training data prep, NDCG evaluation, and root cause analysis), processing over 150 million annotations. It reduced turnaround time from 5 days to 1–3 hours.

Results

Accuracy: The final cascade achieves 89.1% accuracy.
- Fine-tuning alone (single LLaMA-8B) achieves 88.9%, closing a 20.7-point gap over GPT-4 (68.2%).
- Cascading off-the-shelf LLMs (GPT-4 $\to$ Claude-3) only yields a 3.2-point improvement, confirming the accuracy ceiling of uncalibrated routing.
- Per-class calibration improves the cascade accuracy from 88.2% (uncalibrated) to 89.1%.
Cost Efficiency: The production configuration achieves 89.1% accuracy with 50.1% cost savings compared to running the largest model (LLaMA-8B) on every query.
Routing Distribution: The lightweight Cross-Encoder resolves 74.5% of queries with 91.2% accuracy (higher than its standalone performance due to calibration filtering). The ensemble fallback handles only 7.1% of queries.

Significance and Claims
The paper positions AutoRelAnnotator as a building block for scalable, high-quality offline annotation pipelines in search and advertising.

Modularity: The cascade is modular; individual models can be updated without affecting the routing logic.
Practicality: It offers a viable alternative to expensive human labeling and inaccurate generic LLMs for offline tasks where latency is not the primary constraint.
Limitations: The authors note that the 20-point accuracy gain is specific to the fine-tuning of this domain (sponsored search) and that the system requires substantial labeled data (1.2M for fine-tuning, 50K for calibration). The evaluation is limited to a single domain, and proprietary data prevents external reproduction, though the methodology is detailed for replication.

The work concludes that by decoupling accuracy (via fine-tuning) and cost (via cascading), organizations can achieve high-quality annotations at a fraction of the computational cost, enabling faster experimentation cycles in search and advertising systems.