Technical Summary: Unifying Ranking and Generation in Query Auto-Completion via RAG and Multi-Objective Alignment

Problem Formulation

Query Auto-Completion (QAC) is a critical first touchpoint in search systems, designed to suggest completions as users type to reduce keystrokes and clarify intent. Traditional QAC systems rely on a two-stage retrieve-and-rank paradigm: candidates are extracted from historical logs or catalogs, followed by a learning-to-rank model that scores them based on relevance and popularity.

The authors identify three fundamental limitations in this conventional approach:

1. Limited Coverage: Coverage is bounded by historical logs, failing to provide high-quality suggestions for novel or long-tail prefixes with sparse interaction history.
2. Complex Objective Optimization: Balancing competing objectives (relevance, safety, diversity, and groundedness) requires extensive manual feature engineering and heuristic blending, which often fails to capture complex interdependencies.
3. Lack of Holistic Optimization: Decoupled candidate generation and ranking prevent list-level optimization, often resulting in redundant or semantically similar suggestions.

While recent generative approaches using Large Language Models (LLMs) address coverage, they introduce new challenges: hallucination (suggesting non-existent items), lack of groundedness (ignoring actual searchable content), and safety risks (generating harmful completions).

Methodology

The paper proposes a unified framework that reformulates QAC as end-to-end list generation using Retrieval-Augmented Generation (RAG) and Multi-Objective Direct Preference Optimization (DPO).

1. Architecture and RAG Pipeline

The system operates as a multi-stage RAG pipeline (Figure 2):

• Candidate Generation: Candidates are sourced from two complementary channels: a Query Index (fast lookup from historical logs) and a Content Retriever (embedding-based retrieval from the catalog). This ensures high recall and grounds suggestions in actual catalog items.
• Prompt Construction: Contextual information is retrieved for all candidates, including catalog metadata (titles, ratings) and query features (frequency, conversion rates). This context is compiled into a structured prompt to ground the generation.
• List Generation: A fine-tuned LLM (the Generator) processes the prompt to synthesize a complete, ordered list of suggestions in a single forward pass, rather than ranking pre-generated candidates.

2. Training Pipeline

The Generator is trained via a four-stage process (Figure 3):

• Synthetic Data Generation: Due to the scarcity of high-quality labeled data, the authors employ an iterative critique-and-revision process. A "Teacher" LLM generates initial lists, which are analyzed by a "Critic" LLM for issues like redundancy or poor diversity. The Teacher then revises the output. This loop produces a high-quality synthetic corpus (~50k pairs), augmented with a small set of human-labeled data.
• Supervised Fine-Tuning (SFT): The Generator is fine-tuned on the combined synthetic and human-labeled dataset to learn the task structure and output format.
• Verifier Suite: A suite of verifiers is defined to operationalize six specific objectives:
  1. Relevance: Scored via a fine-tuned model using human judgments.
  2. Safety: A binary classifier trained on domain-specific safety policies.
  3. Engagement: Based on conditional conversion probabilities and historical rates.
  4. Catalog Groundedness: A rule-based check ensuring suggestions return non-empty search results.
  5. Context Groundedness: An LLM-as-judge verifying that suggestions are derivable from the retrieved context (preventing hallucinations).
  6. Diversity: An adjusted entropy formula penalizing near-duplicate search results.
  7. Format: A rule-based check for parseable output structure.
• Multi-Objective Direct Preference Optimization (DPO): The SFT-trained model is further aligned using DPO. Preference pairs are constructed by sampling candidate lists, scoring them with the composite reward function (weighted sum of verifier scores), and selecting pairs with significant reward margins. The model is optimized to prefer higher-reward lists while maintaining proximity to the reference policy.

3. Hybrid Serving Architecture

To balance quality and strict latency constraints, the authors deploy a two-tier architecture:

• Offline Pre-computation: A Large Generator pre-computes suggestions for high-frequency prefixes, which are cached for sub-millisecond retrieval.
• Online Real-time Inference: A Compact Generator handles cache misses (long-tail or new prefixes) in real-time, ensuring coverage for emerging queries within acceptable latency limits (~100ms).

Key Contributions

The paper claims four primary contributions:

1. Reformulation of QAC: Shifting from retrieve-and-rank to end-to-end list generation with multi-objective optimization, enabling holistic balancing of competing goals.
2. Verifier Suite: Defining and deploying a comprehensive suite of rule-based, model-based, and LLM-as-judge verifiers tailored to QAC, covering relevance, safety, groundedness, engagement, diversity, and format.
3. Synthetic Data Methodology: Presenting a pipeline combining RAG, multi-objective DPO, and iterative critique-revision to generate high-quality synthetic data for training.
4. Production Validation: Validating the approach on a large-scale commercial search platform with extensive offline, human, and online evaluations.

Results

The framework was evaluated on a proprietary dataset of 50,000 query prefixes and deployed in a live A/B test with 10% of production traffic.

Offline Metrics

Compared to a traditional Learning-to-Rank (LTR) baseline and hybrid baselines:

• Coverage: The full model achieved ~93% coverage, significantly higher than the LTR baseline (84.6%) and competitive with generative baselines.
• Safety & Groundedness: DPO tuning significantly reduced unsafe rates (0.65% vs. 0.79% in baselines) and catalog ungrounded rates (0.49% vs. 0.67%), demonstrating effective mitigation of generative risks.
• Relevance & Diversity: The model achieved relevance scores of ~0.687 and diversity scores in the 74–76 range, outperforming the LTR baseline in relevance and matching or exceeding baselines in diversity.

Human Evaluation

• Item-wise Relevance: The Full model achieved the highest score (0.699) compared to the LTR baseline (0.653).
• Pairwise Preference: Models with DPO showed strong preference scores (+0.40 to +0.69) over the baseline, indicating users perceived the suggestions as more useful and diverse.

Online A/B Test

In a controlled experiment with 10% of traffic:

• Keystrokes: Achieved a 5.44% reduction in characters typed by users.
• Adoption: Achieved a 3.46% increase in suggestion click-through rate (CTR).
• All improvements were statistically significant (p < 0.05).

Significance and Claims

The authors position this work as a paradigm shift for production QAC systems. They claim that unifying ranking and generation through RAG and multi-objective alignment provides an effective solution that overcomes the fundamental trade-offs of traditional systems.

Specifically, the paper asserts that:

• End-to-end generation powered by LLMs, when constrained by RAG and multi-objective DPO, can simultaneously optimize for relevance, safety, groundedness, and diversity without relying on extensive manual feature engineering.
• The proposed framework is production-validated, proving that complex generative models can be deployed under strict latency constraints via hybrid serving architectures.
• The methodology is generalizable beyond mobile app search to other domains (e-commerce, media) where users rely on query suggestions to explore large, fixed catalogs.

The work establishes a framework that moves beyond simple retrieval or unconstrained generation, offering a principled approach to handling the complex interdependencies of modern search auto-completion.