LERA: LLM-Enhanced RAG for Ad Auction in Generative Chatbots

Technical Summary: LERA: LLM-Enhanced RAG for Ad Auction in Generative Chatbots

Problem Statement
The integration of advertising into Large Language Model (LLM)-based chatbots presents a commercialization opportunity but introduces unique challenges in balancing relevance, efficiency, and user experience. Unlike traditional sponsored search or display advertising with predefined slots, chatbot advertising requires determining whether and how to weave advertisements into generated responses. Existing paradigms face significant trade-offs:

1. Retrieve-then-Generate: Methods like those by Hajiaghayi et al. [9] decouple retrieval from generation for efficiency but rely solely on text embedding similarity. This often leads to commercial misinterpretation, failure to capture nuanced intent, and repetitive ad insertions.
2. Generate-then-Retrieve: Approaches leveraging LLM contextual understanding for scoring often require multiple LLM calls or advertiser-specific scoring, incurring prohibitive latency and inference costs.
3. LLM-Only Scoring: Directly using LLMs to score all advertisers is computationally expensive due to the need for parallel pre-filling across large candidate sets in high-throughput inference engines.

Methodology
The authors propose LERA (LLM-Enhanced RAG for Ad Auction), a two-stage retrieve-then-generate auction framework designed to balance accuracy and throughput.

1. Stage 1: Embedding-Based Coarse Filtering

   • To reduce the candidate pool from $N$ advertisers to a small set of size $K$ ($K \ll N$), the system first prompts an LLM to generate concise intent keywords ($w$) from the user query ($x$).
   • A relevance score is computed using the normalized cosine similarity between the keyword embedding and each advertiser’s description embedding: $s^{(1)}_i = \text{Norm}(\text{Similarity}(\text{Emb}(w), \text{Emb}(d_i)))$.
   • This score is weighted by the advertiser’s bid ($b_i$) to form $\text{Score}^{(1)}_i = s^{(1)}_i \cdot b_i$. The top-$K$ advertisers form the candidate set $S$.

2. Stage 2: LLM-Logit Fine Ranking

   • The candidate set $S$ is extended with a "no-insertion" option ($\phi$) to form $\hat{S} = S \cup \{\phi\}$.
   • The LLM is queried with the original query and candidate descriptions to produce logits over these candidates.
   • These logits are normalized via softmax to obtain refined organic relevance scores: $s^{(2)}_j = \frac{\exp(z_j)}{\sum_{k \in \hat{S}} \exp(z_k)}$.
   • The final auction score is $\text{Score}^{(2)}_j = s^{(2)}_j \cdot b_j$ (with $b_\phi = 1$). The winner $i^*$ is the candidate with the highest score. If $i^* = \phi$, no ad is inserted.

3. Critical-Value Payment Rule

   • To ensure incentive compatibility (truthfulness) in this two-stage process, a critical-value payment rule is designed. The winning advertiser pays the minimum bid required to remain the winner, accounting for both thresholds:
     $$p_{i^*} = \max \left( \frac{\text{Score}^{(1)}_{(K+1)}}{s^{(1)}_{i^*}}, \frac{\text{Score}^{(2)}_{(2)}}{s^{(2)}_{i^*}} \right)$$
   • This ensures the winner pays enough to both enter the Stage 1 candidate set and defeat competitors in Stage 2. The paper proves this mechanism is incentive-compatible for utility-maximizing advertisers under independence and unbiasedness assumptions.

4. Extension to Multi-Ad Insertion

   • The framework extends to dynamic dialogue flows by dividing responses into segments. For each segment, the auction is conditioned on the query and response prefix, allowing for diverse ad insertions across turns while maintaining controllable latency.

Key Contributions

• Framework Design: Proposal of LERA, a context-aware ad auction framework combining embedding-based coarse filtering with LLM-logit-based fine ranking.
• Auction Mechanism: Design of a critical-value payment rule tailored to the two-stage allocation process, ensuring truthfulness by accounting for both candidate-set entry and final-stage competition.
• Performance Trade-off: Demonstration that LERA captures nuanced commercial intent and improves insertion diversity while maintaining a favorable accuracy–efficiency trade-off compared to embedding-only and LLM-only alternatives.

Experimental Results
Experiments were conducted on a synthetic dataset of 100 advertisers across 8 categories, using Qwen3 models of varying scales (4B, 30B, 235B) and SGLang for inference.

1. Single Ad Selection Accuracy:

   • LERA consistently outperformed the Embedding-only baseline across complex intent, intent shift, and negative refinement scenarios.
   • For smaller models (4B and 30B), LERA significantly outperformed the "LLM-only" variant (which scores all $N$ advertisers), demonstrating that coarse filtering reduces the difficulty of fine-grained ranking.
   • The 235B LLM-only model achieved perfect accuracy but suffered from severe throughput bottlenecks.

2. Multi-Ad Insertion & Query Satisfaction:

   • In dynamic multi-ad scenarios, LERA improved query satisfaction (proportion of requested categories fulfilled) by 20–40% over the Embedding-only baseline.
   • LERA outperformed ablation variants (Keyword-Match, Cosine-LLM), confirming the necessity of both keyword-aware filtering and LLM-logit ranking.

3. Efficiency and Latency:

   • LERA (LLM-only) reached throughput bottlenecks quickly due to parallel pre-filling costs.
   • LERA incurred controllable latency overhead. While latency increased at higher concurrency levels compared to Embedding-only, the throughput remained viable for real-time applications.
   • Ablation studies on the candidate set size $K$ showed that increasing $K$ improves satisfaction but reduces throughput, highlighting a tunable trade-off.

Significance and Claims
The paper claims that LERA represents a step toward the practical deployment of auction-based ad insertion in LLM-powered chatbots. It addresses the limitations of existing methods by preserving the efficiency of retrieval-based allocation while leveraging LLM contextual judgment only on a small, pre-filtered candidate set. The authors emphasize that this approach allows for the integration of advertising into generative systems in a way that is contextually appropriate, commercially meaningful, and compatible with truthful auction-based pricing, without incurring the prohibitive costs of full LLM-based scoring for all candidates.