Technical Summary: AdsWorldEngine

Problem Statement

Conversational advertising presents a distinct challenge compared to traditional query-based advertising. In sponsored search, user intent is typically explicit and contained within a short, standalone query. In conversational settings, commercial intent is often latent, accumulated, or implied across multiple turns of dialogue. A single user utterance (e.g., "cheaper ones" or "not in the Caribbean") lacks sufficient context to drive ad retrieval without interpreting the dialogue history and the assistant's prior responses.

Consequently, conversational ad serving is not merely a retrieval and ranking problem but a full ad serving problem. The system must simultaneously:

1. Decide suitability: Determine if the current turn is appropriate for ads without being intrusive.
2. Infer intent: Extract commercial intents from the conversation flow.
3. Execute serving: Call advertising tools (retrieval, ranking, pricing) and construct a diverse, relevant slate of ads.
4. Optimize judgment: Handle subjective decisions where errors have asymmetric costs (e.g., an inappropriate ad exposure harms user trust more than a missed opportunity).

Existing approaches often address individual components (e.g., ad generation or temporal placement) rather than the full end-to-end serving system.

Methodology: AdsWorldEngine Framework

The authors propose AdsWorldEngine, an agentic framework designed for production settings. The system operates as a gated decision process involving four core components:

1. Opportunity Gate

This component determines whether an ad should be shown for a specific turn. It combines hard production rules (policy, privacy) with a learned triggering model. The model is trained to abstain in sensitive or purely informational contexts and trigger only when the assistant's response or user query suggests a concrete commercial need.

2. Orchestrator

The Orchestrator acts as the agent that manages the ad serving flow for eligible turns. It performs three sub-tasks:

• State Resolution: Extracts constraints and preferences from the dialogue history.
• Intent Generation: Formulates 1–3 commercial intents (ad queries) based on the resolved state.
• Tool Calling & Slate Selection: Calls advertising tools (retrieval, relevance, ranking) and uses a reflection module to select a final slate of three ads, ensuring diversity and relevance.

3. Tool Set

This includes the downstream infrastructure: retrieval, relevance, and ranking models, as well as pricing and signal models. Unlike traditional systems where these are fixed, AdsWorldEngine treats them as co-evolving components.

4. Evaluator

An offline component that scores delivered ad slates based on user, advertiser, and publisher perspectives. It provides reward signals for training the Orchestrator and preference data for updating the tools.

Core Technical Innovations

A. Iterative Actor-Tool Optimization

The paper introduces a self-improving loop where the Orchestrator (actor) and the Tools co-evolve:

1. Orchestrator Training: The Orchestrator is first trained via Supervised Fine-Tuning (SFT) and then optimized using Agentic Reinforcement Learning (specifically GRPO) based on Evaluator rewards. This aligns intermediate decisions (intent generation, tool calls) with the final slate quality.
2. Tool Training: High-reward and low-reward rollouts from the improved Orchestrator are converted into preference data. For a given intent, ads from high-reward slates serve as positive examples, while ads from low-reward slates serve as hard negatives. This data is used to update the retrieval and relevance tools (e.g., via DPO or ranking losses).
3. Co-adaptation: This creates a feedback loop where the Orchestrator learns to use tools more effectively, and the tools improve to better support the Orchestrator's rewarded behaviors.

B. Label Grounded Judgment Modeling

To address subjective production decisions (triggering and relevance) where errors have asymmetric costs, the authors propose a data generation pipeline:

• Guideline-Based Labeling: Human annotators label data under explicit production guidelines.
• Thinking Traces: For each labeled example, a thinking trace is generated conditioned on the guideline and the human label, explaining the rationale.
• Reflection Filtering: A reflection step verifies that the trace supports the label and follows guidelines. Inconsistent rationales are discarded.
• Cost-Sensitive GRPO: For binary tasks (e.g., trigger vs. no trigger), the authors use a variant of Group Relative Policy Optimization (GRPO) that preserves asymmetric reward gaps. Unlike standard GRPO which scales rewards by group standard deviation (potentially obscuring cost differences), this variant uses group centering without scaling to maintain the magnitude of penalty gaps (e.g., penalizing false positives more heavily than false negatives).

Key Results

Offline Performance

Evaluated against the current production ad delivery system, AdsWorldEngine demonstrates significant improvements in offline metrics:

• Diversity: Improved by 60% (measured by pairwise cosine similarity of ad embeddings).
• Relevance: Improved by 80% (measured by conversation-to-ad relevance judgments).
• Iterative Gains: The iterative training process showed diminishing but consistent gains over three rounds, with the final system achieving a 62.87% diversity gain and 82.26% relevance gain over the baseline.

Online Performance

In a 20-day online A/B test deployed on Microsoft Copilot:

• Revenue Per Mille (RPM): Increased by 22%.
• Ad Coverage: Increased by 74%.

Judgment Modeling Performance

• Opportunity Gate: The cost-sensitive GRPO model reduced the False Positive Rate (FPR) by 39.07% while maintaining the True Positive Rate (TPR), outperforming both prompt-based baselines and SFT-only models.
• Evaluator: The relevance model reduced FPR by 12.71% with minimal TPR loss, achieving a 7.51% gain in balanced accuracy.

Significance and Claims

The paper claims that AdsWorldEngine provides a practical approach for building compound language-model systems where agents and tools co-adapt under production constraints. Its primary contributions are:

1. Formulating Conversational Ad Serving: Defining it as a gated agentic framework that integrates opportunity gating, intent formation, and slate selection.
2. Iterative Actor-Tool Optimization: Demonstrating that rewarded agent rollouts can simultaneously improve the agent's decision-making and the underlying retrieval/ranking tools, creating a self-improving loop.
3. Label Grounded Judgment Modeling: Introducing a pipeline that generates dense, guideline-aligned supervision with reflection filtering and cost-sensitive optimization to handle asymmetric production risks.

The authors emphasize that the system learns not only how to use advertising tools but also how to improve them from rewarded behavior, connecting conversational ad serving to broader research on optimizing compound language model systems.