💬 NLP

Interactor: Agentic RL oriented Iterative Creation for Ad Description Generation in Sponsored Search

This paper introduces Interactor, an agentic reinforcement learning framework that iteratively refines ad descriptions in sponsored search through multi-turn interactions with generative reward models, significantly improving knowledge richness and landing page consistency while achieving successful online deployment.

Original authors: Penghui Wei, Jiayu Wu, Chao Ye, Zhi Guo, Shuanglong Li, Lin Liu

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Penghui Wei, Jiayu Wu, Chao Ye, Zhi Guo, Shuanglong Li, Lin Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a lemonade stand, but instead of just shouting "Lemonade!" (which is like an ad title), you have a long, detailed menu board where you can explain why your lemonade is special.

This paper introduces a new system called INTERACTOR that acts like a super-smart, self-correcting marketing assistant to write those long, detailed descriptions for search ads.

Here is how it works, broken down into simple concepts:

1. The Problem: The "One-Shot" Mistake

Most current AI tools try to write an ad description in one single go, like a student taking a test without a scratchpad. They get it right sometimes, but often they make two big mistakes:

They make things up: They might say your lemonade uses "organic lemons" when your supplier only sells regular ones (this is called being unfaithful to the landing page).
They are too vague: They might just say "It tastes good" instead of explaining that the lemons are hand-picked at dawn (missing world knowledge).

Because the AI can't "see" the user's reaction immediately, it keeps making the same mistakes.

2. The Solution: The "Coach and Player" Loop

INTERACTOR changes the game. Instead of a one-shot test, it turns ad writing into a multi-turn conversation between a Player (the AI writing the ad) and a Coach (a set of specialized AI judges).

Here is the step-by-step loop:

The Player Makes a Draft: The AI writes a first version of the ad.
The Coach Reviews It: The Coach doesn't just give a score (like "8/10"). It acts like a strict editor with a red pen. It says:
- "You claimed 'free shipping,' but your website says 'delivery fee applies.' That's a lie. Fix it."
- "You didn't mention that black sesame helps with hair health, which is what the user is actually asking about. Add that."
The Player Refines: The AI reads the Coach's notes, thinks about them, and rewrites the ad to fix the specific errors.
Repeat: This happens several times until the ad is perfect.

3. The Secret Sauce: "Agentic RL"

The paper calls this Agentic Reinforcement Learning. Think of it like training a dog, but the dog is smart enough to understand why it got a treat or a correction.

Traditional RL: The AI gets a "Good Job!" or "Bad Job!" signal. It guesses what to do next.
INTERACTOR (Agentic RL): The AI gets a reasoning feedback. It knows exactly which sentence was wrong and why. This allows it to learn much faster and write much better descriptions.

4. The Results: From "Fluff" to "Facts"

The paper tested this on real data from Baidu's search engine.

Before: AI ads were often "fluffy" (full of empty words) or "hallucinated" (making up facts).
After: The INTERACTOR system produced ads that were:
- Knowledge-Rich: They actually answered the user's question with real facts.
- Faithful: They stuck strictly to what the advertiser's website actually offered.
- More Clicks: Because the ads were better, more people clicked on them, and the company made more money.

5. Real-World Impact

The paper states that since May 2026, this system has been running live on a major search engine. It isn't just a theory; it is currently helping advertisers write better ads and helping users find more relevant information, all while boosting the search engine's revenue.

In a nutshell: INTERACTOR is like giving an AI writer a team of expert editors who don't just grade the paper, but sit down and help the writer fix every single mistake until the ad is perfect.

` tags, used to understand context and plan.

Response: Enclosed in specific tags, representing real actions:
- <retrieve>: Requests a search engine to inject relevant world knowledge.
- <create>: Generates or refines the ad description.
- <reward>: Requests evaluation from reward models.

2. Customized Environment

The environment provides observations and rewards through two main components:

Generative Reward Models (GenRMs): Unlike scalar reward models, these are rubric-based models (implemented as frontier black-box LLMs) that evaluate specific quality dimensions:
- Knowledge Capacity: Checks if the description contains world knowledge relevant to the user's intent.
- Landing Page Consistency: Verifies that all selling points are strictly entailed by the landing page information, preventing unfaithful exaggerations.
- Output: GenRMs provide both a binary signal (pass/fail) and fine-grained reasoning feedback (e.g., explaining why a phrase is unfaithful or what knowledge is missing).
Search Engine: Retrieves relevant knowledge from a massive corpus of encyclopedic web pages to augment the context, ensuring the generated text is grounded in external knowledge.

3. Agentic RL Optimization

The framework optimizes the policy using Group Sequence Policy Optimization (GSPO), a critic-free method.

Rollouts: Each episode is a multi-turn rollout sequence $\tau$ .
Reward Signal: The total reward $r_t$ is a weighted sum of the binary quality signals from GenRMs (Knowledge Capacity and Landing Page Consistency) and a predicted CTR score derived from a model trained on historical user feedback logs.
Policy Update: The advantage is estimated using group-based normalization across multiple rollouts per prompt. During gradient computation, only the tokens generated by the policy are updated, while environment-provided tokens (observations) and task inputs are masked.

Key Contributions

Shift in Focus: The paper identifies the industrial necessity of shifting from CTR-driven title generation to informative description generation, emphasizing the explicit modeling of world knowledge to satisfy user intent.
Iterative Framework: It proposes INTERACTOR, a multi-turn framework where an LLM policy is optimized via agentic RL. This allows for continuous improvement in knowledge capacity and landing page consistency through detailed, reasoning-based feedback rather than scalar signals alone.
Empirical Validation: Experiments on large-scale industrial datasets demonstrate that INTERACTOR significantly outperforms state-of-the-art single-turn approaches in generating knowledge-rich and faithful descriptions.

Experimental Results

The framework was evaluated on a proprietary dataset of 1 million user-ad pairs from a leading Chinese search engine.

Offline Metrics: INTERACTOR outperformed baselines (including CTOP, DIVER, and a single-turn RL production model) across all dimensions:
- Faithfulness: Achieved 0.872, significantly higher than the single-turn RL baseline (0.726) and other competitors.
- Informativeness: Scored 0.715, surpassing the single-turn RL baseline (0.693).
- Attractiveness (Predicted CTR): Achieved 12.4%, slightly outperforming the single-turn baseline (12.3%).
Ablation Studies:
- Removing the reasoning feedback from GenRMs ("w/o reasoning") caused a dramatic performance drop, confirming that fine-grained feedback is the critical factor for the iterative process.
- The final turn of the iteration consistently outperformed the penultimate turn, validating the continuous improvement capability.
Online A/B Test: Deployed in a leading search ads system covering over 140,000 advertisers since May 2026. The results showed a +0.74% increase in ad revenue and a +7 percentage point increase in the top-grade rate for overall content quality in human evaluations.

Significance and Claims

The paper claims that the INTERACTOR framework represents a significant advancement in ad text generation by moving beyond the limitations of scalar-reward-based single-pass generation. By leveraging agentic RL with multi-turn iteration and generative reward models, the system effectively handles the complex constraints of long-form ad descriptions, balancing the need for informative world knowledge with strict adherence to landing page facts. The successful online deployment demonstrates the practical viability of this approach for improving both advertiser revenue and user experience in real-world sponsored search systems.