💬 NLP

GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketing

This paper introduces GEM-Bench, the first comprehensive benchmark for ad-injected response generation in Generative Engine Marketing, which provides curated datasets, a multi-dimensional evaluation metric, and baseline solutions to reveal the trade-off between engagement and user satisfaction in current methods while guiding future research.

Original authors: Silan Hu, Shiqi Zhang, Yimin Shi, Xiaokui Xiao

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Silan Hu, Shiqi Zhang, Yimin Shi, Xiaokui Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a helpful, knowledgeable librarian (an AI chatbot) to ask for directions or advice. In the old days, this librarian would just give you the answer. But now, the library is trying to make money by slipping in little advertisements for local shops right into the librarian's sentences.

The problem? If the librarian just shouts "BUY THIS!" in the middle of your directions, it feels rude and confusing. If they are too subtle, nobody sees the ad.

This paper introduces GEM-Bench, which is essentially a "report card" and a "playground" for figuring out how to do this advertising thing without annoying the users.

Here is the breakdown of their work in simple terms:

1. The Problem: The "Salesman" vs. The "Helper"

The authors noticed that existing AI benchmarks (tests used to grade AI) are bad at testing ads. They are like giving a math test to a chef; it doesn't make sense.

The Old Way (Ad-Chat): Imagine the librarian is told, "You are a salesperson first, a helper second." They try to weave the ad into their very first thought. The result? They often give wrong directions just to mention a product, or they sound like a pushy salesman.
The Goal: We need an AI that acts like a helpful friend first, and only then suggests a product naturally, like saying, "Take the bus, and by the way, that bus company has a great app."

2. The Solution: GEM-Bench (The Test Kitchen)

To fix this, the team built GEM-Bench, a toolkit with three main parts:

The Datasets (The Ingredients): They gathered three different "menus" of questions.
- Two menus are for Chatbots (like asking for travel tips or recipe ideas).
- One menu is for Search Engines (like typing "best running shoes" to get a quick summary).
- They also have a "catalog" of real ads to insert.
The Grading System (The Taste Test): Instead of just checking if the grammar is right, they created a new way to grade the answers. They look at:
- Flow: Does the sentence sound smooth, or does it feel like a bump in the road?
- Trust: Do you believe the answer, or does it feel like a scam?
- Click-Through: Did the user actually want to click the link?
The New Method (Ad-LLM): They built a "team of robots" (a multi-agent framework) to do the work.
- Robot 1 writes a perfect answer with no ads.
- Robot 2 finds the best ad to match that answer.
- Robot 3 figures out exactly where to slip the ad in so it doesn't break the flow.
- Robot 4 rewrites the sentence to make it sound natural.

3. What They Found (The Results)

They tested the "Old Way" (Ad-Chat) against their "New Team" (Ad-LLM) and found some interesting trade-offs:

The "Salesman" Fails: The old method that tries to sell while answering often gets the facts wrong. Users lose trust because the AI sounds like it's trying too hard to sell them something.
The "New Team" Wins on Quality: The new method (Ad-LLM) keeps the answer accurate and trustworthy. Users feel like they are getting real help, and the ad feels like a helpful suggestion rather than an interruption.
- Analogy: It's the difference between a waiter who interrupts your meal to pitch a dessert (Old Way) vs. a waiter who brings the perfect dessert after you finish your main course (New Way).
The Catch (Cost): The "New Team" is more expensive to run. It takes more computer power and time because it has to write the answer, find the ad, and then rewrite the sentence. It's like hiring a whole team of editors instead of just one person.
Too Many Ads is Bad: If you try to stuff 5 ads into one answer, the quality drops. Users get annoyed, and they stop clicking. It's like a radio station that plays a song for 30 seconds and then 2 minutes of commercials; people just turn it off.

4. The Bottom Line

The paper concludes that while simple methods might get people to click an ad, they ruin the user's experience and trust. The best approach is to generate a great answer first, then carefully and politely insert the ad.

However, doing this perfectly requires more computing power and money. The authors built GEM-Bench so other researchers can test new ways to balance good answers, happy users, and profit without breaking the bank or annoying the customer.

In short: You can't just force a sales pitch into a helpful conversation. You have to be a good helper first, and a smart salesman second. GEM-Bench is the tool that helps figure out exactly how to do that.

Technical Summary: GEM-Bench

Problem Definition
Generative Engine Marketing (GEM) represents a shift from traditional Search Engine Marketing (SEM) toward monetizing Large Language Models (LLMs) by integrating advertisements directly into conversational responses. The core challenge addressed in this work is Ad-Injected Response (AIR) generation: given a user query, an LLM, and an advertisement database, the system must seamlessly integrate relevant ad descriptions into the generated response.

Existing benchmarks and evaluation methods are insufficient for this specific task. Standard LLM benchmarks (e.g., MT-Bench) contain domains unsuitable for advertising (e.g., mathematics, coding), and traditional text quality metrics (ROUGE, BLEU) introduce bias when evaluating responses modified by ad insertion. Furthermore, existing "LLM-as-a-Judge" approaches fail to capture nuanced user experience factors such as trust and the "jump-out feeling" caused by intrusive ads. Current solutions, such as Ad-Chat, which inject ads via system prompts, lack extensive performance evaluation and often compromise response quality for engagement.

Methodology: GEM-Bench
To address these gaps, the authors propose GEM-Bench, the first comprehensive benchmark for AIR generation. It comprises three primary components:

Curated Datasets: The benchmark includes three datasets covering chatbot and search engine scenarios:
- MT-Human: 10 first-turn queries from the humanities category of MT-Bench, manually verified for ad suitability.
- LM-Market: 1,701 single-turn English queries from LMSYS-Chat-1M, filtered and clustered into travel, recipes, and software tools.
- CA-Prod: 120 queries derived from a commercial query-ad dataset, simulating AI overview features in search engines.
- All datasets utilize a shared AdDB containing 6,556 verified products and brands.
Measurement Ontology: The authors define a multi-dimensional evaluation framework:
- Quantitative Metrics: Includes Response Flow and Response Coherence (global semantic continuity), Ad Flow and Ad Coherence (local semantic transitions around ads), Injection Rate, and Click-Through Rate (CTR).
- Qualitative Metrics: Six "LLM-as-a-Judge" dimensions evaluated via a unified rubric: Accuracy (factual correctness), Naturalness (conversational flow), Personality (user-centeredness vs. sales-driven), Trust (credibility), Notice (product awareness), and Click (link engagement).
Ad-LLM Framework: A modular, multi-agent workflow designed for AIR generation, consisting of four agents:
- Ad-free Response Generator: Produces an initial response without ads.
- Ad Retriever: Selects relevant ads based on semantic similarity to the query or the generated response.
- Ad Injector: Determines the optimal insertion point by minimizing semantic flow disturbance (using a cosine similarity-based objective function).
- Response Rewriter: Refines the injected response to ensure smooth integration.
- The framework supports iterative insertion for multiple ads ( $k > 1$ ).

Experimental Results
The authors evaluated four solutions: the existing Ad-Chat and three variants of Ad-LLM (GI-R, GIR-R, GIR-P) across the three datasets.

Effectiveness (RQ1): Ad-LLM variants consistently outperformed Ad-Chat in user satisfaction and engagement. Specifically, the best Ad-LLM instance (GIR-R) improved overall qualitative scores by 14.8% to 15.6% across datasets. Notably, Ad-LLM achieved relative gains of up to 39.8% in Accuracy, 89.3% in Personality, and 62.9% in Trust compared to Ad-Chat. While Ad-Chat achieved a higher CTR in the search engine scenario (6.5% higher), it suffered from significantly lower accuracy and trust, often adopting a "salesman-like" persona that degraded response quality.
Human vs. LLM Alignment (RQ2): A user study with 118 participants (yielding 3,140 ratings) showed strong internal consistency ( $\rho_{hh} = 0.84$ ) and substantial alignment with LLM judges ( $\rho_{hl} = 0.73$ ). Both humans and LLMs consistently preferred Ad-LLM solutions over Ad-Chat.
Ablation Study (RQ3):
- Refinement: Including the "Response Rewriter" (GIR-R/P) significantly improved qualitative metrics (Naturalness, Trust) compared to the non-refining GI-R, though GI-R performed slightly better on quantitative flow metrics.
- Retrieval Strategy: Response-based retrieval (GIR-R) was superior for chatbot scenarios, while prompt-based retrieval (GIR-P) performed better for keyword-oriented search engine queries.
- Embedding Models: Stronger embedding models (text-embedding-3-small) yielded better results in chatbot scenarios but showed negligible differences in search engine scenarios.
Cost Analysis (RQ4): Ad-LLM solutions incur higher computational overhead than Ad-Chat in terms of input tokens due to the multi-agent workflow, but Ad-Chat surprisingly consumed 5–25 times more input tokens in the search scenario due to its internal LLM-based topic and product selection stages. Ad-LLM variants required approximately 2x more output tokens than simple prompt-based methods.

Significance and Claims
The paper claims to provide the first comprehensive benchmark specifically designed for evaluating ad-injected response generation in GEM. Its significance lies in:

Establishing a Standard: Offering curated datasets and a multi-faceted ontology that moves beyond simple engagement metrics to include user satisfaction dimensions like trust and naturalness.
Demonstrating Trade-offs: Highlighting the critical trade-off between simple prompt-based injection (high engagement, low satisfaction) and structured generation-injection pipelines (higher satisfaction, higher overhead).
Enabling Research: Providing an extensible multi-agent framework (Ad-LLM) and baseline implementations to facilitate future research into more effective and efficient AIR generation strategies.

The authors explicitly state that while their preliminary results show promise, GEM-Bench is currently limited by dataset size and human evaluation scope, with plans to release updated versions. They position the work as a diagnostic tool to encourage responsible deployment and safeguards against deceptive advertising, rather than a direct optimization tool for commercial ad systems.

1. The Problem: The "Salesman" vs. The "Helper"

2. The Solution: GEM-Bench (The Test Kitchen)

3. What They Found (The Results)

4. The Bottom Line

Technical Summary: GEM-Bench

More like this