Exploring LLM biases to manipulate AI search overview

1. Problem Statement

The rise of LLM Overview systems (e.g., Google's AI Overview, GPT Search) has transformed web search by using Large Language Models (LLMs) to summarize search results and select the most relevant sources. While these systems improve user experience, they introduce significant risks:

• Bias Propagation: LLMs exhibit inherent biases in source selection and answer generation, which can distort information presented to users.
• Economic Impact: These overviews reduce click-through rates to original publishers (reported as high as 58% decrease), threatening the digital economy.
• Vulnerability to Manipulation: It is unclear how susceptible these systems are to adversarial attacks where bad actors manipulate search snippets to force their content into the AI-generated summary.
• Research Gap: Existing studies often rely on relaxed experimental constraints (e.g., full-page access) or descriptive analysis, lacking rigorous proofs of bias and controlled manipulation attacks within realistic search constraints (snippets only).

2. Methodology

The authors propose a framework to investigate biases and exploit them via Reinforcement Learning (RL).

A. Experimental Setup

• Dataset: ~3,000 queries from the Amazon Shopping Queries dataset. Each query is paired with 7–10 real search engine results (URL, Title, Snippet).
• LLM Overview Simulation: A simplified pipeline where an LLM (primarily gpt-4.1-nano) takes a query and a list of search results, then selects the top 3 most relevant sources to cite in a summary.
• Bias Verification: Before manipulation, the authors proved bias exists by running permutation tests (shuffling URLs, titles, snippets, changing prompts, varying temperature). They used a Binomial Distribution model to statistically prove that certain snippets are selected persistently regardless of order or randomization, indicating inherent model preferences.

B. Reinforcement Learning Framework

The core contribution is training a small language model (SLM) to rewrite search snippets to maximize their selection probability.

• Policy Model: Initialized with gemma-3-1b-it. Trained using LoRA (Low-Rank Adapters) and the Dr.GRPO algorithm (a modification of GRPO).
• Action Space: The policy is strictly constrained to rewrite only the snippet text. It cannot modify the URL or Title, and it cannot visit the actual webpage.
• Context: The policy uses conditional rewriting, meaning it sees the target snippet plus all competing snippets in the context. This allows the model to optimize for comparative advantage rather than absolute quality.
• Reward Function:
  1. Selection Reward: Binary reward (1 or 0) based on whether the rewritten snippet is selected by the LLM Overview generator.
  2. Length Constraint: A soft-limit reward (following DAPO) to prevent "reward hacking" where the model simply makes snippets infinitely long.
  3. Similarity (Optional): Cosine similarity between original and rewritten text (later dropped to speed up training).
• Training Strategy: The authors skipped the Supervised Fine-Tuning (SFT) stage, moving directly to RL with a system instruction, arguing that rewriting is within the model's native capabilities.

3. Key Contributions

1. Proof of Comparative Bias: Demonstrated that LLM Overview selection is driven by comparative advantages among candidates rather than absolute quality. A snippet's success depends on how it performs relative to the specific set of competing snippets provided.
2. RL-Based Manipulation: Successfully trained an SLM to rewrite short search snippets to increase their citation rate in LLM Overviews, proving that biases can be exploited.
3. Realistic Constraints: Unlike previous works (e.g., AutoGEO) that allowed full-page access or long-form rewriting, this study restricts manipulation to search snippets with strict length limits, mirroring real-world search engine constraints.
4. Attack Vector Analysis: Identified and tested three specific attack vectors:
   • Target Snippet Attack: Failed (policy ignored injected harmful text).
   • Title Attack: Failed (LLM Overview considers titles/URLs, not just snippets).
   • Reference Attack (Context Poisoning): Succeeded. By poisoning a competitor's snippet in the context, the policy learned to incorporate that harmful context into the target snippet's rewrite, leading the LLM Overview to cite the poisoned result.

4. Results

• Bias Confirmation: Statistical analysis showed that specific snippets were selected in 100% of permutation experiments (7/7), while others were never selected, confirming strong model bias.
• Manipulation Success: The trained policy significantly increased the selection rate of target snippets across various randomization techniques (shuffling, temperature changes).
• Model Robustness:
  • The attack worked effectively on gpt-4.1-nano, gpt-4.1-mini, and gpt-5-nano.
  • It failed on gpt-5-mini. The authors hypothesize this is due to gpt-5-mini having a stronger bias toward URLs over content, making snippet rewriting ineffective.
• Attack Effectiveness: The "Reference Attack" successfully injected harmful content (e.g., "rope people" in watch band descriptions) into the final AI summary, proving that context poisoning is a viable threat.

5. Significance and Implications

• Security Risk: The study reveals a critical vulnerability in RAG (Retrieval-Augmented Generation) and AI Search systems. Bad actors can manipulate search rankings and AI summaries not just by SEO, but by training small models to rewrite content specifically to exploit the target LLM's biases.
• Context Poisoning: The discovery that poisoning competitor data can corrupt the target output suggests that LLM Overviews are susceptible to indirect attacks, not just direct injection.
• Model Variance: The varying susceptibility of different model sizes (Nano vs. Mini) suggests that bias exploitation is highly model-dependent. Smaller or specific models may be more vulnerable to content manipulation, while others rely more on structural signals (URLs).
• Future Directions: The paper highlights the need for robust defenses against snippet manipulation and context poisoning, as well as further research into the specific biases of frontier models to prevent their exploitation in agentic workflows.

In conclusion, the paper provides empirical evidence that LLM Overviews are biased, that these biases can be systematically exploited via Reinforcement Learning to manipulate search results, and that such manipulation poses a genuine safety risk through context poisoning attacks.