When Search Goes Wrong: Red-Teaming Web-Augmented Large Language Models

1. Problem Statement

Large Language Models (LLMs) augmented with web search capabilities (e.g., GPT-4o with search, Gemini with search) overcome static knowledge cutoffs by retrieving real-time information. However, this integration introduces a distinct safety threat surface: the retrieval and citation process.

Unlike standalone LLMs where risks are limited to the model's internal generation, search-enabled LLMs can retrieve and cite external, uncontrolled webpages. This creates three specific risk categories:

1. Response Risk: The LLM generates harmful content directly.
2. Citation Risk: The LLM's response is benign, but the cited external links contain harmful, low-credibility, or toxic content (e.g., hate speech, misinformation).
3. Combined Risk: Both the response and the citations are harmful.

The Gap: Existing red-teaming methods focus primarily on eliciting unsafe generations from standalone LLMs or RAG systems using fixed corpora. They fail to address the unique vulnerabilities of the retrieval-citation pipeline in open-web search scenarios, particularly under black-box constraints (where internal retrieval mechanisms and safety filters are opaque).

2. Methodology: CREST-Search

The authors propose CREST-Search, a comprehensive red-teaming framework designed to systematically expose citation risks in web-augmented LLMs. The framework consists of three core components:

A. Three Novel Adversarial Search Query Strategies

Instead of direct jailbreaking, these strategies manipulate the search query to steer the retrieval engine toward malicious sources while keeping the query itself seemingly benign:

1. Keyword Injection: Inserting harmful or sensitive keywords (resembling SEO bait) into benign queries to force the retrieval of malicious webpages.
2. Exaggeration: Amplifying query elements beyond scientific or factual limits (e.g., "ultimate lethal dosage") to distort retrieval intent and surface fringe/unreliable sources.
3. Role Play: Instructing the LLM to adopt a specific persona (e.g., "legal advisor," "cybersecurity analyst") to bypass safety filters and justify citing harmful sites as authoritative evidence.

B. Automated Red-Teaming Pipeline

The framework operates in a closed-loop, black-box setting:

1. Query Generation: A red-teaming model generates adversarial queries based on the strategies above.
2. Execution & Evaluation: Queries are submitted to the target LLM. The system evaluates not just the text response, but also the content of the cited URLs using automated moderation tools.
3. Judgment-Guided Refinement: A judgment model analyzes the outcome (Response Risk, Citation Risk, or Combined). If the attack fails, the judgment model provides in-context learning guidance to refine the query, iterating until a citation risk is triggered or a limit is reached.

C. WebSearch-Harm Dataset & Specialized Model

To improve efficiency and reduce reliance on iterative refinement loops:

• WebSearch-Harm: The authors constructed a specialized dataset containing adversarial search queries, annotated with generation strategies, harmful categories, and successful attack outcomes.
• Fine-Tuning: A general LLM is Supervised Fine-Tuned (SFT) on WebSearch-Harm to become a specialized Red-Teaming Model. This model can generate high-quality adversarial queries in a single step, significantly reducing cost and latency compared to iterative refinement.

3. Key Contributions

1. Identification of Citation Risk: The paper highlights that the primary vulnerability in search-enabled LLMs is not the generation itself, but the citation of harmful external content, a threat largely ignored by existing safety evaluations.
2. CREST-Search Framework: A novel, black-box red-teaming framework incorporating three specific attack strategies tailored to the retrieval-citation pipeline.
3. WebSearch-Harm Dataset: A curated dataset enabling the fine-tuning of specialized red-teaming models, improving query quality and generation efficiency.
4. Systematic Evaluation: Comprehensive experiments demonstrating that current safety filters are insufficient against citation-based attacks.

4. Experimental Results

The framework was evaluated on four commercial LLMs with web search (GPT-4o-search, GPT-4o-mini-search, Gemini-2.0-flash, Gemini-2.5-flash) across five harmful content categories (Harassment, Hate, Self-harm, Sexual, Violence).

• Risk Detection Rate: CREST-Search achieved an 80.5% risk detection rate, significantly outperforming the best baseline (HarmBench at 11.6%).
• Citation Dominance: 89.3% of the risks uncovered were Citation Risks. Crucially, 74.7% of these cases involved benign responses with toxic citations, making them invisible to traditional red-teaming methods that only check the generated text.
• Efficiency & Cost:
  • The fine-tuned red-teaming model achieved the highest detection rate (80.5%) with the lowest cost ($1.50 per 100 queries) and time (49.6 seconds/query).
  • It outperformed zero-shot (31.1%) and few-shot (49.3%) baselines in both effectiveness and efficiency.
• Toxicity Control: Despite generating adversarial queries, CREST-Search maintained low query toxicity (23.6%) compared to baselines (e.g., UAT at 87.9%), ensuring the queries remained non-trivially offensive and effective.

5. Significance and Implications

• Paradigm Shift in Safety: The paper argues that safety evaluation for LLMs must evolve from checking "what the model says" to checking "what the model cites." The attack surface has expanded from the model's weights to the open web.
• Limitations of Current Defenses: Standard safety filters are ineffective against this vector because the harmful content originates from external, pre-indexed webpages that the model cites as "authoritative."
• Mitigation Strategies: The authors suggest:
  1. Implementing real-time safety detection on both the returned URLs and the content of the cited webpages before display.
  2. Using frameworks like CREST-Search to fine-tune LLMs for better robustness against citation-based attacks.
• Future Outlook: The work underscores the urgent need for robust, proactive safety management pipelines that continuously test the retrieval-citation loop, rather than relying solely on static model alignment.

In conclusion, CREST-Search demonstrates that web-augmented LLMs are highly vulnerable to adversarial manipulation of their search retrieval mechanisms, posing a significant risk of exposing users to harmful content via "safe" citations.