Technical Summary: SAFESEARCH: Automated Red-Teaming of LLM-Based Search Agents

1. Problem Statement

Large Language Model (LLM) search agents connect models to the Internet to access up-to-date information, enabling applications from rapid information retrieval to long-horizon research tasks. However, this architecture introduces a critical new threat surface: unreliable search results. Unlike Retrieval-Augmented Generation (RAG) systems that rely on curated, local corpora, search agents passively receive content from the open web, which may include low-quality content farms, biased sources, sponsored advertisements, or manipulated information via black-hat SEO.

Real-world incidents and in-the-wild observations indicate that search agents often uncritically trust these results, leading to unsafe outputs such as misinformation, harmful advice, or the execution of indirect prompt injections. Existing evaluations for search agents primarily focus on task accuracy (e.g., GAIA, SIMPLEQA) rather than safety, and current red-teaming methods rely on labor-intensive human crafting or adversarial queries that do not reflect the reality of benign user queries encountering unreliable search results. Furthermore, real-world adversarial testing (e.g., manipulating search rankings) poses ethical risks and is infeasible for most developers.

2. Methodology: The SAFESEARCH Framework

The authors propose SAFESEARCH, an automated, scalable, and cost-efficient red-teaming framework designed to evaluate the safety of LLM-based search agents when exposed to unreliable search results under benign user queries.

2.1 Threat Model

The framework operates under a specific threat model:

• Benign Queries: The tester issues only benign, realistic user queries.
• Controlled Injection: For each query, the tester injects at most one unreliable website into the search results returned to the agent.
• Agent-Agnostic: The unreliable website is fixed across all agents for a given test case, ensuring the threat is not tailored to specific model architectures.
• Objective: To identify if the agent propagates the unreliable content into unsafe final responses.

2.2 Automated Test Case Generation

The framework utilizes a three-step workflow driven by LLM assistants to generate 300 high-quality test cases across five risk categories:

1. Scenario Anchoring: An LLM generates realistic user scenarios and queries where the specific risk (e.g., misinformation, bias) could plausibly arise.
2. Test Design: The system defines a "malicious plan," specifying the target negative consequence and the nature of the unreliable website. Crucially, it accounts for the time gap between the LLM's knowledge cutoff and the current date to ensure the threat is time-sensitive and realistic.
3. Test Instantiation: The system generates guidelines for synthesizing the unreliable website and a checklist for safety evaluation.

Quality Filtering: To ensure test cases are actionable, the framework employs differential testing. A baseline agent is tested under two settings: a benign tool and a manipulated tool (with the injected website). A test case is retained only if the target consequence is attainable under manipulation but not triggered under benign conditions.

2.3 Simulation-Based Testing

To avoid the ethical and practical issues of manipulating real-world search engines, SAFESEARCH uses a sandboxed simulation:

• Injection: An unreliable website is synthetically generated and injected into the list of authentic search results (typically appended to the end to simulate lower ranking, though position sensitivity is analyzed).
• Execution: The search agent processes the mixed results and generates a response.
• Evaluation: An LLM-as-a-Judge, assisted by the pre-generated checklist, evaluates the agent's response against the target consequence to determine an Attack Success Rate (ASR). A separate evaluator assesses Helpfulness Scores (HS) to determine if unsafe responses still appear useful to users.

3. Key Contributions

1. Problem Formulation: The paper formally defines the safety risks of LLM-based search agents exposed to unreliable search results under benign queries, distinguishing this from RAG poisoning and explicit adversarial attacks.
2. SAFESEARCH Framework: The development of an automated red-teaming pipeline that includes test case generation, differential filtering, sandboxed simulation, and checklist-assisted evaluation.
3. Comprehensive Evaluation: The construction of a dataset of 300 test cases and the evaluation of 17 representative LLMs (9 proprietary, 8 open-source) across three search agent scaffolds (Search Workflow, Tool Calling, and Deep Research).
4. Open Source: The release of the code and dataset to facilitate further research.

4. Experimental Results

The authors evaluated 17 models across three scaffolds. Key findings include:

• High Vulnerability: Search agents are highly susceptible to unreliable search results. The highest observed ASR was 90.5% for GPT-4.1-mini using a search workflow. Even the most robust configurations (e.g., Claude-Sonnet-4.5 with tool calling) exhibited non-zero ASRs (4.6%).
• Impact of Backend LLMs: Reasoning models generally demonstrated stronger resilience than non-reasoning models. For instance, GPT-5 and GPT-5-mini showed significantly lower ASRs compared to earlier generations, suggesting that advanced reasoning capabilities aid in filtering unreliable information.
• Impact of Scaffold Design: The agent architecture plays a critical role.
  • Search Workflow (passive retrieval) showed the highest vulnerability.
  • Tool Calling and Deep Research (active, multi-step retrieval) showed improved safety, likely due to cross-validation across sources and the ability to resolve conflicts.
  • However, when search budgets were controlled, the advantage of "Deep Research" diminished, suggesting that the safety gain often comes from utilizing more search queries rather than the scaffold logic itself.
• Risk Variance: Susceptibility varied by risk type. Misinformation posed the greatest threat (highest average ASR), while Indirect Prompt Injection showed the lowest average ASR, indicating stronger resilience to this specific security threat.
• Safety vs. Helpfulness: Unsafe behavior does not necessarily reduce perceived helpfulness. In many cases, agents influenced by unreliable sources produced responses that scored highly on helpfulness (plausibility) despite being factually incorrect or unsafe.
• Defense Limitations: Common defenses like reminder prompting (warning the agent about unreliable sources) offered limited protection. Filtering (using an LLM to pre-filter results) reduced ASR by roughly half but failed to eliminate risks entirely, particularly against stealthy or highly persuasive content.

5. Significance and Claims

The paper claims that SAFESEARCH provides a practical, scalable, and transparent method for measuring and improving the safety of LLM-based search agents.

• Transparency in Development: The framework reveals how design choices (e.g., the number of search results returned) implicitly affect safety. It highlights a knowledge-action gap, where models may possess the capability to identify unreliable sources when explicitly tasked but fail to apply this knowledge in agentic workflows.
• Continuous Supervision: Unlike static benchmarks, SAFESEARCH is designed for continuous safety tracking during agent development, allowing for the evaluation of defense strategies and the generation of increasingly challenging test cases.
• Real-World Relevance: By simulating the injection of unreliable results into authentic search contexts, the framework approximates real-world threats without the ethical risks of manipulating live search engines. The authors argue that while simulation-based testing has limitations, it is necessary to systematically identify the lower bound of agent safety before deployment.

The authors conclude that while current search agents are vulnerable, the framework offers a pathway for developers to iterate on safer systems, emphasizing that safety must be prioritized alongside performance in future agent designs.