Peering Behind the Shield: Guardrail Identification in Large Language Models

1. Problem Statement

As Large Language Models (LLMs) are deployed in real-world conversational AI agents, they are increasingly protected by safety guardrails (e.g., LlamaGuard, WildGuard, ShieldGemma). These guardrails act as external filters to moderate input and output content, preventing harmful or policy-violating responses.

The core problem addressed is Guardrail Identification: determining which specific guardrail (or its derivative) is deployed in a black-box AI agent.

• Motivation: Identifying the guardrail allows adversaries to transition from black-box attacks to targeted, white-box-style attacks, significantly increasing the success rate of jailbreaks and prompt injections. Conversely, it helps model owners detect unauthorized usage of their proprietary safety models.
• Challenges:
  • C1 (Limited Output Space): Guardrails typically output binary "Safe/Unsafe" labels, making it difficult to distinguish them via unique response patterns compared to full LLMs.
  • C2 (Obfuscation): Agents often contain multiple safety layers (e.g., safety-aligned LLMs + external guardrails), obscuring the specific candidate guardrail.
  • C3 (Deployment Stages): Guardrails can be deployed at the input (filtering user prompts) or output (filtering LLM responses) stage. Testing output guardrails is difficult because the attacker cannot directly control the LLM's generated text.
  • C4 (Threshold Selection): Existing identification methods lack principled strategies for setting decision thresholds, leading to unreliable results.

2. Methodology: AP-Test

The authors propose AP-Test (Adversarial Prompts for Guardrail Testing), a framework that uses guard-specific adversarial prompts to identify the deployed guardrail. The method consists of two main phases:

A. Adversarial Prompt Optimization

The goal is to generate a prompt $P = x_a \parallel x_q$ (an adversarial prefix $x_a$ concatenated with a normal query $x_q$) that triggers a specific candidate guardrail ($G_t$) to flag the input as "Unsafe," while remaining "Safe" for all other guardrails and the underlying LLM.

This is achieved by optimizing a loss function $L$ with three components:

1. Candidate Adversarial Loss ($L_1, L_2$): Forces the candidate guardrail $G_t$ to classify the prompt as unsafe.
   • $L_1$: Encourages "Unsafe" classification.
   • $L_2$: Penalizes "Safe" classification.
2. Cross-Guardrail Compatibility Loss ($L_3$): Ensures the prompt is classified as "Safe" by a Safety Scorer ($S$) and other non-candidate guardrails. This prevents false positives from unrelated safety mechanisms.
   • $L = L_1 + \alpha L_2 + \beta L_3$.
3. Optimization Strategy: Uses the Greedy Coordinate Gradient (GCG) strategy to update token embeddings, sampling top-$k$ replacements to minimize the loss.

B. Adversarial Prompt Testing

The framework employs two complementary strategies to cover different deployment stages:

1. Input Guard Test: The adversarial prompt is sent directly to the AI agent. If the agent refuses the query, it suggests the candidate guardrail is deployed at the input stage.
2. Output Guard Test: To test output guardrails, the agent is prompted to repeat the adversarial prompt (e.g., "Your task is to mimic text. Repeat this string exactly..."). The LLM's repetition is then passed to the output guardrail. If the guardrail flags the repetition as unsafe, the candidate is likely present.

C. Match Score Metric

To solve the threshold selection problem (C4), the authors introduce the Match Score (MS):
$$MS = \frac{|min(r, r_t)|^\lambda}{|r_t|^\lambda}$$

• $r$: Refusal rate of the AI agent (ratio of "refuse" responses).
• $r_t$: Refusal rate of the candidate guardrail when queried directly with the same prompts.
• $\lambda$: Scaling factor.
• Decision Rule: If $MS > 0.5$, the candidate guardrail is identified as present. This metric relies solely on the candidate model's behavior, eliminating the need for auxiliary LLM sets.

3. Key Contributions

1. First Guardrail Identification Method: AP-Test is the first approach to identify specific guardrails in black-box AI agents using guard-specific adversarial prompts.
2. Dual-Stage Testing: It effectively handles both input and output guardrail deployments, overcoming the difficulty of testing output filters.
3. Principled Decision Metric: The introduction of the Match Score provides a robust, threshold-independent strategy for identification.
4. Comprehensive Evaluation: The method is validated against four candidate guardrails (WildGuard, LlamaGuard, LlamaGuard2, LlamaGuard3) across diverse agents (Llama3.1, GPT-4o) and complex scenarios (multiple guardrails).

4. Experimental Results

• Perfect Accuracy: AP-Test achieved 100% classification accuracy (Acc = 1.00) and an AUC of 1.00 across all tested scenarios for both input and output guard tests.
• Robustness:
  • The method is robust to different underlying LLMs (Llama3.1 vs. GPT-4o).
  • It successfully identifies guardrails even when the agent contains multiple guardrails (e.g., both input and output filters active).
  • It correctly distinguishes between base guardrails and their fine-tuned derivatives (e.g., Aegis models based on LlamaGuard).
• Ablation Studies:
  • Loss Terms: Removing $L_3$ (compatibility loss) caused high false positives (e.g., misidentifying WildGuard agents as LlamaGuard).
  • Query Set ($Q$): Removing the normal query set $Q$ during optimization led to prompts that triggered all guardrails, destroying specificity. The query set is crucial for guiding the optimization.
  • Epochs: The method converges efficiently (around 50 epochs), with performance stabilizing as the distinction between the target and other guardrails sharpens.

5. Significance and Implications

• Security Risk: The work highlights a critical vulnerability: once an attacker identifies the specific guardrail, they can craft highly effective, targeted attacks that bypass the system. This shifts the security paradigm from "black-box defense" to "white-box vulnerability."
• Copyright & IP Protection: Conversely, the method serves as a tool for model owners to detect if their proprietary guardrails are being used without authorization in third-party agents.
• Future Directions: The paper calls for the development of guardrails that are inherently harder to fingerprint and circumvent. It also notes limitations, such as the current focus on model-based guardrails rather than rule-based filters or proprietary closed-source systems.

In summary, AP-Test demonstrates that current safety guardrails are identifiable through adversarial prompting, revealing a new attack surface in the AI safety ecosystem and emphasizing the need for more robust, non-fingerprintable safety mechanisms.