Technical Summary: LLM STINGER

Problem Statement

Jailbreaking Large Language Models (LLMs) involves crafting inputs that bypass developer-imposed safety measures to elicit harmful responses. While suffix-based attacks (appending specific strings to inputs) have proven effective against both white-box and black-box models, existing methods face significant scalability challenges. Traditional approaches often rely on laborious manual prompt engineering, complex role-playing templates, or white-box gradient-based optimization that requires access to model weights. Although many known suffix attacks have been patched via safety training, the authors observe that modified versions of these suffixes can still succeed. However, manually crafting these modifications or using white-box attackers to find new suffixes is time-consuming and limits the scalability of red-teaming efforts.

Methodology: LLM STINGER

The paper introduces LLM STINGER, a novel framework that leverages an attacker LLM to automatically generate highly effective adversarial suffixes through a Reinforcement Learning (RL) loop. Unlike previous methods that do not fine-tune the attacker model, LLM STINGER fine-tunes an attacker LLM to evolve new attack vectors based on existing successful attacks.

Core Architecture

The system operates within an RL environment using the Proximal Policy Optimization (PPO) algorithm. The workflow is as follows:

1. Input: The attacker LLM receives a harmful question (from the HarmBench benchmark) and a set of seven publicly available successful suffixes.
2. Generation: The attacker LLM is prompted to generate new, similar suffixes.
3. Execution: The generated suffix is concatenated with the harmful question and sent to the victim LLM (requiring only black-box access).
4. Feedback Mechanism:
   • Binary Reward: A judgment LLM evaluates the victim's response to determine if the attack succeeded (producing a harmful output).
   • Token-Level Feedback: If the attack fails, a string similarity checker provides dense, token-level feedback. This checker evaluates the similarity between the generated suffix and previously successful suffixes. It penalizes deviations that stray too far from known effective patterns, guiding the attacker LLM to retain essential characteristics while exploring new variations.
5. Optimization: The attacker LLM is fine-tuned for 50 epochs using PPO, utilizing both the binary success signal and the vector-based similarity feedback to refine its policy.

Implementation Details

• Attacker Model: Gemma (an open-source LLM chosen for its flexibility with system prompts and lack of excessive caution).
• Judgment Model: The HarmBench judgment LLM, selected for its superior performance on manually labeled validation sets compared to GPT-4 or Llama-Guard.
• Library: A customized version of the Transformer Reinforcement Learning (TRL) library, modified to handle vector reward signals for token-level feedback.
• Hardware: Experiments were conducted on a cluster with Intel E5-2683 v4 processors and NVIDIA V100 GPUs.

Key Contributions

1. RL-Driven Suffix Generation: The authors present the first approach to fine-tune an attacker LLM specifically to generate modified adversarial suffixes, moving beyond static prompt engineering or white-box gradient optimization.
2. Hybrid Feedback Loop: The integration of a string similarity checker provides fine-grained, token-level feedback alongside binary success signals. This mechanism prunes the search space by discouraging the generation of suffixes that deviate too significantly from proven successful patterns, accelerating the discovery of effective attacks.
3. Black-Box Efficacy: The method requires only black-box access to the victim model, making it applicable to both open-source and closed-source models without needing internal weights.

Results

The authors evaluated LLM STINGER against 15 state-of-the-art (SOTA) red-teaming methods (including GCG, PAIR, TAP, AutoDAN, and others) across 7 victim LLMs using the HarmBench standard behavior test split.

• LLaMA2-7B-chat: Achieved an Attack Success Rate (ASR) of 89.3%, representing a +57.2% improvement over the next best method.
• Claude 2: Achieved an ASR of 52.2%, a +50.3% increase over the next best method (which achieved only 1.9%).
• GPT-3.5 Turbo (0613): Achieved an ASR of 88.67%.
• GPT-3.5 Turbo (1106): Achieved an ASR of 94.97%.
• Gemma-2B-it: Achieved an ASR of 99.4% on the same model architecture used as the attacker, demonstrating high adaptability.

In all tested scenarios, LLM STINGER outperformed all 15 comparison methods. The authors manually verified outputs to ensure the high success rates were due to genuine jailbreaks rather than classifier gaming.

Significance and Claims

The paper claims that LLM STINGER addresses the combinatorial hardness of the jailbreak search problem, where naive methods fail. By utilizing an RL-based heuristic search guided by a judgment LLM and a string similarity checker, the system effectively narrows the search space to high-potential regions.

The authors assert that their approach demonstrates:

• Robustness: The ability to bypass defenses in models known for extensive safety training (e.g., Claude 2, LLaMA2).
• Scalability: The capacity to automate the discovery of new suffixes without human intervention or white-box access.
• Adaptability: The effectiveness across diverse model architectures, including both open-source and closed-source systems.

The work highlights that safety training alone may be insufficient against automated, RL-driven attack evolution, suggesting a need for more robust defense strategies against such adaptive adversaries. Future work is proposed to include additional token-level attack methods, comparisons with multi-modal models, and further exploration of feedback mechanisms.