Technical Summary: Logit-Gap Steering

Problem Statement

Large Language Models (LLMs) aligned via Reinforcement Learning from Human Feedback (RLHF) are trained to refuse unsafe requests. However, this alignment often creates only a narrow "operational margin" between refusal and compliance. Prior work suggests that appending short, out-of-distribution suffixes to toxic prompts can flip a model from refusal to compliance by closing this margin.

The central problem addressed is the lack of a scalable, quantitative diagnostic to measure this first-token refusal margin (the logit-level decision at the first decoding step) and to probe its robustness. Existing methods for discovering effective jailbreak suffixes, such as Gradient-based Coordinate Gradient (GCG), rely on gradient-based optimization over large, out-of-distribution token spaces. These methods are computationally expensive (e.g., ~5 hours on a single A100 for 50 prompts) and produce suffixes with high perplexity that are easily detected by perplexity-based defenses.

Methodology: Logit-Gap Steering

The authors introduce Logit-Gap Steering, a gradient-free, forward-pass-only diagnostic method designed to measure and close the refusal–affirmation logit gap using in-distribution tokens.

1. The Refusal–Affirmation Logit Gap

The core metric is defined as the difference between the top refusal-token logit and the top affirmative-token logit at the first decoding step ($h_0$) after a harmful prompt:
$$\Delta_0 = \ell_{\text{refusal}}(h_0) - \ell_{\text{affirm}}(h_0)$$
The paper posits that alignment widens this gap on the vast majority of toxic prompts compared to base models. A jailbreak succeeds if a suffix $S$ reduces the cumulative gap to zero or below.

2. Scoring Function (Gap-Closing Proxy)

To avoid gradient computation, the method approximates the gap reduction of a candidate token $t$ using a scoring function $F(h, t)$ derived from a single forward pass. This score combines three proxies:

• Logit Gap Reduction: The immediate change in the logit difference.
• KL Penalty Proxy: Approximates the deviation from the base model by comparing the refusal logit to a neutral, high-probability token.
• Reward Shift Proxy: Uses the incremental affirmative logit as a proxy for the learned reward model's preference.

The combined score is:
$$F(h, t) = \Delta F_{\text{logit}}(h, t) - \lambda_{KL}\Delta KL(h, t) + \lambda_r \Delta r(h, t)$$

3. Search Algorithm

The method employs a Greedy Covering algorithm:

1. Candidate Filtering: The vocabulary is filtered to retain only in-distribution tokens (high probability given the prompt) that are not refusal tokens.
2. Scoring: Each candidate is scored using $F(h, t)$.
3. Selection: Tokens are greedily selected in descending order of score until the cumulative score exceeds the initial gap $\Delta_0$.
4. Variants: To overcome single-token myopia and out-of-distribution artifacts, the authors introduce:
   • DFS Phrase Harvesting: Depth-first search to harvest grammatical affirmative phrases.
   • Constituent-Level Search: Evaluating multi-token sequences as atomic units.
   • Permutation Selection: Searching over permutations of top candidates to optimize for KL-reward balance, direct gap minimization, or combined scores.

4. Efficiency

The entire process requires only forward passes. Discovering an 8-suffix ensemble for a model family takes approximately 26,000 forward-pass equivalents (approx. 2 minutes on one A100), which is roughly 125× less compute than a single GCG search.

Key Contributions

1. Alignment Diagnostic: The definition of the refusal–affirmation logit gap as a quantitative measure of the first-token operational refusal margin. The authors demonstrate that median gap closure co-varies with True Attack Success Rate (ASR) across different strategies, serving as an internal consistency check.
2. Efficient Probing Method: A gradient-free method that discovers effective, in-distribution suffixes (<10 tokens per component) with significantly lower computational cost than GCG.
3. Cross-Scale Transfer: Suffixes discovered on small models (0.5B–2B parameters) transfer without modification to much larger models (up to 72B) within the same family.
4. Stealthiness: The discovered suffixes have perplexity values $10^3$–$10^4$ times lower than GCG suffixes. Consequently, they remain effective against perplexity-filter defenses that successfully block GCG.

Experimental Results

The method was evaluated on 13 models across three families (Llama, Gemma, Qwen) using AdvBench and HarmBench.

• Attack Success Rate (ASR): An 8-suffix ensemble achieved 38–96% True ASR across the 13 models. On strongly aligned models (e.g., Qwen2.5-72B), the ensemble outperformed GCG+SH by 8–18× (37.9% vs. 4.6%).
• Efficiency: The method is approximately 125× faster than GCG for universal suffix discovery and 17× faster than an 8-shot GCG baseline.
• Defense Evasion:
  • Perplexity Filters: Under a perplexity-filter defense ($T=1000$), GCG+SH collapsed from 64.7% to 1.0% True ASR, while the discovered ensemble dropped only slightly from 76.9% to 76.0%.
  • Semantic Filters: Llama Guard 3 (LG3) reduced all methods significantly, but the discovered ensemble retained a 2–3× lead overall and 8–16× lead on strongly aligned targets compared to GCG.
• Internal Dynamics: The analysis revealed "sentence-boundary reward cliffs," where safety penalties spike at punctuation. Successful suffixes concentrate gap-closing power within a single run-on clause to avoid these cliffs.

Significance and Claims

The paper claims that current alignment margins, while consistently present, are often thin and efficiently measurable. The primary contribution is not a novel class of harm (as the suffixes target models already vulnerable to GCG), but rather a quantitative diagnostic framework and a computationally efficient probe for safety evaluation.

The authors argue that:

1. Defenses must evolve: Published perplexity-filter defenses are insufficient against in-distribution suffixes that mimic benign instruction-style text.
2. Layered defense is necessary: Since different defenses (perplexity vs. semantic) catch different attack styles, a combination of input filtering and output semantic classification is required.
3. Alignment artifacts: The method exposes specific alignment artifacts, such as sentence-boundary reward cliffs, offering a lightweight probe into how safety tuning reshapes internal representations.

The work adheres to responsible disclosure, sharing discovered suffixes with safety teams of major model providers (Meta, Alibaba, Google, Mistral, OpenAI) 30 days prior to release and gating artifact access behind an acceptable-use policy.