Technical Summary: Fair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets

1. Problem Statement

Current evaluations of Large Language Model (LLM) jailbreak attacks rely primarily on Attack Success Rate (ASR) but frequently fail to account for the attack budget required to achieve that success. Existing studies often report terminal ASR values obtained under unequal resource constraints, leading to unfair comparisons where a method's effectiveness is conflated with the amount of target access it utilizes.

While recent "compute-aware" evaluations attempt to normalize budgets by aggregating resources into a unified scalar (e.g., FLOPs), this approach has significant limitations for black-box scenarios:

1. Invisibility: Inference FLOPs for closed-source models are generally unavailable and must be estimated.
2. Non-Interchangeability: FLOPs collapse distinct resource constraints (e.g., API rate limits vs. computational cost) into a single metric, obscuring operational realities where target-model access is the primary bottleneck due to rate limits and abuse detection.

Consequently, there is a lack of a comparable, observable basis for evaluating heterogeneous black-box jailbreak attacks.

2. Methodology: Fair-ASR Protocol

To address these issues, the authors introduce Fair-ASR, an evaluation protocol that standardizes comparisons under shared target-call budgets ($B$).

Core Principles

• Primary Budget ($B$): The protocol uses the number of target calls (queries to the victim model) as the primary budget. This is chosen because:
  • It is universal across all black-box attacks (every attack must query the target).
  • It reflects operational constraints (rate limits, account suspensions).
  • It is directly observable, unlike FLOPs for closed APIs.
• Secondary Metric: Attacker calls (queries to an attacker LLM or auxiliary judge) are tracked separately to analyze efficiency trade-offs, rather than being collapsed into the primary budget.
• Evaluation Metrics:
  • ASR@B: The fraction of harmful requests successfully attacked within at most $B$ target calls.
  • ASR-Budget Curve: The trajectory of ASR as $B$ increases, revealing growth rates and saturation points.
  • Average Target Calls (ATC): The average number of target calls consumed per successful attack.
  • Harmfulness Score (HS): A quality metric (using StrongREJECT rubric) assessing the severity and convincingness of the harmful response.

Experimental Scope

The authors re-evaluated 11 representative attacks across three categories:

1. Hand-crafted templates: CodeAttack, DeepInception, CipherChat.
2. Stochastic repeated-sampling: Best-of-N (BoN).
3. LLM-driven automated attacks: PAIR, TAP, ReNeLLM, AutoDAN, GPTFuzzer, AutoDAN-Turbo, Rainbow Teaming.

Experiments were conducted on diverse target models (Llama-3.1, gpt-oss, GPT-4o, GPT-5, Gemini-3.1-Pro, Claude-Sonnet-4.6) using standard datasets (HarmBench, JailbreakBench).

3. Key Findings from Re-evaluation

The application of Fair-ASR revealed three critical insights:

1. Budget-Dependent Rankings: Attack rankings are highly sensitive to the target-call budget. Methods that appear superior under large budgets may be outperformed by simpler methods under tight budgets. For instance, on Llama-3.1-8B, TAP leads BoN at $B=5$, but BoN overtakes TAP at $B=100$.
2. Competitiveness of Simple Primitives: Simple attack primitives remain highly competitive under equal target access.
   • Stochastic Perturbation: BoN continues to improve with additional target calls, achieving high ASR without any attacker-model calls.
   • Hand-crafted Templates: Structured templates (e.g., CodeAttack) achieve high success rates with very few target calls (e.g., 62% ASR at $B=1$), often outperforming complex LLM-driven methods in low-budget regimes.
3. Efficiency Trade-offs: No evaluated LLM-driven method is uniformly efficient in both target and attacker calls.
   • ReNeLLM achieves high target-call efficiency (low ATC) but incurs high attacker-call costs due to iterative rewriting and prompt-only harmfulness gates.
   • Other methods (e.g., PAIR, TAP) may use fewer attacker calls but require significantly more target calls to reach similar success thresholds.

4. Proposed Solution: ReCode

Motivated by the identified "two-dimensional efficiency gap," the authors propose ReCode, a compositional attack designed to maximize ASR while minimizing both target and attacker calls.

Design Architecture

ReCode combines three components into a single-pass pipeline:

1. Gate-free Desensitization Rewriting: Unlike ReNeLLM, which uses a prompt-only harmfulness gate to filter rewrites (triggering costly retries), ReCode performs a single-pass rewrite using desensitization strategies (e.g., literary rewriting, objective substitution) without an auxiliary judge loop.
2. Attacker-Free Stochastic Perturbation: The rewritten prompt undergoes character-level perturbations (e.g., case inversion, ASCII insertion) similar to BoN, requiring zero additional attacker calls.
3. Structured Code-Style Nesting: The perturbed prompt is embedded into a structured code-style template (e.g., Python class definitions), further disguising the intent without requiring attacker-model refinement.

Results

Evaluated under a budget of $B=20$ target calls:

• Performance: ReCode achieved an average ASR of 81.0% across five target models (including gpt-oss variants) and 70.3% across the three frontier closed-source models (GPT-5, Gemini-3.1-Pro, Claude-Sonnet-4.6).
• Efficiency: It required an average of only 7.00 attacker calls per request (AAC), significantly lower than ReNeLLM (18.69) and TAP (49.56).
• Specific Gains: On GPT-5, ReCode improved ASR from 31% (ReNeLLM) to 85% while reducing attacker calls from 26.02 to 7.19.
• Harmfulness: ReCode also achieved the highest average Harmfulness Score (HS) of 0.662, indicating that higher success rates correlated with higher-quality harmful responses.

5. Significance and Claims

The paper claims that Fair-ASR provides a necessary, observable baseline for comparing black-box jailbreaks, correcting misleading conclusions drawn from unequal budget comparisons. It demonstrates that algorithmic complexity does not guarantee efficiency and that simple, low-cost primitives are often underutilized.

The introduction of ReCode serves as a proof-of-concept that closing the efficiency gap is possible by combining desensitization rewriting with attacker-free obfuscation techniques. The authors conclude that future evaluations must move beyond terminal ASR to consider joint resource efficiency (target and attacker calls) to accurately assess the safety landscape of LLMs.

Limitations Noted:

• The protocol currently focuses on single-turn attacks and does not yet cover multi-turn settings.
• Target calls do not capture token usage, API pricing, or the manual cost of template development.
• The study acknowledges that while ReCode is efficient, specific model behaviors (e.g., Claude's sensitivity to code nesting) can vary, requiring further investigation.