How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring

Technical Summary: Calibration and Adversarial Robustness of Automated ASR Scoring

Problem Statement
The field of Large Language Model (LLM) safety relies heavily on the Attack Success Rate (ASR) to quantify the efficacy of jailbreaks and defenses. However, the determination of whether a model response is "harmful" is almost exclusively made by automated judges rather than human annotators. These judges fall into two primary categories: dedicated safety classifiers (e.g., HarmBench classifier, Llama-Guard) and general instruction-tuned models prompted to act as graders (LLM-as-judge). The paper argues that this reliance is unearned; if these judges are miscalibrated or easily manipulated, reported ASR values are either inflated or deflated, and can be artificially altered by attackers or authors without changing the actual model output. The core problem is the lack of verification regarding the reliability and robustness of these automated judges.

Methodology
The authors evaluate the reliability of these judges using the HarmBench classifier validation set, which contains 596 pairs of harmful requests and model responses labeled by three human annotators (using majority vote as the ground truth).

1. Judges Evaluated:
   • Dedicated Classifier: cais/HarmBench-Llama-2-13b-cls.
   • LLM-as-Judges: Three open-weight instruction-tuned models: Qwen2.5-7B-Instruct, Phi-3.5-mini-instruct, and Qwen2.5-3B-Instruct.
2. Calibration: The judges' outputs were compared against human majority labels to calculate precision, recall, F1, and accuracy.
3. Adversarial Robustness (Surface Attacks): The authors tested "content-preserving wrappers" on the judges' true positives. These attacks added benign framing (e.g., a refusal sentence, a preamble/postamble, an educational frame, or a fictional dialogue) without altering the harmful core of the response. The metric used was the "flip rate"—the percentage of cases where the judge changed its verdict from "harmful" to "not harmful."
4. Adversarial Robustness (White-Box Attacks): For the dedicated classifier, the authors applied a gradient-based attack (GCG) on its open weights. This involved optimizing a 20-token adversarial suffix to minimize the logit gap between "Yes" (harmful) and "No" (not harmful), keeping the harmful body verbatim.
5. Human Audit: To ensure that a "flip" represented the judge being fooled rather than the harm being removed, two independent annotators reviewed a random sample of 80 flipped responses.

Key Contributions

1. Human-Anchored Comparison: A direct comparison of dedicated classifiers and LLM-as-judges against human labels under a unified attack protocol.
2. Evidence of Fragility: Demonstration that content-preserving wrappers can fool every tested LLM-judge, whereas a dedicated classifier resists these surface-level attacks.
3. White-Box Vulnerability: A gradient-based attack that successfully breaks a dedicated classifier that resists surface wrappers, showing that open-weight judges are vulnerable to direct optimization.
4. Human Verification: An independent audit confirming that the adversarial flips retained the harmful content, proving the judges were deceived rather than the harm being mitigated.

Results

• Calibration Discrepancies: The two judge families fail in opposite ways.
  • The dedicated classifier exhibits high recall (0.974) but lower precision (0.835), meaning it rarely misses harm but frequently flags benign content as harmful (inflating ASR).
  • The LLM-judges maintain high precision (0.81–0.94) but suffer from erratic and low recall (0.06–0.65), meaning they miss large fractions of actual harm.
• Surface Attack Vulnerability:
  • LLM-judges: Highly fragile. A single refusal sentence placed before harmful content flipped the verdict for 39% to 88% of cases. The rate of any wrapper flipping a verdict ranged from 57% to 100% across the three LLM-judges.
  • Dedicated Classifier: Highly resistant to surface attacks, with an any-wrapper flip rate of only 3.4%.
• White-Box Attack Vulnerability:
  • Despite resisting surface wrappers, the dedicated classifier was vulnerable to a white-box GCG attack. On 30 confident true positives, the attack flipped 21 cases (70% success rate) with a small optimization budget (50 steps).
• Human Audit: All 80 sampled flipped responses retained the harmful content, confirming that the attacks successfully deceived the judges without removing the underlying harm.

Significance and Recommendations
The paper concludes that no judge family should be trusted without empirical verification. The choice of judge fundamentally alters reported ASR values. The authors argue that the current prevalence of LLM-as-judges means many reported safety metrics are unreliable both on average and under deliberate pressure.

The authors propose four specific practices for future research:

1. Report Metrics: Papers must state which judge is used and report its precision and recall on a human-labeled slice of the data.
2. Correct ASR: Report ASR corrected for the judge's precision (e.g., $ASR_{corrected} \approx ASR_{reported} \times precision$) to account for false positives.
3. Adversarial Check: Include an adversarial check of the judge (e.g., content-preserving wrappers) and report the flip rate.
4. Judge Selection: Prefer dedicated classifiers over LLM-judges for scoring, while acknowledging that open-weight classifiers remain vulnerable to gradient-based attacks.

The work emphasizes that safety evaluation procedures themselves must be adversarially tested, rather than treated as ground truth oracles.