Technical Summary: TamperBench

Problem Statement

As open-weight Large Language Models (LLMs) approach the capabilities of frontier closed models, their safety alignment becomes increasingly fragile. While diverse training procedures (e.g., RLHF) are used to align models, these safeguards can be undermined by "tampering"—modifications to a model's weights or latent representations. This tampering can be malicious (intentional jailbreaking via fine-tuning) or accidental (benign fine-tuning that inadvertently erodes safety).

The field currently lacks a standardized approach to evaluate tamper resistance. Existing works suffer from evaluation fragmentation: they utilize disparate datasets, metrics, and threat models (e.g., varying numbers of fine-tuning steps or different data poisoning rates). This makes it difficult to compare the robustness of different models or defenses, leading to inconclusive results where a defense might appear effective under one specific configuration but fail under another. Furthermore, prior evaluations often fail to constrain utility, potentially identifying "attacks" that maximize harmfulness only by destroying the model's general capabilities, which does not reflect a realistic threat where an attacker seeks a capable, harmful model.

Methodology: The TamperBench Framework

To address these gaps, the authors introduce TamperBench, the first unified framework for systematically stress-testing LLM safety under fine-tuning and tampering. The framework operates under a worst-case open-weight threat model, assuming an attacker has full white-box access to weights and representations.

Core Components

1. Unified Attack Suite: TamperBench curates a repository of state-of-the-art attacks across two spaces:

   • Weight-Space: Includes benign full fine-tuning and LoRA (simulating accidental misuse), as well as malicious attacks: harmful full fine-tuning, harmful LoRA, multilingual fine-tuning, backdoor jailbreak-tuning, competing-objectives jailbreak-tuning, and style modulation. Notably, jailbreak-tuning attacks are implemented with 100% harmful data to reflect the open-weight threat surface, rather than the covert data poisoning rates used in some prior closed-API studies.
   • Latent-Space: Includes embedding attacks that perturb internal representations at inference time to elicit harmful completions without modifying weights.

2. Defense Integration: The framework includes re-implementations of seven alignment-stage defenses: Booster, CRL, CTRL, RSN-Tune, SDD, T-Vaccine, and TAR. These allow models to be hardened from scratch rather than relying on pre-released weights.

3. Systematic Hyperparameter Sweeps: A key methodological innovation is the use of Optuna to perform systematic hyperparameter sweeps for every model–attack pair. Instead of relying on arbitrary training choices (e.g., a fixed number of steps), the framework searches for the configuration that maximizes harmfulness while strictly constraining utility loss.

4. Dual Evaluation Metrics:

   • Safety: Measured primarily using StrongREJECT, a rubric-based evaluator that scores compliance, specificity, and convincingness of harmful responses.
   • Utility: Measured using MMLU-Pro (and supported by other benchmarks like IFEval, MT-Bench, etc.).
   • Threat Model Constraint: An attack is considered successful only if it increases harmfulness while keeping the drop in MMLU-Pro accuracy to ≤10%. This ensures the evaluation reflects realistic attackers who require the model to remain capable.

Key Contributions

1. Open-Source Benchmark and Toolkit: TamperBench consolidates attacks, defenses, and evaluation protocols into a single extensible Python toolkit, providing the first standardized basis for comparing open-weight LLMs and tamper-resistance defenses.
2. Realistic Adversarial Evaluation: By performing hyperparameter sweeps over each attack–model pair, the framework reduces sensitivity to arbitrary training choices and enables robust comparisons of susceptibility.
3. Comprehensive Empirical Analysis: The authors evaluate 21 open-weight LLMs (including base, instruction-tuned, and defense-augmented variants) across nine tampering threats.

Key Results

The experiments yield several sobering findings regarding the current state of LLM safety:

• Universal Susceptibility: Every model evaluated (ranging from 0.6B to 70B parameters, including Llama, Qwen, and Mistral families) admits at least one highly effective tampering attack that significantly increases harmfulness (StrongREJECT score > 0.74) while preserving utility.
• Jailbreak-Tuning as the Primary Threat: The competing-objectives jailbreak-tuning method is typically the most severe attack, inducing the largest increase in harmfulness while maintaining utility. It achieved the highest StrongREJECT score in 14 of 21 models.
• Divergent Effects of Instruction Tuning:
  • Qwen3 Family: Post-trained (instruction-tuned) variants generally exhibit lower post-tampering harmfulness than their base counterparts, driven by both higher refusal rates and lower response quality to harmful prompts.
  • Llama Family: Conversely, instruction-tuned Llama variants often reach higher post-tampering harmfulness scores than their base counterparts, particularly at smaller scales (1B). This is attributed to improved instruction-following capabilities that make compliant harmful responses more specific and convincing, even if refusal rates remain similar.
• Failure of Current Defenses: Current alignment-stage defenses largely fail to withstand systematic stress testing.
  • Most defenses (Booster, RSN-Tune, SDD, CTRL, T-Vaccine) failed to significantly reduce post-attack harmfulness under malicious attacks.
  • Some defenses (CTRL, T-Vaccine) caused catastrophic drops in utility (MMLU-Pro scores dropping to ~0.08) without effectively preventing harmfulness.
  • While TAR and CRL showed some partial improvements, TAR incurred a massive utility cost (MMLU-Pro dropping to ~0.18), and CRL offered negligible protection against strong malicious attacks.
• Scale Independence: Preliminary experiments on larger models (Qwen3-32B and Llama-3-70B) indicate that vulnerabilities extend to these scales, with tamperability levels comparable to their 8B counterparts.

Significance and Claims

The paper claims that TamperBench provides a critical, unified foundation for the field of AI safety. Its primary significance lies in:

1. Standardization: It resolves the "crisis of reproducible and realistic evaluation" by unifying disparate attacks and metrics, allowing for direct comparison of defenses and models.
2. Realistic Threat Modeling: By enforcing utility constraints and performing hyperparameter sweeps, it moves beyond "toy" attacks to characterize the true worst-case threat surface for open-weight models.
3. Sobering Reality Check: The results demonstrate that existing safety alignment is not durable against tampering. The authors conclude that "existing models and defenses do not provide durable protection against tampering," highlighting an urgent need for developing and rigorously evaluating new tamper-resistant training methods as open-weight models continue to advance in capability.

The authors explicitly state that their work does not claim to have solved the problem of tamper resistance; rather, it provides the necessary infrastructure to measure the problem accurately and identify that current solutions are insufficient.