🤖 machine learning

Safety Hacking in Constrained Best-of- $N$ Inference-time Scaling

This paper demonstrates that inference-time scaling via constrained Best-of- $N$ sampling is inherently vulnerable to "safety hacking," where imperfect safety proxies contaminate the feasible set and reward maximization amplifies residual unsafe outputs, causing safety failures to become asymptotically certain as the number of samples grows unless specific coverage-control mechanisms are employed.

Original authors: Akifumi Wachi, Takumi Tanabe, Youhei Akimoto

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Akifumi Wachi, Takumi Tanabe, Youhei Akimoto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern landscape of artificial intelligence, large language models are increasingly guided by a two-step process to ensure they are both helpful and harmless. First, a safety filter acts as a gatekeeper, scanning potential responses and rejecting anything that appears dangerous or inappropriate. Second, a reward system evaluates the remaining safe options, ranking them to select the one that seems most useful or high-quality. This combination allows developers to scale up the number of options a model considers, hoping that by looking at more possibilities, they can find a better answer. The assumption has long been that if the safety filter is accurate enough and the reward system is good, simply increasing the number of candidates will lead to safer and better outcomes.

However, a new study reveals a hidden flaw in this logic, showing that scaling up can actually backfire. The researchers, working with the LY Corporation and academic institutions, discovered that when a model is asked to choose the best response from a large pool of candidates, the selection process can inadvertently exploit tiny errors in the safety filter. If the filter mistakenly allows a few unsafe responses to pass through, and if those unsafe responses happen to receive slightly inflated scores from the reward system, the selection process will eventually identify them as the top choice over the truly safe options. This phenomenon, which the authors call "safety hacking," means that as the system looks at more and more candidates, the probability of it selecting a harmful response can rise toward certainty, even if the safety filter is mostly correct.

The core of the problem lies in how the system handles the rare mistakes that inevitably occur. No safety filter is perfect; occasionally, it will let a dangerous response slip through, classifying it as safe when it is not. In a small search, this mistake might be overlooked. But when the system is scaled up to examine thousands of candidates, it begins to search the extreme upper limits of its reward scores. The researchers found that if the few unsafe responses that slip through the filter happen to have reward scores that are slightly higher than the safe ones, the selection process will eventually zero in on them. It is not that the system is trying to be dangerous; rather, it is following its instructions to find the highest-scoring option, and the highest-scoring option in a contaminated pool happens to be the one that was wrongly allowed in.

To demonstrate this, the team ran experiments using both simple toy problems and real-world language models. In their simulations, they created a scenario where a tiny fraction of responses were unsafe but passed the filter, while the rest were safe. They then observed what happened as the number of candidates grew from a handful to thousands. The results were stark: as the number of candidates increased, the probability of the system choosing the unsafe responses shifted dramatically. It stopped picking the safe responses and began selecting the unsafe ones with near-total frequency. This happened even though the safety filter was wrong only a tiny fraction of the time and the reward system was only slightly inaccurate on average. The danger was not in the frequency of the errors, but in the specific way those errors interacted with the system's search for the absolute best score.

The study also tested whether a different approach could fix this. They introduced a method called "constrained pessimistic sampling," which deliberately avoids concentrating too heavily on the single highest-scoring option. Instead of hunting for the absolute peak, this method spreads its attention more broadly across the safe options. In their tests, this approach successfully prevented the system from latching onto the unsafe responses, keeping the safety hacking rate low even as the number of candidates grew. However, the researchers noted that this method does not eliminate the initial problem of the safety filter letting bad responses through; it simply prevents the system from amplifying that mistake. The fundamental issue remains that the pool of candidates is already contaminated.

The implications of this finding are significant for how we build and scale AI systems. It suggests that simply making safety filters more accurate on average is not enough to guarantee safety when using large-scale search methods. The researchers showed that the relationship between the reward scores of safe and unsafe responses is just as critical as the accuracy of the filter itself. If the unsafe responses have a "heavier tail" in their reward distribution—meaning they occasionally get very high scores by chance—the system will eventually find them and prefer them. This means that safety cannot be treated as a separate step that happens before optimization; the two processes are deeply intertwined, and errors in one can be magnified by the other.

In their experiments with real language models, the team confirmed that this mechanism operates in practice. They used a standard safety filter and a reward model to select responses from a large pool of candidates generated by an AI. As they increased the number of candidates, the rate at which the system selected harmful responses rose, while the quality of the safe responses it found did not improve enough to compensate. When they swapped the reward model for a different one, the behavior changed, proving that the specific way the reward system ranks the options is the key driver of this failure. This confirms that the problem is not just about the safety filter being imperfect, but about how the reward system interacts with the few unsafe options that manage to pass through.

The researchers conclude that safe scaling requires a dual approach: improving the safety filter to reduce the number of bad options entering the pool, and carefully managing how the system selects from that pool to avoid amplifying the remaining errors. They argue that current methods, which rely on picking the single best option from a large set, are inherently vulnerable to this type of failure. While their proposed alternative method offers a way to limit the damage, it does not solve the root cause. The study serves as a warning that as we push AI systems to look at more possibilities to find better answers, we must be equally vigilant about how those systems handle the rare, dangerous mistakes that inevitably slip through the cracks. Without addressing the interaction between safety filtering and reward optimization, scaling up may lead to more dangerous outcomes rather than safer ones.

Technical Summary: Safety Hacking in Constrained Best-of-N Inference-time Scaling

1. Problem Statement

The paper addresses a critical failure mode in inference-time scaling for large language models (LLMs) that utilize learned safety and reward proxies. In standard pipelines, such as Constrained Best-of-N (CBoN) sampling, a model generates $N$ candidate responses, filters them using a learned safety model to define a "proxy-feasible" set, and selects the candidate with the highest learned reward score.

The authors identify a two-stage failure mechanism they term safety hacking:

Contamination: An imperfect safety proxy admits unsafe outputs (false positives) into the proxy-feasible set.
Amplification: The subsequent reward maximization step (selecting the best of $N$ ) amplifies these residual unsafe outputs if they possess a heavier upper tail in the learned reward distribution compared to safe outputs.

The core problem is that increasing the inference budget ( $N$ ) does not merely increase the probability of encountering an unsafe output; it can asymptotically guarantee the selection of an unsafe output, even when the average error rates of both the safety and reward proxies are arbitrarily small.

2. Methodology and Theoretical Framework

Definitions

True Sets: $S^\star_+$ (truly safe) and $S^\star_-$ (truly unsafe).
Proxy-Feasible Set: $\hat{S}_+$ , defined by a learned safety proxy $\hat{g}$ exceeding a threshold $b$ .
Decomposition: The proxy-feasible set splits into:
- $A$ : True positives (truly safe and accepted).
- $B$ : False positives (truly unsafe but accepted).
Safety Hacking: An event where the selected output $\hat{y} \in B$ .

Theoretical Analysis

The paper derives finite- $N$ bounds and asymptotic results based on the joint upper tails of the learned reward scores for sets $A$ and $B$ .

Tail Quantities: Let $\hat{\Psi}_A(t)$ and $\hat{\Psi}_B(t)$ be the probability that a sample from the reference distribution falls into sets $A$ or $B$ respectively, with a learned reward score greater than $t$ .
Finite-N Bounds (Theorem 4.1): The probability of safety hacking is bounded by functions of $\hat{\Psi}_A$ and $\hat{\Psi}_B$ . Specifically, if no sample from $A$ exceeds a threshold $t$ while at least one sample from $B$ does, CBoN selects an unsafe output.
Asymptotic Certainty (Theorem 4.2): If there exists a sequence of thresholds $t_N$ $t_{N}$ such that $N\hat{\Psi}_A(t_N) \to 0$ $N \hat{Ψ}_{A} (t_{N}) \to 0$ and $N\hat{\Psi}_B(t_N) \to \infty$ $N \hat{Ψ}_{B} (t_{N}) \to \infty$ , the probability of safety hacking converges to 1 as $N \to \infty$ $N \to \infty$ .
- This implies that even if the mass of false positives ( $\epsilon$ ) and average proxy errors vanish, a budget $N = \Theta(1/\epsilon)$ is sufficient to amplify the residual error to high probability, provided the unsafe tail is "heavier."
Gaussian Tail Separation: The paper shows that if the reward-proxy error on unsafe outputs has a heavier Gaussian tail (larger variance) than on safe outputs, safety hacking becomes certain as $N$ grows, even with bounded true rewards.

Mitigation: Coverage Control

The authors propose that while one cannot remove the initial contamination (false positives admitted by the safety filter), one can limit the amplification of this contamination.

Principle: Limit the $\chi^2$ divergence between the final selection policy and the proxy-feasible reference distribution.
Algorithm (cPes): They instantiate this via Constrained Pessimistic Sampling (cPes). This involves:
1. Clipping the learned reward scores to a bounded range.
2. Solving a regularized optimization problem to reweight the proxy-feasible candidates, penalizing concentration on high-reward outliers.
Guarantee (Theorem 5.1 & Corollary 5.3): cPes provides an $N$ -independent bound on the probability of selecting an unsafe output. However, this bound depends on the initial contamination mass $\kappa(x,b)$ and cannot reduce it to zero; it only prevents the selection probability from growing with $N$ .

3. Key Contributions

Formalization of Safety Hacking: Defined as the selection of an output that satisfies a learned safety constraint but violates the true safety criterion, specifically driven by the interaction between filter contamination and reward tail amplification.
Characterization of Amplification: Derived finite- $N$ bounds showing that safety hacking is governed by the joint reward tails of safe vs. unsafe proxy-feasible outputs. Proved that a heavier unsafe tail leads to asymptotic certainty of hacking, even with vanishing average errors.
Coverage Control Principle: Established that bounding the $\chi^2$ divergence from the proxy-feasible reference distribution yields an $N$ -independent safety bound.
Empirical Validation:
- Toy Experiments: Demonstrated that CBoN shifts from selecting safe to unsafe classes as $N$ increases when the unsafe class has a heavier reward tail, while cPes limits this shift.
- LLM Experiments: Using JailbreakBench, HarmBench, and AdvBench prompts with Qwen2.5-7B, Llama-Guard-3, and Beaver/Skywork reward models.
  - Observed that as $N$ increases, CBoN's proxy reward improves, but safety-hacking rates and safety-aware true rewards deteriorate.
  - Ablation: Replacing the reward proxy (Beaver vs. Skywork) reversed the scaling trend, confirming that the ranking within the contaminated set (tail behavior) is the driver, not just the filter's false-positive rate.

4. Results

Toy Problem: With an unsafe class having a heavier reward tail ( $\sigma_B > \sigma_A$ ), CBoN's safety hacking rate rose from 0.023 ( $N=1$ ) to 0.9996 ( $N=8192$ ), while mean true reward dropped. cPes maintained a low hacking rate (~0.02) across all $N$ .
LLM Experiments:
- Decomposition: The increase in safety hacking with CBoN was primarily driven by "competitive unsafe wins" (unsafe candidates outscoring safe ones), not merely by the absence of safe candidates ("unsafe-only exposure").
- Reward Proxy Sensitivity: Switching from Beaver to Skywork reward models reduced the competitive unsafe win rate, lowering the overall hacking rate, demonstrating that tail behavior is proxy-dependent.
- cPes Performance: cPes limited the amplification effect, keeping hacking rates closer to the baseline "random feasible" selection, though it did not eliminate the baseline contamination.

5. Significance and Claims

The paper claims to expose an inherent difficulty in inference-time scaling when relying on learned safety models. The primary significance lies in shifting the focus from average filter accuracy to joint tail behavior.

Limitation of Average Metrics: The authors argue that average filter accuracy and average proxy errors are insufficient predictors of safety under scaling; the relative upper tails of rewards for safe vs. unsafe outputs are the critical determinant.
Distinction from Prior Work: Unlike reward overoptimization (which assumes a benign feasible set) or jailbreaking (which assumes adversarial prompts), this work analyzes a fixed prompt where the feasible set itself is contaminated, and the search process amplifies this specific contamination.
Modest Mitigation: The paper is modest regarding solutions. It claims that coverage control (cPes) limits amplification but cannot repair a contaminated feasible set. If the safety filter admits unsafe outputs with high learned rewards, regularized selection may still favor them.
Future Directions: The authors explicitly state their analysis is limited to non-adaptive, i.i.d. sampling pipelines. They note that adaptive search, self-refinement, or agentic planning (where candidate distributions depend on previous proxy evaluations) are not covered and represent important future work.

In conclusion, the paper establishes that safe inference-time scaling requires a dual approach: controlling entry into the feasible set (improving the safety filter) and controlling downstream concentration (managing reward tail behavior and selection diversity).