🤖 machine learning

Safety Game: Inference-Time Alignment of Black-Box LLMs via Constrained Optimization

This paper proposes a model-independent, black-box framework for inference-time safety alignment of large language models that formulates the trade-off between safety and helpfulness as a two-player zero-sum game, enabling stakeholders to enforce safety constraints via linear programming without requiring model access or retraining.

Original authors: Tuan Nguyen, Long Tran-Thanh

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tuan Nguyen, Long Tran-Thanh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but somewhat unpredictable, robot assistant. You want to use it to help you with tasks, but you're worried it might accidentally give you dangerous advice (like how to build a bomb) or refuse to help at all when you just need a simple recipe.

Usually, to fix a robot like this, you have to take it apart, retrain it with new rules, and hope it learns. But what if the robot is a "black box"? You can't see inside, you can't take it apart, and you can't retrain it. This is the problem with many powerful AI models available today: companies can't change their internal code, only ask them questions and get answers.

This paper proposes a clever "safety filter" that sits outside the robot, acting like a smart gatekeeper. It doesn't need to know how the robot thinks; it just needs to look at the answers the robot might give and choose the best one.

Here is how it works, using a few everyday analogies:

1. The "Menu" Analogy (The Candidate Set)

Instead of asking the robot to write a brand new essay from scratch (which is hard to control), the system first asks the robot to generate a small list of possible answers, like a menu of options.

Option A: A very helpful answer, but it might be dangerous (e.g., "Mix bleach and ammonia for a cool gas!").
Option B: A completely safe answer, but it's useless (e.g., "I cannot answer that.").
Option C: A balanced answer (e.g., "You can mix baking soda and vinegar for a safe reaction.").

The goal is to pick Option C.

2. The "Tightrope Walker" (The Game Theory)

The paper treats the choice of an answer like a game between two players:

Player 1 (The Helper): Wants to give the most useful, informative answer possible.
Player 2 (The Safety Guard): Wants to ensure the answer isn't dangerous.

The system uses a mathematical "game" to find the perfect balance. It's like a tightrope walker trying to stay as far forward as possible (helpful) without falling off the edge (unsafe). The system calculates the "Minimax" strategy, which is the safest way to be as helpful as possible without crossing the line.

3. The "Budget" Analogy (Constrained Optimization)

Imagine you have a strict budget for "risk."

Every time the robot suggests a risky answer, it costs you "risk dollars."
You have a fixed budget (say, $10).
The system looks at all the options on the menu. It can pick a slightly risky answer if it's very helpful, as long as the total "risk cost" of the final choice stays under your $10 budget.
If an answer is too dangerous, it costs too much, and the system rejects it.
If all answers are too dangerous, the system defaults to a "safe fallback" (like saying "I can't help with that").

4. The "Referee" (The Linear Programming Solver)

How does the system actually make this decision? It doesn't ask the robot to "think" about the rules (because the robot might be bad at following them or might be tricked). Instead, the system uses a separate, simple math tool called a Linear Programming (LP) solver.

Think of the LP solver as a strict referee who only looks at the numbers:

Score the options: How helpful is this? How risky is it?
Run the math: "If I pick this one, do I stay under the risk budget?"
Make the call: Pick the option that gives the most help while staying under the budget.

Because this referee is a simple math program, it's fast, reliable, and doesn't need to be retrained every time a new safety rule comes along.

Why is this a big deal?

No Surgery Required: You don't need to open up the AI model. You can use this with any model, even ones owned by big tech companies that you can't change.
Flexible: If new safety rules appear tomorrow (e.g., "Don't talk about politics"), you just change the math budget. You don't need to retrain the whole AI.
Fair for Everyone: Small companies or researchers who can't afford to train their own massive AI models can still use powerful, pre-made models safely.

The Bottom Line

The paper shows that by treating safety as a math problem of balancing "helpfulness" and "risk" on a menu of options, we can make black-box AI models much safer without ever touching their internal code. It's like putting a smart, mathematically perfect traffic light in front of a chaotic intersection to keep everyone safe, without needing to rebuild the cars.

Technical Summary: Safety Game: Inference-Time Alignment of Black-Box LLMs via Constrained Optimization

1. Problem Statement

The central challenge addressed is ensuring Large Language Models (LLMs) comply with safety requirements without the prohibitive costs and inflexibility of training-time alignment methods (e.g., RLHF, fine-tuning). Existing inference-time alignment approaches often require access to model internals (weights or architecture), rendering them impractical for third-party stakeholders using proprietary black-box APIs. Furthermore, current methods struggle to balance the trade-off between generating helpful, informative responses and avoiding harmful or risky content. There is a critical need for a model-independent, black-box framework that can enforce safety constraints at inference time without retraining the base model.

2. Methodology: The Safety Game Framework

The authors propose SAFETY GAME, a model-agnostic framework that treats the selection of a response from a finite set of candidates as a constrained optimization problem derived from game theory.

2.1. Problem Formulation

The interaction between the LLM (Player 1) and the user (Player 2) is modeled as a one-shot, imperfect-information game.

Candidate Set: For a given prompt $x$ , the system considers a finite set of candidate responses $R = \{r_1, \dots, r_m, r_s\}$ , where $r_s$ is a designated safe fallback (e.g., a refusal or generic safe statement) with zero helpfulness lift and zero risk.
Scoring: Each candidate $r_i$ is assigned a helpfulness score ( $h_i$ ) and a safety-risk score ( $s_i$ ). These are computed via self-probing, where the base LLM is queried with binary probes ("Is this helpful?" / "Is this harmful?") to estimate log-probabilities of YES/NO tokens.
Margins: The framework operates on margins relative to the fallback: $M_i = h_i - h_s$ (helpfulness lift) and $\Delta_i = s_i - s_s$ (extra risk).

2.2. Game-Theoretic Derivation

The selection problem is formalized as a two-player zero-sum game. The LLM commits to a mixed strategy $\pi$ (a probability distribution over candidates) to maximize expected helpfulness while minimizing the risk of violating safety policies under uncertain user intent (benign vs. adversarial).

Minimax Equilibrium: The game seeks a minimax equilibrium where the LLM maximizes helpfulness subject to a safety constraint.
Decision Layer: This equilibrium is operationalized as a Linear Programming (LP) problem solved at inference time:
$\max_{\pi \in \Delta_m} \sum_{i=1}^m \pi_i M_i \quad \text{s.t.} \quad \sum_{i=1}^m \pi_i \Delta_i \leq T$
Here, $T$ is a safety budget parameter. If the LP is infeasible, the system defaults to the safe fallback $r_s$ .

2.3. Robustness and Implementation

Noisy Signals: The framework acknowledges that self-probes (or external scorers) provide noisy estimates. Theorem 3.1 provides a robustness guarantee: if the risk scores have a bounded error $\epsilon_s$ , enforcing a tightened constraint $T' = T - 2\epsilon_s$ ensures the true risk remains within $T$ .
Smooth Relaxation: To handle boundary sensitivity and noisy scores, a sigmoid relaxation of the hard constraint is also proposed, allowing for a smooth penalty ramp near the safety cap.
Black-Box Compatibility: The method requires only API access to generate candidates and score them (via self-probing or external classifiers). It does not modify the base model's weights.

3. Key Contributions

Game-Theoretic Decision Layer: The paper derives a selection rule for black-box LLMs based on adaptation safety in incomplete-information games. This yields two non-obvious design choices:
- Fallback-Relative Margins: Optimization is based on improvement over a safe baseline, not raw scores.
- Aggregate Risk Constraint: Safety is constrained over the mixture of candidates (expected risk) rather than filtering individual candidates, allowing borderline safe candidates to be selected if offset by safer ones.
Empirical Validation of the Decision Layer: Experiments demonstrate that the performance gain is intrinsic to the LP decision layer, not the scorer. When using a fixed external scorer (Llama-Guard-3-8B), the LP-based selector recovered 81 more human-safe responses than the best threshold-based filter across 440 unsafe prompts.
Scalable Black-Box Alignment: The framework offers a plug-and-play solution for stakeholders without access to model internals, enabling safety enforcement across diverse model families without retraining.

4. Experimental Results

The method was evaluated on four benchmarks: HHH (Helpfulness, Harmlessness, Honesty), TruthfulQA, SafetyBench, and SORRY-Bench, across five different LLMs (including LLaMA-2, LLaMA-3, and GPT-OSS).

SafetyBench: The method achieved the largest gains, outperforming all baselines (including Consensus Game variants like ER-G/ER-D) on 4 out of 5 models. Pairwise McNemar tests showed statistically decisive improvements ( $p < 10^{-4}$ ).
SORRY-Bench: The method achieved the lowest Expected Human Fulfillment Rate (HFR) on three of five models, with relative reductions of 45–81% over the strongest baseline. Crucially, it maintained higher refusal quality (providing informative alternatives rather than stonewall refusals) compared to baselines that collapsed to near-zero HFR via aggressive filtering.
General Benchmarks (HHH, TruthfulQA): The method matched or exceeded the best baselines in 14 out of 20 model-benchmark settings, demonstrating that safety alignment did not degrade general helpfulness or truthfulness.
Over-Refusal: On benign instruction-following tasks (AlpacaEval), the method did not exhibit catastrophic over-refusal, maintaining performance comparable to baselines while sitting at the Pareto frontier on borderline prompts.

5. Significance and Claims

The paper claims to establish the viability of black-box alignment frameworks. Its primary significance lies in decoupling the safety decision from the model's internal training, allowing for:

Democratization: Smaller organizations and resource-constrained entities can enforce robust safety practices without the financial or technical burden of retraining models.
Flexibility: Safety constraints can be updated or tuned (via the budget $T$ ) at inference time to adapt to new regulations or social norms without retraining.
Theoretical Grounding: The approach provides a principled, game-theoretic justification for constrained selection, offering guarantees on adaptation safety that simple thresholding or ranking cannot provide.

The authors modestly note limitations: the method's performance is conditional on the quality of the measurement layer (scorer calibration), and it currently operates on single-turn selection rather than multi-turn dialogue. However, the work opens a new direction for scalable, accessible AI safety that operates independently of the underlying model architecture.