Technical Summary: BSO: Safety Alignment Is Density Ratio Matching

Problem Statement

The alignment of Large Language Models (LLMs) faces a fundamental tension between helpfulness (following instructions) and safety (avoiding harmful content). Existing approaches to resolve this tension generally fall into two categories, both of which have significant drawbacks:

1. Constrained Optimization (e.g., Safe RLHF): These methods formulate safety as a constraint in a Lagrangian framework, requiring separate reward and cost models, online Reinforcement Learning (RL), and primal-dual updates. While principled, this pipeline is complex, computationally expensive, and introduces instability across multiple training stages.
2. Direct Preference Optimization (DPO) Extensions: Methods like SACPO, CDPO, and SafeDPO attempt to simplify training by eliminating online RL. However, they often rely on ad-hoc modifications to incorporate safety, such as multi-stage sequential training, auxiliary models, or heuristic margin terms added to the loss function without a unified theoretical derivation.

The field lacks a theoretical framework that systematically derives safety-aware preference optimization from first principles, explaining why existing heuristics work and suggesting principled alternatives.

Methodology: Bregman Safety Optimization (BSO)

The authors propose Bregman Safety Optimization (BSO), a unified framework that recasts safety alignment as a density ratio matching problem.

1. Theoretical Foundation

The core insight is a structural decomposition of the likelihood ratio of the optimal safe policy ($\pi^*_{safe}$). The authors derive a closed-form relationship (Proposition 1) showing that the ratio of the optimal safe policy for a preferred response ($y_w$) versus a dispreferred response ($y_l$) is determined by three factors:

• The reference model ratio ($\pi_{ref}$).
• The helpfulness preference ratio (derived from the Bradley-Terry model).
• A safety correction term based on binary safety labels ($s_w, s_l$) and a penalty constant $C$.

Specifically, the optimal ratio is:
$$\frac{\pi^*_{safe}(y_w | x)}{\pi^*_{safe}(y_l | x)} = \frac{\pi_{ref}(y_w | x)}{\pi_{ref}(y_l | x)} \cdot \left( \frac{p_{help}(y_w \succ y_l | x)}{p_{help}(y_w \prec y_l | x)} \right)^{1/\beta} \cdot e^{-C(s_w - s_l)/\beta}$$

This decomposition implies that safety alignment can be achieved by matching the model's likelihood ratio to a target "data ratio" derived from these components.

2. The BSO Framework

BSO minimizes the Bregman divergence between the target data ratio ($R_{data}$) and the model ratio ($R_\theta$).

• Loss Function: By minimizing the Bregman divergence $D_h(R_{data} \| R_\theta)$, the authors derive a tractable single-stage loss function $L^{BSO}_h$.
• Generality: The framework is parameterized by a strictly convex generator function $h$. Different choices of $h$ yield different training algorithms, all of which provably converge to the optimal safe policy under sufficient model capacity.
• Recovering SafeDPO: The authors demonstrate that SafeDPO is a special case of BSO where the generator corresponds to logistic regression. Crucially, this derivation shows that SafeDPO's "safety margin" is not a heuristic addition but arises naturally from the safety-penalized reward structure.

3. Practical Training Strategies

To address numerical instability caused by large safety penalties ($C$), the authors propose a data transformation strategy:

• Swap: If the preferred response is unsafe and the dispreferred one is safe, swap the labels.
• Drop: If both responses are unsafe, remove the pair.
  This transformation effectively implements the preference reversal required by a large $C$ without introducing unstable exponential terms into the loss function, allowing training with a moderate, numerically stable $C$.

4. Generator Design (SBA)

The authors analyze the gradient weighting behavior of different generators. They identify that an ideal safety-aware generator should:

1. Maintain stable weights for concordant pairs (where safety labels match).
2. Amplify weights for safe-winner pairs (where the safe response is preferred).

They propose the Scaled Basu's Power Divergence (SBA) generator as the default choice. Unlike other generators (like Logistic or KLIEP), SBA decouples the baseline weight from the amplification rate, allowing the model to emphasize safety-conflicting pairs without distorting the baseline helpfulness signal.

Key Contributions

1. Unified Framework: BSO provides the first unified, principled framework for safety-aware preference optimization based on density ratio matching under Bregman divergences.
2. Theoretical Guarantees: The framework proves that minimizing Bregman divergences between the derived ratios recovers the optimal safe policy.
3. Simplicity and Efficiency: BSO requires no auxiliary reward or cost models and operates in a single training stage. It introduces only one hyperparameter beyond standard DPO (the generator choice or its scaling parameter).
4. Unification of Existing Methods: The framework theoretically recovers existing methods like SafeDPO as special cases, explaining their mechanisms through the lens of ratio matching.
5. Empirical Validation: Extensive experiments show that BSO, particularly with the SBA generator, achieves a superior Pareto trade-off between helpfulness and safety compared to baselines (SafeRLHF, SACPO, SafeDPO) across different model backbones (Qwen 2.5 0.5B and Llama 3.2 3B).

Experimental Results

• Safety-Helpfulness Trade-off: On the PKU-SafeRLHF-30K benchmark, BSO consistently shifts the safety-helpfulness frontier outward. It improves safety metrics (harmless ratio) without sacrificing helpfulness, and in many cases, improves both simultaneously.
• Ablation Studies:
  • Safety Penalty ($C$): The authors find a "moderate regime" for $C$. If $C$ is too small, safety is insufficient; if too large, it overwhelms the helpfulness signal, causing performance to degrade.
  • Amplification Parameter ($\lambda$): For the SBA generator, mild amplification ($\lambda \approx 0.2$) yields the best joint performance. Aggressive amplification ($\lambda \ge 0.6$) destabilizes optimization, suggesting that while safety-conflicting pairs need emphasis, concordant pairs remain crucial for stabilizing helpfulness.
• Model Evaluation: Results hold across both model-based evaluations (using Beaver-7B reward/cost models) and LLM-based evaluations (using DeepSeek-V4 as a judge).

Significance and Claims

The paper claims that safety alignment is best treated as a principled ratio-matching problem rather than a collection of heuristic margins or complex multi-stage procedures. By deriving safety-aware objectives from the structural properties of the optimal policy, BSO offers a simpler, more robust, and theoretically grounded alternative to current state-of-the-art methods. The authors emphasize that their approach reduces engineering overhead (removing the need for cost models and online RL) while providing stronger theoretical guarantees for the resulting policy.

The paper modestly notes limitations, acknowledging that the current analysis is restricted to binary safety labels and pairwise preference data, and that the method's performance depends on the quality of the underlying safety signals.