Technical Summary: Implicit Safety Alignment from Crowd Preferences

Problem Statement

Reinforcement Learning (RL) agents rely on reward functions to achieve goals, but specifying rewards that fully capture desired behavior—particularly implicit safety constraints—is notoriously difficult. In complex scenarios, designers often focus on task completion while inadvertently omitting safety considerations (e.g., maintaining safety margins or avoiding aggressive behaviors), leading to hazardous agent behavior.

While Reinforcement Learning from Human Feedback (RLHF) has been developed to align agents with human preferences, most existing methods assume a single, unified set of user values. In practice, preference data often comes from a "crowd" of users with diverse task objectives and behavioral patterns. However, these users often share implicit safety criteria (e.g., avoiding unsafe states) despite their differing goals. The core challenge addressed in this work is how to discover these shared, latent safety criteria from heterogeneous crowd preference data and transfer them to downstream tasks where only safety-agnostic task rewards are available.

The authors identify a limitation in the "naive" approach of simply combining a reward model learned from crowd preferences with a downstream task reward. Theoretical and empirical analyses reveal that this reward combination approach suffers from two critical issues:

1. User-Specific Bias: The learned reward model captures not only shared safety criteria but also user-specific components. If the crowd dataset is imbalanced (e.g., one user contributes the majority of data), the learned reward becomes biased toward that specific user's objectives, degrading performance on other tasks.
2. Sensitivity to Weighting: The performance is highly sensitive to the trade-off weight between the preference-learned reward and the task reward, making the method difficult to tune.

Methodology: Safe Crowd Preference-based RL

To overcome these limitations, the authors propose Safe Crowd Preference-based Reinforcement Learning, a hierarchical framework that composes skills rather than combining rewards. The methodology consists of three main stages:

1. Skill Discovery from Crowd Preferences

Instead of learning a single global reward, the framework models diverse user-preferred behaviors as low-level skills.

• Latent Variable Inference: Since the true user context $z$ (which determines specific preferences) is unobserved, the method uses a Variational Auto-Encoder (VAE) to infer a latent variable $z'$ from the crowd preference data. This $z'$ serves as a proxy for the underlying user context.
• Latent-Conditioned Learning: The framework learns a latent-conditioned reward function $r_\phi(s, a, z')$ or directly learns a latent-conditioned policy $\pi_\theta(a|s, z')$.
  • Safe-VPL: Uses a Variational Preference Learning (VPL) approach with a Bradley-Terry model to learn a reward conditioned on $z'$, followed by offline RL (e.g., IQL) to learn the low-level policy.
  • Safe-CPL: A novel variant that integrates Contrastive Preference Learning (CPL) into the VPL framework. It directly learns the latent-conditioned policy $\pi_\theta(a|s, z')$ without an explicit intermediate reward learning stage, avoiding RL optimization instability.

2. High-Level Policy Composition

For a new downstream task with a safety-agnostic reward $r_{new}$, a high-level policy $\pi_h(z'|s)$ is trained to compose the learned low-level skills.

• Action Selection: The high-level policy outputs a latent variable $z'$, which indexes a specific low-level skill $\pi_l(a|s, z')$ to generate the final action.
• Safety Inheritance: Since all low-level skills are learned from crowd preferences, they inherently encode the shared safety criteria (e.g., avoiding unsafe state-action pairs). By restricting the agent to compose these skills, safety is enforced without explicit safety signals in the downstream task.
• Regularization: To prevent the high-level policy from selecting out-of-distribution skills that could lead to unsafe behavior, a regularization term is applied to keep the selected latent variable close to the prior distribution learned from the crowd data.

3. Downstream Training

The high-level policy is trained to maximize the downstream task reward $r_{new}$ while adhering to the constraints imposed by the low-level skill space. This can be performed in both online (with interaction) and offline (using only existing datasets) settings.

Key Contributions

1. Problem Formulation: The paper formalizes the problem of transferring shared safety objectives from heterogeneous crowd preferences to downstream tasks, identifying the limitations of direct reward combination under preference imbalance.
2. Theoretical Analysis: The authors provide theoretical proofs showing that:
   • A single reward model learned from crowd data will prefer safe trajectories over unsafe ones if the safety penalty is sufficiently large (Theorem 4.2).
   • However, if the crowd data is imbalanced, the learned reward ordering aligns with the majority user's specific objectives, leading to performance degradation (Theorem 4.3).
   • The proposed skill composition approach guarantees safety if the low-level policies are optimal with respect to their conditioned utilities (Corollary A.6).
3. Algorithmic Framework: The proposal of a hierarchical framework (Safe-VPL and Safe-CPL) that decouples skill discovery from task optimization, enabling the reuse of safety-aligned behaviors across diverse tasks.
4. Empirical Validation: Extensive experiments across six Safe RL environments (Reach, Run, Circle, Ant-vel, Swimmer-vel, HalfCheetah-vel) and a proof-of-concept Large Language Model (LLM) conversational task.

Experimental Results

The method was evaluated against several baselines, including Oracle methods (trained with ground-truth safety signals), Task-Only (optimizing only task rewards), Safety-Only Preference Learning (SOPL), and Reward Combination (RC).

• Safety Performance: The proposed methods (Safe-VPL and Safe-CPL) substantially reduced safety costs compared to Task-Only baselines. In the offline setting, average normalized safety costs dropped from 1.0 (Task-Only) to approximately 0.02–0.03, comparable to Oracle methods.
• Task Performance: The approach achieved task performance comparable to Oracle methods and significantly outperformed Reward Combination (RC) methods, especially in imbalanced preference settings.
• Robustness to Imbalance: While RC performance degraded sharply when one user contributed the majority of data (shifting away from the Oracle frontier), the proposed skill composition method remained robust and close to Oracle performance regardless of preference imbalance.
• LLM Evaluation: In a simplified language-based bandit setting, the method successfully avoided harmful responses while achieving strong task performance, whereas Task-Only agents frequently selected harmful responses to maximize reward.

Significance and Claims

The paper claims that its primary significance lies in demonstrating that implicit safety objectives can be discovered and transferred from crowd preferences without requiring explicit safety rewards or demonstrations.

• Reuse of Data: The approach enables the reuse of crowd preference data across tasks, even when those preferences were not collected for a specific downstream task, potentially reducing the cost of repeated human feedback collection.
• Mitigating Reward Design Challenges: By leveraging implicit safety signals, the method addresses the common issue of incomplete or misspecified human-designed rewards.
• Scalability: The framework assumes that crowd preferences reflect a shared consensus on safety. While the authors acknowledge that real-world noise or adversarial feedback could challenge this, they argue that their method offers a robust way to learn safe behaviors in domains where safety specifications are difficult to define formally.

The authors maintain a modest stance, noting that their experiments rely on synthetically annotated preferences generated from ground-truth rewards rather than naturalistic human feedback, and that future work is needed to validate the approach against noisier, real-world human data.