A Descriptive and Normative Theory of Human Beliefs in RLHF

Technical Summary: A Descriptive and Normative Theory of Human Beliefs in RLHF

Problem Statement
Reinforcement Learning from Human Feedback (RLHF) traditionally models human preferences as a function of the human's reward function or the optimal state-action values ($A^*$) of the agent. Standard approaches, including return-based and regret-based models, assume that labelers evaluate trajectories based on what actually occurred or what an optimal agent could achieve. However, this work posits that human labelers also hold beliefs about the actual capabilities of the agent being trained. If a labeler assumes an agent is optimal when it is not (or vice versa), their preferences may lead to catastrophic or suboptimal post-RLHF policies. The paper investigates two core questions: (1) Do human labelers' beliefs about agent capabilities descriptively affect the preferences they provide? and (2) Normatively, what is the ideal set of beliefs labelers should hold to maximize the performance of the resulting policy?

Methodology
The authors propose a belief-based preference model and validate it through theoretical analysis, synthetic experiments, and a human subject study.

1. Theoretical Framework:

   • The authors define Agent Capability Belief as a $Q$-function ($Q^{\pi_{belief}}$) representing the labeler's estimate of the agent's post-RLHF policy.
   • They introduce a new preference model (Equation 3) where the probability of preferring trajectory $\sigma^+$ over $\sigma^-$ is determined by the advantage function $A^{\pi_{belief}}$ rather than the optimal advantage $A^*$.
   • Normative Ideal: The paper defines a normative ideal belief ($Q^{\pi^*_{belief}}$) not as absolute optimality, but as the belief that yields the highest expected return for the best achievable post-RLHF policy given the agent's constraints (e.g., function approximation limits, noisy actuators).
   • Theoretical Bounds: The authors derive a formal bound (Theorem 4.3) on the performance loss of the post-RLHF policy as a function of the "agent-labeler disagreement" ($\delta$), defined as the magnitude of difference between the optimal belief and the labeler's actual belief. The bound shows that expected return degrades linearly with the magnitude of this disagreement.

2. Synthetic Experiments:

   • The authors conducted experiments in a GridWorld domain where the agent's capabilities were limited by an $\epsilon$-greedy noise parameter ($\epsilon$).
   • Labeler beliefs were manipulated by varying the assumed noise level ($\epsilon'$) used to generate synthetic preferences.
   • The study measured the expected return of the trained policy across a matrix of true agent noise ($\epsilon$) and assumed labeler noise ($\epsilon'$).

3. Human Subject Study:

   • Domain: A self-driving car simulation (CARLA).
   • Intervention: Participants were assigned to three conditions: (1) Control (no priming), (2) Safe Priming (watching a video of a car driving safely and competently), and (3) Unsafe Priming (watching a video of a car drifting and crashing).
   • Task: Participants provided preferences over pairs of driving trajectories, choosing between faster/riskier paths and slower/safer paths.
   • Analysis: The study used sentiment analysis (via GPT-4) on free-text feedback regarding the car's capabilities and statistical testing (Kruskal-Wallis and Dunn's test) on Likert-scale preference data to determine if priming shifted beliefs and subsequent preferences.

Key Contributions

1. Illustration of Disagreement: The paper demonstrates that poor agreement between a labeler's beliefs and an agent's actual capabilities can lead to suboptimal policies, even when the labeler acts "rationally" based on their (incorrect) beliefs.
2. Normative Theory: It defines a normative ideal for labeler beliefs that is relative to the agent's achievable performance ceiling, rather than absolute optimality, and provides a theoretical bound on the regret caused by deviations from this ideal.
3. Empirical Validation of Belief Influence: The human study confirms that human beliefs about agent capabilities significantly affect preference generation. Specifically, priming participants with information about an agent's capabilities (safe vs. unsafe) statistically significantly altered their preferences.
4. Suboptimality of Assuming Optimality: Synthetic experiments show that assuming agent optimality (the standard regret-based approach) is often suboptimal when the agent has inherent performance limitations (e.g., high noise).

Results

• Theoretical: The derived bound (Equation 4) establishes that the expected return of the post-RLHF policy is lower-bounded by the optimal return minus a term proportional to the agent-labeler disagreement ($\delta / (1-\gamma)$).
• Synthetic: In the GridWorld experiments, the highest policy returns were achieved when the labeler's assumed noise ($\epsilon'$) closely matched the agent's true noise ($\epsilon$). Assuming an optimal agent ($\epsilon'=0$) when the agent was noisy ($\epsilon=0.3$ or $0.5$) resulted in significantly lower returns, as the agent learned risky behaviors it could not execute safely.
• Human Study:
  • Beliefs: Sentiment analysis of free-text responses showed that participants in the "unsafe priming" group held significantly lower confidence in the car's capabilities compared to the "safe priming" group.
  • Preferences: Statistical analysis revealed a significant difference ($p=0.015$) in preferences between the safe and unsafe priming groups. Participants primed with unsafe behavior were more likely to prefer safer, slower trajectories, while those primed with safe behavior were more willing to accept riskier, faster trajectories.
  • Conclusion: Human preferences are not static; they are influenced by the labeler's beliefs about the agent's current capabilities.

Significance and Claims
The paper claims that reducing the mismatch between human beliefs and agent capabilities is critical for improving RLHF performance. It argues that the standard assumption of labeler rationality based on absolute optimality is insufficient when agents are imperfect. The authors suggest that practitioners should actively manage labeler beliefs to align them with the agent's actual capabilities.

The paper proposes two specific best practices:

1. Informing Labelers: Directly communicating known agent limitations (e.g., data constraints, physical limits) to labelers before data collection.
2. Intermittent Priming: Using online preference collection loops where labelers are exposed to the agent's current policy (via demonstrations) to update their beliefs continuously.

The authors emphasize that this calibration must be "truthful" (representative of actual on-policy behavior) to avoid manipulation. They conclude that while their theoretical bounds are derived in tabular settings and their human study is a proof-of-concept, the results point toward a necessary shift in RLHF best practices: treating human beliefs as a variable that must be aligned with agent reality to ensure safe and effective alignment.