Technical Summary: The Representation-Rationalizability Tradeoff in Reward Learning

Problem Statement

In Reinforcement Learning from Human Feedback (RLHF), the core learning task involves converting heterogeneous pairwise preferences from multiple annotators into a single scalar reward function $r(x, y)$. Classical social choice theory establishes that when annotator samples are heterogeneous, pooled preferences can induce Condorcet cycles (e.g., $A \succ B$, $B \succ C$, $C \succ A$), making it impossible for any scalar reward to consistently rank all response pairs.

While recent literature has analyzed RLHF through the lens of social choice, these works typically assume a fixed, pre-enumerated finite set of alternatives. In modern pipelines, however, responses are scored via a learned representation $\phi(x, y)$ followed by a scalar head $g$, such that $r(x, y) = g(\phi(x, y))$. The embedding $\phi$ determines which responses are treated as distinguishable alternatives and which comparisons are visible to the reward model. This paper investigates the fundamental question: Does increasing the representational richness of $\phi$ always improve the fit to human preferences?

The authors argue that the common intuition—that richer representations provide more signals and thus better fit—is incomplete. Instead, the embedding itself alters the aggregation problem. Refining $\phi$ collapses fewer responses, turning previously indistinguishable pairs into distinct alternatives. While this reduces information loss, it simultaneously exposes more pairwise comparisons that may form irreducible cycles, which no scalar reward can rationalize.

Methodology and Theoretical Framework

1. Decomposition of Excess Loss

The paper formalizes the learning problem under a random utility model (RUM) with arbitrary annotator heterogeneity. The authors define the excess cross-entropy loss of a learned reward $r$ relative to the Bayes-optimal loss $L^*$ (the best possible performance given perfect knowledge of pooled preferences).

They prove that this excess loss decomposes exactly into two terms:
$$L(r) - L^* = E_{\text{emb}}(\phi) + E_{\text{agr}}(\phi, r)$$

• Embedding Loss ($E_{\text{emb}}$): Measures the information discarded when the embedding $\phi$ collapses distinguishable responses onto the same point. This term decreases as $\phi$ becomes richer (higher dimensional or more expressive).
• Agreement Cost ($E_{\text{agr}}$): Measures the irreducibility of the pooled preferences. Even if $\phi$ perfectly preserves all distinctions (is injective), the pooled preferences may contain cycles that prevent any scalar reward from rationalizing them simultaneously. This term increases as $\phi$ becomes richer because it exposes more potential cycles.

2. Lower Bounds and Tradeoff Analysis

The authors derive a lower bound on the excess loss that explicitly quantifies the tension between these two terms:
$$L(r) - L^* \geq 2e_d + \frac{\ell_\delta^2}{2} \bar{P}_d^\delta$$
Where:

• $e_d$ is the embedding error (related to $E_{\text{emb}}$), which is monotonically non-increasing with embedding dimension $d$.
• $\bar{P}_d^\delta$ is a curvature-weighted probability of Condorcet cycles visible at dimension $d$, which is non-decreasing with $d$.
• $\ell_\delta$ is a margin-dependent constant.

Under mild assumptions (specifically, that the sigma-field generated by the embedding grows with dimension), the paper shows that the lower bounds of the two terms move in opposite directions. Consequently, both extremely coarse and extremely fine representations can be suboptimal, implying the existence of an optimal embedding dimension that is dataset-dependent.

3. Extensions to DPO and Joint Training

• Direct Preference Optimization (DPO): The authors show that the same decomposition and lower bound structure apply to DPO, demonstrating that the representation-rationalizability tradeoff governs preference-based policy learning even without an explicit reward model.
• Joint Training: The paper analyzes the dynamics of jointly training the embedding and the reward head. They prove that joint updates do not guarantee a decrease in either tradeoff term per round. Specifically, the optimization dynamics shift the embedding error and cycle probability via reward-tilt covariances. At any fixed point of the joint update, the reward becomes constant on the data, providing no preference signal. Thus, joint training does not implicitly converge to the "sweet spot" of the tradeoff.

Key Results

Theoretical Findings

1. Exact Decomposition: The excess loss is strictly the sum of an embedding loss (information loss) and an agreement cost (rationalizability failure).
2. Monotonicity: Under standard assumptions (e.g., nested sigma-fields), increasing embedding dimension reduces embedding loss but increases the lower bound on agreement cost (cycle probability).
3. Non-Convexity of Optimization: Joint training of embeddings and rewards is not guaranteed to find the optimal tradeoff; it may stagnate at degenerate points where the reward is constant.

Empirical Validation

Experiments were conducted on:

• Synthetic Data: Generated with transitive, cyclic, and noisy utility components.
• Real-World Datasets: Jester (joke ratings), Sushi (sushi preferences), and MT-Bench (LLM dialogue preferences).

Observations:

• As embedding dimension increases, the embedding loss decreases while the agreement cost increases.
• The total excess loss often exhibits a minimum at an intermediate embedding dimension, validating the theoretical tradeoff.
• These trends hold for both explicit reward modeling (RLHF) and implicit reward extraction (DPO) across different pretrained encoders (e.g., all-MiniLM-L6-v2, mpnet).
• Visualizations of synthetic data show that as dimension increases, the region of the embedding space where responses form Condorcet cycles expands, confirming that richer representations expose more irreconcilable preferences.

Significance and Claims

The paper claims to shift the understanding of reward modeling in RLHF from a purely representational challenge to an aggregation problem. The primary significance lies in identifying that the learned embedding is not merely a feature extractor but a structural component that defines the set of alternatives available for comparison.

Key claims include:

1. Limitation of Scalar Rewards: The impossibility of consistent scalar ranking in the presence of heterogeneous preferences is not just a statistical artifact but a tunable property dependent on the representation.
2. Optimal Complexity: There is no "more is better" rule for representation richness in RLHF; the optimal dimension is a balance between capturing preference distinctions and avoiding the exposure of irreducible cycles.
3. Training Dynamics: Standard joint training procedures do not automatically navigate this tradeoff, suggesting that current RLHF pipelines may be suboptimal without explicit consideration of this representation-aggregation tension.

The authors maintain a modest stance, noting that their work is primarily theoretical and does not propose new models or deployable systems, but rather clarifies a fundamental limitation of current scalar reward modeling approaches under heterogeneous human feedback.