Technical Summary: Optimal Use of Preferences in Artificial Intelligence Algorithms

1. Problem Statement

The paper addresses a fundamental architectural choice in the deployment of artificial intelligence (AI) systems: where should downstream preferences (e.g., asymmetric costs, safety priorities, or utility weights) be applied?

The choice lies between two pipelines:

1. Preference Embedding: Training the algorithm directly with a loss function that encodes specific preferences (e.g., class-weighted cross-entropy). The model learns to output a signal optimized for a specific decision rule.
2. Separation (Post-Processing): Training the algorithm with a "preference-free" objective to learn calibrated probabilities (the underlying truth), and applying preferences ex post through a decision rule (e.g., thresholding).

While classical decision theory suggests that preferences belong in the loss function to rank forecasts, this paper argues that when learning is endogenous (i.e., the loss function determines what the algorithm learns), embedding preferences can distort the information acquisition process. The paper investigates whether separating the learning of beliefs from the application of preferences yields superior welfare across a broad class of decision problems.

2. Methodology and Framework

2.1 Information Design and Convex Order

The paper models training as an endogenous choice of an information structure, drawing on Bayesian persuasion and information design (Kamenica and Gentzkow, 2011).

• Representation: A trained predictor is represented by the distribution of posterior beliefs $Q$ it induces about a binary state $Y \in \{0, 1\}$.
• Convex Order: The paper utilizes the convex order ($\succeq_{cx}$) to compare information structures. A distribution $Q$ is more informative than $Q'$ if $Q$ is a mean-preserving spread of $Q'$ (i.e., $Q'$ is a mean-preserving contraction of $Q$).
• Welfare Link: Following Lemma 1, the indirect value function $V(q)$ of any expected-utility decision problem is convex in the belief $q$. Consequently, Lemma 2 establishes that more informative posteriors (in the convex order) weakly dominate less informative ones for any expected-utility decision problem.

2.2 Learning with Frictions

The model treats training as a trade-off between reducing expected Bayes risk and incurring learning frictions ($C(Q)$).

• Assumption 1: The trained algorithm chooses a posterior distribution $Q$ to minimize $E[H(Q)] + C(Q)$, where $H$ is the Bayes risk of the training loss and $C$ represents frictions (e.g., regularisation, capacity limits, optimisation costs).
• Assumption 2 (Monotone Frictions): Learning frictions are monotone in the convex order; more informative signals are weakly more costly to produce.
• Assumption 3 (Diminishing Value of Information): This is the core primitive. It posits that embedding preferences (increasing a preference index $t$) reduces the marginal value of additional informativeness. Formally, the increment in Bayes risk between a preference-embedded loss and a preference-free loss is a convex function of the belief. This implies that preference embedding "flattens" learning incentives, making the reduction of Bayes risk less sensitive to increased dispersion in beliefs.

2.3 Decision-Stage Frictions (Rational Inattention)

To address cases where separation might fail, the paper introduces a decision-stage friction based on Rational Inattention (Sims, 2003).

• Assumption 5: Users face cognitive costs proportional to the mutual information processed. If a user cannot costlessly redesign or garble the signal (a "limited menu"), they may prefer a less informative but cognitively cheaper signal (preference-embedded) over a highly informative but cognitively demanding one (preference-free).

3. Key Results

3.1 The Contraction Result (Theorem 1)

Under Assumptions 1, 3, and 4 (comparability of optimal posteriors), the paper proves that preference embedding induces a mean-preserving contraction of the learned posterior distribution.

• If $t_1 > t_0$ (where $t$ indexes the degree of preference embedding), the optimal posterior $Q_{t_0}$ (preference-free) convex-order dominates $Q_{t_1}$ (preference-embedded).
• Mechanism: Preference embedding reduces the curvature of the Bayes risk, lowering the marginal benefit of generating dispersed (informative) posteriors. Given fixed learning frictions, the optimizer responds by choosing a less dispersed (less informative) signal.

3.2 The Robust Separation Principle (Theorem 2)

Because indirect utility is convex in beliefs, the contraction result implies a robust welfare dominance:

• Result: For any expected-utility decision problem, training with a preference-free strictly proper scoring rule and applying preferences ex post weakly dominates training with a preference-embedded loss.
• Implication: This provides a decision-problem-agnostic foundation for modular AI pipelines. Separation preserves the "option value" of information, allowing the same predictor to serve diverse, changing, or contested downstream objectives.

3.3 Choice Among Preference-Free Losses (Proposition 4)

The paper clarifies that "preference-free" does not imply a unique loss. Among strictly proper scoring rules, the paper establishes a curvature order:

• A loss with a more concave Bayes risk (e.g., Log Loss vs. Brier Loss, under specific normalisations) induces more informative posteriors and weakly dominates for all decision problems, provided the learning friction is held constant.

3.4 When Embedding Dominates (Theorem 3)

The paper identifies conditions where preference embedding is optimal, specifically when decision-stage frictions are binding.

• Reversal Condition: If the cognitive cost of processing a rich probabilistic signal is high (high $\lambda_{cog}$), and the information loss from embedding ($\Delta I$) is small relative to the attention savings ($\Delta MI$), embedding preferences upstream can improve welfare.
• Interpretation: Embedding acts as a form of information compression that reduces the cognitive burden on the user, rationalizing "algorithmic aversion" or the preference for simple heuristics over complex probabilities.

3.5 Application to RLHF (Theorems 4, 5, and Proposition 6)

The framework is applied to Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs):

• RLHF as Embedding: RLHF is interpreted as an exponential tilting of the generator's output distribution toward a specific reward scalarisation.
• Fixed Objectives (Theorem 4): If the deployment objective is stable and aligned with the reward, RLHF improves welfare by shifting the mean acceptability level (First-Order Stochastic Dominance).
• Objective Uncertainty (Theorem 5): If objectives vary (e.g., different users, shifting safety policies), embedding a single scalarisation imposes a "capability tax." The generator becomes specialized for the training objective and loses flexibility (option value) for alternative objectives.
• Reward Misspecification (Proposition 6): Aggressive optimisation (small $\lambda$) in the presence of misspecified rewards amplifies Goodhart's law, pushing the generator toward spurious features that correlate with the proxy reward but not true quality.

4. Significance and Claims

The paper claims to provide theoretical foundations for modular AI design by reconciling two competing intuitions in machine learning practice:

1. Cost-Sensitive Learning: The practice of weighting losses to handle asymmetric errors.
2. Calibration and Thresholding: The practice of learning probabilities and applying thresholds later.

Key Contributions:

• Decision-Problem Agnosticism: Unlike prior work requiring specific downstream objectives, the welfare results apply uniformly across any expected-utility decision problem.
• Endogenous Learning: The paper moves beyond the static view of loss functions (where loss only affects ranking) to show that loss functions fundamentally alter the information acquired by the model.
• Design Guidance:
  • Preserve Optionality: When objectives are uncertain, contested, or changing, separation (learning calibrated probabilities, then post-processing) is optimal.
  • Embed Preferences: When decision-stage frictions (cognitive constraints, latency, inability to implement complex rules) dominate, embedding preferences can be welfare-improving by automating threshold computation and reducing cognitive load.
• Fairness and Safety: The results offer an economic rationale for maintaining a "preference-free" predictive core for fairness and safety, implementing constraints via transparent post-processing layers that can be audited and updated without retraining the core model.

The paper concludes that the optimal design maximizes the division of labor: build general capabilities that preserve information, and apply judgment at the point of choice, unless downstream implementation frictions make separation impractical.