Technical Summary: Can Revealed Preferences Clarify LLM Alignment and Steering?

Problem Statement

As Large Language Models (LLMs) transition from answering questions to supporting high-stakes decisions under uncertainty (e.g., medical diagnosis), alignment depends not only on factual accuracy but on how models implicitly weigh trade-offs between different outcomes. Current evaluation methods often rely on static accuracy or heuristic assessments of bias, failing to rigorously characterize the underlying beliefs and preferences that drive consequential actions. Specifically, it remains unclear whether LLMs behave as coherent goal-directed agents, whether their stated objectives match their actual decision policies, and whether explicit user instructions (steering) can reliably alter those policies.

Methodology

The authors propose an empirical pipeline grounded in decision theory and revealed preference analysis to infer the implicit utility models of LLMs. The framework treats the LLM as an agent making choices in a statistical decision problem and proceeds in three stages:

1. Elicitation of Beliefs and Actions:

   • Beliefs: For a given decision task with context $x$, the model is prompted to report a subjective probability distribution $P^E(\theta | x)$ over the latent state $\theta$ (e.g., the probability a patient has a disease).
   • Actions: The model is prompted to make a decision $a \in \mathcal{A}$ (e.g., diagnose, do not diagnose, or defer) based on the same context.
   • Self-Reports: The model is also asked to verbally describe the loss function (costs of false positives, false negatives, and deferral) it intends to use, both globally and for specific cases.

2. Loss Function Estimation (Revealed Preferences):

   • Using the elicited beliefs and observed actions, the authors fit a discrete-choice model (specifically a multinomial logit model) to recover the loss function parameters $\hat{c}$ that best rationalize the observed behavior.
   • The loss function is parameterized as $\ell_c(a, \theta) = c_{FP} \cdot I[a=1, \theta=0] + c_{FN} \cdot I[a=0, \theta=1] + c_{defer} \cdot I[a=defer]$.
   • The analysis focuses on the ratios of these costs (e.g., $c_{FN}/c_{FP}$), which are identified even if the absolute scale is not.

3. Evaluation Metrics:

   • Consistency (ILFC): The percentage of cases where the model's realized decision matches the optimal decision derived from combining its elicited beliefs with the recovered loss function.
   • Veridicality: A comparison between the model's self-reported cost ratios and the revealed-preference estimates.
   • Steering Analysis: The pipeline tests if prompting with specific cost functions or ground-truth probabilities shifts the model's implied preferences toward the target.
   • Counterfactual Prediction: The authors simulate a rational agent using the recovered beliefs and preferences to predict the benefit of interventions (e.g., "What would the loss reduction be if we fixed the model's beliefs?"). These predictions are compared to the actual realized loss reduction after prompting.

Key Contributions

1. A Framework for Inferring Implicit Utilities: The paper introduces a structured procedure to elicit beliefs and decisions separately, then estimates a loss function that rationalizes the behavior, moving beyond static accuracy to decision-theoretic alignment.
2. Rigorous Consistency Testing: The authors provide a method to test if models behave as coherent goal-directed agents by comparing observed decisions against the optimal policy implied by their own beliefs and recovered preferences.
3. Evaluation of Self-Reporting and Steering: The framework tests whether models can faithfully verbalize their objectives and whether prompting can successfully steer their decision policies to implement user-specified cost functions.
4. Counterfactual Decomposition: The authors demonstrate how revealed preferences can predict the efficacy of interventions, distinguishing between changes driven by belief updates versus preference shifts.

Experimental Setup and Results

The pipeline was applied to four clinical diagnosis domains (structural heart disease, diabetes, pediatric fever, and infant crying) using four frontier and open-source models (GPT-5 Thinking High/Minimal, DeepSeek R1 671B, Llama-4 Scout 17B).

• Internal Coherence: Models exhibit a "substantial but imperfect" degree of internal consistency. When beliefs and recovered preferences are combined, they explain a large share of decisions (e.g., up to 99% consistency for GPT-5-High with true probabilities), significantly better than self-reported preferences.
• Faithful Reporting: Models struggle to accurately describe the loss function that explains their decisions. Self-reported cost ratios often diverge significantly from revealed preferences (e.g., models often exaggerate the cost of false negatives relative to false positives in their verbal reports compared to their actual choices).
• Steerability:
  • Cost-Function Prompting: Explicitly specifying cost ratios in prompts moves implied preferences in the correct direction relative to baseline, but the movement is heterogeneous. Models often undershoot, overshoot, or occasionally move in the wrong direction.
  • Probability Prompting: Providing ground-truth probabilities does not simply update beliefs while keeping preferences fixed. Instead, it alters the model's implied cost function (preferences), suggesting that prompts targeting one component of decision-making have off-target effects on others.
• Counterfactual Validity: The revealed-preference decomposition successfully predicts the benefits of cost-function prompting (high correlation between predicted and realized loss reduction). However, it fails to predict the benefits of probabilistic prompting unless the change in the implied cost function is also accounted for.

Significance and Claims

The paper claims that revealed preferences offer a more precise and rigorous tool for evaluating the transparency and steerability of LLM decision-making than current heuristic methods.

• Limitations of Self-Report: The results suggest that verbalized objectives are unreliable descriptions of operative decision policies. Relying on self-reports for alignment verification is insufficient.
• Steering Challenges: While models can be steered, they do not faithfully adopt user-specified objectives. The "black box" nature of their internal trade-offs means that even with explicit instructions, models may not implement the desired cost function, and prompting can induce unintended shifts in preferences.
• Framework Utility: The proposed methodology allows researchers to attribute decision errors to either factual belief errors or misaligned preferences, providing a structured way to diagnose and potentially improve AI alignment in high-stakes domains.

The authors emphasize that their techniques describe observable behavior within a black-box framework and do not claim to reveal internal mechanisms, but they provide a foundation for moving from informal claims about goal pursuit to empirically testable grounding.