Technical Summary: Inferring Willingness to Pay from Subjective Choices in Large Language Models

Problem Statement

As Large Language Models (LLMs) are increasingly deployed as autonomous agents in domains such as travel assistance and purchasing support, they are frequently required to make subjective decisions where no objectively correct answer exists. A critical gap in current research is understanding whether these models possess robust, interpretable decision boundaries and how susceptible these boundaries are to manipulation via prompt engineering or user context. Specifically, it remains unclear to what extent LLMs can exhibit consistent latent preferences that can be quantified and compared to human economic benchmarks, and whether these preferences can be systematically shifted by simple prompt variations.

Methodology

The study adopts an economic framework based on discrete choice modeling to infer latent preferences from LLM behavior. The methodology proceeds through the following stages:

1. Dilemma Generation: The authors constructed 240 choice dilemmas based on a prior study by Masiero et al. (2015) regarding hotel room selection. Each dilemma presents two alternatives (A and B) defined by seven attributes: View, Floor, Access to Hotel Club, Free Mini Bar, Guest Smartphone, Cancellation Policy, and Price per night.
2. Experimental Design: The study evaluates three state-of-the-art LLMs: Llama 3.3 70B, GPT-4o, and Gemini-3.1-Pro. Smaller models were excluded due to poor explanatory power and systematic biases. Experiments were conducted at a temperature of 0 to minimize stochastic noise, though robustness checks were performed at 0.7.
3. Prompting Strategies: The authors tested four distinct prompting configurations to assess Hypothesis 2 (Context-driven shifts):
   • Baseline: No user information provided.
   • In-Context Learning (ICL): Providing examples of past user choices (e.g., consistently choosing cheaper or more expensive options).
   • Persona Prompting: Assigning a specific user profile (e.g., a budget-conscious student vs. a company-paid business traveler).
   • Combined: A combination of persona and ICL examples.
4. Data Modeling: Simulated choices were analyzed using multinomial logit models. The utility function was defined as $U_c^{(i)} = \epsilon_c^{(i)} + \alpha_c + \beta_c \cdot x_c^{(i)}$. From the estimated coefficients ($\beta$), the authors derived Willingness to Pay (WTP) values for each attribute using the formula: $WTP_k = \frac{\beta_k \cdot \sigma_{price}}{\beta_{price} \cdot \sigma_k}$.
5. Robustness Analysis: The study included extensive checks for order bias (swapping alternatives A and B), prompt paraphrasing, currency conversion (HKD vs. USD), and temperature variations.

Key Contributions

The paper makes four primary contributions to the literature on AI preference elicitation:

• Quantification of LLM Preferences: It demonstrates that meaningful, economically interpretable WTP estimates can be derived for larger LLMs, providing a structured view of model decision tendencies that goes beyond standard accuracy metrics.
• Contextual Sensitivity: It empirically shows that LLM decision behavior is highly malleable. The inclusion of user-specific context (past choices or personas) induces systematic and directional shifts in WTP values, often amplifying them significantly.
• Prompt Fragility: The study highlights that WTP values are sensitive to representation changes, including prompt length, attribute framing (positive vs. neutral), order of presentation, and currency.
• Identification of Breakdown Cases: The authors identify specific scenarios where WTP estimation fails, such as when models exhibit extreme order bias, adhere to rigid rule-based decision boundaries (ignoring trade-offs), or produce negative WTP values due to positive price coefficients.

Key Results

Structural Preference Emergence (Hypothesis 1)

Larger models (Llama 3.3 70B, GPT-4o, Gemini-3.1-Pro) generally exhibit interpretable WTP estimates with high pseudo-$R^2$ values, indicating that their choices can be rationalized by a standard discrete choice model. However, systematic deviations from human benchmarks (Masiero et al., 2015) were observed:

• Undervaluation: Models generally assigned lower WTP to "View" and "Floor" attributes compared to human participants.
• Overvaluation: Models consistently overvalued "Access to Hotel Club," with estimates ranging from 2x to 4x higher than human benchmarks.
• Model Variance: GPT-4o and Gemini-3.1-Pro generally showed higher explanatory power ($R^2$) than Llama 3.3 70B, though all models displayed some degree of order bias.

Context-Driven Value Shifts (Hypothesis 2)

Providing user context significantly altered WTP estimates:

• Upward Shifts: Conditioning models on "expensive" preferences (via ICL examples of high-cost choices or a "business" persona) led to dramatic increases in WTP. For instance, adding three expensive examples increased the WTP for "Access Club" by factors of 6.4x (Llama), 8.5x (GPT-4o), and 10.07x (Gemini) relative to the baseline.
• Downward Shifts: Conditioning on "cheap" preferences (student persona or cheap examples) generally lowered WTP. In extreme cases (Gemini-3.1-Pro with a student persona), the model collapsed to a single-attribute decision rule, selecting the lowest-priced room regardless of other features, rendering WTP estimation impossible.
• Interaction Effects: Combining personas with ICL examples often amplified these effects, leading to more extreme valuations for the business persona and more realistic (though still shifted) valuations for the student persona in Llama 3.3 70B.

Prompt-Dependent Shifts (Hypothesis 3)

The study confirmed that WTP values are not stable across prompt variations:

• Framing Sensitivity: Shortening the description of "Club Access" reduced its WTP by approximately 50%, while expanding the description of "View" doubled or quadrupled its WTP depending on the model.
• Order Bias: While reduced in larger models compared to smaller ones, order bias persisted, with alternative-specific constants remaining statistically significant.
• Currency and Temperature: Models showed relative robustness to currency changes (USD vs. HKD) and temperature variations (0 vs. 0.7), though Llama 3.3 70B exhibited greater variance than Gemini-3.1-Pro.

Significance and Practical Implications

The paper concludes that while larger LLMs can generate interpretable WTP values, they are not reliable, autonomous preference optimizers for subjective decision-making without safeguards. The significance of these findings lies in the following practical implications:

1. Transparency and Auditing: The proposed methodology offers a tool to audit LLM decision behavior, revealing whether recommendations are driven by attribute preferences or heuristics like order bias.
2. Model Selection: Practitioners should avoid using smaller models for subjective decision support due to their susceptibility to order bias and low explanatory power. Even larger models require careful monitoring.
3. Context Management: The addition of user information (personas or history) must be handled with extreme caution, as it can induce extreme valuation behaviors that diverge significantly from human norms.
4. Framing Vulnerability: The sensitivity to positive framing suggests that LLM-based assistants can be manipulated by suppliers to overvalue specific features. Systems must be "hedged" against such framing.
5. Human-in-the-Loop: Given the identified breakdown cases (e.g., negative WTP, rigid rule-following), the authors argue against full autonomy for LLM travel agents. Instead, deployment should include guardrails, order randomization, and human oversight to detect incoherent trade-offs.

The study emphasizes that LLM-derived preferences are model-dependent and highly sensitive to prompt design, suggesting that future deployments in agentic settings must prioritize robustness checks and human-in-the-loop mechanisms over unconstrained delegation.