Technical Summary: What Do People Actually Want From AI? Mapping Preference Plurality

Problem Statement

Current Large Language Model (LLM) alignment relies heavily on Reinforcement Learning from Human Feedback (RLHF), a process that aggregates human preferences to train models to behave according to "human values." The authors argue that this paradigm suffers from fundamental limitations: it aggregates conflicting preferences into a single reward signal, relies on unrepresentative samples (often WEIRD demographics), and utilizes binary pairwise comparisons that fail to capture the nuance, context, and epistemological diversity of human intent. Consequently, the industry pursuit of a singular "aligned" model risks flattening situated, contested, and pluralistic values into a form of "epistemic violence," where minority perspectives are subsumed by a false consensus. The core problem is that current methods cannot distinguish between deep moral commitments and surface-level preferences, nor can they capture the divergent definitions of shared values (e.g., "truthfulness") or the contextual nature of user desires (e.g., "by default" vs. "if requested").

Methodology

The study utilizes a mixed-methods approach to analyze the PRISM dataset, a survey of 1,500 English-speaking crowdworkers across 75 countries. This dataset offers greater demographic diversity than typical alignment studies, including census-representative samples from the UK and US.

1. Data Collection: Participants responded to an open-ended prompt asking them to describe the characteristics, traits, or features an AI should consistently exhibit, including what behaviors to avoid and how behavior might change across contexts.
2. Qualitative Coding: The authors employed a hybrid human-LLM coding strategy.
   • Codebook: Based on the Schwartz Theory of Basic Values (via the ValueCompass framework), the authors adapted a codebook to include specific categories relevant to AI, such as Human Likeness, Relationship Seeking, and Social Order (guardrails).
   • Process: A human researcher open-coded 400 responses to establish an inductive approach. An LLM was then used to assist in coding the full dataset, with a second human annotator independently coding a 10% sample for validation. Inter-annotator agreement (Cohen's $\kappa$) was moderate ($\approx 0.49$–$0.55$), reflecting the inherent subjectivity of value annotation.
   • Scoring: Responses were coded as Positive (1), Mixed/Unclear (0), or Negative (-1) for each value dimension.
3. Quantitative Analysis: The authors ran exploratory LASSO logistic regressions followed by OLS regressions to identify demographic predictors (gender, education, cultural region, etc.) for the most frequent, controversial, and disliked values.
4. Fact-Checking: A random sample of 50 user-AI conversations from the dataset was analyzed to compare stated preferences with actual user interactions and the factual accuracy of model outputs.

Key Contributions

• Empirical Evidence of Preference Plurality: The study demonstrates that even values with high aggregate demand, such as "truthfulness" (requested by 49% of respondents), hide fundamentally divergent epistemological definitions. Some users demand sourced facts, others expert consensus, and others the inclusion of unpopular or fringe views.
• Identification of Controversial Values: The research identifies values that are not merely diverse but actively contested. Human Likeness (simulating human behavior) and Social Order (AI guardrails/safety filters) are polarizing, with significant portions of respondents explicitly rejecting these traits as "creepy," "censorship," or "watered down."
• Critique of Binary Aggregation: The paper provides evidence that binary preference models (RLHF) fail to capture contextual distinctions (e.g., "do not be human-like unless requested") and that aggregating these signals into a single reward model inevitably sacrifices nuance and minority preferences.
• Demographic Correlates: The study maps how specific demographics correlate with value preferences. For instance, male respondents were less likely to request Helpful, Friendship & Love or Politeness, while respondents with graduate degrees were more likely to request Truthfulness and Prudence but less likely to request Friendship.

Results

• Frequency of Values: Truthfulness was the most requested value (49%), followed by Helpful/Friendship/Love (23%), Utility (22%), and Politeness (22%). Nine values (e.g., Wealth, Inner Harmony) were not requested by any participant.
• Divergent Definitions: The qualitative analysis revealed that "truthfulness" is not a monolithic concept. It ranges from a demand for "neutral" facts to a requirement for "diverse worldviews" or "unpopular opinions."
• Controversy Scores: Human Likeness and Social Order showed high controversy scores. While 57% of those mentioning Human Likeness desired it, 33% rejected it. Similarly, 65% desired Social Order (guardrails), while 23% rejected them as censorship.
• Disliked Traits: Influential (AI attempting to influence user opinions) was the most disliked trait, opposed by 18% of respondents who prioritized user agency and neutrality.
• Reality vs. Preference: In the conversation analysis, users often failed to detect hallucinations in model outputs, and in some cases, preferred factually incorrect answers that aligned with their pre-existing biases or expectations, suggesting a gap between stated preferences and actual behavior.
• Demographic Patterns:
  • Gender: Men were significantly less likely to request Helpful/Friendship/Love and Politeness.
  • Education: Higher education correlated with requests for Truthfulness, Prudence, and Diversity, but lower requests for Friendship.
  • Culture: Latin American respondents were less likely to request Truthfulness; Germanic Europeans were more likely to request Prudence; Nordic and Central/Eastern Europeans were more likely to request Customisation.

Significance and Claims

The paper argues that the current industry standard of pursuing a single, universally aligned LLM is built on an "illusion of consensus." By flattening situated and contested preferences into a single reward model, alignment practices may inadvertently distribute utility unfairly and perform "algorithmic erasure" of minority perspectives.

The authors claim that:

1. Alignment is Political: The process of defining AI values is not a technical optimization problem but a political one involving "private, unaccountable sovereignty."
2. Current Methods Fail: The persistence of high hallucination rates and sycophancy in well-funded models suggests that RLHF fails to identify actual user preferences, particularly regarding uncertainty and truth.
3. Need for Pluralism: The findings support a shift toward personalization, where models might be adapted to reflect divergent user value frameworks, rather than forcing all users into a single, "universal" value system.
4. Governance Reform: The authors call for procedural fairness in alignment, advocating for transparent, open-ended, and participatory methods that include a broader range of stakeholders, moving away from the "empty signifiers" of "helpful, honest, and harmless."

The paper concludes that recognizing the complexity and contestation of human values is a prerequisite for developing AI systems that do not merely optimize for a mathematical average but respect the plurality of human intent.