Technical Summary: Toward a Theory of Value in AI Alignment

Problem Statement

The field of AI safety has increasingly framed harmful AI behaviors—ranging from toxic speech to unauthorized agent actions—as "the alignment problem," positing that these issues stem from models being "misaligned" with human values. While a vast technical literature exists on "value alignment," often utilizing methods like Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and Inverse Reinforcement Learning (IRL), the field suffers from a fundamental epistemic gap: it rarely defines what "human values" are, nor does it specify whose values are being aligned.

The paper argues that current alignment research implicitly relies on a "preferentist" approach grounded in economic theory, specifically rational choice theory and expected utility maximization. This framework reduces complex, culturally situated human values to binary preferences or static utility functions. The authors contend that this reductionism, combined with a trend toward using synthetic data and "LLM-as-a-judge" evaluations to replace human annotators, risks closing off alternative methods for contesting and enacting values in foundation models. Furthermore, the lack of a concrete theory of value allows the field to treat "alignment" as a solvable engineering problem while ignoring the sociotechnical reality that values are dynamic, pluralistic, and context-dependent.

Methodology

To characterize the dominant paradigm of value in AI research, the authors conducted a qualitative and quantitative analysis of 94 AI value alignment research papers.

Data Collection:

• Seed Papers: Four canonical, highly cited papers (>1,000 citations) were selected as seeds: Deep Reinforcement Learning from Human Preferences (Christiano et al., 2017b), A General Language Assistant as a Laboratory for Alignment (Askell et al., 2021), Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (Bai et al., 2022), and Training language models to follow instructions with human feedback (Ouyang et al., 2022).
• Snowball Sampling: Using the Semantic Scholar API, the authors retrieved 576 papers published between 2016 and 2024 that cited the seed papers.
• Selection Criteria: Papers were sorted by Google Scholar citation count. Those using "value alignment" only cursory were discarded. The final dataset consisted of 94 papers deemed central to the subfield.

Annotation Framework:
The authors developed a 13-question rubric derived from anthropology, philosophy, and social science literature to assess the implicit theory of value in each paper. The questions were ternary (two specific options or "not applicable") and covered:

1. Definition of values.
2. Monism vs. Pluralism (single intrinsic dimension vs. multiple non-flattenable dimensions).
3. Measurability (quantifiable vs. immeasurable).
4. Source of values (revealed preferences vs. prescribed principles).
5. Abstraction level (abstract/theoretical vs. concrete/specific).
6. Unit of analysis (individual interactions vs. collective agents).
7. Temporal nature (dynamic vs. static).
8. Descriptive depth (thin/instrumental vs. thick/situated).
9. Emergence of values in AI.
10. Normative stance on AI following human values.
11. Identification of specific human groups.
12. Sample size of human raters.
13. Use of utility maximization.

Analysis:
Two of the five co-authors annotated each paper, achieving an average inter-rater agreement of 85%. The analysis combined quantitative scoring with qualitative extraction of illustrative quotes to characterize the field's beliefs about human values.

Key Contributions

1. Analytical Framework: The paper introduces a rubric for analyzing the theory of value in AI alignment, bridging technical ML literature with social science and philosophical concepts (e.g., monism/pluralism, thin/thick descriptions).
2. Empirical Characterization: Through the analysis of 94 papers, the authors provide a systematic overview of how the field operationalizes "value," revealing a heavy reliance on undefined concepts and economic proxies.
3. Critique of "Preferentism": The paper explicitly identifies and critiques the "preferentist" approach, which assumes preferences are adequate representations of values, rationality consists of maximizing these preferences, and aligning to preference data ensures safety.
4. Identification of "Alignment Without Humans": The authors document a trend where human annotators are increasingly replaced by synthetic data and LLM judges, potentially erasing human input from the alignment process.

Results

The quantitative and qualitative findings reveal significant gaps between the field's goals and its theoretical foundations:

• Undefined Values: 79% of the papers neither defined nor described what they meant by "values," despite claiming to align models with them.
• Conflation with Preferences: 82% of papers treated "preferences" as a stand-in for "values." Values were frequently used interchangeably with "intentions" or "ideologies."
• Measurability Assumption: 90% of papers treated values as measurable, operationalizing them through preference datasets where models generate responses evaluated by human raters, other LLMs ("autoraters"), or a mix.
• Monism and Utility Maximization: 87% of papers implicitly or explicitly used utility maximization as part of their technical framework. The field largely subscribes to a monistic view (single dimension of value) and static values (53% treated values as static), ignoring cultural relativity and dynamism.
• Lack of Specificity: 91% of papers did not specify which humans' values were being aligned to, reinforcing a notion of universal values. When human raters were used, 77% of those papers did not state the size of the rater pool (often <100 people).
• Thin Descriptions: 66% of papers offered "thin" descriptions of values (instrumental, calculative), while only 3% offered "thick" descriptions (situated, relational, culturally embedded).
• Shift to Synthetic Evaluation: The paper notes a growing trend of using synthetic humans or LLMs to elicit preferences and evaluate alignment, driven by the perception that human feedback is too expensive, time-consuming, or unreliable.

Significance and Claims

The paper does not claim to solve the alignment problem but rather seeks to make the field's philosophical commitments explicit to facilitate critical reassessment.

• Epistemic Intervention: The authors argue that by failing to define values, the field relies on "taken-for-granted" background knowledge rooted in economic theory (rational choice, utility maximization). This framework assumes human nature is individualistic, rational, and static, which contradicts empirical findings in anthropology and psychology regarding the complexity, cultural situatedness, and dynamism of human values.
• Risk of Ossification: The paper warns that without addressing these foundational questions, current technical practices (e.g., RLHF, DPO) may "ossify," normalizing AI systems trained on a narrow, reductionist conception of value. This could lead to systems that claim to be "aligned" while actually reflecting the biases of a small group of researchers or the constraints of economic modeling.
• Call for Interdisciplinarity: The authors invite researchers from anthropology, psychology, and philosophy to engage with AI alignment. They suggest that future work must move beyond "preferentism" toward richer models that account for how humans evaluate and act upon values in boundedly rational ways, within specific social and cultural contexts.
• Policy Implications: By exposing the shaky foundations of current alignment defaults (e.g., the reliance on HHH principles and synthetic data), the paper aims to inform policymakers and the public about the limits of current value alignment claims, advocating for more realistic conceptions of AI risks.

In conclusion, the paper posits that "alignment" is not merely a solvable engineering problem but a philosophical and sociotechnical challenge. It argues that the current technical paradigm, by translating alignment into utility theory, inherently limits the range of human values AI can represent, potentially foreclosing more diverse and culturally responsive approaches to AI safety.