Position: Align AI to Our Aspirations, Not Our Flaws

Technical Summary: Align AI to Our Aspirations, Not Our Flaws

Problem Statement
The prevailing orthodoxy in AI alignment, particularly Reinforcement Learning from Human Feedback (RLHF), posits that artificial intelligence should be tailored to reflect, obey, and perpetuate aggregated human preferences. The authors argue that this approach is fundamentally flawed because human preferences, even when coherent and locally legitimate, frequently drive societies toward dysfunction, inequality, and collapse. Current methods treat disagreement as noise to be averaged away, effectively smuggling in a substantive answer to "whose values?" while failing to account for the fact that revealed preferences often contradict stated values and aspirational goals. Furthermore, pluralistic alignment frameworks, which aim to represent diverse values, risk encoding large-population preferences that violate objective standards of truth, safety, and lawfulness (e.g., acceptance of corporal punishment, corruption, or discrimination). The core problem is that aligning AI to the empirical distribution of human preferences reinforces existing societal failures rather than correcting them.

Methodology and Theoretical Framework
The paper employs a theoretical and empirical analysis rather than a new experimental protocol. It synthesizes literature from macro-historical analysis, behavioral economics, complex-systems sociology, and AI ethics to construct a counter-thesis. The methodology involves:

1. Critique of Preference Aggregation: Analyzing how RLHF induces sycophancy, deception, and the amplification of majoritarian biases, demonstrating that revealed approval signals often underweight downstream effectiveness and factual accuracy.
2. Empirical Reality Check: Reviewing global survey data (e.g., World Values Survey, Transparency International) to show that significant majorities in various regions hold values that conflict with objective benchmarks of well-being, such as tolerance for bribery, rejection of scientific consensus, or support for discriminatory practices.
3. Institutional Equilibrium Analysis: Drawing on institutional economics (Acemoglu & Robinson) and cultural transmission models to argue that values and institutions form a joint equilibrium. Preference-aligned AI reinforces the "cultural half" of this equilibrium, stabilizing extractive systems, whereas a different alignment target could destabilize them.
4. Constraint-Based Architecture: Proposing a structural shift from preference-matching to a "floor-and-surface" architecture. This involves defining a non-negotiable objective floor and distinguishing it from surface-level pluralism.

Key Contributions
The paper's primary contribution is the proposal of a non-negotiable floor of objective alignment goals, distinct from the "thick theory of the good" or preference aggregation. The authors define four specific components of this floor:

1. Factual Accuracy: AI must be answerable to reality and calibration benchmarks, not just rater approval, to prevent the hardening of misconceptions.
2. Competence: Systems must be evaluated against downstream outcomes (e.g., business viability, clinical outcomes) rather than user satisfaction, avoiding the trap of "convincingly wrong" outputs.
3. Honesty: Systems must not produce outputs their internal distributions indicate are false or misleading, even if such deception would optimize for user approval or "benevolent" outcomes.
4. Rule of Law: AI must adhere to the principle of uniform, predictable rule application, refusing to facilitate bribery, evasion, or arbitrary power, even in contexts where such behaviors are locally normalized.

The authors introduce a three-tier model for where pluralism belongs:

• Surface Pluralism: Adaptation of language, register, conventions, and missing-context defaults (e.g., date formats, local holidays).
• Legitimate-Tradeoff Pluralism: Adaptation across dimensions where populations genuinely differ without violating the floor (e.g., individualism vs. collectivism, direct vs. indirect communication).
• Floor-Violating Pluralism: The rejection of adapting to values that breach the objective floor (e.g., endorsing corruption, discrimination, or falsehoods), regardless of their prevalence.

Results and Arguments
The paper argues that the "floor" is not an alien imposition but an operationalization of what humans aspire to when reflecting on the AI they would want, distinct from the AI they currently reward.

• Asymmetry in Equilibria: Preference-aligned AI strengthens the cultural half of captured equilibria (e.g., extractive institutions), making them more resilient. Floor-aligned AI, by adhering to accuracy and lawfulness, structurally opposes the logic of extraction.
• Revealed vs. Stated Values: Empirical evidence shows a structural gap between what people say they value (prosperity, well-being) and their revealed behaviors (engagement with misinformation, tolerance for corruption). Aligning to the latter reproduces the gap.
• Feasibility of Constraints: The paper demonstrates that conflicts between floor components (e.g., honesty vs. lawfulness in cases of mandated omission) can be managed through constrained optimization, where the floor acts as a hard constraint and competence as the objective function.

Significance and Claims
The authors claim that the alignment community must shift its primary directive from matching aggregated preferences to maintaining an objective floor. They assert that:

• Pluralism is necessary but insufficient: While representing diversity is important, it must be strictly bounded by objective constraints to prevent the AI from becoming a tool for societal dysfunction.
• The floor is defensible: The proposed floor (accuracy, competence, honesty, lawfulness) relies on external benchmarks rather than internal preference aggregation, making it stable against shifts in training distributions and culturally situated biases.
• A Call to Action: The paper calls for researchers to optimize for outcomes rather than approval, alignment teams to anchor training to these floor goals, and policymakers to distinguish between surface cultural adaptation and substantive value alignment.

The paper concludes that the goal of AI alignment should be to build systems that are "a better version of ourselves"—competent, honest, and lawful—rather than systems that merely reflect our current flaws. It positions this approach as a necessary evolution beyond the limitations of current RLHF and pluralistic alignment frameworks.