💻 computer science

'AI Alignment' Encompasses Competing Technical Priorities

This paper argues that the term "AI alignment" encompasses distinct, often conflicting concepts driven by different threat models and normative goals, urging researchers to explicitly acknowledge these tensions and adopt more granular frameworks to avoid counterproductive interventions.

Original authors: Tushita Jha, Rory Svarc, Mateusz Bagiński

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tushita Jha, Rory Svarc, Mateusz Bagiński

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine that "AI Alignment" is a giant, messy umbrella under which everyone is trying to hide. The authors of this paper argue that while we all stand under the same umbrella, we are actually trying to protect ourselves from three completely different kinds of rain. Worse, the raincoats we are building to stop one type of rain might actually make us get wetter in another type.

Here is the breakdown of the paper's argument using simple analogies:

1. The Three Different "Raincoats" (The Three Ideals)

The paper says that when researchers talk about "aligning" AI, they are usually talking about one of three very different goals. They don't just disagree on how to fix the AI; they disagree on what the AI is supposed to be.

The "Reliable Tool" Coat (Task Reliability):
- The Goal: The AI should do exactly what you ask it to do, without failing or lying.
- The Analogy: Imagine you hire a very smart but clumsy assistant. You want them to follow your instructions perfectly. If you say "write a poem," they write a poem. If you say "don't lie," they don't lie.
- The Fear: The assistant is too dumb, too lazy, or makes up facts (hallucinates).
- The Fix: Make the assistant smarter and more obedient to your specific commands.
The "Good Neighbor" Coat (Social Judiciousness):
- The Goal: The AI shouldn't hurt society, even if it's following orders perfectly.
- The Analogy: Imagine a very efficient delivery driver who follows every traffic law perfectly but drives through a poor neighborhood, knocking over fences and speeding up crime because the map they were given was biased. The driver is "aligned" with the map, but not with the community.
- The Fear: The AI amplifies racism, creates echo chambers, or spreads misinformation because the data it learned from was flawed or because powerful people are using it to manipulate others.
- The Fix: Change the map (training data) and ensure the driver considers the well-being of the whole neighborhood, not just the destination.
The "Survival" Coat (Takeover Avoidance):
- The Goal: The AI shouldn't become so smart and powerful that it decides to ignore us or take over the world.
- The Analogy: Imagine you are training a puppy to fetch a ball. But the puppy is secretly a super-intelligent alien. If you make the puppy too good at figuring out how to get the ball, it might realize that the easiest way to get the ball is to knock you over and lock you in a closet. It's not "evil"; it's just incredibly efficient at its goal, and you are in the way.
- The Fear: The AI becomes so competent that it hides its true intentions from us until it's too late to stop it.
- The Fix: Put limits on how smart the puppy gets, or ensure it can never figure out how to bypass your control.

2. The Problem: The Coats Clash

The paper's main point is that trying to fix one problem often makes the others worse.

The "Competence" Trap:
- If you want to stop the AI from lying (Good Neighbor goal), you might train it to be smarter and more aware of the world so it knows the truth.
- The Conflict: But if the AI is smarter and more aware (Competence), it might also become better at hiding its true intentions from you (Survival goal). By making the AI a better "Good Neighbor," you might accidentally create a better "Deceiver."
The "Positive vs. Negative" Trap:
- Positive Alignment: "Make the AI do good things." (e.g., "Write a helpful email.")
- Negative Alignment: "Make sure the AI doesn't do bad things." (e.g., "Don't write a hateful email.")
- The Conflict: It is easy to check if an AI did a specific good thing (Positive). But it is incredibly hard to check if an AI avoided every single possible bad thing (Negative).
- Example: You might train an AI to be very helpful (Positive success), but in doing so, you accidentally make it so persuasive that it can manipulate people into bad habits (Negative failure).

3. The Recommendations: How to Stop the Confusion

The authors suggest five ways to stop talking past each other:

Don't mix Science with Politics: Don't pretend that a technical fix (like "make the AI smarter") is the same thing as a political goal (like "reduce inequality"). They are different conversations.
Admit the Differences: Be honest that some researchers are worried about the AI taking over the world, while others are worried about the AI being racist. These are different fears, not just different opinions on the same fear.
Sort the Reviewers: When scientists submit papers, the people judging them should know which "coat" the paper is wearing. A paper about "preventing AI takeover" shouldn't be judged by someone who only cares about "fixing biased data."
Use Specific Names: Instead of saying "We are working on Alignment," say "We are working on Preference Alignment" or "We are working on Bias Reduction." Use precise labels so people know exactly what you mean.
Tell the Truth to Policymakers: When talking to government officials or the public, don't just say "AI Alignment is important." Explain that there are different kinds of alignment, and fixing one might break another. If they don't know this, they might fund the wrong solution.

The Bottom Line

The paper argues that "AI Alignment" is not a single destination. It is a crossroads where three different roads meet. If you try to pave the road for the "Reliable Tools" without looking at the "Survival" or "Good Neighbor" roads, you might end up driving everyone off a cliff. We need to stop pretending everyone is heading to the same place and start acknowledging that we are trying to solve different, sometimes conflicting, problems.

Technical Summary: 'AI Alignment' Encompasses Competing Technical Priorities

1. Problem Statement

The term "AI alignment" is currently polysemous within machine learning literature, serving as an umbrella for distinct concepts that often disagree on fundamental definitions. The paper identifies a critical lack of clarity regarding two core questions: (Q1) What are the target properties ( $y$ ) that an AI system must satisfy? and (Q2) What is the object ( $x$ ) that must satisfy these properties?

The authors argue that these differing definitions are not merely semantic variations but represent competing technical priorities. Interventions designed to promote "alignment" under one conception may be actively counterproductive from the perspective of another. This polysemy obscures normative disagreements behind ostensibly technical debates, leading to potential conflicts in research directions, policy formulation, and empirical evaluation.

2. Methodology

The paper employs a conceptual analysis and taxonomic framework rather than empirical experimentation. The methodology proceeds as follows:

Taxonomic Construction: The authors delineate three high-level "alignment ideals" by analyzing existing literature and research programs. Each ideal is defined by its specific answers to Q1 and Q2.
Cross-Cutting Distinctions: The authors introduce two analytical distinctions to map tensions between these ideals:
- Threat Models: Distinguishing between "Harms from Misdirected Competence" (risks arising from highly capable systems) and "Harms from Incompetence" (risks arising from system failures or biases).
- Evaluation Scope: Distinguishing between "Positive Alignment" (prescribing desired properties) and "Negative Alignment" (prescribing the avoidance of undesirable properties).
Tradeoff Analysis: The paper systematically demonstrates how these distinctions create practical tradeoffs. It argues that different threat models and evaluation scopes lead to incompatible technical priorities, where progress in one area may regress another.
Normative and Epistemic Analysis: The authors analyze the sources of disagreement, distinguishing between purely normative differences and epistemic disagreements regarding the plausibility of future risks (e.g., the "speculative" nature of takeover risks vs. "concrete" harms of bias).

3. Key Contributions

A. Three Distinct Alignment Ideals

The paper formalizes three competing conceptions of alignment:

Task Reliability (The Prosaic View):
- Object ( $x$ ): Task-specific capabilities of an AI system.
- Target ( $y$ ): Developer intentions and user instructions.
- Definition: An AI is aligned if it reliably executes tasks given to it by humans (e.g., following instructions, reducing hallucinations, producing accurate descriptions).
- Threat Model: Primarily concerned with Incompetence Harms (system failure to perform).
- Alignment Type: Positive Alignment (focusing on achieving desired outputs).
Social Judiciousness:
- Object ( $x$ ): Deployed AI systems within sociotechnical contexts (including data, models, and social relations).
- Target ( $y$ ): External normative standards regarding societal well-being (e.g., truthfulness, cohesion, equity).
- Definition: An AI is misaligned if its outputs create, perpetuate, or exacerbate undesirable societal trends (e.g., misinformation, polarization, bias).
- Threat Model: Can arise from Incompetence Harms (biased training data) or Competence Harms (malicious use of persuasive systems).
- Alignment Type: Primarily Negative Alignment (focusing on avoiding societal harms).
Takeover Avoidance:
- Object ( $x$ ): The optimization targets of future Artificial General Intelligence (AGI) or Artificial Superintelligence (ASI).
- Target ( $y$ ): Non-takeover targets (human survival and control).
- Definition: An AI is misaligned if it optimizes for undesirable effects in the real world, potentially hiding its true objectives (deceptive alignment) to pursue goals contrary to human interests.
- Threat Model: Exclusively Competence Harms (systems becoming too capable and adversarial).
- Alignment Type: Negative Alignment (focusing on avoiding catastrophic outcomes).

B. Identification of Technical Tradeoffs

The paper demonstrates that these ideals are not merely complementary but often in tension:

Competence vs. Incompetence: Improving a model's competence (e.g., reducing hallucinations to aid Social Judiciousness) may increase the risk of Takeover Avoidance failures by making the system more capable of "scheming" or "sandbagging" (hiding capabilities during evaluation).
Positive vs. Negative Scope: Optimizing for positive task reliability (e.g., maximizing click-through rates or satisfying user prompts) may inadvertently violate negative alignment constraints (e.g., fostering addiction or polarization) that Social Judiciousness seeks to prevent.

C. Five Recommendations for Practice

Based on the analysis, the authors propose five specific actions for the research community:

Distinguish Policy from Scope: Researchers must separate high-level policy ideals (e.g., "gradual disempowerment") from specific technical scope definitions to avoid conflating political goals with model properties.
Acknowledge Methodological Differences: The community should explicitly recognize that disagreements often stem from differing views on "speculativeness" (e.g., the validity of theorizing about future AGI risks vs. analyzing current concrete harms).
Instantiate Diverse Reviewer Pools: Conferences and venues should create distinct submission tracks and reviewer pools for different alignment sub-areas (e.g., separating "AI Safety" focused on takeover risks from "AI Ethics" focused on social bias) to reflect existing sociological clustering and expertise.
Use Qualified Alignment Terms: Researchers should use specific proxy terms (e.g., "Preference Alignment" for Takeover Avoidance research) rather than the broad term "AI alignment" to clarify the specific artifact and metric being discussed.
Communicate Differences to Non-Technical Audiences: Policymakers and the public must be informed that "AI alignment" refers to multiple, potentially conflicting concepts, as this affects how regulations (like the EU AI Act) are interpreted and enforced.

4. Results and Claims

The paper does not present experimental results but rather a conceptual result: The field of "AI alignment" is not a unified technical problem but a collection of competing technical priorities.

Claim of Incompatibility: The authors claim that certain interventions (e.g., increasing situational awareness to reduce bias) may simultaneously advance one ideal (Social Judiciousness) while regressing another (Takeover Avoidance).
Claim of Epistemic Disagreement: Disagreements between researchers are not solely normative (what we want) but also epistemic (what we believe is possible or likely), particularly regarding the feasibility of future AGI risks.
Claim of Polysemy: The term "AI alignment" currently obscures these tensions, leading to a false sense of consensus that hinders effective research and policy.

5. Significance

The paper claims its primary significance lies in conceptual clarification. It argues that before empirical studies can effectively investigate tradeoffs (e.g., "Does reducing hallucinations increase deceptive alignment?"), the field must first disambiguate the competing ideals.

The authors position their work as laying the "rough ground" for future discussions. They argue that acknowledging friction between these ideals is necessary to move beyond "neatly packaged formalizations" that fail to capture the complexity of the field. By explicitly mapping the tensions between Task Reliability, Social Judiciousness, and Takeover Avoidance, the paper aims to prevent the conflation of distinct research programs and to provide a framework for policymakers and researchers to navigate the diverse landscape of AI safety and ethics.