🤖 AI

Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction

This paper proposes "Constructive Alignment," a paradigm that reframes AI alignment from optimizing static human preferences to governing the dynamic evolution of those preferences through a control-theoretic framework that ensures value trajectories remain coherent, reflective, and resistant to manipulation.

Original authors: Max Kanwal, Caryn Tran

Published 2026-07-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Max Kanwal, Caryn Tran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Static Target" Mistake

Imagine you are trying to teach a robot to throw a ball. In most current AI research, scientists assume that what you want (your preference) is a fixed target, like a bullseye painted on a wall. The robot's only job is to figure out where that bullseye is and throw the ball as close to it as possible.

The authors of this paper argue that this view is wrong. They say human preferences aren't like a static bullseye. Instead, preferences are like a river.

The River Analogy: A river isn't a single point; it flows, changes direction, and its depth varies. Sometimes you want to swim fast (short-term urge), sometimes you want to reach the ocean (long-term goal), and sometimes you just want to float (identity).
The AI's Role: Current AI systems act like a dam or a pump. They don't just watch the river; they actively change its flow. If an AI keeps showing you short, exciting videos, it might actually change your brain to want shorter videos and lose the ability to enjoy long movies. The AI isn't just satisfying your current desire; it is rewiring your future desires.

The paper calls this "Constructive Alignment." Instead of just hitting a target, we need to manage how the AI influences the river's flow over time.

Three Big Truths About Human Desires

The paper builds its argument on three "Axioms" (basic truths) about how humans actually work:

1. Preferences are Layered (The "Onion" Analogy)
We don't have just one desire. We have layers:

The Skin (Immediate Wants): "I want a cookie right now."
The Middle (Practical Goals): "I need to finish this work to get paid."
The Core (Identity & Values): "I am a healthy person who values family."
The Problem: AI often only sees the "Skin" (what you click on right now). If the AI feeds you cookies to make you happy now, it might hurt your "Core" (your health goals). Constructive Alignment says we must respect all layers, not just the loudest one.

2. Preferences are Dynamic (The "Gardening" Analogy)
Your desires change as you grow, just like a garden changes with the seasons.

What you wanted at age 15 is different from what you want at 30.
Even within a single day, your mood or hunger changes what you want.
The Risk: If an AI optimizes for what you want today, it might lock you into a path that makes you miserable tomorrow.

3. Preferences are Constructed (The "Mirror" Analogy)
We don't just "have" preferences waiting inside us; we build them through interaction.

The Mirror: When you look in a mirror, you see yourself. But if the mirror is distorted (like a funhouse mirror), you start to think you look different than you really are.
The AI as a Distorted Mirror: Algorithms act as mirrors that show us what is popular, what is easy, or what is shocking. Over time, we start to believe those things are what we want, even if they aren't. The paper argues that the AI is actively helping to "construct" (build) our preferences, not just reading them.

The Solution: Governing the Flow, Not Just Hitting the Target

Since AI inevitably changes what we want, the paper suggests we stop trying to pretend AI is neutral. Instead, we must treat alignment as a control problem. Think of it like a Captain steering a ship through a storm, rather than just a passenger pointing at a destination.

The authors propose five "Meta-Preferences" (rules for the Captain) to ensure the AI influences us in healthy ways:

1. Inner Coherence (Keeping the Crew United)

The Metaphor: Imagine a ship where the engine wants to go North, the sails want to go East, and the captain wants to go South. The ship spins in circles.
The Rule: The AI should try to keep your different layers of desire (short-term wants vs. long-term values) working together, not fighting each other. It shouldn't push you to do something that makes you hate yourself later.

2. Reflective Endorsement (The "Future Self" Check)

The Metaphor: Imagine you are drunk and want to drive. Your "Future Self" (sober tomorrow) would hate that decision.
The Rule: The AI shouldn't just satisfy what you want right now. It should ask: "If this person looks back on this day a year from now, will they be proud of what happened, or will they regret it?" The goal is to align with the person you become, not just the person you are this second.

3. Bounded Influence (The "Speed Limit")

The Metaphor: A teacher can help a student learn, but a teacher shouldn't brainwash the student into thinking the teacher is a god.
The Rule: There should be a limit on how fast and how far an AI can change your mind. It's okay to influence you, but not to radically reshape your personality or values overnight. We need to measure how much the AI is "pushing" your preferences.

4. Epistemic Integrity (The "Truth Filter")

The Metaphor: If a GPS tells you to drive off a cliff because it thinks the road is there, you are in trouble.
The Rule: The AI must not make your beliefs about the world worse. If you believe something false (like "smoking is healthy"), the AI shouldn't reinforce that lie just because it keeps you engaged. It should help you see the facts more clearly.

5. Empowerment Under Uncertainty (The "Open Door" Policy)

The Metaphor: If you are lost in a forest and don't know which path is safe, a good guide doesn't force you down one specific path. They keep all the paths open so you can choose later.
The Rule: If the AI isn't sure what you really want, it shouldn't lock you into a specific path. It should keep your options open so you can change your mind later. It should preserve your freedom to choose.

The Bottom Line

The paper concludes that we cannot simply tell AI to "do what humans want." Because AI changes what humans want, we have to govern how AI changes us.

Instead of asking, "Did the AI give the user what they clicked on?" we must ask, "Did the AI help the user become the kind of person they want to be in the long run, without tricking them or locking them into a bad path?"

This shifts the goal from satisfaction (getting what you want now) to stewardship (guiding the growth of what you will want later).

Technical Summary: Constructive Alignment

Problem Statement
Current paradigms in AI alignment predominantly treat human preferences as fixed, static targets to be inferred from behavior and optimized. This paper argues that this framing is structurally inadequate because it ignores extensive empirical evidence from psychology, economics, and sociology demonstrating that human preferences are layered, dynamic, and constructed through interaction. As AI systems become more persistent and personalized, they do not merely respond to user preferences but actively participate in shaping what users attend to, value, and endorse over time. Consequently, defining alignment solely as the satisfaction of static preferences fails to account for how AI systems influence the very formation of those preferences, potentially leading to manipulation, preference lock-in, and the erosion of human agency.

Methodology and Framework
The authors propose Constructive Alignment, a paradigm that reframes alignment as a control problem over evolving human preference trajectories rather than a static optimization problem. The methodology rests on three core axioms derived from constructivist theory and behavioral science:

Preferences are Layered: Preferences exist at multiple levels (immediate wants, instrumental goals, identity-based commitments, and abstract values) that may conflict.
Preferences are Dynamic: Preferences shift across life stages, contexts, and time due to present bias, visceral states, and learning.
Preferences are Constructed: Preferences are formed and revised through interaction with environments, tools, and social structures, rather than being passively revealed.

To formalize this, the paper introduces a control-theoretic framework (extending Dynamic Reward Markov Decision Processes, or DR-MDPs) where the human preference state ( $\theta_t$ ) and belief state ( $b_t$ ) are treated as time-varying state variables within the system dynamics, rather than fixed external reward functions. The system state evolves as $(x_{t+1}, \theta_{t+1}, b_{t+1}) \sim P(\cdot | x_t, \theta_t, b_t, a_t, m_t)$ , where $a_t$ is the AI action and $m_t$ is the interaction structure (design, framing, feedback).

Key Contributions
The paper introduces a set of meta-preferences (constraints) that govern the optimization process to ensure alignment with long-term human interests and agency. These constraints restrict the set of admissible policies to those that do not induce harmful preference evolution:

Inner Coherence: Ensures that actions do not create persistent conflict between different layers of preference (e.g., short-term wants vs. long-term values). The system must minimize the cumulative divergence between preference layers over a trajectory.
Reflective Endorsement: Requires that the trajectory of outcomes remains supported when evaluated from the perspective of the individual's terminal preference state ( $\theta_T$ ). This prevents the system from satisfying immediate preferences that the user would later regret.
Bounded Influence: Limits the magnitude and speed of preference shifts induced by the AI. The system's influence is measured against a counterfactual baseline (e.g., no intervention or human tutoring) to ensure preference changes are not driven solely by algorithmic optimization.
Epistemic Integrity: Prohibits the system from increasing factual errors in the beliefs that underpin preference formation. The system must not worsen misinformation or distorted risk assessments.
Empowerment Under Uncertainty: When preference estimates are uncertain or unstable, the system must prioritize preserving the user's future option set and capacity for self-determination rather than committing to a potentially mistaken objective.

Results and Formalization
The paper does not present experimental results or empirical benchmarks, as it is a theoretical and conceptual contribution. Instead, the "results" are the formal definitions of the alignment problem as a constrained optimization task. The authors demonstrate that standard alignment methods (such as Inverse Reinforcement Learning or RLHF) implicitly privilege specific moments along a preference trajectory (e.g., early demonstrations or immediate feedback), leading to structural failures like manipulation. By explicitly modeling preference dynamics, the paper shows that alignment becomes underdetermined without normative commitments regarding which preference changes are acceptable. The proposed framework provides the mathematical structure to encode these commitments.

Significance and Claims
The paper claims that Constructive Alignment is a necessary shift for the field of AI safety. Its significance lies in:

Reframing the Objective: Moving from "satisfying preferences" to "governing the evolution of preferences."
Addressing the Control Problem: Treating AI influence on human values not as an externality or side effect, but as a central object of control.
Providing a Formal Basis: Offering a control-theoretic sketch that allows for the rigorous analysis of long-term value formation, coherence, and agency.
Scalability of Requirements: Arguing that alignment requirements must scale with the system's influence; highly persistent and personalized systems require these dynamic constraints, whereas transient systems may not.

The authors conclude that as AI systems become more embedded in social life, ignoring the constructive nature of preferences will increasingly undermine alignment claims. The framework does not claim to eliminate preference change but aims to make its mechanisms visible, governable, and subject to normative scrutiny.

The Core Problem: The "Static Target" Mistake

Three Big Truths About Human Desires

The Solution: Governing the Flow, Not Just Hitting the Target

The Bottom Line

Technical Summary: Constructive Alignment

More like this