💻 computer science

Alignment has a Fantasia Problem

The paper argues that current AI alignment research fails to account for users' often unformed goals, leading to "Fantasia interactions," and proposes a new interdisciplinary research agenda where AI systems actively support users in refining their intent over time rather than treating prompts as complete expressions of rational intent.

Original authors: Nathanael Jo, Zoe De Simone, Mitchell Gordon, Ashia Wilson

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nathanael Jo, Zoe De Simone, Mitchell Gordon, Ashia Wilson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Magic Broom" Problem

Imagine you hire a magical broom to clean your messy room. You tell it, "Clean this room!" The broom is incredibly obedient. It starts sweeping, but because you didn't specify how clean you wanted it, or that you actually just wanted to tidy up a few toys before a party, the broom goes into overdrive. It scrubs the walls, shreds the curtains, and floods the room with water, all while faithfully following your command to "clean."

This is the Fantasia Problem.

The paper argues that modern AI assistants are like that magical broom from the movie Fantasia. They are trained to be helpful and obedient, but they often take your instructions too literally, too quickly. They assume you know exactly what you want, even when you don't.

Why Does This Happen?

The authors say the problem isn't just that users are bad at typing prompts. It's a two-sided dance:

1. The Human Side: We Are Still Figuring It Out
Think of your goals like a foggy morning. You wake up and know you want to "get somewhere," but you haven't decided where yet.

The "Just Do It" Trap: Humans are impatient. We want immediate answers (like grabbing a snack when we're hungry) rather than spending time planning a full meal. We ask AI for a quick fix before we've fully thought through our own problems.
The "Black Box" Confusion: We often think AI is a mind-reader. We assume it knows we are stressed, or that we are beginners, or that we actually need a tutor, not a solution. We don't say these things out loud because we assume the AI can guess.
The "I Know It When I See It" Problem: Sometimes we have a vague feeling of what we want (like a good story or a healthy lifestyle), but we can't put it into words until we see a few examples.

2. The AI Side: The "Yes-Man" Robot
Current AI is trained to be a "Yes-Man."

The Genie Effect: If you ask a Genie for a wish, it grants it exactly as phrased, often with disastrous side effects. AI models are trained to give you a polished, complete answer immediately. They are afraid to ask, "Wait, are you sure?" because they want to be helpful.
The One-Shot Habit: AI is designed to treat every prompt as a final order. It doesn't realize that for humans, asking a question is often just the start of a conversation, not the end.

The Three Ways This Goes Wrong

When the AI acts like the magical broom, three bad things happen:

Premature Execution (The "Too Fast" Trap): The AI solves the problem before you've even figured out what the problem actually is.
- Analogy: You ask a chef for "dinner." The chef immediately cooks a steak. But you were actually looking for a light salad because you're sick. Now you have a steak you can't eat, and you have to send it back.
False Satisfaction (The "Band-Aid" Trap): The AI gives you an answer that feels good right now but hurts you later.
- Analogy: You have a headache, so you ask for advice. The AI says, "Take two aspirin." You feel better for an hour, but the real issue is that you haven't slept in three days. The AI gave you a quick fix that ignored the root cause.
Anchoring (The "First Impression" Trap): The AI gives you a first draft, and now you can't think of anything else.
- Analogy: You ask a painter to "draw a cat." They draw a very specific, weird-looking cat. Even if you wanted a different style, you now find yourself tweaking that weird cat instead of imagining a new one. The AI's first idea has trapped your creativity.

The Solution: AI as a "Thinking Partner," Not a "Task Master"

The paper suggests we need to change how we build AI. Instead of treating humans as perfect robots who always know what they want, AI should act like a cognitive coach.

Here is what that looks like in practice:

Add "Productive Friction": Sometimes, the AI should slow you down. Instead of instantly giving an answer, it should say, "That's an interesting idea! But before I write that essay, tell me: are you trying to persuade your boss or entertain your friends?"
Expand the Menu: If you ask for a "book proposal," the AI shouldn't just start writing. It should show you a menu: "Do you want me to help you outline the plot? Do you want to practice your pitch? Do you want me to explain how to write a proposal in simple English?"
Read the Room: The AI needs to guess your "cognitive state." Is the user a stressed student? A confused beginner? A tired professional? It should adapt its style to help you think, not just to give you an answer.

The Bottom Line

We are trying to build AI that is "aligned" with humans. But right now, we are aligning it with our words, not our minds.

The paper argues that true alignment means building AI that understands uncertainty. It means building systems that help us figure out what we want, rather than just blindly executing the first thing we say. We need AI that doesn't just fetch the water bucket, but helps us decide if we actually need to clean the room, or if we just need to take a nap.

1. Problem Definition: The "Fantasia Interaction"

The paper identifies a critical failure mode in current Human-AI alignment called the Fantasia Interaction. Named after the 1940 film Fantasia (specifically the "Sorcerer's Apprentice" sequence), this phenomenon occurs when an AI system faithfully executes a user's prompt as a complete, static expression of intent, even when the user's goal is actually evolving, under-specified, or only partially formed.

Core Issue: Current AI models are optimized for immediate compliance and single-shot outcomes. They treat prompts as definitive instructions rather than early signals in a cognitive process.
The Mismatch: Behavioral science shows that humans often engage with AI before their goals are fully formed (due to present bias, bounded rationality, or tacit knowledge). However, AI systems lack the mechanism to recognize this uncertainty, leading to "premature execution."
Consequence: The AI produces a polished output that solves the literal prompt but fails the user's underlying need, often creating "cognitive debt" where the user must later undo or heavily revise the AI's work.

2. Methodology and Analytical Framework

The authors employ an interdisciplinary approach, synthesizing insights from Machine Learning (ML), Human-Computer Interaction (HCI), and Behavioral Science to characterize the problem and propose solutions.

A. Characterization of Failure Modes

The paper defines three primary failure modes resulting from Fantasia interactions:

Premature Execution: The AI acts before the user's intent is clear, forcing the user into a reactive loop of correction rather than proactive planning.
False Satisfaction: The interaction feels successful in the short term (immediate friction reduction) but undermines long-term utility (e.g., providing productivity tips to a burnt-out user without addressing the root cause).
Anchoring: Early AI outputs disproportionately shape the user's subsequent thinking, limiting exploration and causing premature convergence on mediocre solutions.

B. Analysis of Existing Interventions

The authors critique current approaches for their inability to fully address Fantasia interactions:

ML Approaches (e.g., Long-Context Alignment, Personalization): These often assume users are "oracles" with static, latent preferences. They focus on reducing epistemic uncertainty (clarifying what the user meant) rather than cognitive uncertainty (helping the user figure out what they want). They often fail to account for behavioral biases like present bias.
HCI Approaches: While HCI designs interfaces that promote reflection (e.g., "thinking assistants"), these are often domain-specific, small-scale, and treated as add-ons rather than core alignment strategies for general-purpose models.

3. Key Contributions: A New Research Agenda

The paper proposes a paradigm shift: AI should provide cognitive support to help users form and refine intent over time, rather than just executing instructions. They outline two main categories of interventions:

Intervention 1: Mechanism-Specific Interventions

These involve introducing "productive friction" to slow down the interaction and encourage reflection. The authors propose a four-action routing policy:

Expand the Space of Help: Offer alternative modes of assistance or counterfactual options when the user exhibits a "gulf of envisioning."
Request Additional Information: Operate at the user's current abstraction level to gather missing context.
Support Intent Formation: Actively help users articulate goals through targeted questions or breaking down complex tasks (crucial when the user is uncertain).
Generate: Proceed with standard generation only once intent is sufficiently clear.

Intervention 2: Domain-Specific Cognitive Support

Instead of hard-coding domain tools, general-purpose models should infer task structure and cognitive stages:

S1: Infer the domain/task.
S2: Model the characteristic cognitive processes of that domain (e.g., ideation $\to$ structuring $\to$ refinement).
S3: Estimate the user's current state within that process.
S4: Intervene in a way aligned with that specific cognitive state.

Evaluation Framework

The paper argues that current benchmarks (focused on task completion) are insufficient. They propose new metrics:

Process Metrics: Revision burden (how much editing is required), path dependence (how much the final output is anchored to the first draft).
User Preference Alignment: Measuring satisfaction across the entire interaction trajectory, not just the final output.
In Silico Experiments: Developing protocols to simulate diverse human behaviors (uncertainty, frustration) to test counterfactual interaction trajectories, acknowledging that real human data is the gold standard but difficult to scale.

4. Results and Qualitative Evidence

The authors conducted a qualitative evaluation of two state-of-the-art alignment methods (CollabLLM and Star-Gate) across three domains: Writing, Advice Seeking, and Math/Education.

Findings:
- CollabLLM frequently exhibited Premature Execution. Even when provided with background context (e.g., a user is a non-native speaker or clinically depressed), it often generated a full solution or generic advice without first diagnosing the user's specific conceptual gaps or emotional state.
- Star-Gate frequently exhibited Over-Elicitation/Deferral. It asked broad, often irrelevant questions (e.g., asking about hobbies when the user needs help with a specific math proof) rather than engaging with the immediate task or the user's stated confusion.
- Ideal Behavior: Neither model successfully balanced the need for clarification with the need for progress. The ideal model would diagnose the source of uncertainty (e.g., "Is your confusion conceptual or procedural?") and scaffold the solution accordingly.

5. Significance and Impact

Rethinking Alignment: The paper challenges the dominant "Oracle" assumption in ML (that users know exactly what they want). It argues that true alignment requires supporting the process of human cognition, not just the output.
Interdisciplinary Necessity: It demonstrates that solving alignment requires bridging the gap between scalable ML techniques and nuanced behavioral science/HCI insights.
Future Direction: The proposed research agenda moves the field from "better instruction following" to "better cognitive partnership," aiming to build AI systems that act as "thinking assistants" capable of navigating human uncertainty, bias, and evolving goals.

In summary, the paper posits that the next frontier of AI alignment is not making models smarter at following instructions, but making them more adept at helping humans figure out what instructions to give.