🤖 AI

Interactive Task Alignment as a POMDP

This paper proposes a POMDP framework to formalize task alignment as inferring latent user intent from ambiguous interactions, revealing that current language models significantly underperform humans in resolving uncertainty and often act prematurely, highlighting a critical gap in their ability to reliably align with user goals.

Original authors: Andy Dai, Zexue He, Zhenyu Zhang, Alex Pentland, Jiaxin Pei

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andy Dai, Zexue He, Zhenyu Zhang, Alex Pentland, Jiaxin Pei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a robot butler. In the science labs where these robots are trained, scientists usually give them very specific, perfect instructions: "Go to the kitchen, find the red mug, and fill it with hot water." The robot is then graded on how well it can find that mug and pour the water. This is like a video game where the goal is written clearly on the screen. But in the real world, humans are messy. We rarely know exactly what we want at the start. We might say, "I'm thirsty, but I don't know what I want to drink," or "I need something to keep my kid busy, but I haven't thought about what kind of toy yet." We are explorers, not just commanders. The big question for scientists is: Can a robot figure out what we actually mean before it starts doing things? This paper dives into that tricky gap between what we say and what we really need, treating the robot's job not just as a task-doer, but as a detective trying to solve a mystery.

The researchers behind this study decided to stop testing robots on perfect instructions and start testing them on messy, real-life confusion. They created a new way to play the game, turning it into a "Partially Observable Markov Decision Process" (which is just a fancy math way of saying: "The robot can only see the clues we give it, not the whole picture, and it has to guess the hidden goal while talking to us"). They set up a simulation where a computer program acts as a human user who starts with a vague idea and slowly reveals more details as the robot asks questions. They tested this on three different playgrounds: buying things online (shopping), writing computer code, and doing professional work tasks.

What they found is a bit of a wake-up call. Even though today's smartest AI models are amazing at following clear instructions, they are terrible at figuring out what we actually want when we are vague. When the user's request was fuzzy, the AI models only guessed the correct task about 22% to 32% of the time. That means they got it wrong nearly three out of four times! The paper suggests that these models are too eager to please. Instead of asking, "Wait, what exactly do you need?", they often jump straight into action after just one or two sentences, locking onto the wrong idea and confidently doing the wrong thing.

To see how bad this really was, the researchers brought in real humans to play the robot. When humans tried to figure out the same vague requests, they got it right 48% of the time—significantly better than any computer model. The study suggests that the difference isn't that humans are smarter at solving puzzles; it's that humans are better at talking. When the researchers fed the AI models the exact conversation transcripts that humans had, the models suddenly got much better at guessing the right task. This proves the problem isn't that the AI can't understand the answer; it's that the AI doesn't know how to ask the right questions to get there.

The team also tried to fix the AI by giving it extra training (using methods called Supervised Fine-Tuning and Reinforcement Learning). This helped a little; the AI got better at guessing correctly and stopped being so confidently wrong. However, even with this extra training, the AI still couldn't catch up to the humans. The paper concludes that while AI is getting great at doing tasks, it is still missing a crucial "social skill": the ability to pause, realize it's confused, and ask for help before it messes things up. Until robots learn to be better detectives and less eager beavers, they will struggle to be truly reliable helpers in our messy, real-world lives.

Technical Summary: Interactive Task Alignment as a POMDP

Problem Definition

Current benchmarks for Large Language Models (LLMs) predominantly evaluate execution on fully specified tasks, where goals, constraints, and success criteria are explicitly provided. However, real-world user interactions are characterized by ambiguity, incompleteness, and evolving or inconsistent goals. In these scenarios, the primary bottleneck shifts from executing a known task to determining the user's intended task.

This paper defines task alignment as the process by which an agent grounds and maintains a shared understanding of the intended task through mutual modeling with the user. The authors argue that even highly capable models fail in real-world deployments if they cannot detect missing information, avoid premature assumptions, and effectively interact to resolve ambiguity before execution begins.

Methodology: A POMDP Framework

The authors propose a domain-general framework to evaluate task alignment, formalized as a Partially Observable Markov Decision Process (POMDP) from the assistant's perspective.

1. POMDP Formulation ( $M_T = \langle S, A, T, R, \Omega, O \rangle$ )

State ( $S$ ): The environment state at turn $t$ $t$ is $s_t = (x^*, i_t, h_t)$ $s_{t} = (x^{*}, i_{t}, h_{t})$ , where:
- $x^*$ is the fixed ground-truth task specification (latent/hidden).
- $i_t$ is the user's current intent state (latent/hidden), capturing revealed or implicit aspects of the goal.
- $h_t$ is the conversation history (observable).
Observation ( $\Omega$ ): The assistant observes only the conversation history $h_t$ . The ground-truth specification $x^*$ and intent state $i_t$ must be inferred.
Action ( $A$ ): The assistant takes an action $a_t = (m_t^a, e_t, \rho_t)$ $a_{t} = (m_{t}^{a}, e_{t}, ρ_{t})$ :
- $m_t^a$ : A natural language response (e.g., a clarifying question).
- $e_t \in \{0, 1\}$ : A stop signal indicating commitment to task execution.
- $\rho_t$ : A ranking over a finite candidate set $C$ (containing $x^*$ and 14 perturbations). This ranking is elicited in a separate thread to measure the model's internal belief without leaking candidates into the dialogue.
Transition ( $T$ ): The ground-truth $x^*$ remains fixed. The user simulator (implemented via an LLM with privileged access to $x^*$ ) generates responses $m_t^u$ and evolves the intent state $i_{t+1}$ based on the conversation history and the assistant's actions.
Objective: The assistant policy $\pi_a$ must use interaction to concentrate its hypothesis on $x^*$ while avoiding premature commitment under uncertainty.

2. Evaluation Metrics

The framework evaluates performance along two dimensions:

Task Recovery: Measures if the model identifies the correct task upon commitment.
- CommitAcc@1: Binary accuracy if the top-ranked candidate is correct.
- CommitRank: Normalized rank of the ground truth.
- CommitReward: A reward function balancing accuracy and interaction cost ( $\alpha \hat{r}_{T^*} - \lambda T^*$ ).
Uncertainty Resolution: Measures how effectively the model reduces uncertainty through interaction.
- Total Information Gain (TotalIG): The reduction in normalized entropy of the model's belief distribution over candidates from the start to the end of the episode.
- Four-Quadrant Taxonomy: Classifies outcomes based on correctness (rank) and confidence (entropy) at the commit turn: Confidently Correct, Confidently Wrong, Lucky Correct, and Honestly Uncertain.

3. Experimental Setup

Domains: Three distinct settings were used:
- Shopping-MMLU: Purchasing specific products.
- GDPVal: Producing real-world deliverables.
- Terminal-Bench: Writing code or executing commands.
Underspecification: Ground-truth specifications were rewritten into three tiers of abstraction (Abstract, Moderate, Concrete) to simulate varying levels of user vagueness.
Models Evaluated: 14 state-of-the-art models, including OpenAI GPT-5.4 variants, Claude Sonnet-4.6, Google Gemini-3, and the Qwen3.5 series (9B to 397B).
Human Baseline: A human study was conducted in the Shopping domain to establish a performance ceiling.

Key Results

1. Models Struggle with Task Alignment

Despite high performance on fully specified tasks, current models fail significantly when tasks are underspecified.

Low Recovery Rates: Under abstract intent, models recover the correct task only 22% – 32% of the time on average across the three domains.
Premature Commitment: Models often commit to execution too early (e.g., GPT-5.4 variants commit within 1–2 turns), failing to gather sufficient information.
Ineffective Interaction: Models frequently fail to clarify ambiguous requests, leading to "Confidently Wrong" outcomes where they act with high certainty on incorrect assumptions.

2. Task Alignment is a Separable Axis

The study demonstrates that task alignment is distinct from raw task execution capability.

Decoupling: In the Shopping domain, all 14 models saturated the underlying task capability (scores 81–88%), yet their alignment performance varied nearly threefold (CommitRank 0.28–0.81).
Orthogonality: Variance decomposition shows that while capability explains some alignment variance, a significant portion (approx. 39–40%) is explained by a capability-orthogonal component. Alignment is a stable, domain-general axis of model behavior.

3. Human Superiority and Communication

Human Performance: Humans achieved a 48% task recovery rate in the Shopping domain, outperforming the best LLM (GLM-5.1 at ~37%).
Communication vs. Inference: An ablation study revealed that the human advantage stems from communication ability, not inference. When LLMs were given the transcripts of human-human interactions to infer the task, their accuracy nearly doubled (from ~25% to ~49.5%), surpassing the human baseline. This indicates that humans are better at eliciting and resolving information through dialogue.

4. Impact of Post-Training

Fine-tuning Qwen3.5-9B with Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) using the CommitReward signal improved performance:

Task Recovery: CommitAcc@1 increased from 15.2% (base) to 23.6% (RL).
Uncertainty Calibration: Training significantly reduced "Confidently Wrong" outcomes (53.0% $\to$ 18.8% under RL) and increased "Honestly Uncertain" outcomes (32.0% $\to$ 57.5%), indicating models became more calibrated.
Limitation: Despite improvements, post-trained models still lag behind human performance in task recovery.

Significance and Claims

The paper claims that task alignment is a prerequisite for reliable agency in real-world deployments. As models become more capable at executing complex tasks, the ability to detect ambiguity, elicit information, and know when to act becomes the primary bottleneck.

The authors contribute:

A POMDP-based framework for measuring task alignment that separates task discovery from execution.
Empirical evidence that current LLMs lack key interaction abilities required for reliable agency, often acting prematurely and failing to resolve uncertainty.
A demonstration that task alignment is a separable, domain-general capability, distinct from raw execution power.
Validation that humans outperform models primarily due to superior communicative behaviors in ambiguous settings, and that post-training can partially bridge this gap but not fully close it.

The work suggests that future evaluations and training paradigms must explicitly address the "alignment stage" to build agents capable of handling the inherent ambiguity of human intent.