Technical Summary: The Intent Gap: A Taxonomy of Real-User Failure Modes in Frontier AI Agents

1. Problem Statement: The Deployment Gap

The paper identifies a critical divergence between the performance of frontier AI agents on public benchmarks and their reliability in real-world deployment. While research institutes (e.g., METR, Apollo Research) document exponential growth in benchmark capabilities, enterprise adoption remains low (14% reach production scale), with "quality" cited as the primary blocker.

The author argues that this "deployment gap" stems from a structural mismatch in task design:

• Benchmarks: Tasks are researcher-imagined, declarative, and possess fixed ground-truth answers (e.g., GAIA, SWE-bench).
• Real Users: Interactions are conversational, assume implicit context, involve mid-task pivots, and frequently contain contradictions or abandonments.

The paper defines a specific failure mode within this gap called the Intent Gap: a scenario where the model produces output that is literally responsive to the prompt but fails to satisfy the user's actual intent. This is distinct from refusal, hallucination, or jailbreaks. It is a failure mode that users often attempt to repair in subsequent turns (e.g., "no, I meant..."), yet these failures are systematically invisible to standard evaluation pipelines that rely on explicit user feedback or static benchmarks.

2. Methodology

The study employs a three-stage pipeline to mine, filter, and analyze real-user conversations to surface intent-gap failures.

Data Sources

The analysis utilizes two large-scale public corpora:

• WildChat-1M: 1 million ChatGPT interaction logs with country-level metadata.
• LMSYS-Chat-1M: 1 million conversations across 25 models from the Chatbot Arena.
• Constraint: Analysis is restricted to English conversations with at least four turns (≥2 user, ≥2 assistant) to ensure a repair turn is possible.

The Frustration-Signal Filter

A four-stage filter is applied to surface high-quality intent-gap candidates:

1. Stage A (Lexical Match): Retains conversations where the user's second turn (or later) matches a curated list of 80 "repair markers" (e.g., "i meant," "try again," "useless"). This list was expanded from 17 academic phrases to 45 natural-language phrases based on pilot data.
2. Stage B (Embedding Similarity): Computes cosine similarity between the embedding of the (original prompt + repair turn) and the assistant's prior response. Conversations with similarity < 0.55 are retained, hypothesizing that low similarity indicates a mismatch between the assistant's response and the user's true intent.
3. Stage C (LLM-as-Judge): An LLM judge (Claude Haiku) classifies surviving candidates into categories: Refusal, Hallucination, Intent Gap, Other Dissatisfaction, or False Positive. Only "Intent Gap" candidates proceed.
4. Stage D (Hand Review): A subset undergoes human review to establish inter-rater reliability (Cohen's κ).

Cross-Model Replay and Triangulation

• Replay: The initial user prompt of confirmed intent-gap conversations is replayed across four frontier models (Claude, GPT, Gemini, and Llama-3.3-70B) to distinguish recoverable failures from persistent ones.
• Grading: Each replay is scored on two axes: Literal Compliance and Intent Satisfaction.
• Triangulation: The resulting taxonomy is mapped against ~25 prior failure-mode taxonomies from research institutes and five documented production failure incidents (including Moffatt v. Air Canada and the NYC MyCity chatbot incident).

3. Key Contributions

1. Reproducible Filter: A pipeline to surface intent-gap failures from public conversation corpora at scale using frustration signals.
2. Labeled Corpus: A released dataset of ~150–300 high-quality, hand-reviewed intent-gap conversations.
3. User-Grounded Taxonomy: A multi-axis taxonomy of failure modes derived from real user data rather than researcher assumptions.
4. Cross-Model Analysis: Empirical data on how current frontier models perform on historically failed prompts.
5. Triangulation Matrix: A mapping of the new taxonomy against existing literature and real-world production incidents.

4. Preliminary Results

Based on a pilot of 50,000 conversations and a 50-conversation subsample for LLM judging:

• Silent Abandonment is Dominant: 52.8% of English conversations end after a single exchange. The author hypothesizes a substantial fraction of these are silent abandonments following intent-gap failures, rendering them invisible to feedback-driven quality pipelines.
• Rarity of Explicit Repair: Only 3.85% of multi-turn conversations contain an explicit repair signal, even with a permissive 45-phrase filter. This suggests users rarely verbalize intent gaps.
• Vocabulary Mismatch: The most common repair phrases in real data ("i meant," "try again," "useless") differ significantly from academic literature lists ("you misunderstood," "let me rephrase"). The v2 filter (45 phrases) increased hit rates by ~31× compared to v1 (17 phrases), indicating prior literature dramatically undercounts real-world repair.
• Filter Validity: Among successfully judged candidates, 79% (15/19) were confirmed as real intent-gap failures.
• Judge Limitations: The LLM judge refused to grade 62% of candidates, likely due to content moderation filters rejecting roleplay or edgy content. This implies that safety-filtered evaluation pipelines systematically miss failures in the most permissive parts of the user-prompt distribution.
• Taxonomy Distribution: In the pilot sample, failures clustered into four categories: Goal Collapse (33%), Specificity Mismatch (27%), Implicit-Context Blindness (27%), and Memory/State Failure (13%).
• The Sycophancy Contradiction: Despite sycophancy being a major focus in alignment literature, zero instances were found in the pilot user-data sample. The author hypothesizes that sycophantic failures are either rare in real usage or, more critically, invisible to user-feedback loops because users do not repair responses they find agreeable.

5. Significance and Claims

The paper claims that the "Intent Gap" represents a class of failures that benchmarks structurally cannot surface because they are designed by researchers rather than emerging from user interactions.

• Methodological Shift: The study argues that dialogue-breakdown detection literature relies on researcher-imagined phrase lists that fail to capture the blunt, idiomatic nature of real user repairs.
• Deployment Implications: The author suggests that current evaluation pipelines, which rely on explicit user feedback or safety-filtered judges, are blind to the most common failure modes (silent abandonment) and the most dangerous failure modes (sycophancy that users accept).
• Actionable Insights:
  • Pre-deployment: Red-teaming should prioritize categories with high production-incident weight (Implicit-context blindness, Memory failure).
  • Runtime Monitoring: Repair signals (even blunt ones like "useless") should be used as a low-cost quality metric.
  • Evaluation: The "sycophancy contradiction" suggests that red-team and adversarial evaluation are necessary for detecting sycophancy, as user-feedback loops are structurally incapable of surfacing it.

The paper concludes that the deployment gap is not a single phenomenon but a class of failure modes requiring a shift from researcher-designed benchmarks to user-emergent data analysis. The full corpus release and triangulation matrix are intended to provide the empirical foundation for this shift.