💬 NLP

What Makes a Good Query? Measuring the Impact of Human-Confusing Linguistic Features on LLM Performance

By analyzing 369,837 real-world queries through a 22-dimensional linguistic feature vector, this study demonstrates that specific human-confusing query characteristics, such as deep clause nesting and underspecification, significantly increase the likelihood of Large Language Model hallucinations, thereby establishing a data-driven framework for mitigating these risks through guided query rewriting.

Original authors: William Watson, Nicole Cho, Sumitra Ganesh, Manuela Veloso

Published 2026-02-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: William Watson, Nicole Cho, Sumitra Ganesh, Manuela Veloso

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart, but slightly over-eager, librarian (the AI) for help. Sometimes, this librarian gives you the perfect answer. Other times, they confidently make up facts that sound real but are completely wrong. In the tech world, we call these made-up facts "hallucinations."

Most people think the librarian is just broken or needs better training. But this paper argues something different: Sometimes, the problem isn't the librarian; it's how you asked the question.

Here is the breakdown of the research, explained simply with some analogies.

The Big Idea: The "Bad Question" vs. The "Bad Librarian"

The researchers looked at nearly 370,000 real questions asked to AI models. They wanted to see if the way a question was phrased made the AI more likely to lie.

They treated the question like a recipe. If you give a chef a vague recipe ("Make something tasty"), they might guess and serve you something weird. If you give a specific recipe ("Make a spicy chicken stir-fry with broccoli"), the result is much more reliable.

The 17 "Risk Factors"

The team created a checklist of 17 linguistic features (ways of speaking) that confuse humans. They asked: Do these same confusing features confuse the AI and make it hallucinate?

They found a "Risk Landscape," which is like a weather map for question safety. Some features are "stormy" (high risk), and some are "sunny" (low risk).

🌩️ The Stormy Features (High Risk of Hallucination)

These are the things that make the AI's brain spin and guess:

The "Vague Map" (Lack of Specificity):
- Analogy: Telling a taxi driver, "Take me to a good place."
- Result: The driver will guess. The AI will guess facts.
- Example: "Tell me about Tesla." (Do you mean the car company? The inventor? The stock?)
The "Russian Nesting Doll" (Clause Complexity):
- Analogy: A sentence with too many "if," "when," and "because" clauses stacked on top of each other.
- Result: The AI gets lost in the middle of the sentence and forgets the beginning, leading to made-up connections.
The "Ghost Reference" (Anaphora):
- Analogy: Saying, "He said it was broken," without saying who "he" is or what "it" is.
- Result: The AI has to guess who "he" is, and it often guesses wrong.
The "Double Negative" (Negation):
- Analogy: "I don't want to not go."
- Result: Even though humans are getting better at this, AI still trips over the logic of "not" and "never."

☀️ The Sunny Features (Low Risk of Hallucination)

These are the things that act like guardrails, keeping the AI on the right track:

The "Clear Destination" (Intention Grounding):
- Analogy: "Summarize this article in three bullet points."
- Result: The AI knows exactly what job to do. It doesn't need to guess.
The "Checkable Fact" (Answerability):
- Analogy: Asking "What is 2+2?" instead of "What is the meaning of life?"
- Result: If the answer exists and is clear, the AI is less likely to invent one.
The "Context Clues" (Specific Constraints):
- Analogy: "Tell me about the weather in London last Tuesday."
- Result: The AI is forced to look at a specific time and place, reducing the space for imagination.

The Surprising Twist: Humans vs. AI

Here is the most interesting part of the study.

Some things that drive humans crazy—like using very rare words or complex grammar—don't actually bother the AI much.

Human: "Ugh, I can't understand this sentence with all those big words."
AI: "I know that word. I can still answer the question."

However, things that make humans confused (like vague questions) also make the AI hallucinate. This suggests that while AI is different from humans, it still struggles with the same fundamental logic gaps: ambiguity and lack of context.

The "Risk Score"

The researchers built a system that looks at your question before the AI answers it. It gives the question a "Risk Score":

Safe: Clear, specific, grounded. (Go ahead and ask!)
Borderline: A little vague. (Maybe rephrase it.)
Risky: Full of "who," "what," and "maybe" without enough details. (Stop! Rewrite this first.)

The Takeaway: How to Talk to AI

You don't need to be a computer scientist to get better results. You just need to be a better "prompt engineer" by following three simple rules:

Be Specific: Don't say "Tell me about dogs." Say "Tell me about the dietary needs of Golden Retrievers."
State Your Goal: Don't just ask a question. Say "Compare X and Y" or "Summarize Z."
Clear the Fog: If you use a word that has two meanings (like "Java" the island vs. "Java" the code), tell the AI which one you mean immediately.

In short: If you want the AI to be honest, stop asking it to guess. Give it a clear map, and it will get you to the right destination without making up the scenery along the way.

1. Problem Statement

Large Language Models (LLMs) suffer from hallucinations (producing plausible but factually incorrect outputs), a critical issue in high-stakes domains like finance and law. While existing research focuses on reactive mitigations (e.g., self-verification, logit-based detection) or proactive context enrichment (e.g., RAG), there is a lack of understanding regarding how the linguistic form of the input query itself influences hallucination propensity.

The authors hypothesize that specific linguistic features known to confuse human readers (e.g., ambiguity, complex syntax, underspecification) may similarly destabilize LLMs, leading to higher hallucination rates. They aim to empirically map these features to hallucination risk to enable proactive query optimization.

2. Methodology

A. Data Collection and Feature Extraction

Dataset: The study analyzes 369,837 real-world queries across 13 QA datasets (including SQuAD, TruthfulQA, MMLU, HotpotQA) spanning three scenarios: Extractive, Multiple Choice, and Abstractive.
Feature Taxonomy: The authors operationalized a 17-dimensional binary feature vector based on classical linguistics. These features are categorized as:
- Ambiguity: Lack of Specificity, Polysemous Words, Pragmatic Features.
- Referential Structure: Anaphora Usage.
- Complexity: Clause Complexity, Dependency Depth, Parse Tree Height, Number of Clauses.
- Polarity: Negation Usage.
- Grounding: Answerability, Intention Grounding, Contextual Constraints.
- Others: Domain Specificity, Named Entities, Rare Word Usage, Superlatives, Subjectivity, Excessive Details, Query-Scenario Mismatch.
Extraction: Features were extracted using structured LLM prompts (gpt-4o) with 5-shot examples and operational rubrics, treating detector noise as classical measurement error.

B. Hallucination Risk Labeling (The "Risk Proxy")

To avoid memorization bias in benchmark datasets, the authors used a semantics-preserving paraphrase neighborhood:

Paraphrasing: For each original query, 6 semantically equivalent but lexically perturbed paraphrases were generated.
Scoring: A convex proxy score $\hat{h}(q)$ $\hat{h} (q)$ was computed for each paraphrase, combining:
- $s_{llm}$ : Binary decision by an LLM judge (semantic correctness).
- $s_{fuzz}$ : Fuzzy string similarity.
- $s_{bleu}$ : BLEU-1 lexical overlap.
- Weights: (0.6, 0.3, 0.1) selected via Pareto-optimal ROC-AUC analysis.
Triage Labels: Queries were categorized into three ordinal classes based on the proportion of hallucinated paraphrases:
- Safe: 0/6 hallucinated.
- Borderline: 1–3/6 hallucinated.
- Risky: 4–6/6 hallucinated.

C. Statistical Modeling

Ordinal Logistic Regression: A proportional-odds model was fitted to predict the probability of a query being "Risky" ( $Y \in \{0, 1, 2\}$ $Y \in {0, 1, 2}$ ) based on the 17 features ( $X$ $X$ ) and fixed effects for dataset ( $\alpha$ $α$ ) and scenario ( $\gamma$ $γ$ ).
- Model: $\log \frac{Pr(Y_i \le k)}{Pr(Y_i > k)} = \tau_k - (x_i^\top \beta + \alpha_{d(i)} + \gamma_{s(i)})$
Diagnostics: The study utilized Kolmogorov-Smirnov (KS) distances, $\Delta$ median shifts in Empirical Cumulative Distribution Functions (ECDFs), and propensity score matching (IPW) to assess causal-like uplifts where overlap was sufficient.
Robustness: Leave-One-Dataset-Out (LODO) refits were performed to ensure findings were not driven by a single dataset.

3. Key Contributions

Feature Taxonomy & Extraction: A linguistically grounded framework defining 17 query features that impact human comprehension, adapted to measure their correlation with LLM hallucinations.
Empirical "Risk Landscape": A large-scale distributional map linking specific linguistic markers to hallucination propensity across 369k+ queries, revealing consistent patterns independent of specific datasets.
Proactive Guidance: Identification of actionable, low-effort query rewriting strategies (e.g., adding constraints, clarifying intent) that can reduce hallucination risk before generation occurs.

4. Key Results

A. The "Risk Landscape"

The study identified a consistent correlation between specific linguistic features and hallucination risk:

High-Risk Features (Positive Association):
- Lack of Specificity: The strongest predictor ( $\beta = 0.868$ , Odds Ratio $\approx 2.38$ ). Vague queries significantly increase risk.
- Clause Complexity: Deeply nested clauses and syntactic intricacy correlate with higher risk.
- Negation & Anaphora: Both show positive associations with risk, likely due to interpretive ambiguity.
- Query-Scenario Mismatch: Queries asking for extraction in abstractive settings (or vice versa) increase risk.
Protective Features (Negative Association):
- Answerability: The strongest protective factor ( $\beta = -1.106$ ). Queries with clear, verifiable answers have drastically lower hallucination rates.
- Intention Grounding: Explicitly stating the operation (e.g., "summarize," "compare") reduces risk.
- Syntactic Structure: Surprisingly, features like Query Token Length, Dependency Depth, and Number of Clauses showed negative correlations with risk in some contexts, suggesting that richer syntactic scaffolding can sometimes provide helpful context, provided it is not overly complex.

B. Counter-Intuitive Findings

Human vs. Model Failure Modes: Features traditionally known to confuse humans (e.g., Rare Word Usage, Superlatives, Complex Negation) showed minimal or mixed association with LLM hallucinations. This suggests that LLM failure modes do not perfectly align with human cognitive limitations.
Domain Specificity: The effect of domain-specific terminology was mixed and dependent on the model's familiarity with the domain, rather than being a uniform risk factor.
Presupposition: While common in "Safe" queries, presupposition only increased risk when co-occurring with other ambiguity markers.

C. Task Format Moderation

Abstractive Tasks: Showed the highest baseline risk, which increased sharply with query length.
Extractive Tasks: Remained robust across query lengths.
Multiple Choice: Showed intermediate risk levels.

5. Significance and Practical Applications

Shift from Reactive to Proactive: The paper argues for shifting focus from post-generation detection to pre-generation query optimization.
Automated Triage: Systems can use the derived feature vector to predict $P(\text{Risky})$ in real-time. High-risk queries can be routed to clarification steps or retrieval-augmented paths.
Low-Effort Rewrites: The authors propose three simple rules to mitigate risk:
1. Add Constraints: Specify time, place, or entity to resolve ambiguity.
2. State Intent Explicitly: Use verbs like "summarize," "extract," or "verify."
3. Resolve Polysemy: Clarify ambiguous terms (e.g., "Java the language" vs. "Java the island").
Limitations: The study is observational; while propensity scoring suggests quasi-causal links, features like "Answerability" are inherently semantic and difficult to manipulate without changing the query's meaning. The findings are currently limited to English and a single model family (GPT-4o).

Conclusion

The paper establishes that query form is a significant determinant of LLM reliability. By identifying a "risk landscape" driven by ambiguity and underspecification, the authors provide a roadmap for engineering more robust interactions with LLMs through guided query rewriting, offering a practical path to reduce hallucinations in high-stakes applications.