🤖 AI

What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models

This paper introduces HAERAE-Vision, a benchmark of real-world, under-specified visual queries from Korean communities, revealing that current vision-language models struggle significantly with natural user inputs due to missing context rather than inherent capability gaps, and that explicit query rewriting yields far greater performance gains than web search.

Original authors: Dasol Choi, Guijin Son, Hanwool Lee, Minhyuk Kim, Hyunwoo Ko, Teabin Lim, Ahn Eungyeol, Jungwhan Kim, Seunghyeok Hong, Youngsook Song

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dasol Choi, Guijin Son, Hanwool Lee, Minhyuk Kim, Hyunwoo Ko, Teabin Lim, Ahn Eungyeol, Jungwhan Kim, Seunghyeok Hong, Youngsook Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a brilliant, super-smart detective (the AI) who has been trained on millions of books, movies, and clear, well-written police reports. You are excellent at solving crimes when the clues are laid out perfectly.

Now, imagine a real person walks into your office. They are flustered, holding a blurry photo of a strange object, and they say: "What is this? How do I fix it?"

They don't tell you what the object is, where they found it, or what they were trying to do. They just assume you can read their mind because they are holding the picture.

This paper, HAERAE-Vision, is about testing how well these "super-detective" AIs can handle real people who talk like that, rather than how they handle perfect test questions.

Here is the breakdown of the story:

1. The Problem: The "Perfect Test" vs. Real Life

Most AI tests are like a standardized exam. The questions are clear: "Identify the animal in this picture."
But real life is messy. People ask: "My dog ate this, where do I buy a new one?" (while pointing at a torn piece of wood).

The researchers found that current AI models are like students who ace the textbook but fail the real-world application. They get confused when the user doesn't spell everything out.

2. The Experiment: The "Korean Community" Challenge

The team went to Korean online forums (like Reddit or Quora) and collected 86,000 real questions people asked with photos.

The Filter: They were very strict. They threw out 99.2% of the questions because they were too easy, too weird, or didn't need the photo to answer.
The Result: They ended up with 653 "gold standard" messy questions. These are the real deal: informal, vague, and relying heavily on context.

They created two versions of every question:

The Messy Version: What the user actually typed (e.g., "What's this worm?").
The "Cleaned-Up" Version: A rewrite that fills in the blanks (e.g., "What is the name of this marine snail, Dendropoma maxima, found in Jeju Island waters?").

3. The Big Discovery: "Clarification" is Magic

They asked 45 different AI models (from the biggest giants like GPT-5 to smaller ones) to answer both versions.

The Shocking Result:

On the Messy Version, even the smartest AIs got less than 50% right. They were basically guessing.
On the Cleaned-Up Version, the same AIs jumped up to 57% or higher.

The Analogy:
Think of the AI as a chef.

Messy Query: A customer says, "Make me that thing I had yesterday." The chef has no idea what they want.
Cleaned Query: The customer says, "Make me the spicy beef stew I had at the restaurant on 5th Street."
The Lesson: The chef didn't get better at cooking; the customer just gave better instructions. The difficulty wasn't the AI's brain; it was the user's vague request.

Who benefited most? The smaller, cheaper AI models. They were the ones who got completely lost without the extra details. The big models were a bit better at guessing, but they still struggled.

4. The "Google Search" Trap

The researchers thought, "Okay, if the AI doesn't know, let's give it a Google Search button!"

Result: It helped a little bit, but not enough.
Why? If you ask, "How do I fix this?" without saying what "this" is, Google can't help you. The AI needs to understand the intent first before it can search for the answer. You can't search for a question you haven't fully formed yet.

5. The Hidden Hurdle: Cultural Context

Even after fixing the vague questions, the AIs still got some wrong. Why?

The Cultural Gap: Some questions relied on things only a Korean person would know.
- Example: A photo of orange bags on a rural road. An AI might guess "trash" or "safety cones." A Korean local knows immediately: "Those are sandbags for winter snow removal."
The AI isn't just missing facts; it's missing the "local vibe" and cultural assumptions that humans take for granted.

The Takeaway

This paper tells us that AI isn't as "dumb" as we think, but it's also not as "smart" as we hope.

The Benchmark Lie: Current tests are too clean. They make AI look smarter than it is in the real world.
The User's Job: We can't just expect AI to read our minds. We need to learn how to give better context, or we need tools that help us clarify our own questions.
The Future: To make AI truly useful, we need to teach it not just to "see" and "read," but to understand the messy, cultural, and incomplete way humans actually talk.

In short: The AI isn't failing because it can't see the picture; it's failing because it doesn't know why you're showing it the picture.

1. Problem Statement

Current Vision-Language Model (VLM) benchmarks predominantly rely on well-structured, explicit, and clean prompts. However, real-world user interactions are characterized by informal, under-specified queries where users omit context, rely heavily on images to convey meaning, and use vague references (e.g., "What is this?" or "How do I fix this?").

The paper argues that the difficulty observed in current VLM evaluations often stems not from a lack of model capability, but from the natural under-specification of user queries. Existing benchmarks fail to capture this phenomenon, leading to a gap between benchmark performance and real-world deployment. Furthermore, it is unclear whether current retrieval-augmented generation (RAG) or web search capabilities can compensate for missing context in these ambiguous queries.

2. Methodology: HAERAE-Vision Benchmark

To address this, the authors introduce HAERAE-Vision, a new benchmark constructed from authentic Korean online community data.

Data Collection & Filtering:
- Source: 86,052 question-image pairs collected from nine diverse Korean platforms (e.g., Naver KnowledgeIn, Ruliweb, BRIC, coding forums).
- Pipeline: A rigorous six-stage filtering process was applied:
  1. Data Collection: Prioritizing high-engagement, accepted answers.
  2. Appropriateness Filtering: Removing unsafe, subjective, or time-sensitive content (using GPT-4o).
  3. Difficulty Calibration: Removing questions too easy for strong models (GPT-4o, Gemini-1.5, Claude-3.5).
  4. Image Dependency Verification: Ensuring the question cannot be answered without the image (comparing text-only vs. image+text responses).
  5. Checklist Generation: Converting answers into structured checklists (1–5 criteria) using LLMs for granular evaluation.
  6. Human Validation: Three-phase human review to ensure quality, resulting in 653 high-quality items (0.76% survival rate).
Dataset Composition:
- HAERAE-Vision (Original): 653 under-specified, natural queries.
- HAERAE-Vision-Explicit: A parallel dataset where each original query is rewritten to explicitly state missing context, domains, and entities while preserving the original intent.
- Domains: Covers 13 categories including Natural Objects, Gaming, Science, Coding, Daily Life, and Transportation.
- Cultural Grounding: ~23.7% of items require specific Korean cultural knowledge (e.g., local interfaces, regional objects, Korean media).

3. Evaluation Framework

Models Evaluated: 45 VLMs, including proprietary models (GPT-5 series, Gemini 2.5 Pro/Flash, Grok-4, Perplexity), open-source models (Qwen, InternVL, Ovis, Gemma), and Korean-specialized models (VARCO-Vision, HyperCLOVA).
Scoring Mechanism:
- LLM-as-a-Judge: GPT-5-Mini acts as the primary judge, scoring responses against the generated checklists.
- Granularity: Scores are assigned on a 0.0 (not met), 0.5 (partially met), or 1.0 (met) scale for each checklist item.
- Reliability: Inter-judge agreement was verified across four different LLM judges (Krippendorff's $\alpha$ = 0.867) and validated against human annotators.

4. Key Results

A. Performance on Under-Specified Queries

Low Baseline: Even state-of-the-art (SOTA) models struggle significantly. GPT-5 and Gemini 2.5 Pro achieved <50% accuracy (48.0% and 48.5%, respectively) on the original under-specified queries.
Model Scale: Larger models generally outperform smaller ones, but the gap is not solely due to scale; proprietary models consistently outperform open-weight counterparts.
Korean Models: Korean-specialized models (VARCO, HyperCLOVA) did not show a clear advantage, suggesting that local training data has not yet addressed the specific challenge of under-specification.

B. Impact of Query Explicitation

Significant Gains: Rewriting queries to be explicit yielded massive improvements, ranging from 8 to 22 percentage points.
- GPT-5: 48.0% $\rightarrow$ 57.6% (+9.6).
- Gemini 2.5 Pro: 48.5% $\rightarrow$ 56.7% (+8.1).
- GPT-5 Nano: 21.2% $\rightarrow$ 43.0% (+21.7).
Implication: Smaller models benefit disproportionately, indicating they lack the capacity to infer implicit context from images alone. The difficulty in HAERAE-Vision is largely driven by query ambiguity rather than pure reasoning capability.

C. Limitations of Web Search (Retrieval)

Search vs. Explicitation: Adding web search to original under-specified queries improved performance (e.g., GPT-5: +7.6%), but still underperformed compared to explicit queries without search.
- Original + Search: 55.6%
- Explicit (No Search): 57.6%
Conclusion: Retrieval systems cannot compensate for under-specification because models fail to generate the specific search queries needed (e.g., product names, error codes) without first understanding the user's intent. Search is only effective after the query is clarified.

D. Error Analysis & Cultural Knowledge

Error Shift: After explicitation, "Lack of Explicitness" errors dropped significantly (-14.6%), but Cultural Knowledge errors increased (+6.4%).
Root Cause: Once surface ambiguity is resolved, the remaining bottleneck is Korean-specific cultural knowledge (e.g., identifying winter road sandbags, specific Korean phone models). This accounts for ~19% of errors in the explicit condition, compared to <3% in an English pilot study.
Visual-Text Grounding: Explicitation "unmasked" latent grounding errors. Models that gave vague answers to ambiguous queries were forced to engage with specific image regions when the query was explicit, revealing failures in visual-text grounding that were previously hidden.

E. Cross-Lingual Validation

An English pilot study (100 samples from Stack Exchange) confirmed that under-specification affects English models too, but the performance gap between original and explicit queries was smaller (+3.2 to +6.6) compared to Korean (+7.8 to +21.7). This is attributed to the lower prevalence of culture-specific knowledge barriers in English datasets.

5. Significance and Contributions

New Benchmark: HAERAE-Vision provides the first large-scale, rigorously curated benchmark for under-specified, culturally grounded visual questions, moving beyond "clean" academic datasets.
Redefining Difficulty: The paper challenges the assumption that VLM failure on real-world tasks is solely due to model capability. It demonstrates that query under-specification is a primary source of difficulty, accounting for up to 22 points of performance drop.
Retrieval Limitations: It highlights a critical flaw in current RAG paradigms: retrieval cannot fix ambiguity if the model cannot formulate the right query based on visual context.
Cultural Gap: The study underscores that even SOTA models struggle with culturally specific knowledge, a gap that is exacerbated by under-specification and not addressed by standard English-centric training.
Practical Implication: For real-world deployment, systems must prioritize query clarification and intent inference before attempting retrieval or reasoning, rather than relying on search to fill in missing context.

Conclusion

The paper concludes that the "sanitized" nature of current benchmarks creates a false sense of security regarding VLM capabilities. Real-world deployment requires models that can handle the "unsaid" in user queries, infer implicit context, and possess deep cultural grounding. The authors call for future benchmarks and model training strategies that explicitly address these under-specified, culturally nuanced interactions.