Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

Technical Summary: Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

Problem Statement
In human dialogue, interlocutors actively maintain mutual understanding by recognizing uncertainty and initiating clarification requests (repair) when ambiguity arises. While Large Language Models (LLMs) and Vision-Language Models (VLMs) demonstrate fluency, it remains an open question whether they can effectively assume a listener role that involves recognizing internal uncertainty and translating it into appropriate clarification behavior. Current models often mask comprehension problems with fluent output, leading to overestimation of their competence and misaligned confidence. Furthermore, existing research indicates that LLMs rarely proactively initiate repair and often fail to ground interactions effectively. The challenge lies in evaluating this capability in open dialogue settings, where ground truth for when clarification is warranted is absent and the space of interpretations is unbounded.

Methodology
To address these challenges, the authors propose using reference games as a controlled, self-contained testbed. In this setup, a "speaker" describes a target object from a set of candidates (e.g., color grids), and a "listener" must identify it. This environment makes clarification needs explicit and measurable because ambiguity arises strictly from the mismatch between the description and the candidate set.

The study evaluates three VLMs on the color-grid reference game dataset (McDowell and Goodman, 2019), which includes 197 games with 60 rounds each, varying in difficulty based on color similarity (far, split, close). The models tested are:

• Qwen2.5-VL-7B and Qwen2.5-VL-72B (selected for their strong prior performance on this task).
• GPT-5-mini (as a representative commercial model).

The experimental design consists of three phases:

1. Baseline Experiment: Models act as listeners to predict the target grid based solely on the speaker's description and the visual input. To estimate uncertainty, the models are sampled five times per round; the majority vote determines the prediction, and the proportion of matching samples serves as a Baseline Confidence metric.
2. Clarification Experiment: Models are explicitly instructed to request clarification if they are uncertain about the referent. Performance is measured by CR-Rate (frequency of clarification requests), Accuracy (correctness of non-clarification responses), and Relaxed Accuracy (correctness or successful clarification).
3. Interaction Experiment: A human-in-the-loop provides answers to the model's clarification requests. This phase evaluates the task-relevance of the generated questions and measures whether the subsequent human feedback improves the model's final accuracy and confidence.

Key Results

• Overconfidence: All models exhibited uniformly high confidence scores (ranging from 87% to 99%) even in difficult conditions, indicating a disposition toward overconfidence that is poorly calibrated to actual accuracy.
• Clarification Behavior:
  • GPT-5-mini demonstrated the most aligned behavior, generating clarification requests in 13% of cases. Its request rate increased with task difficulty (6% in "far" to 17% in "split/close"), and its accuracy on non-clarification responses (94%) exceeded its baseline accuracy (91%), suggesting it successfully withheld answers when uncertain.
  • Qwen2.5-VL-72B generated clarification requests in 24% of cases, but this behavior was not systematically aligned with task difficulty or internal uncertainty. Its confidence only dropped marginally (from 91% to 87%) when generating a request.
  • Qwen2.5-VL-7B almost never generated clarification requests (< 0.1%).
• Effectiveness of Clarification: In the interaction experiment, only 42% of the clarification questions generated by Qwen2.5-VL-72B were deemed task-relevant. Crucially, providing human answers to these requests did not improve the model's end-to-end accuracy; in fact, accuracy slightly decreased in most conditions. This suggests that the models either failed to utilize the provided information or that the requests themselves were not seeking the necessary information to resolve ambiguity.

Key Contributions

• Testbed Proposal: The paper establishes reference games as a rigorous, controlled environment for evaluating the alignment between model uncertainty and clarification behavior, overcoming the limitations of open-ended dialogue evaluation.
• Empirical Evidence of Misalignment: The study provides empirical evidence that current state-of-the-art VLMs struggle to translate internal uncertainty into appropriate interactional responses. Even in simple, constrained tasks, models often fail to recognize when they are uncertain or fail to ask relevant questions.
• Ineffectiveness of Current Clarification: The findings highlight that current models' clarification attempts are often uninformative or inappropriate, failing to function as genuine repair mechanisms that improve mutual understanding.

Significance
The paper argues that reference games offer a valuable benchmark for probing the pragmatic competence of language models. The results demonstrate a significant gap in the ability of VLMs to act as cooperative listeners who can manage uncertainty. The authors conclude that even in settings where ambiguity is quantifiable and the need for clarification is explicit, models struggle to convert internal uncertainty into adequate clarification behavior. This underscores the value of reference games for diagnosing interaction qualities in (vision and) language models, revealing that fluency does not equate to robust, self-aware communication.