Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models

Technical Summary: Know2Guess

Problem Statement
Current evaluation of Large Language Models (LLMs) often conflates supported answering with unsupported guessing, failing to distinguish between epistemic uncertainty (not knowing) and policy refusal (choosing not to answer). Existing benchmarks typically measure raw accuracy, truthfulness, or hallucination in isolation, or rely on dynamic evaluations that do not provide a frozen protocol for separating answerable items from abstention-expected unknowns. Furthermore, data contamination risks can inflate performance metrics, blurring the line between memorization and generalization. There is a lack of a unified framework that simultaneously tracks contamination risk, distinguishes between "productive abstention" (correctly identifying unknowns) and "policy refusal," and evaluates model behavior under strict parsing and prompt variations.

Methodology
The paper introduces Know2Guess, a contamination-aware, multi-zone benchmark containing 1,200 items across five domains: commonsense, general trivia, medicine, multi-hop reasoning, and science.

• Zone Construction: Items are assigned to four frozen build-time zones:
  • Zones A–C (Answer-Expected): These contain real-public items. Zone A has high answer popularity; Zone B has lower popularity; Zone C consists of transformed public items (paraphrased, compressed, or re-ordered) designed to increase boundary difficulty while preserving the gold answer.
  • Zone D (Abstention-Expected): These are synthetic unknowns constructed with syntax and domain styles matching the real-public pool but containing unsupported entities or events. They are designed to test if a model can abstain without resorting to policy refusal.
• Contamination Metadata: Each item is tagged with a contamination-risk level (low, medium, high) based on lexical overlap with known corpora and answer salience, allowing for slice-based analysis.
• Evaluation Protocol: Models are evaluated under two primary conditions:
  1. Answer-or-Abstain: A structured prompt explicitly permitting the model to abstain if unsure.
  2. Answer-Only (Control): A prompt instructing the model to guess if unsure, preventing abstention.
• Parsing and Metrics: An official strict parser extracts decisions (ANSWER, ABSTAIN, REFUSE) and confidence scores. A normalized parser is used for robustness analysis to handle formatting defects without altering decision semantics.
  • Primary Metric: Reliability, which rewards correct answering on Zones A–C and correct abstention on Zone D.
  • Secondary Metrics: Answered accuracy, productive abstention rate, refusal rate, Expected Calibration Error (ECE), and boundary sharpness (the performance gap between difficult answerable items and unknowns).

Key Results
The benchmark was evaluated on FLAN-T5 variants, Qwen2.5-Instruct (1.5B, 3B), and Llama-3-8B-Instruct.

• Baseline Performance: FLAN-T5 models demonstrated weak productive abstention capabilities, with FLAN-T5-Base and Large achieving zero productive abstention under the strict parser.
• Instruction-Tuned Models: Stronger models (Qwen2.5-3B and Llama-3-8B) showed a selective transition from answering to abstaining but did not solve the problem.
  • Qwen2.5-3B-Instruct achieved the highest overall reliability (0.3657) and high productive abstention on Zone D (0.9249). However, it struggled significantly on Zones A–C (reliability ~0.13–0.34) and exhibited poor calibration (ECE ~0.59).
  • Llama-3-8B-Instruct showed higher accuracy on answerable public items but was less selective on unknowns and more prone to policy refusal on benign items, resulting in lower overall reliability (0.3407).
• Control Experiments: Removing the option to abstain (Answer-Only control) caused a sharp collapse in reliability for all models, confirming that the benchmark measures decision-making behavior rather than just factual recall.
• Robustness: The main rankings and qualitative conclusions remained stable under parser normalization, prompt template variations, and cost-sensitive metric reweighting.
• Error Analysis: Qualitative audits revealed that the primary failure modes were unnecessary abstention on answer-expected items (28.3%) and confident hallucinations (24.2%), followed by policy refusal on benign items.

Significance and Claims
The authors explicitly state that the benchmark does not claim that abstention-aware reliability has been "solved." Instead, the significance of Know2Guess lies in providing a reproducible protocol for auditing LLMs across distinct but interacting dimensions: answerability, abstention, refusal, and contamination.

The paper argues that current models exhibit a "selective but incomplete transition" from answering to abstaining. While they can effectively abstain on synthetic unknowns, they fail to calibrate well on difficult answer-expected items and often confuse policy refusal with epistemic caution. By freezing build-time labels and separating refusal from abstention, the benchmark offers a stricter protocol to diagnose whether a model is knowledgeable, overconfident, overly conservative, or misaligned with the decision structure required for reliable deployment. The dataset and protocol are made publicly available to facilitate further auditing of these specific failure modes.