Technical Summary: Q-DAPS (Question Difficulty based on Answer Plausibility Scores)

Problem Statement

Estimating question difficulty is essential for evaluating and improving Large Language Models (LLMs) in Question Answering (QA) systems. Existing approaches typically rely on readability formulas, retrieval-based signals, or popularity statistics. However, these methods often fail to capture the specific reasoning challenges posed to modern LLMs, as they are frequently grounded in human readability or retrieval system metrics rather than the model's internal uncertainty. Furthermore, current methods do not explicitly address the risk of hallucination, where harder questions are more likely to elicit fabricated answers. There is a need for an LLM-oriented difficulty metric that reflects how convincing incorrect answers appear to the model itself.

Methodology: Q-DAPS

The authors propose Q-DAPS, a method that estimates question difficulty by computing the entropy of plausibility scores over a set of candidate answers. The method operates in three distinct stages:

1. Candidate Generation: An LLM is prompted to generate $N$ unique candidate answers for a given question (excluding the ground truth) along with a plausibility score (0–100) and a justification for each. The system validates these outputs to ensure no duplicates, valid score ranges, and the correct number of candidates. If validation fails, the generation is retried with increased temperature.
2. Popularity Debiasing: To mitigate popularity bias (where LLMs tend to generate or favor more popular entities), the method adjusts plausibility scores using Wikipedia page view statistics. The debiased score ($DePls_i$) is calculated as:
   $$DePls_i = Pls_i \times (1 - \alpha \times Pop_i)$$
   where $Pls_i$ is the original plausibility score, $Pop_i$ is the normalized popularity of the candidate, and $\alpha$ is a hyperparameter controlling the debiasing strength.
3. Scoring: The final difficulty score is derived from the entropy of the normalized, debiased plausibility scores.
   • First, scores are normalized to form a probability distribution.
   • Entropy ($H(q)$) is computed: $H(q) = -\sum DePls^{norm}_i \times \log_2 DePls^{norm}_i$.
   • The entropy is normalized to a [0, 1] range by dividing by $\log_2 N$ (where $N$ is the number of candidates).
   • Interpretation: High entropy indicates that many candidate answers are similarly plausible to the LLM, signifying a difficult question with high reasoning uncertainty. Low entropy indicates a skewed distribution where only a few candidates are plausible, signifying an easier question.

The authors evaluated three estimation paradigms for generating plausibility scores: Pointwise (scoring candidates individually), Pairwise (comparing pairs), and Listwise (generating candidates and scores jointly). The Listwise approach was found to be the most effective and computationally efficient ($O(1)$ complexity).

Key Contributions

1. Novel Metric: Introduction of Q-DAPS, an LLM-oriented difficulty metric based on the entropy of answer plausibility scores, shifting focus from surface-level text properties to reasoning uncertainty.
2. Debiasing Strategy: Identification of popularity bias in candidate answer generation and the introduction of a lightweight debiasing mechanism using Wikipedia view counts to improve estimation accuracy.
3. Comprehensive Validation: Extensive experiments across four diverse QA datasets (TriviaQA, Natural Questions, MuSiQue, QASC) covering simple and complex reasoning tasks. The method was tested against readability, popularity, retriever-based, prompt-based, and uncertainty-based baselines.
4. Robustness Analysis: Ablation studies demonstrating that Q-DAPS remains effective even without access to the gold answer, without popularity debiasing, and across different underlying LLMs (ranging from 7B to 70B parameters).
5. Human Alignment: Human evaluation studies confirming that Q-DAPS difficulty estimates align well with human judgments and that questions labeled as "hard" by Q-DAPS are indeed harder for humans to answer correctly.

Experimental Results

• Performance: Q-DAPS consistently outperformed all baseline methods (including Flesch-Kincaid, Gunning-Fog, PopQA, and uncertainty-based methods) across all four datasets. It achieved the highest Cohen's $d$ (separating easy and hard questions) and Spearman's $\rho$ (correlation with model performance) scores.
• Entropy vs. Average: The entropy-based scoring approach significantly outperformed using the average plausibility score, confirming that the distribution of plausibility (uncertainty) is a stronger signal for difficulty than the mean.
• Ablation Findings:
  • Gold Answer: While providing the gold answer improves performance, Q-DAPS remains superior to baselines even when the gold answer is withheld, making it applicable to real-world scenarios where answers are unknown.
  • Debiasing: Removing the popularity debiasing component reduced performance but did not invalidate the method, suggesting debiasing is a beneficial refinement rather than a strict requirement.
  • Model Size: The method performed robustly across different model sizes, though larger models (e.g., LLaMA 3.3 70B) yielded slightly better results.
• Error Analysis: The authors identified specific divergence patterns where entropy-based estimates differ from exact-match evaluation outcomes (e.g., "entropy-overestimated difficulty" in list-based factual questions with many semantically related answers). They argue these cases reflect answer-space richness rather than model failure.

Significance and Claims

The paper claims that Q-DAPS provides an interpretable, scalable, and bias-resilient approach to question difficulty estimation. Its significance lies in its operational perspective: it directly links difficulty to hallucination risk detection. By measuring how plausible incorrect answers appear to an LLM, Q-DAPS can serve as a practical tool for:

• Model Selection: Choosing stronger LLMs for domains where questions are consistently estimated as difficult.
• Question Routing: Directing high-difficulty questions to human reviewers in knowledge bases.
• Safeguard Triggering: Requiring citations or user confirmation for questions estimated to have high uncertainty.

The authors emphasize that Q-DAPS captures the intrinsic ambiguity of the answer space from the model's perspective, offering a more nuanced view of difficulty than traditional readability or retrieval metrics. Future work is suggested to explore multilingual and multimodal extensions.