Technical Summary: Reassessing Extractive QA Datasets at Scale

Problem Statement

Extractive Question Answering (QA) tasks are traditionally evaluated using Exact Match (EM) and F1-score. However, these metrics frequently fail to reflect true model performance, particularly when predicted answers are semantically equivalent to gold answers but phrased differently (e.g., "EPA" vs. "Environmental Protection Agency"). While recent studies have proposed using Large Language Models (LLMs) as judges (LLM-as-a-judge) to address this, critical aspects of their performance remain unexplored. Specifically, it is unclear how these judges handle diverse answer types, whether they exhibit biases (such as self-preference or sibling-preference), and how robust their judgments are under varying prompting strategies.

Methodology

The authors conducted a systematic study across four diverse extractive QA datasets: Quoref (coreferential reasoning), DROP (numerical reasoning), HotpotQA (multi-hop reasoning), and 2WikiMultiHopQA.

• Data Curation: From the development sets of these four datasets, the authors curated a representative subset of approximately 1,000 samples per dataset (totaling ~4,200 samples), stratified by eight distinct answer types: Place, Name, Job, Date, Number, Year, Bool, and String.
• Models:
  • QA Models: Eight models from four families (Mistral, Qwen, Gemma, Llama) ranging from 7B to 72B parameters were used to generate predictions.
  • Judges: Three model families (Mistral 7B v0.3, Llama 3.3 70B, and Qwen 2.5 72B) served as the "LLM-as-a-judge."
• Evaluation Protocol:
  • Human Baseline: 161 samples (1,288 predicted answers) were manually annotated by human experts to establish a ground truth for correlation analysis.
  • Bias Analysis: The study investigated self-preference bias (a model judging its own output favorably) and sibling-preference bias (a model favoring outputs from the same family).
  • Prompt Robustness: The authors tested various prompting configurations, including different shot counts (zero-shot, 2-shot, 6-shot), the inclusion/exclusion of context, and variations in prompt wording.

Key Results

1. Correlation with Human Judgment

LLM-as-a-judge significantly outperforms traditional metrics in correlating with human evaluations.

• EM and F1: Correlated with human judgments at 0.22 and 0.40, respectively.
• LLM-as-a-Judge: Achieved correlations up to 0.85 (specifically with Qwen 2.5 72B), indicating a much stronger alignment with human assessment of answer correctness.

2. Performance by Answer Type

The effectiveness of LLM-as-a-judge varies by answer type:

• High Performance: The approach performs exceptionally well on Number (0.899 correlation) and Date (1.000 correlation) answers.
• Challenges: Performance drops significantly for Job titles (0.352 correlation), largely due to the ambiguity of multiple valid job descriptions in gold answers versus predicted answers.
• General Trend: For most other types (Name, Place, String), correlations remain high (0.77–0.86).

3. Bias Analysis

Contrary to findings in other NLP tasks, the study found no evidence of significant bias:

• Self-Preference Bias: When the same model served as both the QA generator and the judge, no significant self-preference bias was observed.
• Sibling-Preference Bias: Similarly, no bias was detected when a judge from the same model family evaluated outputs from a sibling model.

4. Prompt Robustness

The evaluation results were highly stable across different prompt variations:

• Zero-Shot & Context-Free: Surprisingly, zero-shot prompting and evaluations without context often yielded the best correlation scores (e.g., Qwen 2.5 72B reached 0.909 without context).
• Stability: Variations in wording and setup had minimal impact, with very low variance in correlation scores (0.0007 to 0.0037), suggesting the approach is robust to prompt engineering nuances.

Key Contributions

The paper makes four primary contributions:

1. Comprehensive Evaluation: A large-scale evaluation of multiple open-weight models across four diverse QA datasets, with code and data released for reproducibility.
2. Fine-Grained Analysis: The first detailed analysis of how judgment quality varies across specific answer types, identifying strengths in numerical reasoning and weaknesses in complex job titles.
3. Bias Investigation: A systematic investigation into self-preference and sibling-preference biases, finding no evidence of these effects in extractive QA tasks.
4. Prompt Robustness Study: A demonstration that LLM-as-a-judge outcomes are stable across prompt variations, with zero-shot, context-free judging often proving superior.

Significance and Claims

The authors claim that their findings offer a practical and reliable alternative to EM and F1 scores for evaluating extractive and short-form QA datasets. They argue that LLMs can serve as robust evaluators that better capture semantic correctness, particularly for non-exact matches. The study suggests that future research can rely on LLM-as-a-judge without extensive prompt engineering or fear of inherent model biases, provided the task involves clear gold answers. The paper concludes that while LLM-as-a-judge is not perfect (notably struggling with complex job titles), it represents a significant improvement over traditional metrics for assessing model performance in reading comprehension tasks.