LLMs as Assessors: Right for the Right Reason?

Technical Summary: LLMs as Assessors: Right for the Right Reason?

1. Problem Statement

While recent research has explored using Large Language Models (LLMs) as "judges" to replace human assessors in evaluating Information Retrieval (IR) systems, most studies focus on document-level relevance (binary or graded labels). This approach fails to address a critical question: Is the LLM making the correct judgment for the correct reasons?

A model might label a document as "relevant" due to superficial lexical overlaps (keyword matching) rather than a true understanding of the information need. This paper seeks to move beyond document-level labels by evaluating LLMs on their ability to perform rationale generation—specifically, extracting the exact passages within a document that justify its relevance.

2. Methodology

The authors utilize the INEX 2009 and 2010 Wikipedia-based collections, which provide high-quality, human-annotated "ground truth" consisting of specific highlighted passages (excerpts) within documents.

• Task Formulation: The researchers implement a two-stage evaluation:
  1. Document-level Relevance Prediction: A zero-shot prompt asks the LLM to classify a document as relevant or non-relevant.
  2. Rationale Extraction (The Core Task): Using a few-shot (k=6) In-Context Learning (ICL) approach, the LLM is prompted to extract the specific text segments that correspond to the human-highlighted passages.
• Models Evaluated:
  • Llama-3.1-8B-Instruct (Open-source, smaller).
  • GPT-4.1-mini and GPT-4o-mini (Closed-source, larger/more advanced).
• Evaluation Metrics:
  • For rationale extraction, they use Precision, Recall, and F1 scores by mapping LLM outputs to the document via Longest Common Subsequence (LCS) to ensure they are measuring actual text overlap rather than paraphrasing.
  • For document-level prediction, they use accuracy.
• Experimental Variables: The study tests different exemplar selection strategies (how the few-shot examples are chosen) and examines performance across different document lengths and "hard" data points (where models typically fail).

3. Key Contributions

• Granular Evaluation Framework: The study is among the first to evaluate LLMs as judges at the passage level rather than just the document level, providing a proxy for "explainability."
• Identification of "Needle-in-a-Haystack" Failure: The authors quantify the difficulty LLMs face when relevant information is a very small fraction of a large document.
• Exemplar Selection Analysis: They demonstrate that the choice of few-shot examples significantly impacts performance, specifically finding that selecting examples with a low proportion of relevant content (teaching the model to find "needles") improves precision.
• Comparative Model Analysis: They provide a nuanced comparison between smaller models (which tend to over-highlight/over-predict) and larger, more "conservative" models.

4. Key Results

• Document-Level Divergence: There is a notable trend where larger models (GPT-4o-mini) are much more conservative (predicting "non-relevant" more often) than smaller models (Llama-3.1-8B), which produce many false positives.
• High Recall, Low Precision: In rationale extraction, LLMs generally achieve high recall but struggle with precision. They tend to over-highlight content, often including tangential or background information in their "explanations."
• The "Needle" Problem: Precision drops significantly as document length increases and the fraction of relevant content decreases. LLMs struggle to isolate small, specific relevant chunks within large texts.
• Discontiguous Chunks: LLMs perform significantly worse when the relevant information is split into multiple, non-adjacent passages compared to a single contiguous block.
• Reasoning vs. Scale: On "hard" data points (where precision is near zero), moving to a reasoning-focused model (GPT-4o-mini) yielded substantial improvements in precision, even if absolute scores remained low.
• Exemplar Impact: Using a "rel-chunk-prop" strategy (selecting exemplars where the relevant part is a small proportion of the document) significantly improved precision on difficult cases.

5. Significance

The paper serves as a cautionary note for the IR community. While LLMs are promising tools that can reduce the human labor required to build benchmark datasets, they cannot yet replace human assessors.

The findings suggest that LLMs often lack the discriminative nuance required for high-quality "gold standard" creation. They are prone to over-generalization and struggle with precision in complex retrieval scenarios. However, the study also provides a roadmap for judicious use: by using principled exemplar selection and leveraging high-reasoning models, the utility of LLMs as automated annotators can be significantly enhanced.