Technical Summary: SAGE (Search-AuGmented Evaluation)

Problem Statement

The evaluation of Large Language Models (LLMs) on free-form Question Answering (QA) faces a critical bottleneck. Traditional evaluation relies on static, pre-annotated reference answers compared via lexical metrics (e.g., Exact Match, F1). However, these methods fail to capture the diversity of open-ended responses, are costly to scale due to human annotation requirements, and are fundamentally misaligned with non-deterministic, context-dependent LLM outputs.

Conversely, the emerging "LLM-as-a-judge" paradigm, which uses models to evaluate outputs without external references, proves unreliable for objective, fact-centric tasks. Without external grounding, these judges rely solely on frozen parametric knowledge, leading to specific failure modes:

1. Knowledge Staleness: Judges confidently endorse answers that are outdated relative to their training cutoff.
2. Length Bias: Longer, more verbose answers are often overrated even when factually incorrect.
3. Prompt Sensitivity: Verdicts can flip based on minor prompt variations or the order of candidate answers.
4. Hallucinated Rationales: Judges fabricate supporting evidence to justify incorrect decisions.

The paper argues that objective correctness cannot be assessed solely through an LLM's internal knowledge or preferences, necessitating a framework that bridges the gap between scalable automation and factual grounding.

Methodology: Search-AuGmented Evaluation (SAGE)

SAGE is a reference-free framework designed to assess LLM outputs by equipping an LLM judge with the ability to actively retrieve, synthesize, and reflect on external evidence. Unlike single-pass search-augmented methods, SAGE operates as an iterative agent.

Core Components

The framework utilizes an LLM judge ($J$) that plays four distinct roles via one-shot prompt templates, interacting with a web search tool ($S$) and a short-term memory buffer ($M$):

1. Query Generation ($Q$): At each iteration $i$, the generator produces a search query $q_i$ conditioned on the input question $x$ and the accumulated memory $M_{i-1}$. The initial query is topic-level; subsequent queries are refined based on prior findings to target specific information gaps.
2. Web Search ($S$): Queries are submitted to a search engine (e.g., Serper API), returning a set of snippets (title, text, URL) as raw external evidence.
3. Summarization ($\Sigma$): Retrieved results are condensed into a focused evidence segment $E_i$, filtering redundancy and extracting salient factual content.
4. Reflection ($\Phi$): The summarizer assesses the evidence $E_i$ against the question $x$ and candidate answer $\hat{y}$. It determines if the evidence supports, contradicts, or is inconclusive regarding the answer, and flags missing information to guide the next query.
5. Short-Term Memory ($M$): An append-only buffer records the trace of interactions $(q_i, E_i, r_i)$ across iterations, conditioning future query refinement and providing the full context for the final judgment.
6. Final Judgment ($J$): After a fixed budget of $N$ iterations (default $N=3$), the judge synthesizes the full memory $M_N$ to produce a binary verdict ($v \in \{0, 1\}$) and a natural language rationale ($\rho$).

The process is iterative: the reflection module explicitly drives query refinement, allowing the system to correct course if initial searches yield insufficient or irrelevant evidence.

Key Contributions

1. A Reference-Free Framework: SAGE eliminates the need for costly human-annotated ground truth by dynamically generating output-specific web queries to verify facts.
2. Iterative Evidence Synthesis: Unlike prior search-augmented evaluators that typically perform a single-pass retrieval, SAGE employs a "summarize–reflect–refine" loop. This iterative process allows the system to adaptively target information gaps and resolve ambiguities.
3. Grounding in Verifiable Data: By relying on retrieved external evidence rather than parametric knowledge, SAGE mitigates hallucinations, knowledge staleness, and length biases inherent in unguided LLM judges.
4. Scalability and Cost-Efficiency: The framework offers a scalable alternative to human annotation, with costs orders of magnitude lower than manual evaluation while maintaining high alignment with human judgments.

Experimental Results

The authors evaluated SAGE on five free-form QA benchmarks (AmbigQA, HotpotQA, TriviaQA, Natural Questions, and FreshQA) using various candidate and judge models (including GPT-3.5, GPT-4o, Gemini, and smaller open-weight models like Mistral 7B and Qwen).

• Agreement with Reference-Based Metrics: SAGE significantly outperforms parametric-only judges. For instance, on AmbigQA, a GPT-3.5 judge using SAGE achieved 0.80 accuracy against RefGPT (a reference-based baseline), compared to 0.67 without external evidence.
• Alignment with Human Evaluators: On datasets with human annotations (AmbigQA and HotpotQA), SAGE achieved substantial to perfect agreement with human majority votes.
  • Cohen's $\kappa$: SAGE reduced the gap between LLM judges and humans. For example, on HotpotQA, a GPT-4o judge without search achieved $\kappa = 0.38$, whereas the same judge under SAGE reached $\kappa = 0.70$.
  • Macro-F1: SAGE consistently delivered the highest Macro-F1 scores across models and tasks, correcting the tendency of reference-free judges to over-estimate correctness (high recall, low precision).
• Detection of Outdated Knowledge: On FreshQA, SAGE successfully identified and rejected outdated answers (e.g., identifying the correct current location of EMNLP 2025 as Suzhou, China, whereas candidates often provided pre-training cutoff locations).
• Ablation Studies:
  • Iterations: Performance improves with iterations, with 3 iterations offering the best trade-off between performance and cost.
  • Query Generation: Removing the query refinement step (using the input question directly) significantly degraded performance, particularly on ambiguous datasets like AmbigQA.
  • Search Engine Robustness: SAGE's gains persisted across different search engines (Serper, Brave, Tavily), though absolute performance varied slightly based on retrieval quality.
  • Judge Capability: While SAGE improves performance across all judge sizes, more capable judges (e.g., GPT-4o) yield higher agreement with humans than smaller models (e.g., Mistral 7B), though even small models benefit significantly from the search augmentation.

Significance and Claims

The paper claims that SAGE represents a significant advancement in the automated evaluation of LLMs for objective tasks. By shifting from static reference matching or ungrounded parametric judgment to an active, evidence-based verification process, SAGE addresses the core limitations of current evaluation protocols.

The authors assert that SAGE provides a scalable, adaptive, and cost-effective alternative for evaluating the factuality of LLMs. It is particularly effective for:

• Free-form QA: Where answers are open-ended and lack a single definitive ground truth.
• Dynamic Topics: Where information evolves rapidly and pre-trained knowledge is insufficient.
• Resource-Constrained Settings: Offering a viable alternative to human annotation without sacrificing the reliability required for objective correctness.

The paper concludes that while SAGE is not a perfect substitute for human review in high-stakes scenarios, it effectively bridges the gap between the scalability of automated evaluation and the reliability of human fact-checking, establishing a new standard for reference-free evaluation of LLM factuality.