DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation

Technical Summary: DoGMaTiQ – Automated Generation of Question-and-Answer Nuggets for Report Evaluation

Problem Statement
The evaluation of long-form, citation-backed reports generated by Retrieval-Augmented Generation (RAG) systems has become a critical challenge, particularly in cross-lingual settings where user queries are in English but source documents are multilingual. Traditional evaluation frameworks rely on "nuggets"—atomic units of information—to assess a report's coverage. However, existing approaches face significant limitations:

1. Manual Curation Bottleneck: Creating high-quality nugget sets requires laborious human effort, scaling poorly to new topics and multilingual collections.
2. Representation Limitations: Traditional nuggets are often short factual statements. These suffer from leniency biases, score compression, and difficulty in handling complex information needs where multiple valid answers exist or where the information need (question) must be decoupled from diverse, multilingual answers.
3. Circularity Risks: Automated evaluation systems often use the same Large Language Models (LLMs) for both generation and scoring, potentially inflating results through circular judgments.

Methodology: The DoGMaTiQ Pipeline
The authors introduce DoGMaTiQ (Document-Grounded, Merged and Top-Criteria Question-based Nuggets), a three-stage pipeline designed to automatically generate high-quality, document-grounded Question-and-Answer (QA) nugget sets. The pipeline operates on retrieved documents for a given topic:

1. Stage 1: Document-Grounded QA Generation

   • Summarization: To avoid over-indexing on specific document details, the system first prompts an LLM to summarize each retrieved document in its original language.
   • QA Generation: Using the summary and the user query, the LLM generates 1–6 English QA pairs per document. Answers are generated before questions to minimize hallucinations, ensuring every nugget is grounded in a specific source document.

2. Stage 2: Paraphrase Clustering and Refinement

   • Question Clustering: Since QA pairs are generated independently per document, semantic overlap is high. The system identifies paraphrased questions using a fine-tuned embedding model (bge-large-en-v1.5) followed by LLM verification on high-similarity candidates. These are merged into clusters via single-link clustering.
   • Refinement: For each cluster, a single "canonical" question is selected. Answer sets are validated to remove uninformative or contradictory responses. Finally, an LLM assigns an aggregation logic (OR or AND) to determine if a nugget is satisfied by any single answer or requires all answers.

3. Stage 3: Final Nugget Selection

   • The pipeline ranks the merged nuggets to select the top 20 for the final "nugget bank." Three ranking methods are evaluated:
     • Most Common: Ranks by frequency of occurrence across documents.
     • Quality Criteria (DoGMaTiQ): Represents each nugget as a vector of 19 explicit linguistic and utility criteria (e.g., fluency, clarity, criticality, goal match). A Support Vector Machine (SVM), trained on human-written nuggets as positive examples, scores and ranks candidates.
     • Sample: Random selection (used as an ablation).

Key Contributions

• QA Representation for Evaluation: The paper argues that QA pairs are superior to statement-based nuggets because they decouple information needs from diverse answers, prevent the over-weighting of duplicate statements, and enable fine-grained assessment via answer aggregation (OR/AND).
• Automated Pipeline: DoGMaTiQ provides a fully automated method to generate nugget sets that are grounded in source documents, eliminating the need for manual curation.
• Robustness to Circularity: By grounding nuggets in source documents and utilizing a different LLM family for generation (Claude) versus the generation models being evaluated (often Llama-based), the system mitigates the risk of circular judgments.
• Integration with Auto-ARGUE: The generated nuggets are integrated with the Auto-ARGUE framework to enable fully automatic report evaluation.

Experimental Results
The authors evaluated DoGMaTiQ on two cross-lingual TREC shared tasks: NeuCLIR (2024) and RAGTIME (2025).

• Correlation with Human Judgments:
  • On the RAGTIME track, system rankings based on DoGMaTiQ-generated nuggets showed strong rank correlations with rankings based on human-written nuggets (Spearman's $\rho = 0.899$, Kendall's $\tau = 0.734$).
  • When compared to fully manual judgments (both nugget creation and alignment), DoGMaTiQ achieved a moderate but significant correlation ($\tau = 0.505$), outperforming baseline automated methods like GINGER and random sampling.
• Comparison to Baselines:
  • DoGMaTiQ significantly outperformed the GINGER system (which uses statement-based nuggets) and random sampling variants.
  • The "Quality Criteria" selection method (SVM) proved superior to simple frequency-based ("Most Common") or random selection methods.
• Robustness Analysis:
  • Outlier Systems: The system rankings induced by DoGMaTiQ remained stable even when evaluating a subset of high-priority systems, whereas other methods showed inflated correlations due to the presence of many low-performing systems.
  • Circularity: Experiments revealed that using the same LLM family for generation and evaluation (e.g., Llama for both) led to inflated scores for systems using that same family (Crucible), confirming the importance of decoupling generation and evaluation models.

Significance and Claims
The paper claims that DoGMaTiQ represents a significant step toward scalable, fully automatic evaluation of RAG-based report generation. By shifting from statement-based to QA-based nuggets and employing a principled, multi-stage filtering process, the system achieves rankings that closely mirror human evaluations without requiring manual nugget curation.

The authors maintain a modest tone, emphasizing that DoGMaTiQ is intended to complement, not replace, expert human effort. They note that while correlations are strong, per-topic scores still exhibit variance due to the inherent subjectivity of information needs. The work addresses the urgent need for evaluation tools as generative AI reshapes information consumption, providing a mechanism to assess how well systems satisfy information needs in complex, multilingual environments. The code and artifacts are publicly released to facilitate future research.