Technical Summary: Grounding Generative Evaluations of Language Models in Unsupervised Document Corpora

Problem Statement

Language Models (LMs) are advancing rapidly, yet the construction of evaluation benchmarks to assess their domain-specific capabilities has not kept pace. Traditional benchmarks (e.g., MMLU) are static, public, and limited in scope; their static nature makes them susceptible to data contamination, where models encounter benchmark questions during training, inflating performance metrics. Furthermore, the human effort required to curate high-quality, expert-validated benchmarks for every emerging domain is impractical given the scale of modern models. There is a critical need for automated methods that can generate accurate, comprehensive, and diagnostically relevant evaluations grounded in authoritative, domain-specific corpora without relying solely on the internalized knowledge of the models being tested.

Methodology

The authors propose a novel automated methodology for generating synthetic benchmarks grounded in authoritative documents (e.g., textbooks, academic papers, government reports). The pipeline operates as follows:

1. Document Ingestion and Chunking: Source documents are converted to Markdown and split into context chunks. While modern LMs support long contexts, chunking ensures questions are distributed throughout the document rather than clustered in one section.
2. Topic Extraction: For each context chunk, an LM extracts relevant topics suitable for generating difficult, expert-level questions. This step leverages the same LM used for generation, with empirical findings suggesting that LM-based topic generation effectively handles de-duplication.
3. Question and Answer Generation: Given a context chunk and a specific topic, the generating LM constructs a high-quality evaluation question. The model is prompted to produce:
   • The question itself.
   • The correct answer.
   • A justification/explanation for the answer (used internally to guard against incorrect generation but not included in the final benchmark).
   • The system supports both Multiple Choice Questions (MCQ) and Open-Ended (OE) formats.
4. Evaluation Framework: The generated benchmarks are evaluated using the Inspect-AI framework.
   • MCQs: Graded by checking the selected option against the ground truth.
   • OE Questions: Graded by a separate "LM-Judge" that compares the candidate model's output to the ground truth answer to determine correctness.
5. Validation Strategy: To validate the synthetic benchmarks, the authors compare them against human-curated benchmarks derived from existing NLP datasets (e.g., SQuAD, HotpotQA, PubMedQA). They also replicate subsets of the MMLU-Pro benchmark using Creative Commons textbooks as grounding documents.

Key Contributions and Results

1. Automated Benchmark Generation

The paper demonstrates that LMs can reliably extract information from documents and generate diagnostically relevant questions. The approach reduces human effort significantly, enabling rapid adaptation to new domains by using only grounding documents as input.

2. Correlation with Human-Curated Benchmarks

The study validates the synthetic approach by comparing its performance metrics against human-written questions.

• Rank Correlation: The ensemble of generating models achieved a Spearman rank correlation of 0.91 with human-curated benchmarks, indicating that the synthetic benchmarks preserve the relative ordering of model capabilities effectively.
• Accuracy Correlation: The ensemble achieved a Pearson accuracy correlation of 0.74 (0.82 for specific model configurations) with human benchmarks, suggesting strong alignment in absolute performance measurement.
• MMLU-Pro Replication: When replicating MMLU-Pro subsets using OpenStax textbooks, the generative approach achieved a Pearson accuracy correlation of 0.78 and a Spearman rank correlation of 0.83 against the original human benchmarks.

3. Analysis of Question Quality and Failure Modes

The authors characterized the quality of generated questions using LM-based metrics (clarity, groundedness, answer correctness).

• Grounding Failures: Two primary failure modes were identified: (1) hallucinating content absent from the context (e.g., phi-4), and (2) using parametric knowledge to expand on context information (e.g., o4-mini), which can lead to questions that are not fully grounded in the provided text.
• Question Length and Over-Performance: A significant finding is that longer, more detailed synthetic questions can artificially inflate model performance. The authors hypothesize that excessive detail in the question places the evaluated model in a favorable "state of mind" or representation space, making it easier to retrieve memorized patterns. This effect was particularly pronounced with models like gpt-4.1-mini, which generated overly easy questions, resulting in low correlation scores (Pearson 0.276).
• Model Self-Preference: The study found no evidence of "self-preference," where models perform better on questions they generated themselves. Performance differences were driven by the quality of the generating model rather than the identity of the generating vs. evaluating model.

4. Domain-Specific Application

The methodology was applied to three recent documents, including two published after the knowledge cutoffs of the evaluated models. In these open-ended (OE) evaluations, Gemma-3 models (12B and 27B parameters) surprisingly outperformed other models, demonstrating the approach's utility in assessing capabilities on post-training data.

Significance and Claims

The paper claims that this generative benchmarking approach offers a scalable solution to the evaluation bottleneck in AI. By grounding evaluations in specific document populations, practitioners can create targeted benchmarks that reveal domain-specific strengths and weaknesses not apparent in general-purpose evaluations.

The authors modestly note that while the approach is effective for many applications, it is not yet suitable for evaluating the absolute frontier of model capability due to limitations in question nuance and the reliance on text-only grounding (excluding tables, charts, and images). They identify a "noise floor" of approximately 7.5% in generative benchmarking (based on human review of MMLU-Pro replications), primarily caused by grounding failures where the generator relies on external knowledge rather than the provided text.

Ultimately, the work provides a framework for automating the creation of both MCQ and OE synthetic data, enabling deeper exploration of LM capabilities in complex knowledge work while mitigating the risks of data contamination inherent in static public benchmarks.