Technical Summary: Anticipatory Evaluation of Language Models

Problem Statement

The progress of large language models (LLMs) is increasingly constrained by an evaluation bottleneck. Traditional benchmarking requires building datasets, running experiments, and iterating only after results are obtained. This paper investigates whether evaluation outcomes can be forecast before any experiments are conducted. Specifically, the authors define a text-only performance forecasting task: given only a natural-language description of a proposed task, dataset provenance, and experimental configuration (without access to dataset instances or seed measurements), predict the expected performance metric normalized to a 0–100 scale.

This setting differs from prior work in performance prediction, which typically relies on partial experimental results, small pilot runs, or structured tables of observed data. The goal here is to support early-stage experiment planning by estimating difficulty and discriminative power prior to annotation or resource commitment.

Methodology

Data Curation: The PRECOG Corpus

To enable systematic study, the authors curated PRECOG, a corpus of 2,290 description–performance pairs derived from 1,519 arXiv papers.

• Source: Experimental records were harvested from the scientific literature, pairing a "result paper" (reporting scores) with a "dataset paper" (describing the task).
• Construction: An LLM (GPT-5-mini) extracted schema-aligned, identity-masked descriptions ($x_i$) summarizing the task definition, data provenance, evaluation protocols, difficulty cues, and prompting strategies.
• Target: Reported scores were normalized to a 0–100 scale ($y_i$) to handle heterogeneous metrics (e.g., Accuracy, F1, BLEU, ROUGE, EM).
• Leakage Control: The corpus includes a test split of papers published after the knowledge cutoff of the evaluated models (2025), ensuring predictions are not based on memorized training data. A streaming setting was also established where models predict on newly released papers before they are indexed by search engines.

Task Formulation

The problem is framed as a regression task where a model $f_\theta$ predicts $\hat{y}_i \in [0, 100]$ given a description $x_i$.

• Input: A free-form text specification of the benchmark design and experimental setup.
• Output: A numeric performance estimate and a self-reported confidence score.
• Retrieval: Models can optionally use a tool-augmented approach (ReAct-style) to query a time-bounded literature corpus for evidence, though the primary signal is derived from the description and the model's internal knowledge.

Evaluation Metrics

Performance is measured using:

1. Mean Absolute Error (MAE): Lower is better; measures absolute accuracy across heterogeneous metrics.
2. Pearson Correlation ($r$): Higher is better; assesses whether predictions preserve the relative ordering of tasks.

Key Results

Model Performance

• Feasibility: Text-only forecasting is challenging but feasible. GPT-5 achieved the best overall performance with an MAE of 14.6 on the leakage-controlled 2025 test split, significantly outperforming text-embedding baselines (e.g., E5-Mistral + XGBoost, which scored ~21.4 MAE).
• Reasoning Capability: Enabling reasoning modes (Chain-of-Thought) improved performance. GPT-5 in reasoning mode outperformed its non-reasoning variant and open-source counterparts like Qwen3-32B (reasoning) and Qwen2.5-72B.
• Confidence Calibration: Self-reported confidence is a useful signal. When filtering for high-confidence predictions, GPT-5's MAE dropped to 9.9, indicating that the model can reliably identify when its forecasts are likely to be accurate.
• Streaming Evaluation: In a real-time setting predicting on newly released arXiv papers before search indexing, GPT-5 achieved an MAE of 14.0 and a Pearson correlation of 0.78, closely matching offline performance.

Human Comparison

In a study involving NLP experts (graduate students and a professor), human forecasters achieved an MAE of 19.6 on a subset of accuracy predictions. GPT-5 achieved 13.6 on the same subset, suggesting that current reasoning LLMs can outperform human experts in this specific zero-shot forecasting task.

Ablation and Analysis

• Metric Heterogeneity: Prediction error varies by metric. Accuracy and Exact Match (EM) were predicted with moderate error, while Recall and Precision showed higher errors, likely due to fewer training examples for these specific metrics in the corpus.
• Dataset Transfer: Performance remained stable across "established" datasets (where the result paper differs from the dataset paper) and "new" datasets, suggesting the predictors generalize well to unseen benchmarks.
• Retrieval Impact: While retrieval tools improved GPT-5's performance slightly, they did not significantly help Qwen3-32B. The authors note that most forecasting signal comes from reasoning over the curated descriptions and the model's pretraining, rather than retrieving specific source papers.

Significance and Claims

The paper claims that anticipatory evaluation offers a practical, lightweight planning tool for the machine learning community. By predicting difficulty upfront, researchers can:

• Prioritize Experiments: Identify which benchmark designs are likely to be too easy (saturated) or too hard (degenerate) before committing to large-scale annotation.
• Optimize Resources: Guide the design of pilot studies and baseline selection more efficiently.
• Inform Design: Answer "what-if" questions regarding design choices (e.g., the impact of increasing multiple-choice options from 5 to 20) without running initial experiments.

The authors emphasize that this approach is not intended to replace empirical benchmarking. Instead, it serves as an initial step toward open-ended, data-efficient experimental design, supporting smarter resource allocation in an era where evaluation costs are rising. The release of the PRECOG corpus and associated code aims to foster further research into predictive evaluation methods.