💻 computer science

Mind the Gap: Explicit Cross-Benchmark Calibrations for Estimating Missing LLM Evaluation Scores

The paper introduces "benchgap," a pipeline that estimates missing large language model evaluation scores by explicitly calibrating benchmarks within capability groups using monotone curve fitting and multivariate mappings, achieving significantly lower error rates than matrix completion baselines while quantifying evaluation setup discrepancies known as the "harness tax."

Original authors: Patrik P. Süli, György Eigner, Roland Hollós

Published 2026-10-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Patrik P. Süli, György Eigner, Roland Hollós

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, new language models are being released faster than they can be tested. These models are designed to solve problems, write code, and reason through complex tasks, but their true capabilities are only known when they are put through standardized tests called benchmarks. Think of a benchmark as a specific exam paper; a model's score on that paper is a snapshot of its performance under strict conditions. However, the landscape of these exams is vast and fragmented. Different research labs and companies choose different sets of tests to showcase their models, and evaluation services run their own subsets of models on their own schedules. The result is a massive, incomplete picture where most models have been tested on only a handful of exams, leaving the vast majority of potential scores missing. This gap makes it difficult to compare models fairly or to understand their full range of abilities, as analysts are forced to look at tables where more than ninety percent of the data is empty.

To solve this puzzle, researchers Patrik P. Süli, György Eigner, and Roland Hollós have developed a method called benchgap, a system designed to estimate the missing scores without actually running the tests. Instead of trying to predict a model's performance from scratch or using a single global formula for all models, this system acts like a translator between specific exams. It looks at models that have already taken two different tests and maps the relationship between their scores. If a model scores high on one exam, the system uses the historical data of other models to estimate what that same model would likely score on a different, untested exam. This process is not a guess; it is a precise calibration that fits a smooth curve to the data, ensuring that the estimate respects the natural limits of the tests and the capabilities of the models. The system is strict about its rules: it never allows a model to be estimated on a test it is fundamentally unsuited for, such as predicting a vision test score based on text-only results, and it refuses to make estimates that stretch too far beyond the range of data it has seen.

The researchers tested this approach on a dataset containing 401 different models and 146 versions of benchmarks. They found that the system could successfully fill in 71 percent of the missing scores with a high degree of accuracy. When they compared their estimates to the actual, measured scores that were hidden away for testing, the system's average error was just 4.6 percentage points. This was significantly better than a competing method that tried to fill the gaps using a broad, global pattern, which made larger errors on the same cells. The system is also transparent about its own uncertainty; every estimate it produces comes with a confidence level and a specific error margin, telling the user exactly how reliable that number is. For instance, estimates based on strong, clear relationships between tests are marked as high confidence, while those based on fewer data points or more uncertain connections are flagged as lower confidence, allowing users to weigh the information appropriately.

A major discovery in this work was the quantification of what the authors call the "harness tax." This is the disagreement that arises when the same benchmark is run by different evaluation setups. Even when two services are testing the exact same model on the exact same set of questions, the scores can differ significantly depending on the software and protocols used to run the test. The researchers found that for benchmarks involving agents that can use tools or navigate computer terminals, these differences were substantial, with scores differing by an average of 8.7 percentage points between setups. In contrast, for simpler, tool-free knowledge tests, the disagreement was much smaller, at just 2.3 percentage points. This finding highlights that a benchmark score is not just a property of the model itself, but also a property of the specific environment in which it was tested. The system accounts for this by calibrating only between tests that share the same capability group and by recording the source of every score, ensuring that the estimates reflect the reality of how these tests are conducted.

The system operates as a living pipeline that updates daily, pulling in new scores as they are published and recalculating the estimates. It does not rely on hidden training data or the internal architecture of the models, making it applicable to any model, including those that are closed-source and only release their final scores. By filling in the gaps with explicit, traceable calibrations, the system provides a more complete and honest view of the artificial intelligence landscape. It acknowledges that while no estimate is perfect, a carefully constructed prediction based on real-world data is far more useful than a blank space. The result is a tool that helps researchers, developers, and the public navigate the complex world of model evaluation with greater clarity, turning a sparse and confusing matrix of data into a coherent map of capabilities.

Technical Summary: Mind the Gap: Explicit Cross-Benchmark Calibrations for Estimating Missing LLM Evaluation Scores

Problem Statement
The rapid release of Large Language Models (LLMs) outpaces the capacity for comprehensive evaluation, resulting in a sparse "model × benchmark" matrix. In a dataset comprising 401 models and 146 benchmark versions, only 8.3% of cells contain measured scores. Existing methods for predicting missing scores, such as global matrix completion or latent factor scaling laws, often operate in a latent space, treating the entire matrix as a single entity. While these methods demonstrate that scores are largely redundant, they obscure the specific provenance of predictions and may fail to respect the distinct capability groups or evaluation setups (harnesses) that define benchmark scores. Furthermore, global methods do not provide per-cell error estimates or confidence levels, making it difficult to audit individual predictions in public leaderboards.

Methodology: The benchgap Pipeline
The authors present benchgap, a pipeline that estimates missing scores using explicit, local calibrations rather than global latent factorization. The system operates on the principle that benchmark scores can be mapped to one another through monotone curves, provided they share the same capability group.

Univariate Calibrations: For every ordered pair of same-capability benchmarks with at least five shared models, the system fits six monotone curve families (Hill, linear, Michaelis–Menten, offset logistic, inverse MM, and MM + offset).
- Selection: The best curve is selected based on the lowest Leave-One-Out (LOO) cross-validated error.
- Constraints: Only rising fits are retained (discarding anti-correlated fits which often indicate insufficient data coverage). Fits must achieve $R^2 \ge 0.3$ and an LOO error $\le 15$ percentage points (pp) to be stored.
- Extrapolation: Predictions are bounded to a range of one "fitting width" ( $w = x_{max} - x_{min}$ ) beyond the observed data.
Multivariate Mappings: To improve accuracy, target benchmarks are predicted from multiple source benchmarks simultaneously.
- Feature Selection: An elastic net (with a lasso component) or a greedy forward search selects the most informative source benchmarks.
- Functional Form: The system compares a linear elastic net (with non-negative coefficients) against a multivariate Michaelis–Menten curve.
- Quality Gate: Multivariate mappings are stored only if they pass the same quality gates as univariate calibrations and demonstrate lower LOO error than the best univariate alternative.
Gap Filling and Constraints:
- No Recursion: Estimates are derived strictly from measured scores; an estimate never feeds another estimate.
- Capability Gating: Calibrations never cross capability boundaries (e.g., text benchmarks do not predict vision scores).
- Confidence Levels: Every estimate is assigned a confidence level (High, Medium, Low) based on its LOO error and three potential warnings: extrapolation, small sample size ( $<8$ models), or uninformative fit ( $R^2 < 0.5$ ).
Harness Tax Quantification: The system quantifies the "harness tax"—the disagreement between different evaluation setups (sources) for the same benchmark. This is measured by comparing per-model score deltas across verified same-item-set pairs.

Key Contributions

Transparent Gap-Filling Pipeline: A system that provides traceable estimates where every prediction links to a specific source score, curve family, parameters, and error metric via an API.
Multivariate Mappings: A mechanism to combine multiple source benchmarks for prediction, utilizing feature selection to eliminate uninformative sources.
End-to-End Evaluation: A rigorous holdout test masking 15% of measured scores to validate the pipeline's published error labels against realized errors.
Harness Tax Measurement: The first study to quantify evaluation setup disagreement per model across benchmark families, distinguishing between agentic, tool-free, and protocol-layer variations.
Running System: A daily-refreshing, MIT-licensed system (benchgap.net) that publishes scores, calibrations, and provenance records via a public API.

Results

Accuracy vs. Baselines: In a 15% holdout test, benchgap filled 71% of masked cells with a Mean Absolute Error (MAE) of 4.6 pp. This is 23% lower than a rank-2 matrix completion baseline (5.9 pp MAE) on the same cells. On the 29% of cells the pipeline left empty (due to lack of valid mappings), the baseline's error was significantly higher (11.8 pp).
Confidence Calibration: Realized error correlates with confidence levels: High (3.1 pp MAE), Medium (5.1 pp), and Low (7.5 pp). However, the realized Root Mean Square Error (RMSE) exceeds the claimed LOO error labels by a factor of 1.4–1.7, indicating the labels are optimistic minima.
Harness Tax: Agentic benchmarks exhibit a high harness tax, with verified same-item-set pairs showing an average disagreement of 8.7 pp. In contrast, tool-free benchmarks show only 2.3 pp disagreement. Enabling tools on "Humanity's Last Exam" increased scores by an average of 10.0 pp across all 11 models measured both ways.
Sparse Data Performance: For models with very few measured scores (1 or 2), the pipeline fills 19–26% of remaining cells with an MAE of roughly 5.0 pp, comparable to the random-mask results.

Significance and Claims
The paper positions benchgap as a complementary approach to global latent factor methods. While global methods assume a shared structure across the entire matrix, benchgap prioritizes explicitness and auditability. By keeping calibrations local and explicit, the system ensures that every estimate can be traced to its source and that its limitations (e.g., extrapolation, small sample size) are visibly flagged.

The authors claim that this explicitness comes at the cost of coverage (filling 71% vs. 99% of cells) but yields higher accuracy on the filled cells and provides a mechanism to detect and quantify evaluation setup inconsistencies (the harness tax). The paper acknowledges that, like all score-prediction methods, it cannot detect models with capability profiles unlike those in the training set, but it makes known risk factors visible through confidence demotions and error labels. The system is presented not as a final solution but as a transparent, auditable snapshot of the current evaluation landscape, updated daily to reflect the live state of LLM benchmarking.

Technical Summary: Mind the Gap: Explicit Cross-Benchmark Calibrations for Estimating Missing LLM Evaluation Scores

More like this