Deployment-complete benchmarking

Technical Summary: Deployment-Complete Benchmarking

Problem Statement
Current machine learning benchmarks increasingly guide critical deployment decisions, such as model procurement, material screening, and compound triage. However, a fundamental disconnect exists: a benchmark score provides evidence only for the specific response it records (e.g., prediction accuracy on a test set), not necessarily for the deployment action required (e.g., safety, robustness, or a specific physical threshold). A model may perform perfectly on a benchmark while the action derived from that score remains underdetermined because the deployment depends on responses outside the benchmark's scope (e.g., toxicity, stability, or out-of-distribution robustness). This paper argues that standard metrics like accuracy, calibration, and uncertainty do not guarantee that a benchmark score supports a specific deployment claim.

Methodology
The authors introduce deployment-complete benchmarking, a framework that treats the transition from benchmark evidence to deployment action as a mathematical factorization problem.

• Formal Definitions:

  • Let $S$ be the space of candidates, $Z$ the space of recorded evidence (benchmark scores), and $A$ the space of deployment actions.
  • A benchmark records an evidence map $E: S \to Z$. A deployment claim is an action $D: S \to A$.
  • Claim-Completeness: A benchmark is complete for a claim $D$ if and only if $D$ is constant on every evidence fiber $E^{-1}(z)$. Equivalently, there must exist a map $\phi: Z \to A$ such that $D = \phi \circ E$.
  • Mixed Fibers: If a fiber $E^{-1}(z)$ contains candidates requiring different deployment actions (e.g., one candidate should be deployed, another rejected), it is a "mixed fiber." This serves as a mathematical witness that the reported evidence is insufficient to determine the action.

• Diagnostic Tools:

  • Mixed-Fiber Audit: A diagnostic that counts the fraction of candidates residing in mixed fibers.
  • Completion Curves: These quantify the cost (in terms of additional response probes or budget) required to resolve ambiguity. They plot the "certified fraction" (candidates with pure fibers) against the "response budget."
  • Certification Types: The paper distinguishes between exact certification (finite sets with known labels), calibration certification (empirical risk on held-out data), and predicted certification (interval bounds based on modeling assumptions).

• Experimental Design:

  • Controlled Response Spaces: Synthetic experiments where the geometry of the response space is known, allowing precise calculation of "residual size" (the component of the deployment response invisible to the benchmark).
  • Public Audits: Analysis of existing datasets (Tox21, Matbench, JARVIS, Vision) to measure the prevalence of mixed fibers in real-world benchmarks.
  • Held-out Replays: Operational simulations where a "certify-then-acquire" workflow is tested. Candidates with pure fibers are decided immediately; those in mixed fibers are sent for additional measurement (acquisition).

Key Results

1. Benchmark Accuracy $\neq$ Deployment Completeness:

   • In controlled experiments, benchmark-channel conformal prediction achieved 94.98% coverage on measured labels but only 10.07% on the unmeasured deployment channel.
   • Even with zero benchmark error (perfect prediction on the benchmark), deployment coverage collapsed (to 45.4% at a specific residual size) because the deployment direction had a component orthogonal to the benchmark span.
   • Leaderboard Inversion: The model with the best benchmark Mean Absolute Error (MAE) was often the worst at certifying deployment candidates. In some runs, the best-MAE model certified only 6% of its top candidates, while a model with worse MAE certified 100% because its response span covered the deployment probe.

2. Systemic Incompleteness in Public Benchmarks:

   • Tox21: 97.9% of fibers (groups of compounds with identical 7-assay profiles) were mixed regarding the SR-p53 deployment endpoint.
   • Matbench & JARVIS: In the main 20-quantile audits, the median certifiable fraction for stability and band-gap claims was 0%.
   • Vision: Clean prediction fibers left 22.7% of corruption-robustness claims ambiguous.

3. Operational Impact of Completion-Aware Policies:

   • Reduced False Decisions: In Tox21 held-out replays, a "certify-then-acquire" workflow reduced empirical false decisions from 1.19% (benchmark-majority policy) to 0.027%. In JARVIS, false band-gap decisions dropped from 20.3% to 0.128%.
   • Model Selection: Completion-aware workflows selected different models than benchmark-MAE workflows. In a representative JARVIS split, the benchmark-MAE model made 613 false band-gap decisions, whereas the response-selected workflow made none.
   • Probe Selection: "Response-rank" and "residual-greedy" policies successfully identified deployment-relevant probes (e.g., specific toxicity assays or band-gap calculations) that reduced ambiguity, outperforming uncertainty-based or diversity-based acquisition strategies.

Significance and Claims
The paper claims that deployment-ready benchmarks must shift their reporting standard from "scores alone" to an auditable evidence package. This package should include:

1. The score.
2. The evidence map used.
3. The specific action the score is meant to support.
4. The fraction of candidates whose action is already determined (pure fibers).
5. The ambiguity fraction (mixed fibers).
6. The completion cost (budget required to resolve ambiguity).

The authors argue that accuracy, calibration, and uncertainty estimation remain valuable for ranking and reliability but answer different questions. Deployment completeness specifically asks whether the evidence package determines the action. The framework provides a diagnostic for when evidence is insufficient, a limit on what statistics can repair missing information (none, if they rely solely on the same evidence), and a design rule for acquiring the necessary information to support deployment decisions.

Limitations and Scope
The paper does not claim that all benchmarks are useless, but rather that they are often incomplete for specific deployment claims. The "certify-then-acquire" workflow treats ambiguous candidates not as failures, but as requests for further measurement. The framework is evidence-relative; adding descriptors, constraints, or new measurements changes the evidence map and can lower completion costs. The results are presented as empirical risk evaluations on held-out data and mathematical diagnostics on finite sets, rather than formal population guarantees for unobserved distributions.