Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents

Title: Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents

Problem Statement
The evaluation of Large Language Model (LLM) agents has outgrown the limitations of static, aggregate-score leaderboards. While agents today perform complex tasks involving planning, tool calling, multi-turn artifact reuse, and inter-agent coordination, they are currently ranked by single or aggregate metrics inherited from single-shot benchmarks. The authors argue that this approach systematically underspecifies the dimensions relevant to deployed agents, leading to "rank instability." Empirical evidence from a recent 149-team competition (Patel et al., 2026) demonstrates that rankings derived from public leaderboards often fail to transfer to hidden, out-of-distribution (OOD) evaluation sets. For instance, the Spearman correlation between public and hidden rankings was statistically indistinguishable from zero ($\rho = -0.13$) on the execution track, indicating that high in-sample scores do not reliably predict out-of-sample performance. Furthermore, the reliance on "LLM-as-Judge" scoring introduces reflexivity, where the leaderboard measures the biases of the judge model rather than the system's true capability, and lacks an anchor against judge drift.

Methodology
This paper is a position paper grounded in synthesis rather than new controlled experiments. The methodology involves:

1. Aggregation of Evidence: The authors consolidate findings from seven existing agent benchmarks (including AssetOpsBench, MCP-Bench, TaskBench, and Exgentic) and fourteen parallel implementation studies conducted on the AssetOpsBench framework.
2. Architectural Sensitivity Analysis: The fourteen studies, conducted by independent teams, modified specific architectural variables (e.g., reasoning modes, retrieval strategies, orchestration patterns, infrastructure) while holding others constant. The authors treat the convergence of findings across these diverse implementations as evidence of "convergent architectural sensitivity."
3. Synthesis of Dimensions: The authors map the findings from these studies and benchmarks into a proposed "twelve-tier measurement apparatus." This apparatus is designed to expose orthogonal dimensions that current leaderboards collapse into single scores.
4. Proposed Ranking Criterion: The paper proposes shifting the ranking criterion from "in-sample mean" to "predictive validity," defined as the correlation between in-sample rank and out-of-sample rank.

Key Contributions
The paper makes five primary contributions:

1. A Twelve-Tier Measurement Apparatus: A synthesis of evaluation dimensions organized into two categories:
   • Core Capability Tiers (T1–T7): Success rates, tool-call hygiene, planning quality, capability axes, cost-efficiency Pareto, failure-mode taxonomies, and reproducibility.
   • Deployment Extension Tiers (T8–T12): Deployment infrastructure (latency decomposition, protocol overhead), multi-turn dialog (artifact reuse), reasoning-mode adaptivity, knowledge augmentation (retrieval strategies), and evidence grounding (judge-independent verification).
2. A Position Argument: The assertion that aggregate-score leaderboards fail to capture the complexity of deployed agents, and that the field requires a multi-tier apparatus ranked by predictive validity.
3. Three Falsifiable OOD Criteria: Concrete criteria to test the position:
   • Criterion A (Held-Out): Mild distributional shift via stratified splits.
   • Criterion B (Cross-Subset): Moderate shift (e.g., ranking on HVAC chillers vs. hydraulic pumps).
   • Criterion C (Adversarial): Strong shift via semantic paraphrasing, identifier renaming, time-window shifting, and distractor injection.
4. A Predictive-Validity Score Schema: A proposed composite score $PV(c) = \alpha \bar{Y}_c - \beta \sigma_{Y_{c,OOD}} - \gamma IQR(Y_c)$ that balances mean performance with out-of-sample reliability and variance.
5. A Pre-Registered Pilot Design: A roadmap for an empirical study to validate the position, including specific thresholds for falsification (e.g., if in-sample to OOD rank correlation $\rho \ge 0.85$, the position is refuted).

Results and Evidence
The paper does not present new experimental results but synthesizes convergent evidence from the fourteen implementation studies and seven benchmarks to support its claims:

• Rank Instability: Analysis of the CODS-2025 competition shows that public-to-hidden rank correlations are often weak or non-existent, particularly in execution tracks.
• Dimensional Collapse: The synthesis reveals that current benchmarks report on at most four or five of the proposed twelve tiers. For example, single-turn benchmarks miss multi-turn artifact reuse dynamics, and aggregate scores hide trade-offs between reasoning quality and latency.
• Convergent Findings: Independent teams identified the same bottlenecks and failure modes despite different starting architectures. Examples include:
  • Reasoning Modes: Enabling "extended thinking" improved clarity by 31 percentage points but increased latency by 21.5%, a trade-off invisible to aggregate scores.
  • Retrieval Strategies: Agentic multi-hop retrieval achieved ~90% accuracy but with 4.5x–10x token inflation compared to single-pass RAG (50–68% accuracy).
  • Infrastructure: MCP-stdio subprocess overhead was identified as a dominant latency floor (up to 5–6 seconds per call) across multiple studies, conflating protocol overhead with reasoning ability.
  • Judge Reflexivity: LLM-as-judge inter-rater reliability ($\alpha = 0.61$) was found to be lower than human-human reliability ($\alpha \in [0.74, 0.82]$), highlighting the need for judge-independent anchors like rule-based pipelines or DAG oracles.

Significance and Claims
The paper claims that the current paradigm of "aggregate-score leaderboards" is insufficient for guiding deployment decisions because it fails to predict out-of-distribution performance. The significance of this work lies in:

• Operationalizing Predictive Validity: Proposing a concrete, falsifiable framework for ranking agents based on their ability to generalize rather than their ability to overfit a specific benchmark set.
• Exposing Hidden Dimensions: Demonstrating that the "measurement space" of deployed agents is roughly twelve-dimensional and orthogonal, whereas current leaderboards collapse these into single metrics.
• Shifting the Field: Moving the community toward a "design science" of interactive evaluation that requires declared configurations (architecture, reasoning mode, retrieval strategy), layered presentation of results, and judge-independent verification.

The authors are explicit about the paper's limitations: it is a synthesis and position paper, not an empirical validation. The predictive validity claim is a hypothesis supported by convergent architectural sensitivity but requires the controlled randomized trial outlined in their research agenda to be confirmed. They acknowledge that the twelve-tier apparatus is currently asserted as orthogonal but not empirically tested for independence.