Technical Summary: Two AI Metrics Diverged: Will it Make All the Difference?

Problem Statement

As AI development scales exponentially in compute, a critical question arises for governance and economics: Will the capabilities of frontier models (those with massive, exponentially growing compute budgets) permanently outstrip "meek" models (those with fixed or slowly growing compute budgets that still benefit from shared hardware and algorithmic efficiency gains)?

Existing empirical literature offers conflicting answers. Some studies suggest the performance gap will widen, leading to oligopolistic concentration of capability. Others, specifically Gundlach et al. (2025b), argue that the gap will shrink, allowing meek models to eventually "inherit the earth." This paper posits that these divergent conclusions are not necessarily contradictory but are artifacts of the specific performance metrics chosen to measure AI capabilities. The central problem is that different metrics, even when measuring ostensibly similar capabilities, possess different functional forms relative to compute, leading to fundamentally different long-term convergence properties.

Methodology

The authors employ a theoretical framework to categorize performance metrics based on their mathematical behavior under exponential scaling of effective compute.

1. Definitions:

   • Normal Performance Metric: A differentiable, weakly monotonically increasing function mapping training compute to performance.
   • Meek Metric: A metric where the performance gap between a frontier model (exponential investment growth) and a meek model (constant investment growth, but benefiting from shared efficiency gains) converges to zero as time approaches infinity.
   • Mighty Metric: A metric where the performance gap does not converge to zero (it may persist as a constant or diverge).

2. Analytical Approach:
   The paper generalizes the argument from Gundlach et al. (2025b) regarding validation loss. It analyzes the limit of the difference $P(C_{frontier}) - P(C_{meek})$ as $t \to \infty$, where $C$ represents effective compute. The authors derive tight mathematical conditions to determine whether a metric is meek or mighty based on its growth rate and boundedness.

3. Classification Framework:
   The authors construct a decision tree (Figure 1) to classify metrics:

   • If a metric is bounded (has an upper limit), it is always meek.
   • If a metric is unbounded, it is meek if and only if it grows slower than $\log \log C$ (specifically, $\lim_{C \to \infty} \frac{P(C)}{\log \log C} = 0$). Otherwise, it is mighty.

Key Contributions

1. Theoretical Classification of Metrics: The paper provides the first rigorous mathematical conditions distinguishing "meek" metrics (where gaps close) from "mighty" metrics (where gaps persist). It establishes that boundedness is a sufficient condition for meekness.
2. Reinterpretation of Common Benchmarks: The authors demonstrate that many standard metrics are "meek" only under specific interpretations, while closely related transformations of the same underlying capability are "mighty."
   • Example: Benchmark accuracy (0-100%) is bounded and thus meek. However, the underlying error rate $\epsilon$ or the "number of nines" of reliability ($\log(1/\epsilon)$) is unbounded and mighty.
   • Example: Win-rates in games are bounded (meek), but the underlying Bradley-Terry strength (ELO) is unbounded (mighty).
3. Extension to Inference Scaling: The framework is extended to inference-time compute budgets, showing that while benchmark accuracy remains meek under inference scaling, the corresponding "march of nines" (error reduction) remains mighty.
4. Policy Relevance: The paper argues that determining the "apt" metric for a specific domain is a prerequisite for effective policy. Bounded and unbounded metrics suggest opposing policy responses regarding compute controls and regulation.

Key Results

• Convergence of Bounded Metrics: For any bounded metric (e.g., classification accuracy, pass@k, validation loss), the performance gap between frontier and meek models will eventually vanish. In the long run, meek models will achieve performance indistinguishable from frontier models on these scales.
• Divergence of Unbounded Metrics: For unbounded metrics that grow faster than $\log \log C$ (e.g., ELO strength, task horizon length, error rate orders of magnitude), the gap persists or widens. Frontier models will maintain a permanent advantage.
• Sensitivity to Utility Functions: The choice between a meek and a mighty metric often depends on the user's utility function.
  • If a user cares about whether a task is completed within a fixed threshold (bounded utility), meek models suffice.
  • If a user cares about how much better the performance is (e.g., reducing error rates by orders of magnitude, extending task horizons), the capability is unbounded, and frontier models dominate.
• Positional Metrics: The paper notes that in "winner-takes-all" scenarios (positional metrics), even if the underlying capability is bounded, the reward may always go to the frontier model due to infinitesimal advantages, creating a "mighty" outcome in terms of economic reward despite "meek" underlying convergence.

Significance and Claims

The paper claims that the debate over whether AI capabilities will concentrate or diffuse is often a debate about which metric is being used, not necessarily about the underlying physics of AI scaling.

• For Governance: If the relevant societal capability is measured by a bounded metric (e.g., "can the model write a functional script?"), regulations may need to account for the eventual proliferation of these capabilities to many actors (meek models). If the capability is measured by an unbounded metric (e.g., "how efficiently can it synthesize a novel protein?"), capabilities will likely remain concentrated in the hands of wealthy actors with massive compute budgets.
• For Policy: Compute controls (export restrictions) may act as a delay rather than a permanent ceiling for capabilities measured by meek metrics, as meek models will eventually catch up via shared hardware/algorithmic progress. However, for mighty metrics, such controls could sustain a durable gap.
• For Safety: The paper highlights that dangerous capabilities often have a "threshold" nature. If a capability is bounded (meek), the threshold may eventually be crossed by many actors, not just frontier labs. Conversely, if the degree of danger scales unboundedly (e.g., speed of exploitation), the risk may remain concentrated.

The authors conclude that analysts and policymakers must move beyond naively reading convergence or divergence from a single metric. Instead, they must explicitly define the utility function of the domain to determine whether the relevant metric is meek or mighty, as this distinction dictates whether the future of AI is one of concentration or proliferation.