Token Arena: A Continuous Benchmark Unifying Energy and Cognition in AI Inference

1. Problem Statement

The paper identifies a critical gap in the current landscape of Large Language Model (LLM) evaluation. While public benchmarks compare models at the model or provider level, real-world deployment decisions are made at the endpoint level. An endpoint is defined as a specific tuple of (provider, model, SKU, precision, decoding strategy, region).

The authors argue that current leaderboards suffer from three structural opacities:

1. Endpoint Identity: The same model name (e.g., Llama 3.3 70B) served by different providers or in different SKUs (e.g., FP8 vs. BF16, Turbo vs. Reference) exhibits significant, often undisclosed, variations in behavior, speed, and quality. Aggregating at the provider level hides these differences.
2. Workload Identity: Standard pricing benchmarks often assume a static 3:1 input-to-output ratio (typical for chat). However, production workloads vary wildly (e.g., RAG at 20:1, reasoning at 1:5). A 3:1 pricing model fails to reflect the true cost of diverse production tasks.
3. Reliability and Energy Invisibility: Existing metrics focus on median throughput or uptime, ignoring the "long tail" of latency that impacts production. Furthermore, no major public leaderboard tracks the energy cost per correct answer, despite energy becoming a binding constraint for the industry.

2. Methodology: The Token Arena Framework

Token Arena introduces a continuous, endpoint-level benchmarking framework that unifies energy and cognition.

A. Unit of Analysis: The Endpoint

The framework treats the endpoint (not the model) as the primary unit of analysis. It measures 78 distinct endpoints across 12 model families and 33 providers.

B. Core Metrics & Composite Scores

The framework synthesizes five core axes into three headline composite metrics:

1. Five Core Factors:

   • Output Speed ($S$): Tokens per second (streaming).
   • Time to First Token ($T$): Latency (P50, P95, P99), including Time to First Visible Token for reasoning models.
   • Blended Price ($P_\pi$): Workload-aware pricing based on configurable input:output ratios (e.g., 20:1 for RAG).
   • Effective Context: The maximum context length retaining $\ge$90% accuracy on long-context retrieval tasks.
   • Quality ($Q$): Performance on a live suite of benchmarks (MMLU-Pro, AIME 2025, HumanEval+, etc.) run directly against the endpoint.
   • Endpoint Fidelity ($R$): A measure of output-distribution similarity to a first-party reference to detect silent quantization or weight substitution.

2. Headline Composites:
   The framework calculates two primary efficiency metrics and one fidelity metric:

   • Joules per Correct Answer (JCA): $JCA = \frac{j_e \cdot T_e}{A_e}$ (Energy cost per successful task).
   • Dollars per Correct Answer (CCA): $CCA = \frac{p_e \cdot T_e}{A_e}$ (Monetary cost per successful task).
   • Endpoint Fidelity: A score based on symmetrized KL divergence between the endpoint's output distribution and a high-fidelity reference.

3. Energy Modeling:
   Since direct metering is impossible, energy is modeled using public data: Thermal Design Power (TDP), utilization rates, Power Usage Effectiveness (PUE), and regional grid intensity (ElectricityMaps). The model biases assumptions toward higher energy consumption to provide a conservative lower bound on variance.

C. Data Pipeline

• Probing: Continuous, geographically distributed probes (US-East, EU-Central, APAC) measuring latency, throughput, and error rates.
• Evaluation: Daily and weekly runs of uncontaminated eval suites (e.g., AIME 2025, LiveCodeBench) against all endpoints.
• Fingerprinting: Comparing output distributions against a "first-party" reference to detect undisclosed quantization (e.g., FP8 vs. BF16).

3. Key Contributions

1. Measurement Framework: A novel methodology defining the endpoint as the unit of analysis, with headline metrics focused on efficiency (Joules/$ per correct answer) rather than raw throughput.
2. Continuous Data Plane: A live system covering 78 endpoints, featuring rotating eval splits to prevent contamination and fingerprinting to detect silent model drift.
3. Empirical Analyses:
   • Endpoint Divergence: Demonstrating that the same model on different endpoints can vary by up to 12.5 points in accuracy (on AIME 2025) and 6.2x in energy efficiency.
   • Fingerprint Detection: Successfully identifying undisclosed quantization (FP8 Turbo SKUs) via output distribution divergence (99.5% fidelity threshold) before quality degradation is obvious to users.
   • Workload-Aware Re-ranking: Showing that changing the input:output ratio (e.g., from Chat 3:1 to RAG 20:1) completely reshuffles the leaderboard, with only 30–40% overlap in top-10 lists.
4. Open Release: Full release of the schema, probe/eval harness, energy models, and a v1.0 leaderboard snapshot under CC BY 4.0.

4. Key Results

• Variation at the Endpoint Level: For the model gpt-oss-120B across 19 endpoints:
  • Throughput: Varies by 12x.
  • Price: Varies by 3.3x.
  • Accuracy (AIME 2025): Varies by 9.5 points (41.5% to 51.0%).
  • Energy Efficiency: Varies by 6.2x in Joules per correct answer.
• Quantization Detection: FP8/Turbo SKUs showed a fidelity drop to ~92% compared to BF16 references, correlating with a 5–9 point drop in math/code benchmarks, even when standard "smoke test" benchmarks (like MMLU) showed no gap.
• Workload Sensitivity:
  • Chat (3:1): Favors fast, cheap open-weight models.
  • Reasoning (1:5): Elevates frontier closed models (e.g., Claude Opus, GPT-5.5) because their high accuracy outweighs high token costs when calculating "cost per correct answer."
  • RAG (20:1): Favors providers with aggressive input pricing (e.g., DeepSeek, discount serverless providers).
• Stability: Sensitivity analysis shows the top rankings are robust to weight perturbations ($\pm$10%), with the leader of each preset remaining invariant.

5. Significance

Token Arena shifts the paradigm of AI evaluation from "Does this model solve the benchmark?" to "How efficiently does this specific endpoint convert energy and money into correct answers for a specific workload?"

• For Practitioners: It provides a data-driven method to select inference providers based on actual production constraints (latency tails, energy costs, and specific workload ratios) rather than marketing claims.
• For the Industry: It highlights the "energy bottleneck" in AI, providing the first public framework to quantify the energy cost of intelligence.
• For Transparency: It exposes "silent" practices like undisclosed quantization and model substitution, forcing providers to be more transparent about SKU differences.

The paper concludes that as AI inference becomes the dominant cost driver and energy becomes the binding constraint, evaluation must evolve to be endpoint-level, workload-aware, and energy-inclusive. Token Arena provides the methodology and infrastructure to achieve this.