Technical Summary: Every Eval Ever

1. Problem Statement

AI evaluation results are currently fragmented, inconsistent, and difficult to interpret, hindering reproducibility, cross-benchmark comparison, and systematic meta-analysis. The core issues identified are:

• Format Incompatibility: Results are scattered across leaderboards, research papers, blog posts, and custom repositories in 31 distinct reporting formats.
• Metadata Omission: Critical context—such as generation parameters (temperature, stop sequences), data provenance, evaluation harness settings, and model access modes—is frequently omitted or scattered across unstructured logs.
• Reproducibility Gaps: Identical benchmark names often mask divergent evaluation settings. For instance, the same model (e.g., LLaMA 65B) can receive significantly different scores (63.7 vs. 48.8 on MMLU) depending on the evaluation harness used, yet these discrepancies are often invisible without deep forensic analysis.
• Lack of Standardization: Unlike datasets (standardized by Datasheets for Datasets, Croissant) or models (Model Cards), evaluation results lack a unified schema to capture the runtime context necessary to determine if two scores are comparable.

2. Methodology: The Every Eval Ever (EEE) Framework

The authors propose Every Eval Ever (EEE), a community-crowdsourced repository and a unifying schema designed to standardize AI evaluation reporting. The methodology consists of three primary components:

A. The Unifying Schema

EEE utilizes a source-agnostic, versioned JSON schema that captures the full context of an evaluation run. The schema is modular, organized into five metadata blocks:

1. Source Metadata: Identifies the producer (developer, third-party, self) and origin (paper, leaderboard, local run).
2. Model Information: Records the model identifier and access mode (e.g., hosted API vs. local inference engine like vLLM), addressing hidden confounds where different providers yield different results.
3. Generation Configuration: Captures first-class fields for parameters like temperature, number of samples, and stop sequences. Missing parameters are explicitly recorded as omitted rather than silently defaulted.
4. Evaluation Results: Stores metrics with explicit semantics (score direction, type, range) to prevent ambiguity (e.g., distinguishing between "lower is better" toxicity scores and "higher is better" pass@1 scores).
5. Instance-Level Data: An optional companion file (_samples.jsonl) storing per-instance outputs, prompts, references, and scores. This supports single-turn, multi-turn, and agentic interaction types (including tool-call traces).

Design Principles:

• Partial Record Acceptance: Required fields are minimal to include existing results that would otherwise be excluded by strict standards.
• Run Identity Preservation: Each run receives a UUID rather than a canonical fingerprint, allowing conflicting or underspecified runs to remain visible for later deduplication.
• Separation of Concerns: Aggregate scores are separated from instance-level data to enable granular reanalysis.

B. Converters and Validation Pipeline

To lower the barrier to entry, EEE provides:

• Automatic Converters: Tools to parse outputs from major evaluation harnesses (HELM, lm-eval-harness, Inspect AI) and scrape leaderboards into the EEE schema.
• Validation: A Pydantic-based validation pipeline (accessible via CLI and GitHub Bot) that enforces schema compliance, checks data types, and flags duplicate records before data is merged into the repository.

C. Community Governance

EEE operates as a public good with a governance model involving core maintainers, contributors, and community reviewers. Records are immutable once accepted; corrections are handled via explicit retraction and correction mechanisms to preserve reproducibility.

3. Key Contributions

The paper outlines four specific contributions:

1. A Shared Schema: A community-governed, versioned JSON schema (with an optional instance-level companion) that standardizes source provenance, model access, generation config, and metric semantics.
2. Conversion Tooling: Automatic converters for popular frameworks and a validation pipeline ensuring schema compliance at the point of contribution.
3. Crowdsourced Repository: A Hugging Face-hosted database (evaleval/EEE_datastore) containing 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats (as of May 2026).
4. Enabling Meta-Analysis: The unified format enables research previously infeasible due to data fragmentation, including cost-accuracy tradeoff analysis, perplexity version dependency detection, reproducibility forensics, and Item Response Theory (IRT) meta-analysis.

4. Results and Empirical Findings

The authors demonstrate the utility of EEE through four case studies:

• Cost–Accuracy Tradeoffs in Agentic Evaluation: By aggregating metadata on cost and time, EEE reveals that scaffold choices (e.g., Codex vs. OpenClaw) can significantly alter costs without changing accuracy, and that scaffold effects are dependent on the model backbone.
• Version-Dependent Perplexity: The schema exposes that "perplexity" is ambiguous; different implementations (e.g., GPTQ scripts vs. LM Evaluation Harness) normalize loss differently (tokens vs. words vs. bytes), leading to non-comparable scores for the same model. EEE makes these normalization conventions explicit.
• Reproducibility Forensics: By converting official HELM records and local reproductions to EEE, the authors identified specific discrepancies, including row-order changes in data processing, empty completions in official records, and stochastic sampling variations.
• Meta-Analysis via IRT: Using instance-level data, the authors applied Item Response Theory to compare datasets like GPQA Diamond and Wordle Arena, revealing differences in item difficulty distributions and model ability estimates that are invisible at the aggregate score level.

Data Statistics:
The repository currently spans over 200,000 aggregated results. Analysis shows a long-tail distribution of benchmarks and a heavy concentration on US-based models (particularly GPT variants). Crucially, critical metadata like the inference platform is missing in 98% of rows in the raw data, highlighting the necessity of the schema to capture these details.

5. Significance and Claims

The paper claims that EEE addresses a critical gap in the AI ecosystem by providing the "common language" for evaluation reporting. Its significance lies in:

• Reusability: Transforming static, isolated scores into reusable, context-rich data.
• Transparency: Making hidden confounds (like inference engine differences or generation parameters) visible and auditable.
• Scalability: Enabling large-scale meta-analyses that were previously impossible due to the manual effort required to wrangle heterogeneous data.
• Governance: Establishing a model for community-driven standardization that preserves immutability while allowing for corrections.

The authors acknowledge limitations, noting that coverage is currently strongest for text-based, single-model evaluations, and that the schema's value is contingent on broad community adoption. They do not claim to replace evaluation harnesses but rather to sit above them as a translation and aggregation layer.