🤖 AI

Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

The paper introduces Messier, a unified high-resolution corpus of nearly one million standardized records spanning 30 benchmarks and diverse domains, which enables comprehensive cross-benchmark agent evaluation, reveals uneven progress across task types, and provides a foundational infrastructure for auditing and scaling AI agent capabilities.

Original authors: Stefan Krsteski, Charlotte Meyer, Guillaume Allegre, Tony O'Halloran, Alexandre Sallinen

Published 2026-07-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Stefan Krsteski, Charlotte Meyer, Guillaume Allegre, Tony O'Halloran, Alexandre Sallinen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out which video game character is the absolute best at everything. You have a character who is amazing at jumping, another who is a wizard at solving math puzzles, and a third who can navigate a maze perfectly. But here's the catch: every game measures "skill" differently. One game gives you a gold star for jumping over a single pit, while another requires you to solve a complex algebra problem to open a door, and a third demands you collect every single coin in a level without touching a single enemy. If you just look at the final score—a single number like "95%"—you might think they are all equally good. But that number hides the messy details. Did the character fail because they couldn't jump, or because they got stuck on a door? Did they get 95% because they did everything perfectly, or because they failed one tiny thing and the game counted the whole run as a zero?

This is the exact problem facing scientists who study AI agents. These are not just chatbots; they are digital workers that can use tools, write code, browse the web, and make decisions in real-time environments. Right now, there are hundreds of different tests (benchmarks) for these agents, but they all speak different languages. Some tests are strict: if you miss one step in a 10-step recipe, you get a zero. Others are more forgiving. Because of this, it's incredibly hard to compare an agent that is great at coding with one that is great at legal research. It's like trying to compare a marathon runner to a chess grandmaster using only a single "athleticism" score. To understand how AI is actually improving, we need a way to look under the hood of these scores and see exactly where the agents succeed and where they stumble.

Enter MESSIER, a massive new project that acts like a giant, universal translator for AI testing. The researchers behind MESSIER didn't just build a new test; they built a giant library that collects the results from 30 different existing tests, covering 11,891 specific tasks and 74,205 different ways of checking if an agent succeeded. They gathered data on 714 different AI agents and even ran new, fresh tests on six difficult areas where data was missing, like legal reviews and quantum circuit design. In total, they organized nearly one million trial records into one neat, standardized format. Think of it as taking a chaotic pile of receipts from 30 different stores, each written in a different currency and with different math rules, and turning them all into a single, clear spreadsheet that lets you see exactly what was bought, how much it cost, and which store had the best deals.

The most exciting thing MESSIER found is that the way we count points changes the story. The paper shows that many tests use a "strict" rule: if an agent fails even one small part of a task, the whole thing is marked as a failure. The researchers showed that if they relaxed this rule and looked at how many parts the agent did get right, the picture of progress looks very different. For example, on a legal benchmark, the strict "all-or-nothing" score said agents were succeeding only 0.2% of the time. But when they looked at the details, they found that on average, agents were actually satisfying 28.8% of the criteria. The strict rule was hiding a lot of partial progress. This suggests that while AI is getting better, the way we measure it might be too harsh, making it look like agents are failing more often than they really are.

The study also mapped out where AI is truly "winning" and where it is still struggling. They found that AI is almost perfect at "function calling" (using tools), with a success rate of 0.97. It is also getting very good at programming, improving rapidly over the last two years. However, AI is still having a tough time with "enterprise workflows"—complex, multi-step office jobs like managing a business process. In these areas, the success rate is only 0.57, meaning agents still fail more often than they succeed. The researchers also proved that you can use this massive library to create a new "skill scale" for AI that matches up very closely with other major rankings (with a correlation of 0.81), without needing to spend millions of dollars running new tests.

Ultimately, MESSIER is a tool for clarity. It suggests that by looking at the tiny details of how an agent fails, rather than just the final score, we can get a much truer picture of what AI can actually do. It doesn't say AI has solved everything, but it does suggest that our current measuring sticks might be too blunt. By standardizing these results, the authors hope to stop the waste of money on repeating the same tests and help researchers understand exactly which jobs AI is ready for and which ones still need a human hand. The data is now open for anyone to use, turning a confusing mess of numbers into a clear map of the AI landscape.

Technical Summary: MESSIER – A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

Problem Statement

The evaluation of AI agents in interactive environments is currently hindered by fragmentation. Existing efforts are characterized by:

Heterogeneity: Benchmarks vary significantly in domain definitions, agent scaffolds, environments, action spaces, and success conditions.
Incomparability: Final scores often compress complex, multi-step behaviors into single numbers, obscuring the specific nature of success or failure. A "pass" in a binary math check differs fundamentally from a "pass" in a multi-verifier enterprise workflow.
Data Scarcity and Cost: High-quality, per-task outcome data is often unavailable or requires costly reruns (estimated at ~$40,000 for large-scale sweeps) to generate comparable datasets.
Aggregation Artifacts: Standard aggregation rules (e.g., "all-pass" requirements) can obscure partial progress, leading to misleading capability rankings and hiding the distribution of criterion-level outcomes.

Existing meta-collections either focus on narrow agentic settings (e.g., tool use), static non-agentic benchmarks, or lack the scale and granularity required for fine-grained analysis.

Methodology

The authors introduce MESSIER, a unified corpus designed to reconcile heterogeneous agent evaluation data into a standardized schema.

Data Construction

MESSIER consolidates 957,253 trial outcomes spanning:

30 Benchmarks: Including established public releases (e.g., SWE-bench, LiveCodeBench, BFCL) and six new benchmarks targeting underrepresented professional domains (legal, clinical, scientific, etc.).
714 Agents: Distinct model-scaffold combinations.
11,891 Tasks: Individual problem instances.
74,205 Verifiers: The specific checks used to score trials.

The corpus integrates data from three primary sources:

Public Releases: Ingesting per-task outcomes from existing releases (METR, BRIDGE, Agent Psychometrics, General AgentBench) and author-released artifacts (Hugging Face datasets, GitHub repositories).
New Evaluations: The authors contributed new, uniform five-agent runs on six benchmarks (HarveyAI-Lab, MedAgentBench, DABStep, QCircuitBench, ScienceAgentBench, ReplicationBench) using the OpenHands scaffold via the Harbor harness.
Reconciliation Pipeline: A four-stage process to normalize the data:
- Standardization: Casting records into a uniform data model where every environment contains tasks, and every task contains typed verifiers (script, exact-match, LLM judge, human) with defined aggregation rules.
- Normalization: Mapping model identifiers, scaffolds, and release dates to standardized categories (following Epoch ECI conventions).
- Verification: Running deterministic tests for identifier hygiene, date validity, and deduplication, followed by manual spot checks.
- Classification: Assigning SOC (occupational) and NAICS (industry) codes to tasks using an LLM ensemble.

Analytical Framework

The paper defines agents as a pair of (model, scaffold) and tasks as (instruction, initial state). It explicitly models the reconciliation problem: how different aggregation rules (e.g., all-pass vs. threshold vs. soft scoring) map multiple verifier outcomes ( $v_1, \dots, v_N$ ) to a single task score ( $S$ ). MESSIER preserves the raw verifier outcomes $\{v_i(\tau)\}$ rather than just the aggregated score $S$ , enabling counterfactual analysis.

Key Contributions

Unified Corpus: A foundational, reusable infrastructure containing macro- and micro-scale views of agent performance across 30 benchmarks and 714 agents.
Counterfactual Rescoring Capability: By retaining verifier-level records, the corpus allows researchers to re-aggregate scores under different rules (e.g., changing from "all-pass" to "majority-pass") to observe how aggregation choices alter measured capability and agent rankings.
Open-Data Capability Indices: The derivation of capability scales from open data that align with established rankings (Epoch ECI) without requiring new evaluation campaigns.
Granular Taxonomy: The inclusion of SOC/NAICS classifications and action space/environment state definitions supports slicing analysis by occupation, industry, and task type.

Results and Analysis

1. Inconsistent Frontier Progress

Analysis of frontier progress (2024–2026) reveals asymmetric improvement across benchmark groups:

Function Calling: Approaches saturation with a frontier score of 0.97.
Programming: Shows the fastest improvement, rising from a low baseline to 0.82 (net growth of +0.64).
Enterprise Workflows: Remains the most challenging domain with a frontier score of 0.57, indicating multi-step professional workflows are still difficult for agents.
Research/Reasoning & GUI: Occupy intermediate bands (0.68 and 0.71, respectively).

2. Impact of Aggregation Rules

Counterfactual rescoring on benchmarks with per-criterion outcomes (HarveyAI-Lab, $\tau^2$ -bench, TheAgentCompany) demonstrates that aggregation rules significantly distort measured capability:

Information Loss: Strict "all-pass" grading can report near-zero success (e.g., 0.2% on HarveyAI-Lab) even when agents satisfy a significant fraction of criteria on average (28.8%).
Ranking Instability: Changing aggregation rules alters agent rankings. Spearman correlations between "all-pass" and "soft" (mean fraction passed) scores vary from $\rho=0.667$ to $\rho=0.981$ , indicating that comparative conclusions are sensitive to aggregation choices.

3. Capability Scale Alignment

Using a 1PL Item Response Theory (IRT) model on the MESSIER data, the authors derived agent ability estimates ( $\theta$ ) that correlate strongly with Epoch's Evaluation Capability Index (ECI):

General Capability: Spearman $\rho = 0.81$ .
Programming: Spearman $\rho = 0.77$ .
Mathematics: Spearman $\rho = 0.84$ .
This suggests that high-quality capability ordering can be derived from existing open data without new, costly evaluation campaigns.

4. Task Difficulty Prediction

The paper tests the predictability of task difficulty ( $\beta$ ) using metadata (task instructions, environment descriptions, action spaces).

Metadata containing full context (task + environment + action + gold answer) yields modest but consistent improvements in predicting difficulty compared to raw instruction text alone.
Benchmark identity explains a significant portion of raw difficulty variance, but task descriptions contain signal for relative difficulty within benchmarks.

Significance and Claims

The paper claims MESSIER provides a foundational infrastructure for the field of agent evaluation. Its significance lies in:

Reducing Redundancy: By standardizing and sharing evaluation data, it aims to reduce the need for costly, redundant reruns of agent evaluations.
Enabling Fine-Grained Analysis: It shifts the focus from aggregate scores to the distribution of criterion-level outcomes, allowing for the study of evaluation failures, verifier design, and aggregation artifacts.
Supporting Auditing: The open, inspectable nature of the corpus supports the auditing of agent behavior and the detection of issues like reward hacking or sandbagging.
Democratizing Access: By providing a large-scale, pre-computed dataset, it lowers the barrier to entry for researchers lacking the capital to run large-scale agent campaigns from scratch.

The authors maintain a modest stance, acknowledging that the corpus reflects the current landscape (Western, English, technical domains) and relies on the validity of upstream benchmarks, which remain an ongoing community effort to validate.