One-Eval: An Agentic System for Automated and Traceable LLM Evaluation

One-Eval is an agentic system that automates the end-to-end evaluation of large language models by converting natural-language requests into traceable, customizable workflows through integrated components for benchmark planning, dataset resolution, and decision-oriented reporting, thereby reducing manual effort and enhancing reproducibility.

The Gist

Imagine you are a chef who just invented a new, revolutionary recipe. Before you can serve it to the world, you need to know: Is it actually good? Does it taste like a pizza or a pancake? Did it burn the kitchen down?

In the world of Artificial Intelligence (AI), this "taste test" is called Evaluation. But right now, testing a new AI model is like trying to run a restaurant kitchen where every ingredient comes from a different country, every recipe is written in a different language, and you have to manually build your own stove, oven, and cutting board before you can even start cooking. It's slow, confusing, and prone to mistakes.

Enter One-Eval. Think of One-Eval as an AI Sous-Chef (a highly skilled assistant) that you can talk to in plain English. Instead of giving it complex code or spreadsheets, you just say, "Hey, I need to check if my new AI is good at math and can tell the truth."

Here is how One-Eval works, broken down into three simple steps:

1. The Translator (NL2Bench)

The Problem: You say, "Check my math skills." But the AI doesn't know which math test to use. Is it elementary school math? College calculus? Tricky riddles?
The One-Eval Solution: This is the Translator. It listens to your casual request and turns it into a professional shopping list.

• Analogy: Imagine you tell a travel agent, "I want a relaxing beach vacation." The agent doesn't just book a random flight; they figure out you probably want Hawaii, not a desert, and they book the specific resorts that fit your budget.
• What it does: It takes your sentence, figures out exactly what you mean, and picks the perfect "tests" (benchmarks) from a giant library of thousands of options. It even asks you, "Did you mean this specific test?" so you can tweak it before it starts.

2. The Logistics Manager (BenchResolve)

The Problem: Even if you know which test to use, getting the test materials is a nightmare. One test is on a website, another is in a zip file, and a third uses a weird format that your computer doesn't understand. You'd have to download them, rename files, and fix broken links manually.
The One-Eval Solution: This is the Logistics Manager. It does all the heavy lifting behind the scenes.

• Analogy: Imagine you order a custom-built house. Instead of you having to go to the lumber yard, the hardware store, and the plumbing supply shop to buy bricks and pipes, the builder goes out, buys everything, and delivers it all to your driveway, already sorted into neat piles.
• What it does: It automatically finds the right data files, downloads them, and "translates" them into a standard format so the AI can actually take the test. It makes sure the "stove" is built and the "ingredients" are prepped.

3. The Critic & Reporter (Metrics & Reporting)

The Problem: Usually, when you test an AI, you get a single number back, like "85%." That's like a teacher giving you a grade of "B" without telling you why you got it, or which questions you missed. It doesn't help you improve.
The One-Eval Solution: This is the Critic. It doesn't just give you a score; it gives you a full report card with advice.

• Analogy: Instead of just saying "You got a B," a great coach says, "You ran fast, but you tripped on the last hurdle. Also, your form was great on the left side but weak on the right. Here are three specific drills to fix it."
• What it does: It looks at how the AI failed. Did it lie? Did it get confused by a long sentence? Did it hallucinate (make things up)? It generates a colorful, easy-to-read report that tells you exactly where the AI is strong and where it needs to go back to school.

The "Human-in-the-Loop" Safety Net

One-Eval is smart, but it knows it's not perfect. It has a Safety Checkpoint.

• Analogy: Think of it like a pilot flying a plane. The autopilot (One-Eval) can handle 99% of the flight, but before it lands, it asks the pilot (you), "Hey, I'm about to land on Runway 4. Is that okay?" If you say, "No, use Runway 2," the system instantly changes course.
• What it does: At key moments, it pauses and shows you its plan. You can approve it, edit it, or tell it to start over. This ensures you are always in control, even though the system does the boring work.

Why Does This Matter?

Before One-Eval, testing an AI was like trying to assemble IKEA furniture without the instructions, using tools you had to build yourself. It took days and required a PhD in carpentry.

With One-Eval, you just say, "Build me a table," and the system brings you the wood, the screws, the instructions, and the finished product, while pointing out any wobbly legs. It turns a chaotic, manual nightmare into a smooth, automated conversation, making it much easier for companies to build safer, smarter, and more reliable AI.

Technical Summary

Here is a detailed technical summary of the paper "One-Eval: An Agentic System for Automated and Traceable LLM Evaluation."

1. Problem Statement

Despite the rapid adoption of Large Language Models (LLMs) in industrial settings, model evaluation remains a bottleneck due to its reliance on substantial manual effort. Current workflows suffer from several critical limitations:

• High Manual Overhead: Practitioners must manually identify benchmarks, reproduce heterogeneous codebases, configure dataset schema mappings, and interpret aggregated metrics.
• Rigidity: Existing frameworks (e.g., lm-eval-harness, OpenCompass) rely on static configurations. They assume predefined tasks and benchmarks, leaving users to manually map abstract evaluation goals to concrete setups.
• Lack of Actionability: Outputs are often limited to isolated scalar scores (e.g., accuracy), which provide insufficient guidance for real-world decision-making, risk assessment, or debugging.
• Traceability Issues: It is difficult to audit why a specific evaluation was chosen or to reproduce results due to a lack of preserved evidence trails for configuration and schema choices.

2. Methodology: The One-Eval Framework

One-Eval is an agentic system designed to convert natural-language evaluation requests into executable, traceable, and customizable workflows. It operates through a three-stage pipeline, integrating human-in-the-loop (HITL) checkpoints for review and refinement.

A. Stage 1: NL2Bench (Intent Structuring & Planning)

This module translates unstructured user requests into structured evaluation plans.

• Intent Structuring: Parses natural language to extract target domains (e.g., math reasoning), explicit benchmark requests, execution constraints, and preferences.
• Candidate Retrieval: Uses a hybrid retrieval strategy:
  • Local Gallery: A curated registry of 77 verified, error-free benchmarks.
  • Live Search: Falls back to HuggingFace Hub for long-tail or new benchmarks if local matches are insufficient.
  • Retrieval Modes: Supports both embedding-based semantic similarity and lightweight TF-IDF keyword matching.
• Selection & Normalization: Selects a compact subset of benchmarks based on intent alignment and constraints (cost, redundancy). It normalizes heterogeneous benchmark identifiers and collects metadata (splits, column mappings) to ensure executability.
• HITL Integration: Presents the plan to the user for approval, editing, or rejection before proceeding.

B. Stage 2: BenchResolve (Resolution & Configuration)

This stage orchestrates the conversion of the benchmark plan into a runnable configuration.

• Hierarchical Resolution:
  • Local-First: Loads expert-validated configurations for high-frequency benchmarks from a local registry to ensure stability.
  • Dynamic Fallback: For unknown benchmarks, it dynamically resolves identifiers, downloads datasets, and inspects metadata (dataset cards) to infer splits and configurations.
• Schema Normalization: Converts heterogeneous data schemas (e.g., question/answer vs. instruction/output) into a unified internal BenchInfo object. This includes mapping columns to a standardized input-output interface.
• Traceability: Persists resolved IDs, cache paths, and configuration choices as artifacts for auditability.

C. Stage 3: Metrics & Reporting (Analysis & Decision Support)

This module transforms raw model outputs into decision-oriented insights.

• Dual-Track Metric Recommendation:
  • Static Control: Respects explicit metric configurations provided in benchmark metadata.
  • Dynamic Adaptation: Uses an LLM agent to perform semantic reasoning over dataset context (prompts, few-shot samples) to recommend metrics if none are predefined.
• Execution Engine: A unified runner (MetricRunner) handles parallel execution, input normalization, and score calculation.
• Hierarchical Diagnostic Reporting: Generates reports beyond scalar scores:
  • Macro View: Radar/sunburst charts for holistic capability profiling.
  • Diagnostic View: Root-cause analysis (e.g., hallucinations vs. instruction following) and failure mode attribution.
  • Micro View: Case-level inspection linking aggregate metrics to specific failure instances.

3. Key Contributions

1. End-to-End Agentic Evaluation: One-Eval is the first framework to treat the entire evaluation lifecycle (from intent to report) as an agent-driven process, bridging the gap between abstract goals and executable pipelines.
2. Automated Schema Normalization: It solves the "heterogeneity problem" by automatically resolving dataset schemas and column mappings, allowing seamless integration of diverse community benchmarks without manual configuration.
3. Decision-Oriented Reporting: Moves beyond simple accuracy scores to provide multi-granular diagnostic reports that support specific industrial decisions (e.g., model selection, release gating).
4. Traceability and Auditability: By preserving intermediate artifacts (plans, resolved IDs, schema mappings, and per-sample traces), the system ensures that every evaluation result is reproducible and auditable.
5. Human-in-the-Loop Control: Integrates strategic checkpoints where users can review, edit, or rollback agent decisions, balancing automation efficiency with human oversight.

4. Experimental Results

The authors evaluated One-Eval on 100 diverse natural-language evaluation requests across six domains (reasoning, math, code, safety, retrieval, factual QA).

• End-to-End Success Rate:
  • Plan Executable Rate: 99% (99/100 requests successfully parsed and retrieved at least one benchmark).
  • Auto-Complete Rate: 85% (Successfully inferred correct splits, subsets, and key mappings without human intervention).
  • Full Plan Rate: 84% (Successfully generated a complete, runnable plan including metric recommendations).
• Efficiency: The system generates an executable evaluation plan in a median time of ~11.4 minutes per request.
• Feature Comparison: Unlike existing frameworks (e.g., lm-eval-harness, OpenCompass, HELM) which lack native automation for intent-to-plan conversion and metric recommendation, One-Eval supports Customization, Automation, Benchmark Recommendation, and Metric Recommendation simultaneously.

5. Significance

One-Eval represents a paradigm shift in LLM evaluation from static, manual configuration to dynamic, intent-driven automation.

• Industrial Impact: It significantly reduces the engineering overhead required to evaluate models, making rigorous evaluation accessible to non-experts and scalable for continuous integration/continuous deployment (CI/CD) pipelines.
• Reproducibility: By enforcing traceable evidence trails and standardized configuration artifacts, it addresses the "reproducibility crisis" often seen in LLM benchmarking.
• Actionability: The shift from scalar scores to diagnostic reports empowers stakeholders to make informed decisions about model deployment, iteration, and risk management based on why a model failed, not just that it failed.

The framework is publicly available at https://github.com/OpenDCAI/One-Eval.