Technical Summary: Benchmark Agent

1. Problem Statement

The construction of benchmarks for Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) is currently hindered by three primary challenges:

• Labor Intensity and Lack of Reusability: Existing benchmark construction is largely human-driven, requiring substantial effort in task design, data collection, cleaning, and annotation. These pipelines are often bespoke to specific benchmarks, leading to repeated manual labor and slow iteration cycles.
• Performance Saturation: Benchmarks frequently reach performance saturation shortly after release. As models improve rapidly, accuracy scores often exceed 80%, leaving insufficient headroom to discriminate between state-of-the-art methods. This necessitates the continuous introduction of new benchmarks, which themselves tend to saturate quickly.
• Scalability and Sustainability: The heavy reliance on human labor and non-standardized pipelines raises concerns about the long-term sustainability and scalability of current evaluation practices, particularly as real-world applications demand more comprehensive and fine-grained evaluations.

2. Methodology: Benchmark Agent

To address these challenges, the authors introduce Benchmark Agent, the first fully autonomous agentic system designed for benchmark building and customization. The framework is inspired by the brain-cerebellum hierarchical architecture and consists of two tightly coupled components:

A. Benchmark Planner (High-Level Decision Module)

The Planner translates abstract user evaluation requirements into concrete, feasible benchmark specifications. It employs a multi-agent collaboration system:

1. Design Agent: Converts informal human intentions into structured subtasks ($S$). It uses a Proposer tool to brainstorm candidates and Revising/Discarding tools to refine or eliminate subtasks based on well-posedness and contribution.
2. Grounding Agent: Validates whether each subtask can be supported by real data and executable transformations.
   • Dataset Search: Identifies candidate datasets ($D_i$) matching the subtask's modality and domain preferences.
   • Transformability Validation: Constructs and scores candidate transformation plans ($t_{i,j}$) based on alignment with intent, process robustness, and preservation of evaluation signals. A subtask is only accepted if at least one valid grounding $(d_{i,j}, t_{i,j})$ exists.
3. Allocation Agent: Determines feasible allocations under global quota and resource constraints. It uses a closed-loop mechanism involving an Allocation tool, a Diagnose tool (to identify bottlenecks), and an Adjustment tool to revise allocations until a feasible plan is found.

B. Benchmark Executor (Operational Module)

The Executor transforms the validated specifications into runnable benchmarks:

1. Sample-Level Realization:
   • Orchestration: An LLM specializes the dataset-level transformation plan to the current sample state, determining the next action.
   • Execution: Actions are executed using a tool pool. This includes content synthesis tools (e.g., TTS, image resizing, audio mixing) and script-based processing tools (e.g., file conversion, metadata editing).
2. Quality and Quota Control:
   • Generated samples undergo continuous verification for semantic validity and structural compliance.
   • Invalid samples are discarded or routed for localized correction.
   • The system monitors quota fulfillment, reprocessing raw samples if valid yields fall short of targets.

3. Key Contributions

1. Autonomous Framework: The proposal of Benchmark Agent, a system that automates the entire benchmark construction pipeline from user query analysis to quality-controlled data generation, reducing reliance on human labor.
2. Customization and Efficiency: The system enables user-oriented customization (adjusting formats, domains, and criteria) and rapid iteration, addressing the sustainability issues of static, human-built benchmarks.
3. Comprehensive Evaluation: The authors implement the agent to produce 15 representative benchmarks across diverse scenarios (text, audio, image, and multimodal reasoning). They validate the system through:
   • Human Evaluation: Assessing correctness, clarity, and relevance.
   • LLM-as-a-Judge: Scoring alignment, format quality, coherence, grounding, and challenge.
   • Consistency Checks: Verifying that benchmarks produce coherent performance rankings across model scales (e.g., Qwen3.5 series).
   • Ablation Studies: Confirming the necessity of each component (Design, Grounding, Allocation, Planning, Verification).

4. Experimental Results

• Quality: Benchmarks generated by Benchmark Agent achieved high human acceptance rates (96–98%) and strong LLM-as-Judge scores. The system demonstrated high Format & Schema Quality (FSQ) and Question–Answer Coherence (QAC).
• Discriminative Power: The generated benchmarks successfully differentiated between model scales. For instance, on the Art-Reasoning benchmark, Qwen3.5 models showed a clear scaling trend from 40.96% (2B) to 56.38% (27B).
• Cost Efficiency: The agent significantly reduced annotation costs compared to human annotation. For example, generating an Audio-based Reasoning sample took 0.3 minutes with the agent versus 6 minutes for humans.
• Ablation Findings:
  • Direct LLM generation (without the agentic workflow) resulted in significantly lower scores, particularly in User-Intention Alignment (UIA) and Skill-Specific Challenge (SSC).
  • The Design Agent was critical for broad requirements, while Transformability Checking was essential for complex multimodal tasks.
  • Verification was crucial for filtering ambiguous samples and ensuring reliability.
• Model Insights: Evaluation of MLLMs on the generated benchmarks revealed that while models excel in general vision and reasoning (e.g., geography, math), they struggle with fine-grained perception in specific domains like art and animal breeds.

5. Significance and Claims

The paper claims that Benchmark Agent represents a shift from static, labor-intensive evaluation to a dynamic, autonomous, and sustainable paradigm. Its significance lies in:

• Solving the Saturation Problem: By enabling rapid, continual benchmark updates, the system can keep pace with model evolution, preventing evaluation stagnation.
• Democratizing Custom Evaluation: It allows researchers and practitioners to generate high-quality, domain-specific benchmarks tailored to unique requirements without prohibitive manual costs.
• Reliability: The system produces benchmarks that are not only usable but also discriminative, providing meaningful signals for comparing model capabilities across different scales and families.

The authors conclude that while current models show strengths in general tasks, the ability to rapidly generate fine-grained benchmarks highlights specific limitations in fine-grained perception and domain-specific reasoning, offering a pathway for future research directions.