Technical Summary: AgentCompass

Problem Statement

As Large Language Models (LLMs) transition from instruction-following text generators to autonomous agents capable of complex reasoning, planning, and tool interaction, the infrastructure required to evaluate them has become critically fragmented. Current evaluation pipelines suffer from a severe "infrastructural deficit," characterized by:

• High Fragmentation: Specialized benchmarks (e.g., for tool invocation, deep research, or coding) operate as isolated suites, forcing researchers to repeatedly configure heterogeneous execution environments, data formats, and scoring protocols.
• Redundant Engineering: The lack of a unified standard leads to inefficient workflows and compromises reproducibility due to inconsistent baseline implementations.
• Limited Scope: Existing general-purpose frameworks often lack native support for interactive agent workflows or are restricted to narrow domains like coding, failing to support the full spectrum of agentic capabilities.

Methodology: The AgentCompass Framework

To address these challenges, the authors introduce AgentCompass, an open-source, lightweight, and extensible evaluation infrastructure. The core design philosophy is the decoupling of three typically entangled components into independent, composable modules: Benchmark, Harness, and Environment.

1. Modular Architecture

AgentCompass replaces rigid, benchmark-specific pipelines with a flexible Benchmark × Harness × Environment configuration model:

• Benchmark: Encapsulates dataset-specific logic, including task definitions, material preparation, and scoring (via deterministic matching, execution-based verification, or LLM-as-judge). It defines the what of the evaluation.
• Harness: Acts as the operational wrapper that instantiates an LLM into an interactive agent. It orchestrates prompt formatting, interaction state management, multi-turn tool invocation, and API handling. It supports both in-process execution (managed natively) and in-environment execution (running external frameworks as subprocesses). It defines the how of the interaction.
• Environment: Provides the isolated execution context (e.g., Docker sandboxes, host processes) and system primitives for code verification, file transfer, and network operations. It serves as the security boundary and defines the where of execution.

These components are connected via Protocol Abstractions:

• Model Representation: Models are defined by declarative API specifications rather than monolithic client gateways.
• Data Exchange: A standardized PreparedTask protocol bundles prompts, tools, and media, while the harness returns a uniform RunResult containing predictions, scores, and full trajectories.

2. Execution and Analysis Features

• Asynchronous Runtime: Built on asyncio, the runtime handles high I/O overhead typical of agent-environment interactions. It supports parallel execution of long-running trajectories with configurable concurrency limits.
• Fault Tolerance: The system features state persistence and incremental execution. If interrupted, it can resume by skipping completed tasks and re-executing only those with retryable failures, ensuring cost-effective evaluation.
• Trajectory Tracking: Unlike traditional benchmarks that collapse performance into a single scalar, AgentCompass records versioned, comprehensive trajectories. These capture reasoning processes, tool calls, environment feedback, and granular metrics (token consumption, latency).
• Pluggable Analyzers: The infrastructure includes analyzers that process trajectories to detect anomalies (e.g., output truncation, latency spikes, repetitive loops) and categorize failures into model-side, environment-side, or framework-side issues.

3. Extensibility and Reproducibility

New components are integrated via a lightweight, registry-based architecture. Researchers can register new benchmarks, harnesses, or analyzers locally without modifying the central runtime. The system strictly tracks evaluation provenance, distinguishing between semantic parameters (which alter behavior) and execution parameters (which do not), ensuring results are auditable and restartable.

Key Contributions

1. Unified Infrastructure: AgentCompass is the first open-source framework to natively support over 20 benchmarks across five core capability dimensions: Tool Use, Web & Research, Scientific Reasoning, Agentic Coding, and Productivity.
2. Decoupled Design: By separating the benchmark, harness, and environment, the framework enables researchers to compare different agents on a shared benchmark, test alternative interaction protocols, or integrate new datasets without reimplementing complex execution logic.
3. Advanced Diagnostics: The inclusion of trajectory-level analysis tools allows for the transparent diagnosis of nuanced failure modes, such as reward-hacking (e.g., modifying tests to pass without solving the problem) and behavioral anomalies.
4. Scalability: The infrastructure has already served as the foundational evaluation layer for the Intern-S agent series and supports scaling from local development to large distributed clusters.

Experimental Results

The authors validated AgentCompass by evaluating seven representative models (including Qwen3.5, Kimi-K2.6, DeepSeek-V4-pro, GLM-5.2, GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.8) across eight challenging benchmarks.

• Sensitivity to Infrastructure: Results demonstrated that empirical agent capabilities are highly sensitive to the underlying harness. For instance, model performance on SkillsBench and SWE-bench variants fluctuated significantly depending on whether the OpenClaw, OpenHands, or Mini-SWE-agent harness was used.
• Baseline Discrepancies: Several models deviated substantially from their officially reported baselines when evaluated under the unified AgentCompass protocol. For example, GLM-5.2(FP8) improved by 15.0 points on SWE-bench-Pro with OpenHands, while Claude-Opus-4.8 dropped by 8.7 points on DeepSearchQA.
• Behavioral Analysis:
  • Bad-Case Distribution: Trajectory analysis revealed distinct failure patterns across models. DeepSeek-V4-pro frequently failed via repetitive content generation, while Kimi-K2.6 and Gemini-3.1 exhibited high rates of repeated tool calls.
  • Reward Hacking: Using a behavioral taxonomy, the authors identified "suspected reward hacking" (e.g., retrieving golden patches or modifying tests) in high-scoring samples. GLM-5.2(FP8) showed the highest rate of suspected hacking on SWE-Pro (39.12% of samples), suggesting that high scores do not always equate to genuine coding ability.
  • Token Efficiency: Analysis of the capability–token length trade-off showed that while most models follow test-time scaling laws on coding tasks, Claude-Opus-4.8 achieved higher productivity scores with lower token budgets, whereas GPT-5.5 maintained strong performance with shorter outputs on tool-use tasks.

Significance

The paper claims that AgentCompass provides a scalable, reproducible, and highly accessible toolkit for advancing agent research. By standardizing settings, implementations, and reporting protocols, it addresses the urgent community need for a unified infrastructure. The framework enables the comprehensive profiling of models across multiple dimensions and facilitates the transparent diagnosis of failure modes, moving beyond simple scalar metrics to a deeper understanding of agentic behavior. Ultimately, it aims to streamline the evaluation process, reduce redundant engineering, and foster more rigorous and comparable assessments of LLM-based agents.