Technical Summary: Results and Retrospective Analysis of the CODS 2025 ASSETOPSBENCH Challenge

Problem Statement

The paper addresses the critical challenge of evaluating multi-agent AI systems in real-world industrial settings, specifically within the "Industry 4.0" domain of physical asset management. While recent advances in Large Language Model (LLM) agents have demonstrated capabilities in reasoning and tool use, moving these systems from laboratory benchmarks to deployment reveals significant evaluation gaps. Static benchmarks often favor narrowly specified, easily optimized tasks, failing to capture essential deployment behaviors such as tool-use robustness, privacy-preserving execution, and multi-step orchestration. The authors argue that competition-based evaluation offers a principled alternative to static benchmarks by exposing failure modes (e.g., progressive reasoning failures) that only emerge under iterative, blind evaluation. The specific problem tackled is the design and retrospective analysis of a competition that combines agentic evaluation, a physical-asset industrial domain, and privacy-constrained deployment, aiming to determine what such an evaluation design actually measures and where its incentives may misalign with true capability.

Methodology

The authors conducted a retrospective analysis of the CODS 2025 ASSETOPSBENCH challenge, hosted on the Codabench platform. The methodology involved a multi-dimensional analysis of competition artifacts rather than a simple leaderboard report.

• Competition Design: The challenge utilized a "hidden-scenario" framework with two complementary tracks:
  • Track 1 (Planning): Participants optimized the planning prompt and Directed Acyclic Graph (DAG) construction (track1_planning.py) while the executor and domain agents remained frozen.
  • Track 2 (Execution): Participants optimized the workflow execution logic, including parallel paths, context aggregation, and fallback mechanisms (track2_execution.py), while the planning prompt and agents remained frozen.
• Evaluation Protocol: Submissions were evaluated against 11 public development scenarios and 11 hidden evaluation scenarios (Phase 1 and Phase 2). The scoring formula combined public scores ($S_{pub}$), hidden scores ($S_{priv}$), and a semantic t-match signal ($\tau$): $C_t = 0.6 S_{pub}^t + 0.3 S_{priv}^t + 0.1 \tau_t$.
• Data Collection: The analysis aggregated data from 149 registered teams (349 member slots), a 300-row server log of submission attempts, and 331 accessible source-level artifacts.
• Analytical Dimensions: The authors analyzed the corpus across five dimensions:
  1. Participation: Funnel statistics and team composition.
  2. Submission Behavior: Learning curves, score progression, and reliability.
  3. Ranking Robustness: Sensitivity of rankings to metric composition and public-hidden alignment.
  4. Computational Cost: Token usage, API calls, and wall-clock duration fingerprints.
  5. Strategy Attribution: Clustering of implementation patterns (archetypes) and their correlation with performance.
• Validation: The infrastructure was cross-referenced against a recently published "Agent-Eval Checklist" for trustworthy benchmarks to identify design gaps.

Key Contributions

1. First Industrial Agentic Benchmark with Privacy Constraints: The paper documents the first competition-track benchmark to jointly combine agentic multi-step tasks, a physical-asset industrial domain (chillers, air-handling units), and privacy-constrained deployment (using containerized isolation to prevent data leakage).
2. Nine-Dimensional Retrospective Framework: The authors propose a structured, nine-dimensional analytical template for future agentic competitions, moving beyond aggregate leaderboard metrics to include participation funnels, score saturation analysis, computational fingerprints, and strategy clustering.
3. Empirical Evidence of Public-Hidden Misalignment: The study provides concrete statistical evidence that public leaderboard standings do not predict hidden robustness, particularly in execution tasks.
4. Strategy Taxonomy and Archetype Analysis: The paper introduces a taxonomy of agent strategies, distinguishing between "planning" (prompt engineering) and "execution" (guardrail engineering) archetypes, revealing that top performers often rely on robust error handling rather than novel architectures.

Results

• Participation and Attrition: Out of 149 registered teams, only 11 achieved fully ranked status across both tracks. The attrition rate was significant, with a 17.7% failure rate in submission attempts due to platform conformance issues (packaging/workflow format violations), establishing a fixed infrastructure overhead for agent evaluation.
• Public vs. Hidden Score Correlation:
  • Planning Track: Moderate correlation between public and hidden scores ($\rho = 0.69$), though public scores showed systematic optimism.
  • Execution Track: No correlation between public and hidden scores ($\rho = -0.13, p = 0.71$). Public execution scores were statistically indistinguishable from random noise regarding hidden performance.
• Score Saturation and Sensitivity: The public planning leaderboard saturated at 72.73% for four teams, making the public signal unable to discriminate between top performers. Furthermore, the final ranking was highly sensitive to the scoring formula's composition. The "t-match" term, intended to contribute 10% weight, was numerically inert due to scale differences (0–1 vs. 0–100), contributing at most 0.05 points. Rescaling this term or adjusting track weights would have swapped the top two teams.
• Computational Cost Inversion: Analysis of computational footprints revealed that the "cheapest" domain (Time Series Forecasting, TSFM) was the most prompt-sensitive (highest strategy variance), while the "most expensive" domain (Work Orders, WO) was the most robust. This suggests that weighting benchmarks by token cost could inadvertently measure prompt sensitivity rather than agent capability.
• Strategy Patterns:
  • Planning: Strategies were diffuse, with high noise in clustering. Top submissions often achieved saturation through prompt elaboration (e.g., longer examples, agent catalogs) without architectural changes.
  • Execution: Strategies were highly concentrated into three tight clusters. Top performers were identified as "guardrail engineers" who improved response selection, cleanup, and fallback logic rather than introducing new agent architectures.
• Failure Modes: Hallucination was found to be a coupled failure mode strongly negatively correlated with overall task quality ($r \approx -0.93$), serving as a leading indicator of broader failure. Work-order tasks involving long-horizon reasoning over historical records consistently formed the capability ceiling.

Significance and Claims

The paper claims that the CODS 2025 ASSETOPSBENCH challenge demonstrates why agent competitions must be analyzed as evaluation instruments rather than merely ranking mechanisms. The authors assert that their analysis reveals critical flaws in current benchmark design, specifically:

• Leaderboard Saturation: Public signals can saturate quickly, failing to discriminate between high-performing systems.
• Public-Hidden Misalignment: Public iteration can reward strategies that overfit to specific scenario sets, leading to a disconnect between public standing and hidden robustness.
• Metric Design Flaws: Composite scoring functions are sensitive to numerical scale choices (e.g., the t-match term), which can arbitrarily alter rankings.
• Guardrail-Centric Success: In complex industrial settings, top performance is often driven by robust error handling and context control ("guardrail engineering") rather than novel architectural innovations.

The authors conclude that future agentic competitions should isolate capability dimensions across tracks, surface skill-aware subscores, ensure scoring components are numerically commensurate, and treat team metadata and versioned code as core deliverables. They position their nine-dimensional retrospective as a reusable template for future benchmarks to ensure that competitions function as rigorous scientific instruments capable of publishing evidence on what was actually measured.