Technical Summary: CTBench

Problem Statement

Network operations and maintenance (NetO&M) in telecommunications is a complex domain requiring engineers to interpret heterogeneous telemetry, issue device-specific commands, and diagnose faults across multiple layers under strict safety constraints. While Large Language Model (LLM) agents have demonstrated capabilities in multi-step reasoning and tool use, existing benchmarks fail to accurately model the realities of telecom environments. Current evaluations often assume full observability, rely on single-domain topologies, utilize standardized interfaces, and focus solely on final answers. Consequently, they cannot distinguish between an agent that correctly identifies a fault through rigorous evidence and one that returns a correct label without justification. This lack of process-level transparency hinders the deployment of AI agents in operational settings where diagnoses must be justified before remediation.

Methodology

The authors introduce CTBench, a public benchmark designed to evaluate the troubleshooting capabilities of AI agents in realistic telecom network operations.

Dataset Construction

CTBench comprises 234 expert-curated tasks divided into two operational families:

1. Root Cause Analysis (RCA): 126 tasks where agents must identify affected nodes/objects and assign normalized root-cause labels.
2. Path Restoration: 108 tasks where agents must reconstruct service forwarding paths from source to destination.

The dataset is constructed via an expert-in-the-loop process:

• Task Generation: 15 senior telecom experts (average 20 years experience) abstract tasks from real maintenance cases, sanitizing sensitive data.
• Independent Review: An independent expert solves each task without access to ground truths to verify solvability and consistency.
• Golden Evidence: Unlike standard benchmarks, CTBench annotates each task with golden evidence steps—a deterministic set of key diagnostic actions required by experts to solve the problem.

Task Characteristics

Tasks are annotated with rich metadata to characterize structural difficulty:

• Observability: Tasks are labeled as O1 (Full) where evidence is directly observable, or O2 (Partial) where agents must infer faults from indirect observations.
• Heterogeneity: Tasks vary in the number of vendors (e.g., Huawei, Cisco, H3C) and device types (routers, switches, firewalls) involved.
• Complexity: Metadata includes fault-propagation chain length, root-cause count (single vs. multi-fault), protocol complexity, and the length of the golden solution path.

Evaluation Framework

CTBench evaluates agents using a reactive sandbox environment that exposes permitted diagnostic interfaces. The evaluation goes beyond final answer accuracy to include:

• Localization: Intersection over Union (IoU) of predicted vs. gold affected nodes/objects.
• Identification/Restoration: IoU of predicted vs. gold root-cause labels or path edges.
• Evidence Acquisition: F1 score measuring whether the agent collected the specific diagnostic steps found in the golden evidence.
• Cost Metrics: Latency, interaction rounds, and token consumption.

Key Contributions

1. CTBench Benchmark: A public benchmark covering 234 expert-curated RCA and path-restoration tasks that simulate partial observability, network heterogeneity, and multi-vendor environments.
2. Expert-Grounded Metrics: A novel evaluation protocol that separately assesses localization, identification, restoration, and evidence acquisition, ensuring that correct answers are supported by the necessary diagnostic process.
3. Task Metadata Schema: A comprehensive annotation system covering observability, heterogeneity, protocol complexity, and fault propagation, enabling fine-grained analysis of agent performance across different operational conditions.
4. Comprehensive Evaluation: An assessment of five representative agent-model combinations (e.g., Codex+GPT-5.5, HermesAgent+DeepSeek-V4-Pro) revealing specific gaps in current agentic capabilities.

Experimental Results

The authors evaluated five agent-model combinations on the CTBench dataset. Key findings include:

• Performance Gap: State-of-the-art agents perform significantly better on Path Restoration (identifying endpoints) than on Root Cause Analysis. For instance, Codex+GPT-5.5 achieved 95.28% Path Restoration accuracy (Path-Res) and 99.38% Path Localization, but only 47.62% RCA accuracy.
• Evidence Deficit: Even when agents produce plausible or correct final answers, they frequently fail to provide the evidence-grounded diagnoses required in practice. For example, Codex+GPT-5.5 achieved high path localization but only 47.84% Path Evidence coverage, showing that producing a correct answer does not imply the agent collected the necessary diagnostic evidence.
• Difficulty by Category: Agents struggle most with Interface State and Link-Layer faults (C1) (0.00%–10.71% accuracy) and Service/Management faults (C5). Conversely, they perform relatively better on Security/NAT faults (C2).
• Impact of Constraints:
  • Partial Observability: Performance drops significantly under partial observability (O2). While agents can still collect some indirect evidence, they struggle to convert it into precise localization.
  • Heterogeneity: High network heterogeneity (multiple vendors/devices) drastically reduces path restoration accuracy (dropping from ~29% to ~4% for some agents).
• Cost vs. Capability: Higher computational costs (more tokens, longer latency) do not necessarily correlate with better diagnostic capabilities. Some agents consumed significantly more resources but achieved lower scores.
• Human Baseline: Human experts achieved 92.6% accuracy on path restoration and 56.4% on RCA, completing tasks in 40–60 minutes, highlighting the current gap between AI agents and human operators.

Significance and Claims

The paper claims that CTBench provides a necessary shift from evaluating "what" an agent answers to "how" it arrives at that answer. The authors assert that:

• Process Transparency is Critical: For operational trust, a diagnosis must be justified by evidence before it can be acted upon. Current agents often fail to provide this justification even when the final label is correct.
• Realistic Constraints Matter: Partial observability and multi-vendor heterogeneity are not merely implementation details but core dimensions of telecom-agent capability that current models struggle to handle.
• Diagnostic Reasoning is the Bottleneck: The primary limitation of current agents is not tool execution but evidence-grounded diagnosis, causal reasoning under partial information, and the ability to distinguish between independent root causes and downstream symptoms.

The authors conclude that while agents show promise in endpoint identification, they remain far from reliable operators for complex telecom troubleshooting, particularly in evidence-grounded RCA and partially observable settings. They position CTBench as a foundation for developing trustworthy, domain-grounded, and operationally safe AI agents for communication networks.