💬 NLP

TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents

This paper introduces TelcoAgent-Bench, a multilingual benchmarking framework designed to evaluate the reliability and operational consistency of telecom LLM agents in English and Arabic, revealing that while current models understand telecom problems, they struggle with consistent troubleshooting execution and stability across scenario variations.

Original authors: Lina Bariah, Brahim Mefgouda, Farbod Tavakkoli, Enrique Molero, Louis Powell, Merouane Debbah

Published 2026-04-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lina Bariah, Brahim Mefgouda, Farbod Tavakkoli, Enrique Molero, Louis Powell, Merouane Debbah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have hired a brilliant, multilingual assistant named "Agent" to fix your home's Wi-Fi. You tell Agent, "The internet is slow," and you expect them to:

Understand exactly what's wrong (is it the router? the cable? the signal?).
Follow a strict recipe to fix it (check the router lights before restarting the modem).
Use the right tools (a screwdriver, not a hammer).
Write a clear report at the end explaining what they did.

Now, imagine this "Agent" isn't fixing a home router, but a massive, complex global telephone network (like the one your phone uses). And instead of just English, this Agent needs to speak both English and Arabic fluently.

This paper introduces TelcoAgent-Bench, a giant "training camp" and "test scorecard" designed to see if these AI Agents are actually ready to do this job, or if they are just good at talking but bad at working.

Here is a breakdown of the paper using simple analogies:

1. The Problem: Smart Talkers, Clumsy Workers

We know Large Language Models (LLMs) are great at chatting. They can write poems, answer trivia, and even explain how a network works. But in the real world of telecom, chatting isn't enough.

If an AI agent guesses the wrong problem or uses the wrong tool, it could accidentally shut down a city's phone service. The authors realized that existing tests (which check if an AI can navigate a website or play a game) don't work for telecom. Telecom needs precision, order, and stability.

2. The Solution: The "TelcoAgent-Bench" (The Training Camp)

The authors built a massive simulation called TelcoAgent-Bench. Think of this as a "flight simulator" for network engineers, but the pilot is an AI.

The Scenarios (Blueprints): They created 15 different types of network problems (like "slow internet" or "dropped calls").
The Variations: For each problem, they created hundreds of slightly different versions. One might be a slow connection in Dubai; another might be a slow connection in Cairo, with slightly different numbers.
The Language: The whole test happens in both English and Arabic, because telecom networks in the Middle East need to speak both.
The Tools: The AI is given a toolbox. Some tools are essential (like a "signal checker"), and some are "distractors" (like a "subscriber profile reader" that isn't needed for this specific problem). The AI must ignore the distractors.

3. The Scorecard: TelcoAgent-Metrics (How We Grade Them)

Just giving the AI a test isn't enough; we need a way to grade it fairly. The authors created four specific grades:

Grade 1: Did they understand the job? (Intent Recognition)
- Analogy: If you say, "My car won't start," does the mechanic know if you mean the battery is dead or the engine is broken?
- The Test: The AI has to guess the problem type just from a vague description. The paper found that while AI is okay at this, it struggles more in Arabic than in English.
Grade 2: Did they follow the recipe? (Sequence Alignment)
- Analogy: If a recipe says "Preheat oven, then mix ingredients," but the AI mixes ingredients before preheating, the cake will fail.
- The Test: Telecom troubleshooting has a strict order. You can't fix the antenna before checking if the power is on. The AI often gets the order wrong or adds unnecessary steps (like checking the weather when it's raining).
Grade 3: Did they write a good report? (Resolution Accuracy)
- Analogy: After fixing the car, the mechanic writes a note. Is the note clear, accurate, and does it match what actually happened?
- The Test: The AI writes a summary of the fix. Surprisingly, the AI is actually good at writing the summary, even if it messed up the actual fixing steps! It's like a student who writes a perfect essay about how to solve a math problem but gets the wrong answer on the test.
Grade 4: Are they consistent? (Blueprint Reliability)
- Analogy: If you ask the same mechanic to fix the same car three times, do they do it the same way every time? Or do they guess differently each time?
- The Test: This is crucial for safety. If an AI fixes a network issue one way today and a different way tomorrow, it's dangerous. The paper found that most AI agents are unstable; they act differently every time they face the same problem.

4. The Results: The "Smart but Clumsy" Reality

The paper tested several popular AI models (like Llama, Qwen, and Granite). Here is what they found:

They are good talkers: The AI can understand the problem and write a nice summary report.
They are bad workers: They struggle to follow the strict, step-by-step rules required to actually fix the network.
They are inconsistent: If you ask the same AI to solve the same problem twice, it might give you two different (and potentially wrong) solutions.
Language matters: The AI performs significantly worse in Arabic than in English, showing a gap in technical training for non-English languages.

The Big Takeaway

The authors are saying: "We can't just trust AI to run our phone networks yet."

While the technology is impressive, it's like having a brilliant intern who knows all the theory but keeps forgetting to turn off the lights or uses the wrong screwdriver. Before we let AI agents run our critical networks, we need to train them to be reliable, consistent, and obedient to strict rules, not just good at chatting.

TelcoAgent-Bench is the tool we need to measure exactly how far we are from that goal.

1. Problem Statement

The integration of Large Language Model (LLM) agents into telecommunications networks offers the potential for autonomous network operations. However, current LLM agents face significant challenges when deployed in this domain:

Domain Specificity: General-purpose benchmarks (e.g., AgentBench, GAIA) focus on web navigation or general tool use and fail to capture the strict operational constraints, reliability requirements, and deterministic logic flows required in telecom troubleshooting.
Process Alignment: In telecom, the order of actions is as critical as the actions themselves. Incorrect sequencing (e.g., changing parameters before verifying coverage) can lead to network instability.
Multilingual Gaps: Telecom networks operate globally, requiring agents to function effectively in multiple languages (specifically English and Arabic in this study), yet current models often struggle with domain-specific terminology in non-English languages.
Stability & Consistency: LLMs are probabilistic, leading to inconsistent behavior across repeated variations of the same scenario, which is unacceptable for critical network operations.

2. Methodology

The authors propose a comprehensive framework consisting of a new dataset (TelcoAgent-Bench) and a specialized evaluation suite (TelcoAgent-Metrics).

A. The Dataset: TelcoAgent-Bench

The dataset simulates interactions between a network engineer and an LLM agent to troubleshoot network issues.

Structure: It is built around 15 high-level intents (e.g., Beam Misalignment, Downlink Throughput Drop) categorized into 7 network management areas.
Blueprints: Each intent is expanded into 3–5 "blueprints" that define parameter ranges (e.g., PRB utilization, throughput values). This allows for the generation of ~1,470 unique dialogue instances by sampling parameters within these constraints.
Bilingual Support: All components (problem statements, conversations, and gold summaries) are generated in both English and Arabic.
Guardrails: The dataset enforces strict rules: KPI values must stay within defined ranges, tool calls must follow a "gold" reference order, and only predefined tools are permitted to prevent hallucinations.
Tool Set: The environment includes Core Tools (essential for the task, e.g., oss_query, kpi_timeseries) and Distractor Tools (valid but irrelevant tools, e.g., subscriber_insight) to test the agent's decision-making.

B. Evaluation Metrics: TelcoAgent-Metrics

The framework introduces four key metrics to evaluate agent performance beyond simple task success:

Intent Recognition Accuracy (IRA):
- Measures the agent's ability to identify the underlying troubleshooting intent from a free-text problem description.
- Uses cosine similarity on sentence embeddings to compare the agent's generated intent description against the gold label, allowing for semantic flexibility rather than rigid label matching.
Sequence Alignment Score (SAS):
- Evaluates the correctness of the order of tool execution.
- Composed of two sub-metrics:
  - Mandatory Step Coverage (MSC): The length of the Longest Common Subsequence (LCS) between the agent's actions and the gold path, normalized by the gold path length.
  - Extra Action Penalty (EAP): A penalty for invoking unnecessary or distractor tools.
- $SAS = MSC \times EAP$ .
Resolution Accuracy (RA):
- Measures the semantic similarity between the agent's final resolution summary (root cause, actions, ticket logging) and the gold summary.
- Uses cosine similarity on sentence embeddings to ensure the final report is accurate even if intermediate steps varied slightly.
Blueprint Reliability Score (BRS):
- Quantifies the agent's stability across multiple variations of the same scenario (blueprint).
- Combines three sub-metrics:
  - GPC-0: Exact adherence to the gold path.
  - GPC-1: Tolerance for minor deviations (Levenshtein distance $\le$ 1).
  - Sequence Dispersion (SD): Measures variability between different runs; lower is better.
- $BRS = \alpha_1 \cdot GPC\text{-}0 + \alpha_2 \cdot GPC\text{-}1 + \alpha_3 \cdot (1 - SD)$ .

3. Key Contributions

First Telecom-Specific Agent Benchmark: Introduces a domain-specific framework that moves beyond "answering questions" to "solving network issues" with strict process constraints.
Multilingual Capability: Specifically addresses the need for bilingual (English/Arabic) evaluation, highlighting gaps in current models' technical Arabic capabilities.
Process-Oriented Metrics: Develops metrics (SAS, BRS) that prioritize the sequence and stability of operations over simple task completion, which is crucial for safety-critical telecom environments.
Open Dataset: Released a structured dataset of intents, blueprints, and dialogues to facilitate reproducible research in telecom AI.

4. Experimental Results

The authors evaluated several state-of-the-art instruct-tuned models (e.g., Llama-3, Qwen, Granite, Gemma, Mistral) on the benchmark.

Intent Recognition (IRA): Models performed reasonably well when provided with a list of intents but struggled in unconstrained settings. English performance consistently outperformed Arabic, with larger gaps observed in smaller or Mixture-of-Experts (MoE) models.
Sequence Alignment (SAS):
- There is a clear trade-off: Models that strictly avoided distractors (high EAP) often missed mandatory steps (low MSC), and vice versa.
- Granite-3.3-8B achieved the highest overall SAS, demonstrating the best balance between executing required steps and avoiding irrelevant tools.
- Qwen-7B had high coverage (MSC) but very low EAP, meaning it frequently used unnecessary tools.
Blueprint Reliability (BRS):
- Most models scored extremely low on GPC-0 (exact path adherence), indicating a lack of strict compliance with predefined flows.
- Granite-3.3-8B again led with the highest BRS, showing stable and repeatable behavior.
- Models with high Sequence Dispersion (SD), such as Granite-3.1-MoE-3B, showed unstable behavior across scenario variations.
Resolution Accuracy (RA):
- Most models achieved high RA scores (0.80–0.88), suggesting they can generate coherent final summaries.
- Crucial Insight: High RA does not guarantee correct process execution. Models could produce good summaries even when they failed to follow the correct troubleshooting steps or tool order.

5. Significance and Future Work

Operational Relevance: The paper highlights that while LLMs can understand telecom problems semantically, they currently lack the operational consistency required for autonomous network management.
Multilingual Needs: The performance gap in Arabic underscores the necessity for domain-adapted, bilingual LLMs for global telecom deployment.
Evaluation Shift: The study argues that future evaluations must move beyond "task success" to "process alignment" and "stability" to ensure safety in critical infrastructure.
Future Directions: The authors plan to extend the framework to model closed-loop reasoning, where agents interpret tool outputs, implement configuration changes, and re-evaluate network states dynamically, rather than just generating a static plan.