Technical Summary: SADE – Symptom-Aware Diagnostic Escalation for LLM-Based Network Troubleshooting

1. Problem Statement

Modern computer networks face costly and slow-to-diagnose failures, often triggered by operator or automation errors. While Large Language Models (LLMs) have been applied to network tasks like intent translation and configuration generation, their application to active fault diagnosis in live, degraded networks remains limited. Existing LLM-based agents often rely on free-form deliberation that conflates evidence acquisition with hypothesis commitment. Consequently, they struggle to localize root causes on public benchmarks like NIKA, performing well below practical deployment thresholds. The core challenge is not merely producing configurations from intent, but actively gathering evidence, revising hypotheses, and navigating multi-layered, cascading symptoms to identify the root cause under time and query constraints.

2. Methodology: The SADE Framework

The authors propose SADE (Symptom-Aware Diagnostic Escalation), an LLM-based agentic framework that encodes the classical Cisco troubleshooting methodology into an explicit, multi-stage policy. SADE separates general diagnostic methodology from fault-specific expertise, enforcing a disciplined workflow that moves from symptoms to root causes.

The framework operates through four primary stages:

1. Initial Scan: The agent executes get_reachability() to perform a basic all-pairs ping scan. It identifies observable symptoms (e.g., packet loss, timeouts, refused connections). To resolve ambiguities caused by DNS failures (which may mimic path failures), SADE performs direct-IP probes to distinguish resolver issues from actual network path failures.
2. Deep Network Scan: If the initial scan yields no clear symptoms, SADE performs a systematic, bottom-up scan following Cisco's methodology. This phase inspects the network from Layer 2 up to the application layer in four ordered sub-phases:
   • Infrastructure/L2: Checks for firewall drops, missing IPs/routes, ARP mismatches, and duplicate MAC addresses.
   • Control Plane/Routing: Scans OSPF and BGP states, neighbor adjacencies, and convergence.
   • Host-Local: Analyzes kernel paths, DHCP events, and link flaps.
   • Service/Resource: Inspects DNS, HTTP timing, CPU/socket spikes, and daemon presence.
   • Implementation: This stage utilizes a library of Python helper scripts (e.g., infra_sweep.py, ospf_snapshot.py) to gather structured evidence.
3. Symptom-to-Fault-Family Mapping: Detected symptoms are mapped to high-level fault categories (fault families) using a Fault-Index. This index employs a rule set and logical reasoning to disambiguate symptoms that may originate from multiple layers (e.g., distinguishing a link fault from a routing control-plane fault). This step selects the appropriate diagnostic skill from a library.
4. Fault Detection and Localization: The agent invokes a Skill Bank containing 15 specialized "Claude Skills" (declarative SKILL.md documents). Each skill defines canonical failure modes, leading signals, specific probes (e.g., nft list ruleset, show ip ospf neighbor), and guardrails. The agent executes these probes on candidate devices, matches the evidence against the skill's "fingerprints," and identifies the specific faulty device and canonical fault label.

SADE interacts with the NIKA benchmark (built on the Kathará network emulator) via the Model Context Protocol (MCP), adhering to a strict 20-turn budget per incident.

3. Key Contributions

• SADE Framework: An LLM-based agentic system that implements a multi-stage troubleshooting approach (Initial Scan, Deep Scan, Symptom Mapping, Skill-Driven Analysis) grounded in industry-standard methodologies.
• Performance Gains: SADE achieves a root-cause F1 score of 0.77 on a held-out 523-incident slice of the NIKA benchmark. This represents a 37 percentage point improvement over the ReAct + GPT-5 baseline (0.40) and a 22 percentage point improvement over a Claude Code baseline (0.55) using the same underlying model backend.
• Scalability and Robustness: SADE successfully troubleshoots faults across topologies of varying sizes (up to 100+ nodes), achieving a task completion rate approximately 5% higher than the best-performing baseline.
• Efficiency: Despite higher token usage for systematic evidence collection, SADE requires significantly fewer tool calls per correct submission (27.2) compared to ReAct (49.5) and Claude Code (51.5). It also reduces "no-submission" failures (failing to produce a valid diagnosis within the budget) from 15.3% (Claude Code) to 4.2%.
• Open Source: The authors released the SADE source code, diagnostic skill library, and utility scripts to support reproducibility.

4. Results and Analysis

The evaluation on the NIKA benchmark (523 matched test cases) highlights several critical findings:

• Cross-Stack Diagnostics: SADE excels in "cross-stack" scenarios where the visible symptom (e.g., service unreachability) does not immediately reveal the root cause (e.g., a specific ACL blocking BGP traffic or a port filter). While baselines often stop at the symptom layer or provide incomplete explanations, SADE's structured workflow forces the agent to trace the failure from the symptom to the specific routing or filtering mechanism, resulting in higher judge scores and accuracy.
• Topology Sensitivity: As network size increases (from ~11 to ~101 nodes), the performance of the Claude Code baseline declines, whereas SADE maintains high accuracy. This suggests SADE's symptom-driven narrowing of the search space is more effective than the baselines' uniform exploration in complex environments.
• Attribution of Gains: A model-controlled comparison against the same Claude Sonnet backend without the SADE policy attributes 22 of the 37 percentage point gains to the diagnostic policy alone. This confirms that the improvement is driven by the structured workflow rather than just the underlying LLM capabilities.
• Injection Validity: The authors noted that the standard NIKA injector sometimes fails to inject the intended fault. A manual verification pipeline on the training set showed that correcting these injection failures would further improve SADE's reported scores, suggesting the current metrics are conservative.

5. Significance and Claims

The paper argues that the primary barrier to effective LLM-based network troubleshooting is not the lack of tool access, but the lack of a disciplined diagnostic procedure. By encoding the Cisco methodology into an explicit policy that separates evidence acquisition from hypothesis commitment, SADE demonstrates that agentic networks can move beyond free-form deliberation to achieve practical root-cause localization.

The authors claim that SADE's success proves that structuring the agent's diagnostic process is as critical as the model itself. The framework effectively bridges the gap between high-level intent and low-level fault localization, offering a pathway toward reliable, automated network troubleshooting that can handle the complexity of modern, multi-layered network failures. The work concludes that while SADE is currently evaluated in a simulated environment (Kathará), its structured approach provides a foundation for deployment in live networks, provided specific wrappers for telemetry and configuration systems are developed.