Technical Summary: A Reproducible Semantic Benchmark for Multivendor DSM-to-CLI Translation

1. Problem Statement

Network automation in heterogeneous, multivendor environments faces a persistent challenge: translating high-level network intents into correct, vendor-specific configurations. While syntactically valid outputs can be generated, they often fail to implement the intended operational state, leading to "silent failures." Recent advances in Large Language Models (LLMs) have renewed interest in intent-driven automation, yet the field lacks reproducible semantic benchmarks for rigorous cross-vendor evaluation. Existing approaches predominantly frame the problem as text generation, relying on prompt engineering, lexical similarity, or single-pass execution success. These methods provide limited guarantees regarding semantic fidelity, stability across repeated executions, and comparability across different models and vendors. Consequently, progress is hindered by a lack of semantically grounded benchmarks that can distinguish between syntactic plausibility and true operational correctness.

2. Methodology

The paper introduces dsm2cli-bench, a reproducible evaluation framework designed to systematically compare cloud LLMs in translating Desired State Models (DSMs) into vendor-specific Command Line Interfaces (CLI).

Experimental Design

• Scope: The benchmark covers five cloud LLM translators (gpt-5, claude-opus-4-6, gemini-2.5-pro, deepseek-chat, grok-4-1-fast-reasoning) across three vendors (Cisco NX-OS, Huawei VRP, Arista EOS) and five representative use cases (L2 access, L2 trunk, SVI IPv4, SVI IPv6, Dual-stack SVI).
• Repetition: Each of the 75 experimental cells (5 translators × 3 vendors × 5 use cases) is executed 10 times, resulting in 750 total runs.
• Evaluation Protocol:
  • Fixed Judges: A panel of three independent LLMs (gpt-5.4-mini, claude-haiku-4-5, gemini-2.5-flash-lite) evaluates the generated CLI against the DSM.
  • Separation of Concerns: The framework strictly separates translation generation from semantic verification.
  • Failure Taxonomy: Failures are explicitly classified into semantic categories (omission, divergence, structural) and pipeline categories (timeout, provider error, invalid output, parsing failure, empty response).
• Metrics: The study measures semantic quality (success rate, vote stability), consistency (vote dispersion), and performance (latency, token usage).
• Statistical Rigor: Hypotheses are tested using planned proportion contrasts, Wilson confidence intervals, and regression analysis. A pilot phase was conducted to refine semantic equivalence handling and pipeline error labeling.

3. Key Contributions

The paper makes four primary contributions to the state of the art:

1. Reproducible Benchmark Design: It establishes a reproducible semantic benchmark for DSM-to-CLI translation in multivendor environments, moving beyond single-pass, single-vendor evaluations.
2. Controlled Experimental Protocol: It introduces a protocol combining structured DSM inputs, fixed judges, repeated executions, and an explicit failure taxonomy to ensure comparability.
3. Comprehensive Comparative Evaluation: It provides a multidimensional evaluation of five cloud LLMs across three distinct vendor ecosystems.
4. Multidimensional Analysis: It presents findings on semantic correctness, execution stability, inter-evaluator agreement, latency, and token usage, demonstrating that aggregate accuracy alone is insufficient for rigorous assessment.

4. Key Results

The analysis of 750 executions yields several critical findings:

• Orthogonality of Quality and Reliability: Semantic quality and operational reliability are distinct dimensions. For instance, claude-opus-4-6 achieved 100% semantic correctness on valid runs but suffered significant end-to-end performance degradation due to provider-side outages. Conversely, grok-4-1-fast-reasoning achieved the highest end-to-end correctness (99.3%) with moderate latency.
• Dominance of Vendor Effects: Vendor identity is the primary source of performance variation. The performance gap between Huawei VRP and the other vendors (Cisco NX-OS and Arista EOS) is 17.6 percentage points, which is more than three times larger than the variation caused by use-case complexity (5.3 percentage points).
• Hidden Failure Modes: Aggregate metrics can mask systematic failures. deepseek-chat performed perfectly on Cisco and Arista but collapsed on Huawei VRP, exhibiting both semantic failures and pipeline errors. This vendor-specific collapse would be obscured in a single-vendor or aggregate summary.
• Stability and Dispersion: There is a strong negative correlation between the dispersion of binary outcomes across repetitions and vote stability. Cells with higher outcome dispersion exhibit lower vote stability, suggesting that repeated-run analysis is essential for identifying unstable models.
• Inter-Judge Agreement: Agreement is lowest for Huawei VRP, which concentrates the majority of judge splits. This indicates that Huawei's CLI semantics are intrinsically harder to evaluate consistently than Cisco or Arista.
• Cost vs. Performance: No direct correlation exists between correctness and latency. However, gpt-5 incurred significantly higher token costs and latency compared to other models with similar correctness rates.

5. Significance and Claims

The paper argues that the current state of LLM-based network configuration evaluation is methodologically immature. Its significance lies in demonstrating that:

• Semantic benchmarks are essential: Rigorous progress requires benchmarks that are semantically grounded, independent of generation-only workflows, and capable of detecting failures hidden by syntactic validation.
• Multivendor and repeated-execution designs are non-negotiable: Single-pass, single-vendor evaluations fail to capture critical failure modes and stochastic instabilities. The paper claims that future evaluations must explicitly separate semantic correctness from infrastructure robustness and decompose results by vendor.
• Benchmark calibration is part of the scientific method: The pilot phase, which refined semantic equivalence handling for specific CLI syntaxes, proved critical to the validity of the final results, highlighting that benchmark design requires iterative refinement.
• Limitations: The authors modestly note that their claims are limited to semantic consistency with the DSM under a fixed judging protocol, not production-level service correctness. The study is bounded by the specific vendors and use cases tested, and long-term reproducibility is subject to cloud provider model updates.

In conclusion, the paper asserts that for LLM-based network configuration systems to be scientifically rigorous and operationally safe, the field must adopt multidimensional evaluation frameworks that account for vendor sensitivity, execution stability, and semantic fidelity simultaneously.