Technical Summary: Evaluating Agentic Configuration Repair for Computer Networks

Problem Statement

Network misconfigurations remain a primary cause of critical Internet outages, affecting enterprise and ISP networks. While formal verification and synthesis have advanced, they often fail in real-world, open-ended deployments due to the difficulty of accurately modeling complex network behaviors. Consequently, human error persists as the root cause of costly downtime.

Large Language Models (LLMs) have emerged as a flexible alternative for automating network operations, with major hyperscalers already deploying them in production. However, state-of-the-art models struggle with complex, large-scale network tasks when using monolithic (single-turn) prompting. Three recurring challenges limit their efficacy:

1. Noise: Configurations span thousands of lines, obscuring relevant signals.
2. Single-shot Limitations: One-off attempts hinder error correction, causing models to halt at partial diagnoses.
3. Semantic Gap: The disconnect between configuration edits and their emergent forwarding behavior makes network-wide impact difficult to predict.

Existing benchmarks often focus on short diagnostics or single-command tasks, failing to address end-to-end iterative configuration repair.

Methodology

The authors propose and evaluate an agentic architecture for network configuration repair, building upon the CORNETTO benchmark. This benchmark evaluates LLM-driven repair across 231 misconfiguration scenarios derived from real-world topologies (Topology Zoo), involving up to 8 simultaneous faults and configurations reaching 200,000 lines.

The Agentic Pipeline

Unlike monolithic approaches that receive the full problem context in a single prompt, the proposed system utilizes a ReAct-style agent equipped with three core capabilities:

1. Dynamic Context Retrieval: The agent does not receive the full network configuration upfront. Instead, it uses tools to list routers, retrieve specific configuration files, and inspect the network topology on demand, guided by violated specifications.
2. Iterative Repair: The agent proposes search-and-replace edits on a working copy of the configuration. It can inspect changes after each modification, allowing for a step-by-step diagnosis and repair process rather than a single pass.
3. Formal Verification as Feedback: The agent interacts with a network verifier (Batfish) to validate intermediate repairs. This tool returns a decomposition of specifications into fixed, unresolved, and regressed sets, enabling the agent to diagnose remaining conflicts or detect safety regressions before final submission.

Evaluation Setup

The study benchmarks four models (GPT-5 MINI, GLM-4.7, QWEN3.5-9B, and GEMMA 4 E4B) on the CORNETTO test set. The evaluation compares the agentic setup against monolithic baselines using two primary metrics:

• Fix Score: The proportion of initially violated specifications restored by the fix.
• Regression Rate: The proportion of previously healthy specifications broken by the fix.

A repair is considered strictly successful only if it achieves a perfect fix score with zero regressions.

Key Results

The experimental results demonstrate that agentic architectures significantly outperform monolithic LLMs in both efficacy and safety:

• Overall Performance: The agentic system improves repair efficacy by an average of 12% and reduces safety regressions by 17% compared to base LLMs.
• Impact on Model Capability: The benefits are particularly pronounced for open-source models. For instance, QWEN3.5-9B saw its fix score increase by up to 7× under the agentic framework, while its regression rate dropped from 34.2% to under 6%. This suggests agentic scaffolding disproportionately compensates for weaker base-model capabilities.
• Design Trade-offs:
  • Verification Feedback: Enabling iterative verifier feedback generally improves safety (lowering regressions) but can make models more conservative, potentially lowering fix scores for stronger models like GPT-5 MINI.
  • Context Retrieval: Dynamic context retrieval significantly benefits smaller models (e.g., QWEN3.5-9B), which struggle to extract signals from large, prefilled contexts. Conversely, models with strong long-context capabilities (GPT-5 MINI) performed better when the entire problem description was prefilled, as they could process the full scenario in a single pass without spending steps on retrieval.
• Cost-Efficiency: The agentic approach incurs higher inference costs (approximately 4–5× for GPT-5 MINI at 30 steps). However, analysis shows diminishing returns beyond 20 steps, suggesting an optimal step budget exists to balance cost and performance.

Significance and Claims

The paper claims that agentic architectures offer a viable path toward safer and more effective automated network management. By decoupling context management from the model's context window and introducing iterative validation, the approach addresses the specific limitations of LLMs in complex network environments.

The authors emphasize that while the agentic pipeline improves safety and efficacy, it introduces trade-offs between cost, step budgets, and model capabilities. They position this work as a step toward the safe adoption of LLM-based agents for critical infrastructure, noting that future work must address richer interfaces (e.g., live routing tables), proactive safety guardrails, and fine-tuning methodologies that leverage step-level feedback.

The study concludes that while frontier models struggle with monolithic prompting in network repair, equipping them with agentic tools for dynamic context retrieval, iterative editing, and formal verification significantly enhances their ability to resolve misconfigurations without introducing catastrophic errors.