Collaborative AI Agents and Critics for Fault Detection and Cause Analysis in Network Telemetry

1. Problem Statement

The paper addresses the challenge of managing multimodal tasks (e.g., text, sensor data, images) in complex network environments using Federated Multi-Agent Systems (MAS). Specifically, it targets network telemetry, where the goal is to perform fault detection, severity assessment, and root cause analysis.

Key challenges identified include:

• Scalability and Coordination: Managing a large number of AI agents and critics without inter-agent communication overhead.
• Privacy: Agents and critics must keep their local cost functions and derivatives private while collaborating to minimize a global system cost.
• Resource Optimization: Determining the optimal number of active agents and critics for specific modalities at any given time to balance performance and computational cost.
• Hallucination and Accuracy: Ensuring that Generative AI (Foundation Models) provides accurate, reliable outputs, particularly in critical infrastructure like network management.

2. Methodology

The authors propose a stochastic iterative algorithm for the collaborative control of AI agents and critics within a federated architecture.

System Architecture

• Federated Multi-Agent System: Consists of $N$ AI agents and $C$ critics collaborating with a Central Server.
• Roles:
  • AI Agents: Access foundation models (LLMs, LVMs) or classical ML models to generate responses (e.g., fault detection).
  • AI Critics: Evaluate agent responses based on criteria (accuracy, relevance, completeness) and provide feedback/scores.
  • Central Server: Tracks the number of active agents/critics per modality and broadcasts feedback signals to regulate activity.
• Privacy: Agents and critics do not share their cost functions or gradients with each other or the server. They only share their "willingness" to participate (active state).

Algorithmic Framework

The system is formulated as a constrained optimization problem to minimize the total cost function of agents and critics subject to capacity constraints (desired number of active participants per modality).

• Stochastic Approximation: The authors use multi-time-scale stochastic approximation techniques.
  • Fast Time Scale: Agents update their active states ($X_{ji}$) based on local cost derivatives and feedback signals.
  • Slow Time Scale: The Central Server updates feedback signals ($\Theta_j$) based on the aggregate active states to drive the system toward the optimal capacity.
• Convergence Mechanism:
  • Agents and critics calculate a probability of being active ($\sigma$) based on their time-average active states, cost function derivatives, and the server's feedback signal.
  • A Bernoulli trial determines the actual active state (0 or 1).
  • The feedback signal is updated using a decaying step size ($\eta(k)$) to ensure convergence to the optimal solution.
• Theoretical Guarantees: The paper provides mathematical proofs (using martingale difference sequences and Hoeffding's inequality) showing that the time-average active states of agents and critics converge almost surely to their unique optimal values, minimizing the global cost.

Application to Network Telemetry

The framework is applied to a real-world dataset (Network Telemetry Dataset 4) involving:

1. Fault Detection: Classifying network events as "Fault" or "No Fault."
2. Severity & Cause Analysis: Classifying faults as "Critical" or "Non-critical" and generating natural language summaries of the cause.

• Hybrid Approach: The system compares Classical ML (XGBoost) for speed/accuracy in detection against Generative AI (LLMs) for nuanced reasoning and cause analysis.
• RAG Integration: For LLMs, Retrieval-Augmented Generation (RAG) is used with ChromaDB to embed network logs and provide context, reducing hallucinations.

3. Key Contributions

1. Novel Control Algorithm: Development of stochastic iterative algorithms for collaborative control of agents and critics in a federated setting without inter-agent communication.
2. Convergence Guarantees: Rigorous mathematical proof that the system converges to a unique optimal solution using multi-time-scale stochastic approximation, ensuring the time-average active states match the desired capacities.
3. Low Communication Overhead: The communication cost is $O(m)$ (where $m$ is the number of modalities), independent of the number of agents or critics, making it highly scalable.
4. Privacy Preservation: Agents and critics maintain private cost functions while achieving global optimization.
5. Empirical Validation: Extensive evaluation on network telemetry data comparing Classical ML (XGBoost) vs. Generative AI (LLMs) across multiple models (Llama3.2, Mistral, DeepSeek-R1, Granite, Phi4, Llava).

4. Experimental Results

The authors evaluated the system on a dataset containing port state changes (optical transceiver, admin state, link state).

• Fault Detection Performance:

  • XGBoost (Classical ML): Achieved superior performance with 96.44% accuracy, 0.9656 F1-score, and significantly faster processing times.
  • LLMs (Generative AI): Without RAG, performance dropped (F1 ~0.50). With RAG, F1 improved to 0.6957, but was still lower than XGBoost and much slower.
  • Conclusion: Classical ML is preferred for high-speed, high-accuracy fault detection.

• Severity and Cause Analysis:

  • LLMs were used to classify fault severity (Critical vs. Non-critical) and generate cause summaries.
  • Model Variance: Different LLMs showed varying tendencies. Llama3.2 classified the highest percentage of faults as "Critical" (25.8%), while Granite3.2 classified the least (8.7%).
  • Qualitative Output: The agents successfully generated human-readable cause summaries (e.g., "High data rate... indicates potential interface issue"), though some models required RAG to avoid hallucinations.

• Convergence:

  • Simulation results confirmed that the time-average active states of agents and critics converged to the optimal values calculated by a centralized CVX solver.
  • The ratio of the algorithm's total cost to the optimal cost converged to 1.
  • Feedback signals ($\Theta$) stabilized over time, demonstrating the effectiveness of the multi-time-scale control.

5. Significance

• Scalable AI Management: The proposed framework offers a scalable solution for managing large fleets of AI agents in distributed systems without the bottleneck of peer-to-peer communication.
• Hybrid AI Strategy: The study validates a pragmatic "Hybrid AI" approach: using fast, accurate classical ML for initial detection and powerful, reasoning-capable LLMs (with RAG) for complex analysis and explanation.
• Real-World Applicability: By demonstrating success on network telemetry, the paper paves the way for applying similar collaborative AI architectures to other critical domains like smart energy grids, industrial IoT, and healthcare diagnostics.
• Theoretical Foundation: It bridges the gap between theoretical distributed optimization (stochastic approximation) and practical Generative AI applications, providing a robust mathematical basis for autonomous agent coordination.