PromCopilot: Simplifying Prometheus Metric Querying in Cloud Native Online Service Systems via Large Language Models

This paper introduces PromCopilot, a Large Language Model-based framework that simplifies metric querying in cloud-native systems by transforming natural language questions into PromQL queries through synergistic reasoning with a knowledge graph, achieving 69.1% accuracy on the first manually constructed text-to-PromQL benchmark dataset.

The Gist

Imagine you are the captain of a massive, high-tech spaceship (your Cloud Native Online Service System). This ship is made of thousands of moving parts: engines, fuel pumps, navigation computers, and life support systems. To keep the ship running smoothly, you have a dashboard full of thousands of gauges, blinking lights, and digital readouts. This is your Prometheus monitoring system.

The problem? To check a specific gauge or figure out why an engine is sputtering, you have to speak a very difficult, alien language called PromQL. It's like trying to ask the ship's computer, "Show me the fuel pressure of the third engine on the starboard side, but only if the temperature is above 200 degrees and the last 5 minutes of data is averaged," but you have to type it in a code that looks like a math equation written by a robot. If you make one typo, the computer gives you nothing, or worse, the wrong answer.

PromCopilot is the solution the authors built. Think of it as a super-smart, bilingual translator sitting right next to you in the cockpit.

How PromCopilot Works (The Magic Trick)

Instead of forcing you to learn the alien language (PromQL), PromCopilot lets you speak in plain English.

1. The "Brain" (Large Language Model): This is the translator. It understands your question: "Which engine is running the hottest right now?"
2. The "Map" (Knowledge Graph): Here is the secret sauce. A generic AI translator might guess the wrong engine because it doesn't know your specific ship. PromCopilot has a special, living 3D map of your entire system. It knows exactly which engines are connected to which fuel lines, where the sensors are, and what they are named.
   • Analogy: If you ask a regular AI "Where is the library?", it might guess a library in a different city. But PromCopilot's map knows your library is in the blue building next to the cafeteria.
3. The "Reasoning" (Synergy): When you ask a question, PromCopilot doesn't just guess. It looks at your question, checks its Map to find the right sensors and engine names, and then uses its Brain to write the perfect alien code (PromQL) to get the answer.

Why Was This Hard? (The Old Way)

Before PromCopilot, engineers were like detectives trying to solve a crime without a crime scene.

• The Context Problem: If you asked, "Why is the order service slow?" a regular AI wouldn't know that "order service" is actually running on three different servers in a specific data center. It would just guess.
• The Memory Problem: Engineers had to memorize thousands of metric names and how they were connected. It was like trying to remember every single street name in a city the size of New York just to ask for directions.
• The "History" Trap: Some older tools tried to learn by looking at past questions. But if a new server was added today, the old tool wouldn't know about it, and it would fail. PromCopilot looks at the live map, so it always knows the current state of the ship.

What Did They Prove?

The team built a test ship (a real microservice system) and asked 280 different questions in plain English.

• The Result: PromCopilot got the right answer 69% of the time on its first try.
• Comparison: Without the "Map" (just using a standard AI), the success rate was less than 5%. With the Map, it skyrocketed.
• Human Test: They asked real engineers to use the tool. With PromCopilot, the engineers finished their tasks in 100 seconds. Without it, using old tools, it took them 380 seconds. That's nearly 4 times faster!

The Bottom Line

PromCopilot is like giving your spaceship a voice-activated assistant that knows the entire ship's blueprint by heart. It takes your simple question, finds the right data on the complex map, and writes the complex code for you. It turns a job that used to require a PhD in computer science into something anyone can do by just asking, "Hey, what's going on with the system?"

This is a huge step forward for keeping our digital world (online shopping, banking, social media) running smoothly without needing a team of experts to decode the dashboard every time something acts up.

Technical Summary

Here is a detailed technical summary of the paper "PromCopilot: Simplifying Prometheus Metric Querying in Cloud Native Online Service Systems via Large Language Models."

1. Problem Statement

Modern cloud-native online service systems are highly complex, involving thousands of microservices, containers, and nodes. To ensure reliability, engineers rely on metric monitoring systems like Prometheus. However, interacting with these systems requires writing queries in PromQL (Prometheus Query Language), a domain-specific language (DSL).

Key Challenges:

• High Learning Curve: Writing PromQL requires deep knowledge of syntax, operators, and functions.
• Context Dependency: Accurate queries require specific knowledge of the system's current state (e.g., metric names, label values, service deployment topology, and node IDs), which is often dynamic and not static.
• Limitations of Existing Solutions:
  • Generic LLMs: Lack private domain knowledge (specific metric names and system topology) and often hallucinate context.
  • Fine-tuning/Few-shot Learning: Rely on historical data, which is insufficient because system components (pods, nodes) change dynamically. Historical queries may not match the current system state.
  • Standard RAG: Simple retrieval often fails to support the multi-hop reasoning required to connect a user's intent (e.g., "order service") to specific infrastructure details (e.g., specific nodes hosting that service).

2. Methodology: PromCopilot

The authors propose PromCopilot, a framework that translates natural language questions into PromQL queries using a synergistic reasoning approach between Large Language Models (LLMs) and a System Context Knowledge Graph (KG).

A. System Context Knowledge Graph Construction

PromCopilot constructs a dynamic KG to represent the system's topology and metric relationships.

• Schema: The KG includes two main entity types:
  1. System Components: Nodes, Deployments, Pods, Services, Containers, APIs, etc.
  2. Metric Data: Metrics and Label-Value Pairs.
• Data Sources: The graph is automatically built by fusing data from:
  • Prometheus: Metric metadata (names, types, labels).
  • Kubernetes: Resource metadata and relationships (e.g., Pod $\to$ Node).
  • Traces: Service dependencies and API invocation paths.
  • Documents: Service and API descriptions to aid intent understanding.
• Storage: Stored in a graph database (Neo4j) for efficient traversal.

B. Query Generation Pipeline

The process involves four main steps:

1. Question Parsing:

   • The LLM parses the natural language input to extract Component Relation Paths (e.g., Service $\to$ Pod $\to$ Node) and Metric-Component Pairs (e.g., "available memory" $\to$ "Node").
   • Uses Chain-of-Thought (CoT) and few-shot prompting.

2. System Component Knowledge Retrieval:

   • The extracted relation paths are used to query the KG.
   • Path Retrieval: A Breadth-First Search (BFS) algorithm retrieves specific "reasoning paths" (instances of the schema) that match the user's intent, resolving placeholders (e.g., finding the actual pod-1 for a generic "order service").
   • Handles entity name variations via similarity search (BM25).

3. Metric Knowledge Retrieval:

   • Based on the retrieved component types, the system retrieves candidate metrics from the KG.
   • Semantic Matching: Uses BM25 to rank metrics based on the description in the user's question.
   • Label-Value Retrieval: Identifies relevant label-value pairs (e.g., specific node IDs) by traversing paths from the metric to the retrieved components.

4. Query Generation:

   • The LLM receives the original question plus the retrieved reasoning paths and metric details as context.
   • Using CoT prompting, the LLM generates the final PromQL query.

3. Key Contributions

1. PromCopilot Framework: The first framework designed specifically for text-to-PromQL, utilizing a knowledge graph to bridge the gap between natural language and complex, context-dependent DSLs.
2. System Context Knowledge Graph: A novel schema and automated construction method to capture the dynamic topology and metric relationships of cloud-native systems.
3. Benchmark Dataset: The creation of the first public text-to-PromQL benchmark dataset, containing 280 manually constructed question-PromQL pairs based on the TrainTicket microservice system.
4. Synergistic Reasoning: A demonstrated approach where the KG handles multi-hop reasoning and context retrieval, while the LLM handles natural language understanding and code generation, overcoming the limitations of using LLMs alone.

4. Experimental Results

The authors evaluated PromCopilot using GPT-4-Turbo and DeepSeek-Coder-V2-236B against baselines (Basic Prompt and Few-shot Learning).

• Accuracy (QueryAcc): PromCopilot achieved 69.1% accuracy with GPT-4-Turbo, significantly outperforming the best baseline (Few-shot with 10 examples at 37.4%).
• Metric Accuracy (MetricAcc): Achieved 91.3%, indicating the system correctly identifies the necessary metrics even if the final query syntax varies slightly.
• Retrieval Performance:
  • Metrics Retrieval: Recall of 90.3% and Precision of 90.7%.
  • Reasoning Paths Retrieval: Recall of 90.8%, demonstrating the KG's ability to find the correct system topology.
• Ablation Study: Removing either metric knowledge or system component knowledge caused a drastic drop in QueryAcc (down to ~27.8% and ~12.2% respectively), proving both are essential.
• User Study: In a pilot study with 8 industry engineers, PromCopilot reduced the average time to write a query from 382 seconds (using other tools) to 101 seconds, with high user satisfaction regarding usefulness (3.75/4.0).

5. Significance and Future Work

• Significance: PromCopilot addresses a critical bottleneck in AIOps (Artificial Intelligence for IT Operations) by democratizing access to complex monitoring data. It allows engineers to query system states without memorizing complex DSL syntax or manually hunting for topology details.
• Limitations: The current implementation is specific to Kubernetes and Prometheus. Failure analysis showed that ~59% of errors occur in the final query generation step (e.g., incorrect label usage), suggesting a need for PromQL-specific fine-tuning of LLMs.
• Future Directions:
  • Extending the framework to other monitoring platforms (e.g., InfluxDB, Azure Monitor).
  • Fine-tuning LLMs specifically for PromQL generation.
  • Implementing feedback loops to correct generation errors automatically.

In conclusion, PromCopilot represents a significant step forward in making cloud-native observability more accessible and efficient by combining the reasoning power of Knowledge Graphs with the generative capabilities of LLMs.