🤖 AI

Bian Que: An Agentic Framework with Flexible Skill Arrangement for Online System Operations

Bian Que is an agentic framework deployed on KuaiShou's search engine that addresses the orchestration bottleneck in large-scale system operations by introducing a unified operational paradigm, flexible skill arrangement for dynamic data and knowledge retrieval, and a self-evolving mechanism, ultimately reducing alert volumes by 75% and cutting mean time to resolution by over 50%.

Original authors: Bochao Liu, Zhipeng Qian, Yang Zhao, Xinyuan Jiang, Zihan Liang, Yufei Ma, Junpeng Zhuang, Ben Chen, Shuo Yang, Hongen Wan, Yao Wu, Chenyi Lei, Xiao Liang

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bochao Liu, Zhipeng Qian, Yang Zhao, Xinyuan Jiang, Zihan Liang, Yufei Ma, Junpeng Zhuang, Ben Chen, Shuo Yang, Hongen Wan, Yao Wu, Chenyi Lei, Xiao Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive, high-speed spaceship (like Kuaishou's search engine) that carries hundreds of millions of passengers every day. The ship is so complex that it has thousands of moving parts, and every single day, the engineering crew makes dozens of tiny adjustments to the engines, navigation systems, and life support.

In the past, keeping this ship running safely was a nightmare. Every time an engineer made a change, alarms would start blaring. The human crew had to run around, check thousands of gauges, read endless logs, and guess what was wrong. It was exhausting, slow, and often led to mistakes because there was just too much information to process.

Enter BIAN QUE.

Think of BIAN QUE not as a single robot, but as a super-smart, self-improving "Chief of Staff" for your engineering team. It doesn't just react to problems; it anticipates them, diagnoses them, and learns from every single event to get better at its job.

Here is how it works, broken down into three simple concepts:

1. The Three Lines of Defense (The "Shifts")

Instead of waiting for a disaster to happen, BIAN QUE organizes the work into three specific shifts, much like a hospital:

The Gatekeeper (Release Interception): Before a new update is even launched, this agent checks it. It's like a strict bouncer at a club who checks IDs and bags before you enter. If something looks risky, it stops the launch immediately, preventing problems before they start.
The Health Inspector (Proactive Inspection): This agent wanders the ship regularly, checking the engine temperature and fuel levels even when no alarms are ringing. It finds "silent" issues that might cause a crash later, fixing them before anyone even knows there's a problem.
The Detective (Alert Root Cause Analysis): If an alarm does go off, this agent is the detective. Instead of panicking, it instantly gathers the right clues (logs, metrics) and asks, "What actually broke?" It gives the human crew a clear answer, saving them hours of guessing.

2. The "Flexible Skill" System (The "Toolbox")

This is the paper's biggest innovation. In the past, you had to manually write a rulebook for every single part of the ship. If the ship changed, the rulebook became useless.

BIAN QUE uses Flexible Skills. Imagine a magical toolbox where every tool knows exactly what job it's for.

If the problem is with the "Search Engine," the agent pulls out the "Search Tool."
If the problem is with the "Video Player," it pulls out the "Video Tool."

How does it know which tool to use?
It doesn't rely on a static manual written by humans. Instead, the AI writes its own instructions based on the situation.

Generation: If a new part of the ship is added, the AI looks at what data is available and writes a new "Skill" (a set of instructions) on the fly to handle it.
Correction: If the AI makes a mistake, a human engineer can just say, "Hey, you forgot to check the fuel gauge," in plain English. The AI instantly rewrites its own instructions to include that step next time. It's like a student who gets a correction on a test and immediately updates their study guide.

3. The Self-Evolving Loop (The "Memory")

Most computer systems are static; they do exactly what they were programmed to do, even if the world changes. BIAN QUE is different because it has a dual-memory system that learns from every single event.

When an engineer gives feedback (e.g., "That diagnosis was wrong"), BIAN QUE uses that one piece of feedback to improve in two ways simultaneously:

Updating the Knowledge Base: It saves the lesson learned into a permanent "encyclopedia" of how the ship behaves.
Refining the Skills: It updates the specific "tool" (the Skill) used for that job so it doesn't make the same mistake again.

Think of it like a chef who tastes a dish. If the dish is too salty, the chef doesn't just throw the dish away; they update the recipe book (Knowledge) and adjust their specific technique for salting (Skill) for next time.

The Results: What Happened on Kuaishou?

The team tested this system on Kuaishou's real-world search engine (a massive platform with hundreds of millions of users). The results were like turning a chaotic emergency room into a well-oiled machine:

Fewer Alarms: The number of alarms that actually required human attention dropped by 75%. The system stopped the noise before it reached the humans.
Fewer False Alarms: The "noise" (alarms that were actually false or unimportant) dropped by about 95%. The humans only got paged when it truly mattered.
Faster Fixes: When a real problem did occur, the time it took to fix it (Mean Time to Resolution) was cut by more than 50%.
Accuracy: The system correctly identified the root cause of problems 80% of the time.

Why This Matters

The paper argues that the hardest part of using AI for operations isn't making the AI "smarter" at reasoning; it's figuring out what information to feed it. If you feed an AI everything (all logs, all metrics), it gets confused and hallucinates. If you feed it nothing, it can't solve the problem.

BIAN QUE solves this by acting as a smart filter. It dynamically decides exactly which data and which rules are needed for the specific problem at hand, and it teaches itself how to do this better every time a human gives it a little nudge.

In short, BIAN QUE turns a team of overworked engineers into a team of highly efficient supervisors, letting the AI handle the heavy lifting of data sorting and initial diagnosis, while the humans focus on the big decisions.

1. Problem Statement

Operating large-scale online engine systems (search, recommendation, advertising) requires significant human effort for release monitoring, alert response, and root cause analysis (RCA). While Large Language Model (LLM) agents offer a path to automation, current deployments face a critical bottleneck: orchestration, not reasoning capability.

The Context Gap: Feeding all available system signals (metrics, logs, change events) indiscriminately into an LLM causes information dilution and hallucinations.
The Knowledge Gap: Operational knowledge (handbooks, failure patterns) is often undocumented, shifts with every system iteration, and varies by business line.
The Scalability Issue: Manually curating the mapping between operational events and the specific data/knowledge required for diagnosis is intractable given the volume of daily releases and the combinatorial complexity of business modules.
The Feedback Loop: Existing systems lack a mechanism where a single feedback signal can simultaneously update the retrieval logic (what data to fetch) and the knowledge base (what to learn from the event).

2. Methodology: The BIAN QUE Framework

BIAN QUE is an agentic framework designed to abstract, orchestrate, and evolve operational workflows. It consists of three core pillars:

A. Unified Operational Paradigm

The framework abstracts heterogeneous operational tasks into three canonical patterns, each handled by a specialized agent:

Release Interception: Monitoring system behavior during deployments to prevent incidents before they occur.
Proactive Inspection: Periodic, comprehensive health checks to identify latent risks.
Alert Root Cause Analysis (RCA): Diagnosing and remediating issues after alerts are triggered.

All agents share a common computational structure: $o = f_{LLM}(D(e), K(e), p_s)$ , where $D(e)$ is retrieved data, $K(e)$ is operational knowledge, and $p_s$ is the prompt.

B. Flexible Skill Arrangement

To solve the orchestration problem, BIAN QUE introduces Skills, which are structured documents acting as orchestration units for specific business-module contexts.

Structure: A Skill contains a LoadDataSchema (JSON specifying which metrics/logs/changes to fetch), a Prompt (reasoning steps), and Meta (tags for matching).
Lifecycle:
- Generation: When a new business-module combination is encountered, an LLM generates a Skill based on seed descriptors (available data sources and capabilities). It undergoes a validate-and-retry loop to ensure data sources are valid.
- Update: Practitioners can correct a Skill via natural language. The system automatically identifies whether the error was due to missing data (updating LoadDataSchema) or flawed reasoning (updating Prompt).
Orthogonality: Agents handle the scenario logic (e.g., "is this release safe?"), while Skills handle the specific data/knowledge retrieval for that module. This allows independent iteration of agents and skills.

C. Unified Self-Evolving Mechanism

BIAN QUE implements a dual-pathway feedback loop driven by a single practitioner correction signal:

Memory-to-Knowledge Distillation: Case records are written to a short-term memory store (for immediate pattern recognition) and distilled into long-term persistent knowledge (failure patterns, causal relationships). This knowledge base is seeded with handbooks and updated via daily consolidation.
Targeted Skill Refinement: The same feedback signal triggers the Skill Update process to refine the data retrieval logic or reasoning prompts.

Result: The knowledge base and the context-assembly mapping co-evolve, eliminating the need for disconnected update pipelines.

3. Key Contributions

Unified Paradigm: Abstracts the entire O&M lifecycle into three defense lines (Interception, Inspection, RCA), moving beyond the traditional "alert-first" approach to proactive prevention.
Flexible Skill Arrangement: Replaces static configuration with LLM-generated, human-correctable Skills that dynamically map events to relevant data and knowledge.
Co-Evolutionary Feedback: A unified mechanism where one feedback signal simultaneously improves the agent's retrieval strategy (Skills) and its domain knowledge base.
Production Validation: Successfully deployed on Kuaishou's e-commerce search engine, demonstrating significant real-world impact.

4. Experimental Results

The framework was evaluated over a six-month period on Kuaishou's search engine (hundreds of millions of users).

Production Metrics:

Alert Volume Reduction: Reduced fired alerts by 75% (via release interception and proactive inspection).
Noise Reduction: Reduced non-actionable alerts reaching practitioners by ~95%.
RCA Accuracy: Achieved 80% accuracy in root cause analysis for alerts that required attention.
MTTR: Reduced Mean Time to Resolution by >50%.

Offline Evaluation (104 Real Events):

Skill Initialization: The automated generation achieved a 78.8% pass@1 rate. With 5 retry attempts, this rose to 94.2%.
Human-in-the-Loop: A single round of natural language correction on the remaining failures pushed the overall pass rate to 97.1%, and three rounds reached 99.0%.
Ablation Studies:
- Removing Flexible Skills (using static configs) dropped pass@5 from 94.2% to 83.7%, proving the necessity of dynamic skill generation.
- Removing historical knowledge retrieval dropped pass@5 to 86.5%, showing the critical role of accumulated domain knowledge.
- Feedback Loop: Without the feedback pathway, system accuracy degraded from ~75% to ~32% over 13 days. With feedback enabled, accuracy remained stable above 80%.

LLM Backbone Sensitivity:

Models with ~35B parameters (e.g., Qwen3.5-35B, GLM5, DeepSeek-V3.2) performed comparably and robustly.
Smaller models (e.g., 9B, 4B) showed catastrophic degradation in operational reasoning tasks, suggesting a minimum threshold for production deployment.

5. Significance

BIAN QUE represents a shift from static, rule-based AIOps to dynamic, self-evolving agentic systems.

Scalability: It solves the "combinatorial explosion" of mapping events to data by using LLMs to generate and maintain the mapping logic (Skills) rather than hard-coding it.
Sustainability: The self-evolving mechanism ensures the system adapts to rapid system changes without requiring constant manual re-engineering of the knowledge base or retrieval logic.
Proactive Operations: By prioritizing release interception and proactive inspection, it shifts the operational paradigm from reactive firefighting to preventive maintenance.
Generality: While deployed on a search engine, the framework is business-agnostic, relying on standard data-lake descriptors and modular Skill definitions, making it applicable to other large-scale online systems.

The paper concludes that the primary challenge in LLM-based O&M is not the reasoning capability of the model, but the context assembly (selecting the right data and knowledge), which BIAN QUE solves through its Flexible Skill and self-evolving architecture.