Help Without Being Asked: A Deployed Proactive Agent System for On-Call Support with Continuous Self-Improvement

This paper introduces Vigil, a deployed proactive agent system that integrates into human-analyst on-call dialogues to offer unsolicited assistance and continuously self-improve by learning from resolved cases, thereby reducing support workloads on ByteDance's Volcano Engine platform.

The Gist

The Problem: The "One-and-Done" Robot

Imagine you have a very smart, helpful robot assistant named Reactive. You call the support line for your cloud server, and Reactive answers.

• You: "My database won't connect."
• Reactive: "Have you checked your settings?"
• You: "Yes, I tried that, but it's still broken."
• Reactive: "Oh no, that's too hard for me. I'm hanging up now. Good luck!"

In this scenario, the moment the problem gets slightly complicated, the robot gives up and hands you over to a human expert. Once the human takes over, the robot disappears completely. It doesn't listen to the conversation, it doesn't help with the next question, and it doesn't learn from how the human fixed the problem. It's like a lifeguard who jumps in to save you, but the second you start swimming toward the shore, they swim away and leave you to finish the swim alone.

The Solution: Meet Vigil

The authors of this paper built a new kind of robot called Vigil. Instead of being a "first responder" that quits when things get tough, Vigil is a Shadow Assistant that stays right next to the human expert the whole time.

Think of Vigil as a super-attentive intern sitting next to a senior doctor during a surgery.

• The senior doctor (the human analyst) is doing the heavy lifting.
• Vigil is watching the patient (the customer) and the doctor.
• If the patient asks a simple question like, "Will this hurt?" or "What time is the next check-up?", Vigil jumps in and answers immediately without the doctor needing to ask.
• If the patient asks something complex, Vigil stays quiet and lets the doctor handle it.
• Crucially, Vigil never leaves the room. Even after the doctor takes over, Vigil keeps listening, ready to help with the next simple question.

How Vigil Works (The Magic Tricks)

1. The "Eagle Eye" (Context Awareness)

Vigil doesn't just talk; it watches. It knows exactly when to speak and when to stay silent.

• The Analogy: Imagine a butler at a fancy dinner party. If the host is busy explaining a complex wine pairing, the butler doesn't interrupt. But if a guest asks, "Is there a napkin?", the butler instantly slides one over. Vigil does the same in chat: it knows the difference between a question it can answer and one that needs a human.

2. The "Instant Library" (Continuous Learning)

This is the most powerful part. In the old system, if a human expert solved a weird, new problem, that knowledge died with that conversation.

• The Analogy: Imagine a chef who invents a delicious new sauce to fix a burnt steak. In the old system, the recipe is lost. With Vigil, the robot instantly writes down the recipe and adds it to the master cookbook.
• The next time a customer has a burnt steak, Vigil doesn't wait for the chef; it pulls out the new recipe and says, "I know how to fix this!" This happens automatically, 24/7, without anyone needing to manually update a database.

3. The "Double-Check" (Self-Improvement)

Vigil isn't perfect. Sometimes it guesses wrong. But it has a built-in "second opinion" system.

• The Analogy: If Vigil suggests a solution and the customer says, "That didn't work," or the human expert gives a different answer, Vigil doesn't get defensive. It quietly goes back, checks its notes, and realizes, "Oh, I was missing a detail." It then updates its own brain to ensure it never makes that specific mistake again. It turns every failure into a lesson.

Why This Matters (The Results)

The team tested Vigil on Volcano Engine, a massive cloud platform used by ByteDance (the company behind TikTok).

• The Stats: Over 10 months, Vigil handled over 130,000 support calls.
• The Impact: It answered 136,000 questions proactively.
• The Efficiency: In about 40% of cases, Vigil was able to answer questions even after a human had already joined the chat. This means the human expert didn't have to stop and type basic answers; they could focus on the hard stuff.
• The Learning: Vigil taught itself from over 270,000 new pieces of information, constantly getting smarter.

The Big Takeaway

The paper argues that we shouldn't just build AI that waits to be asked. We should build AI that watches, learns, and helps without being told.

Think of it like the difference between a flashlight and a night-vision guide.

• A flashlight (Reactive Agent) only shines where you point it. If you stop pointing, it goes dark.
• A night-vision guide (Vigil) sees everything in the dark, anticipates where you might trip, and gently nudges you before you fall, all while learning the terrain better every time you walk together.

This system makes customer support faster, cheaper, and much less stressful for the humans who have to deal with angry or confused customers. It's not about replacing the human; it's about giving them a super-powered sidekick that never sleeps.

Technical Summary

1. Problem Statement

Large-scale cloud platforms (specifically ByteDance's Volcano Engine) face a critical bottleneck in on-call support due to the high volume of daily customer tickets. While recent research has introduced reactive agents (LLM-based chatbots) to handle initial inquiries, these systems suffer from two major limitations in real-world deployment:

• Premature Disengagement: Reactive agents typically operate only until a query is escalated to a human analyst. Once a human intervenes, the agent disengages, missing the opportunity to assist with subsequent sub-questions, track resolution progress, or reduce the human analyst's workload during the collaborative phase.
• Static Knowledge & Lack of Learning: Existing agents rely on manually curated knowledge bases (RAG). In dynamic cloud environments, new symptoms, workarounds, and fixes emerge rapidly during live incidents. Reactive agents cannot learn from these ongoing human-led resolutions in real-time, leading to duplicated troubleshooting efforts and delayed responses for similar future issues.

2. Methodology: The Vigil System

The authors propose Vigil, a novel proactive agent system designed to operate throughout the entire on-call lifecycle, specifically targeting the phase where human analysts are already involved. Vigil acts as an autonomous assistant integrated directly into the dialogue between the customer and the human analyst.

The system architecture consists of two primary modules:

A. Online Proactive Response

Unlike reactive agents that wait for explicit prompts, Vigil continuously monitors the dialogue stream.

1. Context-Aware Question Identification: Vigil analyzes incoming messages to classify them into three categories: Within Scope (cloud platform issues), Out of Scope (subjective/complex decisions), or No Assistance Needed (greetings/resolved). This ensures the agent only intervenes when it can add value without being intrusive.
2. Answer Generation: For identified questions, Vigil rewrites the query to resolve coreferences, then employs a Multi-path Knowledge Retrieval strategy. This combines:
   • Domain-specific document knowledge bases.
   • Historical Q&A pairs from past on-calls.
   • Tool invocation (e.g., retrieving logs, alerts, and diagnostic metadata).
   • A reranking model ensures the most relevant context is used for generation.
3. Answer Deduplication: To prevent repetitive responses in multi-turn conversations, Vigil calculates semantic similarity between new answers and historical answers within the current session. If similarity exceeds a threshold ($\theta=0.7$), the response is suppressed.
4. Interface Design: Responses are delivered via a distinct "Proactive Response Card" that visually distinguishes the AI from the human analyst, includes citations for verifiability, and features an "Accept" button for feedback.

B. Continuous Self-Improvement Mechanism

Vigil autonomously updates its knowledge base by learning from every interaction, reducing reliance on manual curation.

1. Learning from Unanswered Questions: When Vigil fails to answer and a human analyst provides a solution, Vigil parses the dialogue to extract the solution, structuring it as a new Q&A pair for the knowledge base.
2. Learning from Answered Questions:
   • Positive Feedback: If a user/analyst clicks "Accept," the Q&A pair is saved.
   • Implicit Review: Since explicit negative feedback is rare and ambiguous, Vigil employs an automated review agent. If a human analyst later provides a different or more detailed answer, the system compares the two.
     • Keep: If the human answer aligns with Vigil's.
     • Update: If the human answer adds necessary context (e.g., specific SDK versions), Vigil refines the entry.
     • Delete: If Vigil's answer is proven incorrect or the reference is outdated, the entry is removed.
3. Learning from External Documents: Vigil parses links to official documentation shared during chats, extracting structured knowledge to ensure answers remain current with API changes and versioning.

3. Key Contributions

1. Proactive Paradigm Shift: Introduced a system that extends the agent's action cycle beyond the initial triage, allowing it to collaborate with human analysts rather than being replaced by them.
2. Continuous Self-Improvement: Designed an automated loop that extracts knowledge from both successful and failed human-led resolutions, enabling the agent to evolve without manual knowledge base updates.
3. Real-World Deployment: Successfully deployed Vigil on the Volcano Engine production environment for over 10 months, validating the system in a high-stakes, high-volume industrial setting.

4. Experimental Results

The system was evaluated using 300 non-sensitive on-call sessions and production statistics from 131,433 on-calls.

• Extended Support Coverage (Q1): In 40.5% of on-calls where humans intervened, Vigil continued to provide valuable assistance, addressing sub-questions that would otherwise wait for the analyst.
• Question Identification (Q2): The dedicated identification module significantly improved performance across various foundation models. For example, using GPT-5-mini, the F1-score improved by 203.2% compared to a baseline without the module.
• Answer Accuracy (Q3): Ablation studies showed that the full self-improvement mechanism is critical. On the Seed-1.6 model (used in production), the accuracy of proactive answers reached 59.1%. Removing the self-improvement module dropped accuracy to 15.4%, demonstrating that continuous learning is essential for reliability.
• Deduplication (Q4): A threshold of $\theta=0.7$ for semantic similarity provided the optimal trade-off, maximizing the weighted F1-score at 0.752 while effectively suppressing redundant responses.

5. Significance and Lessons Learned

• Operational Efficiency: Vigil reduces the workload of human analysts by handling routine sub-questions during the "human-involved" phase, shortening end-to-end resolution times.
• Dynamic Knowledge Adaptation: The system solves the "knowledge lag" problem common in cloud support, where new fixes often appear in chat logs before they are documented. Vigil captures these immediately.
• Latency vs. Reasoning: The authors note that in on-call scenarios, speed is critical. Complex, multi-step reasoning that introduces latency is less valuable than a timely, accurate answer. Vigil prioritizes low-latency proactive intervention.
• Feedback Ambiguity: The system overcomes the ambiguity of user feedback by using dialogue context to validate answers rather than relying solely on explicit "thumbs up/down" signals.

Conclusion: Vigil demonstrates that proactive agents can effectively complement human experts in high-stress, dynamic environments. By staying active after escalation and continuously learning from live interactions, it transforms the on-call process from a linear handoff into a collaborative, self-improving ecosystem. The open-source version is available at github.com/volcengine/veaiops.