Technical Summary: Towards Agentic Runtime Healing

Problem Statement

Runtime errors are a critical challenge in software reliability engineering. While developers strive to anticipate and handle exceptions through static analysis, testing, and verification, unanticipated runtime errors inevitably occur. These uncaught errors can lead to severe consequences, including program crashes and data exposure. Traditional self-healing systems attempt to mitigate these issues using predefined heuristic strategies, such as restarting processes, rolling back to checkpoints, or reusing existing error handlers. However, these rule-based approaches struggle to adapt to the diverse and complex nature of runtime errors. Defining adaptation rules for every potential scenario is impractical, limiting the flexibility and effectiveness of current self-healing mechanisms in real-world applications.

Methodology

The paper proposes Healer, a framework that leverages Large Language Models (LLMs) to dynamically generate error-handling strategies in real-time. Unlike traditional patching methods that modify source code, Healer aims to recover the program by revising its runtime state (e.g., variable values, environment configurations) to allow execution to continue.

The Healer framework operates through three primary components:

1. Runtime Information Collector: Upon detecting an unhandled runtime error, this component gathers essential context, including the source code, the specific error message, the error location, and the current program state (variable values, stack traces, etc.).
2. Large Language Model (LLM): The collected context is formatted into a structured prompt and fed to an integrated LLM. The prompt includes task instructions, the error context, and examples. The LLM generates a snippet of "healing code" designed to resolve the specific error and produce a corrected program state.
3. Healing Runtime: The generated healing code is executed in a dedicated runtime environment (which can be a sandbox, virtual machine, or new session) using the program state existing immediately before the error occurred. If the healing code executes successfully, the new state is applied to the original program, allowing it to proceed from the point of failure. If the healing code fails, the system reverts to traditional error reporting.

The authors implemented a prototype for Python and evaluated it across four datasets: CodeNet (human-written code), HumanEval+, MBPP+, and LiveCodeBench (LLM-generated code), and DebugBench (code with implanted bugs). The evaluation utilized three state-of-the-art LLMs: GPT-3.5, GPT-4, and CodeQwen-7B.

Key Contributions

• Proposal of Agentic Runtime Healing: The paper introduces the concept of using LLMs as autonomous agents to handle runtime errors dynamically, marking a shift from static, rule-based healing to adaptive, context-aware recovery.
• Framework Design (Healer): The authors designed and implemented a proof-of-concept framework that integrates LLMs into the runtime error handling loop, capable of generating and executing bespoke healing code without human intervention.
• Empirical Evaluation: The study provides a comprehensive evaluation of LLMs in this domain, demonstrating that even zero-shot LLMs can effectively handle runtime errors. Furthermore, the paper investigates the impact of fine-tuning, creating an instruction-tuning dataset to improve model performance.
• Risk Analysis: The authors empirically analyzed the behaviors of LLM-generated healing code, identifying 19 distinct patterns of behavior (both recognizable and unrecognizable) to highlight potential risks and the need for safety constraints.

Results

The evaluation yielded the following key findings:

• Zero-Shot Performance: Without fine-tuning, GPT-4 demonstrated the highest capability, successfully allowing programs to proceed with execution in 72.8% of instances and producing correct results in 39.6% of instances across all benchmarks. GPT-3.5 and CodeQwen showed lower but significant success rates in the zero-shot setting.
• Impact of Fine-Tuning: Fine-tuning significantly improved performance. When fine-tuned on a dataset of successful (prompt, healing code) pairs, GPT-3.5's performance increased by 31.7% in "proceeding" instances and 26.8% in "correct" instances, achieving results comparable to the zero-shot GPT-4. CodeQwen also saw substantial gains, with a 65.8% increase in correct instances.
• Comparison to Traditional Strategies: Traditional strategies like retrying or rolling back were shown to be ineffective for the deterministic runtime errors in the benchmarks (yielding 0% success in proceeding). Simple "error steamrolling" (ignoring errors) allowed execution to continue but failed to correct the program state, resulting in a very low correct result rate (4.3%). Healer bridged the gap between simple crash avoidance and functional recovery.
• Behavioral Analysis: An analysis of 204 incorrect healing code samples revealed that 75% exhibited "recognizable" behaviors (e.g., value changes, variable initialization, library imports) that could potentially be constrained via static analysis. However, "unrecognizable" behaviors (e.g., context omission, function replacement) posed significant risks, highlighting the need for safety mechanisms.

Significance and Claims

The authors position this work as the first step toward agentic runtime healing. They claim that the results demonstrate the feasibility of using LLMs to dynamically generate error-handling strategies, offering a promising pathway toward more adaptive, resilient, and self-healing software systems.

The paper modestly acknowledges that while the results are promising, significant challenges remain before practical deployment. Specifically, the authors emphasize the trustworthiness of LLM-generated code as a primary hurdle. They note that LLMs may produce code that ignores key logic or introduces unexpected behaviors. Consequently, the paper argues that future work must focus on mitigation strategies, such as:

• Safety Checks: Implementing static checkers and whitelists to restrict LLM actions to safe, predefined behaviors.
• Healer-aware Programming: Designing software systems that accommodate the uncertainty of Healer, such as confining its interventions to non-critical modules.
• Reliable Skillsets: Providing LLMs with restricted toolkits of safe functions to ensure predictable outcomes.

Ultimately, the paper concludes that while the technology is not yet ready for fully autonomous, risk-free deployment, it represents a significant advancement in the field of software reliability, paving the way for future research into robust, AI-driven self-healing systems.