FlyCatcher: Neural Inference of Runtime Checkers from Tests

Technical Summary: FlyCatcher: Neural Inference of Runtime Checkers from Tests

1. Problem Statement

Complex software systems are prone to silent failures—violations of intended semantics that do not trigger explicit errors (e.g., crashes or exceptions). While runtime checkers (semantic monitors) are a powerful way to detect these failures, they are rarely used in practice because writing them manually is labor-intensive, requires deep domain expertise, and is difficult to scale to complex systems.

Existing automated approaches, such as T2C, attempt to derive checkers from existing tests. However, they face three critical challenges:

• Workload Generalization: Tests are written for specific inputs (e.g., a specific name for a data structure), whereas checkers must work for arbitrary inputs.
• Semantic Understanding: Traditional static analysis struggles to distinguish between "magic values" (constants) and actual data values, leading to incorrect generalizations.
• Statefulness: Many meaningful properties (e.g., list length, session counts) depend on the history of operations. Prior tools lack a mechanism to track and reason about the system's evolving internal state.

2. Methodology

The authors propose FlyCatcher, an automated approach that transforms existing tests into generalized, stateful runtime checkers using a combination of Large Language Models (LLMs), static analysis, and dynamic validation.

The process is divided into two phases:

A. Offline Phase (Inference & Validation):

1. Identifying State-Changing Methods: FlyCatcher uses an LLM to analyze test code and method implementations to identify which method calls actually modify the system's state (e.g., add, remove, set).
2. Checker Synthesis (LLM-based): Instead of using rigid templates, FlyCatcher prompts an LLM (e.g., Claude 3.5 Sonnet or GPT-4o) to "understand" the intent of a test. The LLM is provided with the target test, context tests, and strict guidelines to generate a Java method that:
   • Maintains a Shadow State: A hierarchical mapping that acts as a lightweight, isolated abstraction of the system's internal state.
   • Updates this shadow state whenever a state-changing method is invoked.
   • Performs assertions by comparing the real system state against the shadow state.
3. Iterative Refinement (Feedback Loop): To ensure correctness, the generated code undergoes:
   • Static Validation: Checking for syntax errors, missing assertions, or incorrect method signatures.
   • Dynamic Validation: Running the checker against a suite of validation tests. If the checker fails a test or causes a timeout, the error message is fed back to the LLM to refine the code iteratively.

B. Online Phase (Deployment):
The validated checker is deployed via instrumentation. The system's source code is modified (source-to-source transformation) to wrap state-changing methods in a try-finally block that invokes the checker, passing the current operation and the shadow state.

3. Key Contributions

• First LLM-based Test-to-Checker Approach: Leverages the semantic reasoning of LLMs to move beyond low-level event correlations to high-level semantic properties.
• Shadow State Mechanism: Introduces a way for inferred checkers to track and reason about complex, stateful system behaviors.
• Automated Refinement Pipeline: A robust feedback loop that uses both static and dynamic signals to self-correct generated code.

4. Results

The approach was evaluated on 400 tests from four major Java systems (Zookeeper, Cassandra, HDFS, and HBase).

• Effectiveness: FlyCatcher (using Claude 3.5 Sonnet) inferred 334 checkers, of which 300 were correct via cross-validation. This is 2.6x more correct checkers than the state-of-the-art T2C approach.
• Bug Detection: In mutation testing (using Zookeeper), FlyCatcher-generated checkers detected 5.2x more mutants than T2C, successfully catching bugs that the original test suites missed.
• Low False Positives: In distributed system stress tests (Jepsen), the false positive rate was extremely low (e.g., 0.00003 in Zookeeper).
• Efficiency & Cost: Generating a checker takes approximately 15 seconds and costs roughly $0.60 USD (using Claude). The runtime overhead is modest, ranging from 2.7% to 40.3% depending on the system.

5. Significance

FlyCatcher bridges the gap between "testing" (which is often too specific to catch subtle bugs) and "runtime monitoring" (which is often too expensive to implement manually). By automating the derivation of high-quality, stateful semantic checkers from existing assets (tests), it makes continuous, production-level monitoring a practical and scalable defense against silent failures in complex software.