Call-Chain-Aware LLM-Based Test Generation for Java Projects

Technical Summary: Call-Chain-Aware LLM-Based Test Generation for Java Projects (CAT)

1. Problem Statement

While Large Language Models (LLMs) have shown promise in generating unit tests, existing state-of-the-art (SOTA) approaches (such as PANTA) primarily rely on execution-path feedback to guide prompt construction. This approach is often insufficient for large-scale, project-level Java software due to:

• Complex Inter-class Dependencies: Methods often require specific object instances that depend on external classes or third-party libraries.
• Intricate Initialization Requirements: Simply knowing an execution path does not tell the LLM how to instantiate the necessary objects (e.g., which constructors to call or which concrete implementations of an abstract class to use).
• Deep Call Chains: To reach a "focal method" (the method under test), a specific sequence of method calls across multiple modules may be required.
• Method Overloading: Real-world Java code frequently uses overloaded methods, which can confuse LLMs if the specific parameter types and contexts are not clearly defined.

2. Methodology

The authors propose CAT, a novel approach that explicitly incorporates call-chain and dependency contexts into LLM prompts through a dedicated static analysis framework. The workflow consists of two main phases:

A. Static Analysis Framework (Preprocessing):
Before generation begins, CAT performs a one-time analysis to build a rich context:

• Call Graph Construction: Uses Class Hierarchy Analysis (CHA) to build a call graph and identify potential invocation paths from public entry points to the target method.
• Call-Chain Extraction: Employs a depth-limited Depth-First Search (DFS) to find feasible sequences of method calls that lead to the focal method.
• Dependency Resolution: Systematically identifies all classes involved in the call chains and transitively expands this set by analyzing public constructors. This ensures the LLM knows exactly which classes need to be instantiated and how.
• Filtering: Refines paths to ensure only public, non-abstract classes/methods are suggested as entry points, making the generated tests executable.

B. Iterative Generation and Fixing Phase:

• Generation Phase: CAT uses a structured prompt containing the Call-Chain Context (to guide direct or indirect invocation) and the Initialization Context (to guide object construction). It prioritizes direct invocation but allows for indirect invocation via the call chain if the focal method is hard to reach directly.
• Fixing Phase: If generated tests fail to compile or pass assertions, CAT enters a repair loop. It feeds the failed test code and the specific error messages (compiler or runtime) back into the LLM to perform targeted repairs.

3. Key Contributions

• Context-Aware Prompting: The first approach to explicitly integrate call-chain and dependency-aware execution contexts into LLM-based project-level test generation.
• Dedicated Static Analysis Tool: A specialized framework (implemented via SootUp) that automates the extraction of complex initialization and invocation metadata.
• Iterative Repair Mechanism: A robust feedback loop that leverages execution errors to improve test validity and compilability.
• Open-Source Toolkit: The authors released their toolkit to support reproducibility in the research community.

4. Experimental Results

The authors evaluated CAT on the Defects4J benchmark and four unseen GitHub projects (released after the LLM's training cutoff to prevent data leakage).

• Defects4J Performance: CAT outperformed the SOTA approach (PANTA) by 18.04% in line coverage and 21.74% in branch coverage. Even when compared against a time-bounded version of PANTA (to ensure a fair comparison of computational budget), CAT maintained significant leads.
• Generalization (Unseen Projects): On more complex, multi-module GitHub projects, CAT improved line coverage by 25.09% and branch coverage by 25.91% over PANTA. This demonstrates that the approach is robust against "memorization" and works on real-world, complex architectures.
• Ablation Study: Results showed that removing the call-chain or dependency context (the CATvanilla variant) led to a massive drop in coverage, proving that both components are critical to the system's success.

5. Significance

This paper shifts the paradigm of LLM-based testing from path-guided (focusing on where to go) to context-aware (focusing on how to get there). By bridging the gap between high-level LLM reasoning and low-level program structure through static analysis, CAT provides a scalable solution for generating high-quality unit tests in complex, industrial-grade software systems. It highlights that for project-level testing, the "context of how to instantiate and invoke" is just as important as the "logic of the code itself."