Technical Summary: MocklessTester

Problem Statement

While Large Language Models (LLMs) have demonstrated potential for automated test generation, existing approaches for Java primarily rely on mocking frameworks (e.g., Mockito) to isolate the method under test (MUT) from its dependencies. This reliance introduces two fundamental limitations:

1. Shallow Coverage: Mocked tests exercise the MUT in isolation, leaving the actual behavior of dependency classes untested.
2. Mock Fragility: Tests become tightly coupled to assumed API behaviors, breaking when dependencies evolve even if the real behavior remains compatible.

Mockless testing, which instantiates and executes real dependencies, offers a solution but faces significant challenges. Generating valid mockless tests requires strict adherence to language constraints, correct object instantiation, and valid API call sequences. Current LLM-based approaches struggle here due to two root causes of hallucination:

• Not Knowing: The LLM lacks sufficient project-specific context (e.g., valid constructors, factory methods, or usage patterns) to generate correct code.
• Not Following: Even when constraints are provided, the LLM fails to comply with them (e.g., violating symbol validity or API call protocols), often due to a lack of enforcement mechanisms.

Methodology: MocklessTester

The authors propose MocklessTester, a mockless unit test generation approach for Java that employs an iterative plan–generate–validate–fix workflow. The system is built around two core strategies to address the identified root causes: Context-enriched Generation and Constraint-enforced Fixing.

1. Preparation Stage

Before test generation begins, MocklessTester analyzes the target repository to construct three project-specific artifacts:

• Code Property Graph (CPG): Built using Joern, this graph captures dependency usage, enabling the retrieval of real object construction and API-call examples via program slicing.
• ClassIndex: A record of all project-visible classes, methods, constructors, and imports. It serves as a ground truth for validating symbols and resolving types during repair.
• Markov Typestate Model: A probabilistic model capturing valid method-call orders for stateful APIs. It is initialized from source code evidence and updated dynamically based on test outcomes to detect illegal call sequences.

2. Iterative Multi-Agent Loop

The generation process involves five agents: Initializer, Planner, Generator, Validator, and Fixer.

• Context-enriched Generation: To mitigate "Not Knowing," the Generator does not rely solely on the CUT source code. Instead, a program slicer mines real instantiation and usage patterns of dependencies from the project's production and test code. These patterns are injected into the prompt to guide the LLM in creating plausible object instantiations.
• Constraint-enforced Fixing: To mitigate "Not Following," the Fixer employs a two-stage repair mechanism for failing tests:
  • Stage I (Fixer I): Generates an initial fix based on execution feedback (compilation/runtime errors).
  • Stage II (Fixer II): If the initial fix violates constraints, a second repair is triggered. The LLM must generate a fix that satisfies three levels of constraints and provide a structured justification explaining how the fix resolves the failure and adheres to the rules:
    • Symbol-level: Ensures all classes, methods, and imports exist in the ClassIndex.
    • Protocol-level: Ensures API call sequences respect the Markov typestate model (e.g., initializing state before usage).
    • Iteration-level: Leverages Experience Memory, which stores successful repair patterns ("gold tests"), fix recipes, and anti-patterns to avoid repeating known failures.

Key Contributions

1. Context-enriched Prompting: The authors identify dependency usage patterns as critical context. They propose a strategy that augments the LLM with real dependency instantiation examples mined from the target project, significantly reducing hallucinations caused by insufficient knowledge.
2. Constraint-enforced Fixing Mechanism: A novel two-stage repair process that enforces symbol, protocol, and iteration-level constraints. Crucially, the second stage requires the model to generate a structured justification for its corrections, directly targeting "not following" hallucinations.
3. First Mockless LLM Approach for Java: MocklessTester is presented as the first LLM-based approach for Java that explicitly targets external dependencies, advancing automated testing from mocked unit testing toward realistic execution.
4. Comprehensive Empirical Evaluation: The study introduces Dependency Line Coverage (DepLC) as a new metric to measure the extent of real project code exercised beyond the Class Under Test (CUT).

Experimental Results

The approach was evaluated on Defects4J (14 projects, 130 classes) and a new benchmark, Deps4J (5 recent, multi-module projects, 30 classes), against the state-of-the-art baseline PANTA.

• Effectiveness:

  • On Defects4J, MocklessTester improved average line coverage by 19.99% (68.83% → 88.82%) and branch coverage by 24.90% (58.84% → 83.74%). The mutation score increased by 13.67%.
  • On Deps4J, it improved line coverage by 22.69% and branch coverage by 15.78%, while maintaining comparable mutation scores.
  • Dependency Coverage: MocklessTester covered significantly more real dependency code, increasing project DepLC by 378 lines on Defects4J and 55 lines on Deps4J compared to the baseline.
  • Test Efficiency: Despite generating fewer tests per project on average, MocklessTester achieved higher coverage, indicating higher quality per test.

• Efficiency:

  • MocklessTester incurred higher total token and time costs due to performing more iterations to reach higher coverage.
  • However, the cost remains practical: averaging 108.97 seconds and 26.59k tokens per method on Defects4J.
  • Notably, MocklessTester was substantially faster per iteration (approx. 58% less time) than the baseline, suggesting its iterations are more time-efficient even if more numerous.

• Ablation Study:

  • Removing Usage Retrieval caused the largest drop in performance, confirming the importance of context.
  • Removing Protocol-level constraints had the second-largest impact, highlighting the difficulty of managing stateful objects without explicit enforcement.
  • Symbol-level and Iteration-level constraints also provided significant, though slightly smaller, improvements.

Significance and Claims

The paper claims that MocklessTester successfully addresses the limitations of mock-based testing by enabling the generation of tests that exercise real project dependencies. By systematically addressing "not knowing" through context enrichment and "not following" through constraint enforcement, the approach achieves state-of-the-art results in line coverage, branch coverage, and mutation score on both established and new benchmarks.

The authors emphasize that while the approach requires higher computational resources than baseline methods, the trade-off yields tests that are not only more comprehensive in coverage but also more robust against dependency evolution. The introduction of DepLC provides a new dimension for evaluating test suites, moving beyond the CUT to measure the actual execution of the broader system. The work suggests that future improvements could involve richer memory mechanisms and grammar-aware input generation for parser-centric components.