Improving LLM-Driven Test Generation by Learning from Mocking Information

1. Problem Statement

While Large Language Models (LLMs) have shown promise in automated unit test generation, existing approaches typically rely only on the source code of the class under test (CUT) and its documentation. They often fail to leverage test doubles (mocks) found in existing developer-written test suites.

• The Gap: Mocks contain rich, implicit knowledge about how dependent components should behave, including specific method invocations, argument values, return values, and interaction expectations (stubbings and verify operations).
• The Challenge: Current LLM-based generators do not utilize this "mocking information," potentially missing complex usage patterns, paging mechanisms, or specific edge cases that developers have already encoded in their test doubles. This leads to generated tests that may be syntactically correct but lack behavioral depth or fail to cover critical code paths.

2. Methodology: MOCKMILL

The authors propose MOCKMILL, an LLM-based technique and tool designed to generate tests for components that are replaced by test doubles in existing tests. It operates through a four-phase workflow:

1. Project Analysis:
   • Parses the Abstract Syntax Tree (AST) of existing test files (specifically JUnit 5 and Mockito in Java).
   • Identifies components replaced by test doubles and filters out third-party libraries to focus on project-specific code.
2. Mock Extraction (Novelty):
   • Extracts specific structural data regarding the mocked components, including:
     • Stubbings: when(...).thenReturn(...) calls (defining inputs and return values).
     • Verify Operations: verify(...) calls (defining expected interactions).
   • This data is structured into a JSON format to avoid overloading the LLM's context window with full test files.
3. Test Generation:
   • Constructs a prompt for the LLM containing:
     • Instructions to target mutation-based adequacy.
     • The full source code of the target component.
     • The extracted mock data (stubbings and verify operations).
     • Few-shot examples of domain-agnostic test generation.
   • The LLM is instructed to generate tests that instantiate the real CUT (not a mock) and exercise the behaviors implied by the extracted mock data.
4. Post-Generation Repair:
   • Implements an iterative loop where generated tests are compiled and executed.
   • If compilation or runtime errors occur, error logs are fed back to the LLM with specific repair instructions until the test passes or a retry limit is reached.

3. Key Contributions

• First of its Kind: MOCKMILL is the first approach to explicitly leverage existing developer-defined mocking information (stubbings and verifies) to guide LLM-driven test generation.
• Complementarity: It targets components that are mocked in existing tests, generating new tests for the actual implementation of those components, thereby filling gaps left by both developer tests and standard LLM generators.
• Tool Implementation: A prototype tool supporting Java projects (Maven/Gradle) using JUnit 5 and Mockito, with static analysis via JavaParser.
• Empirical Validation: A comprehensive evaluation across six open-source Java projects and four different LLMs.

4. Experimental Results

The authors evaluated MOCKMILL on 10 classes from 6 open-source Java projects using four LLMs (GPT-4o Mini, GPT-5 Mini, GPT-5, and Claude Sonnet 4.5).

• Effectiveness (RQ1):
  • MOCKMILL successfully generated compilable and executable tests with high reliability (up to 100% eventual compilation rate).
  • GPT-5 Mini emerged as the most practical model, offering the best trade-off between cost, compilation success, and high median mutation scores (84%) and line coverage (93%).
• Complementarity (RQ2):
  • Unique Fault Detection: MOCKMILL killed unique mutants that were missed by both baseline LLMs (without mock info) and random testing (Randoop). For example, on the RRA component, MOCKMILL killed 25.64% unique mutants, compared to 12.82% for the baseline LLM.
  • Unique Coverage: It covered unique lines of code (e.g., 2.61% on RRA) that other approaches missed, particularly in complex scenarios like paging mechanisms.
  • Comparison: It outperformed Randoop significantly in fault detection and showed better complementarity than standard LLM generation.
• Cost (RQ3):
  • The inclusion of mock information increased the prompt length, resulting in a 5–15% cost overhead compared to baseline LLM generation.
  • Despite the slight cost increase, the gain in unique fault detection and code coverage was deemed a favorable trade-off for practical adoption.

5. Significance and Conclusion

• Paradigm Shift: The study demonstrates that mocking information is not just a tool for isolation but a valuable knowledge base for generating realistic tests. It proves that "learning from mocks" allows LLMs to infer complex usage patterns that are otherwise invisible when only looking at source code.
• Complementary Approach: MOCKMILL does not replace existing testing methods but enhances them. It suggests a future framework where LLM-driven testing combines both "mock-informed" and "non-mock-informed" prompts to maximize test suite diversity and coverage.
• Practical Impact: The tool provides a low-cost mechanism to improve test quality, specifically for components that are heavily dependent on external services or complex dependencies, areas where traditional random testing often fails.

In summary, the paper establishes that leveraging the implicit behavioral specifications found in existing mocks significantly improves the quality, coverage, and fault-detection capabilities of LLM-generated unit tests.