Automated Test Suite Enhancement Using Large Language Models with Few-shot Prompting

1. Problem Statement

Unit testing is critical for software quality but is labor-intensive to create manually. Traditional automated testing tools (e.g., Search-Based Software Testing or SBST) often generate tests that lack readability, naturalness, and practical usability. While Large Language Models (LLMs) have shown promise in code generation, their application in unit test generation faces challenges:

• Zero-shot limitations: Generating tests without examples often results in low correctness and coverage.
• Hallucination & Non-determinism: LLMs frequently produce syntactically invalid code or varying outputs for the same prompt.
• Underexplored Few-shot Potential: While few-shot prompting (providing examples in the prompt) is known to improve performance, it remains unclear how the source of these examples (Human, SBST, or LLM) and the selection strategy (Random vs. Retrieval-based) impact the quality of generated tests and their ability to enhance existing test suites.

2. Methodology

Experimental Setup

• Model: GPT-4o (integrated into GitHub Copilot).
• Datasets:
  • HumanEval: Isolated coding exercises.
  • ClassEval: Tasks requiring cross-file and class-level references (more complex).
• Data Preparation: The authors created three distinct sets of "demonstration" examples for few-shot prompting:
  1. Human-written: Converted from the benchmark datasets to pytest format.
  2. SBST-generated: Created using Pynguin (a search-based tool).
  3. LLM-generated: Created using GPT-4o with zero-shot prompting.
• Prompt Engineering:
  • Structure: System prompt (Persona: Python test expert) + User prompt (Class definition + 5 selected examples).
  • Optimization: Preliminary experiments determined that 5 examples yielded the best balance of coverage and correctness.

Research Questions (RQs)

1. RQ1: How does the source of few-shot examples (Human, SBST, LLM) affect the quality of generated tests?
2. RQ2: How does the selection technique (Random vs. Retrieval-based) affect test quality?
3. RQ3: How do different example sources affect the enhancement of existing test suites (coverage improvement)?
4. RQ4: How does retrieval-based selection affect the enhancement of existing test suites?

Evaluation Pipeline

The study employs an Assured LLM-based Software Engineering pipeline:

1. Generation: LLM generates tests based on few-shot prompts.
2. Validation: Checks for syntactic correctness (AST), compilation, and execution.
3. Rule-Based Repair: A critical step where 9 specific rules fix common LLM errors (e.g., missing import pytest, missing class imports, removing self from standalone functions, fixing syntax errors).
4. Optimization: New tests are evaluated against the initial suite. Only tests that improve line/branch coverage are retained; redundant or faulty tests are discarded.
5. Metrics:
   • Functional: Syntactic correctness, Compilation, Execution pass rate.
   • Coverage: Line and Branch coverage.
   • Code Quality: Cyclomatic complexity, Cognitive complexity, Test smells, Maintainability (Technical Debt).

3. Key Contributions

• Empirical Analysis of Example Sources: First systematic comparison of using Human, SBST, and LLM-generated examples as few-shot prompts for test generation.
• Retrieval Strategy Evaluation: Evaluation of various similarity-based retrieval methods (Problem definition, Source code, Combined) versus random baselines.
• Hybrid Human-AI Workflow: A practical framework demonstrating how LLMs can be used to enhance (rather than replace) existing test suites, focusing on coverage gaps.
• Repair Mechanism Validation: Demonstration that lightweight, rule-based repair can significantly recover the usability of LLM-generated tests.

4. Key Results

Functional Correctness & Repair

• Initial State: SBST-generated examples produced the highest initial pass rates (6.07% for HumanEval, 51.53% for ClassEval) compared to Human or LLM examples.
• Post-Repair: After applying rule-based repairs, all sources achieved high pass rates (~76–84%).
• Common Fixes: The most frequent repair was adding missing imports (Rule 3), indicating LLMs often overlook dependencies.

Impact of Example Source (RQ1 & RQ3)

• Coverage: Human-written examples consistently yielded the highest line and branch coverage in the final optimized suites.
• Enhancement Potential: SBST-generated examples provided the most significant improvement to low-coverage initial suites (e.g., +26.9% branch coverage on ClassEval). This is because human-written initial suites were already near-optimal, leaving little room for improvement, whereas SBST suites had large gaps.
• Complexity: Tests generated using SBST examples resulted in higher cyclomatic complexity, suggesting they may be more complex but less readable.

Impact of Selection Technique (RQ2 & RQ4)

• Best Strategy: The "Problem + Source Code Similarity" retrieval method (combining natural language problem descriptions with code similarity) consistently produced the highest coverage and most diverse test cases.
• Random Baselines: Random selection from the entire test suite performed reasonably well, but random selection only from the class under test performed poorly, likely causing the LLM to generate repetitive tests.
• Correctness: Selection techniques had minimal impact on the functional correctness of generated tests (which was dominated by the repair phase).

Code Quality

• Readability: Human-written examples produced tests with better cognitive complexity scores (easier to understand).
• Technical Debt: Differences in technical debt and code smells between sources were generally negligible, suggesting LLM-generated tests are maintainable if guided correctly.

5. Significance and Implications

• Practical Utility: The study confirms that LLMs are most effective as augmentation tools for existing test suites rather than standalone generators. They excel at filling coverage gaps in under-tested code.
• Prompt Engineering Best Practices:
  • Use 5 examples in few-shot prompts.
  • Select examples based on combined problem description and code similarity.
  • Human examples are best for final quality and readability, but SBST examples are highly effective for boosting coverage in poor initial suites.
• Repair is Essential: Simple, rule-based post-processing is sufficient to make LLM-generated tests production-ready, reducing the need for complex iterative repair frameworks.
• Future Research: Highlights the need for diverse benchmarks (beyond simple functions) and the importance of evaluating maintainability metrics (cognitive complexity, smells) alongside traditional coverage metrics.

In conclusion, the paper demonstrates that a hybrid approach—combining retrieval-based few-shot prompting with rule-based repair—can effectively generate high-quality, high-coverage unit tests that integrate seamlessly into human-maintained codebases.