💻 computer science

Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation

This paper proposes a Retrieval Augmented Generation (RAG) pipeline to enhance Large Language Models in automating software testing and inspection, demonstrating that integrating external context effectively mitigates hallucinations, improves V&V efficiency, and reduces project costs.

Original authors: Zoe Fingleton, Nazanin Siavash, Armin Moin

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zoe Fingleton, Nazanin Siavash, Armin Moin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a brilliant, super-fast intern to help you write software. This intern has read every book, manual, and code snippet ever written. They are incredibly smart, but they have a major flaw: they are a confident liar.

When they don't know the answer, instead of saying "I don't know," they make up a plausible-sounding story. In the world of computer science, this is called hallucination.

This paper is about how a team of researchers fixed this problem by giving the intern a smart librarian to work with.

Here is the breakdown of their work using simple analogies:

1. The Two Big Jobs: The Detective and the Architect

In software development, there are two critical jobs to make sure the code works:

Code Inspection (The Detective): Looking at code to find bugs (errors) before the software is released.
Test Case Generation (The Architect): Writing a checklist of "what if" scenarios to prove the software works correctly.

Usually, humans do these jobs. But humans get tired, bored, and make mistakes. So, researchers started using Large Language Models (LLMs)—the AI brains—to do these jobs instead.

2. The Problem: The Confident Intern

The researchers found that while the AI is fast, it often gets things wrong.

Without help: If you ask the AI to find a bug in a specific piece of code, it might confidently say, "No bugs here!" when there actually is one. Or it might invent a bug that doesn't exist. It's like a detective who guesses the culprit just to fill out the report.

3. The Solution: The "RAG" Librarian

To fix this, the researchers introduced Retrieval-Augmented Generation (RAG).

Think of RAG as giving the AI intern a personal librarian who stands right next to them.

How it works: Before the AI answers your question, the Librarian quickly searches a massive library of "Golden Standards" (perfectly written code examples and rules).
The Process:
1. You ask the AI: "Is this code broken?"
2. The Librarian shouts: "Wait! I found three examples of this exact code pattern in our library. Here is how they are supposed to look."
3. The AI reads the examples, realizes its mistake, and gives you the correct answer.

This stops the AI from making things up because it has to stick to the facts found in the library.

4. The Experiment: Did the Librarian Help?

The researchers tested this "AI + Librarian" team on two massive tasks:

Task A: Finding Bugs (Code Inspection)

The Test: They gave the AI thousands of code snippets, some with bugs and some without.
The Result:
- AI alone: Got about 67% to 84% of the bugs right (depending on which AI model they used).
- AI + Librarian (RAG): Jumped to 90% accuracy.
- The Analogy: It's like a student taking a test. Without a textbook, they guess. With the textbook open on the desk, they ace the exam.

Task B: Writing Test Checklists (Test Generation)

The Test: They asked the AI to write a list of tests to check if a program works.
The Result:
- AI alone: Wrote tests that covered about 85-89% of the code.
- AI + Librarian (RAG): Covered about 90-94% of the code.
- The Analogy: The AI with the librarian didn't just write more tests; it wrote better tests that actually checked the tricky parts of the code that the lonely AI missed.

5. The Catch: Speed vs. Accuracy

There is one trade-off.

The Librarian takes time. Searching the library takes a few seconds.
The Result: For some AI models, adding the librarian made them slightly slower. For others, it actually made them faster because the AI didn't have to "think" as hard or correct its own mistakes later.

The Bottom Line

This paper proves that if you want an AI to do serious, high-stakes work like checking software for safety, you can't just let it "wing it." You have to give it access to a trusted knowledge base (the Librarian).

In short:

AI alone = A genius who sometimes lies.
AI + RAG = A genius who double-checks their facts with a reference book.

The result is software that is safer, cheaper to build (because humans spend less time fixing AI mistakes), and more reliable. The researchers showed that this "AI + Librarian" team is now better at finding bugs than the average human inspector!

1. Problem Statement

The paper addresses two critical challenges in the Software Development Lifecycle (SDLC), specifically within Verification and Validation (V&V):

LLM Hallucinations: Large Language Models (LLMs) often generate confident but factually incorrect outputs (hallucinations), which is detrimental to software testing and code inspection where precision is paramount.
Manual V&V Bottlenecks: Traditional software inspection (code review) and test case generation are labor-intensive, time-consuming, and prone to human error. While LLMs offer automation potential, their reliability without external context is insufficient for industrial adoption.
Contextual Gaps: Standard LLMs lack specific domain knowledge or access to external reference materials (like coding standards or specific project documentation) during generation, leading to inconsistencies between code, requirements, and test artifacts.

2. Methodology

The authors propose a Retrieval-Augmented Generation (RAG) pipeline to enhance LLM performance in two specific V&V tasks: Automated Code Inspection and Automated Test Case Generation.

A. System Architecture

The framework consists of three main stages:

Knowledge Base Preparation:
- Dataset: The MBPP (Mostly Basic Python Programming) dataset is used as the external knowledge base. It contains 1,000 crowd-sourced, well-written Python code snippets serving as "golden standards."
- Embedding: Documents are encoded into vector embeddings using the MiniLM-L6-V2 sentence transformer model.
- Storage: Embeddings are stored in an in-memory vector store.
Retrieval Stage:
- When a query (code snippet or function description) is received, it is converted into a vector.
- A similarity search (using cosine similarity via NumPy) identifies the top- $K$ most relevant documents from the knowledge base.
Generation Stage:
- The retrieved context is fused with the user's original prompt.
- The enriched prompt is fed into the LLM to generate the output (code analysis or test cases).

B. Experimental Setup

Models Evaluated: OpenAI's GPT-3.5 Turbo, GPT-4, and GPT-4o.
Datasets:
- Code Inspection: Bug In the Code Stack dataset (4,010 tasks, including bug-free and buggy code variants).
- Test Generation: TestEval benchmark dataset (210 tasks from LeetCode).
Configuration: Experiments were run with Temperature set to 0 (deterministic) and a max token limit of 256. Comparisons were made between Standard LLMs (no RAG) and RAG-enhanced LLMs.
Evaluation Metrics:
- Inspection: Accuracy (Match/Mismatch against ground truth), specific bug type classification rates.
- Test Generation: Compilation success (syntax/execution), Line/Branch coverage, and cov@k (coverage of top- $k$ generated tests).
- Efficiency: Runtime duration.

3. Key Contributions

Novel RAG Framework for V&V: The paper introduces a specific RAG pipeline tailored for software engineering tasks, integrating external code knowledge (MBPP) to mitigate hallucinations in both inspection and testing.
Empirical Validation of RAG in SE: It provides a rigorous comparative study demonstrating that RAG significantly improves LLM performance in software V&V compared to baseline models.
Dual-Task Evaluation: The study covers two distinct but related V&V activities:
- Code Inspection: Detecting and classifying bugs (e.g., missing colons, mismatched brackets).
- Test Case Generation: Synthesizing executable unit tests with high coverage.
Efficiency Analysis: The authors analyze the trade-off between accuracy gains and computational overhead (runtime) introduced by the retrieval process.

4. Experimental Results

A. Automated Code Inspection

Accuracy: RAG significantly improved bug detection accuracy across all models.
- GPT-3.5 Turbo: Accuracy increased from 66.78% (without RAG) to 90.57% (with RAG).
- GPT-4: Improved from 73.39% to 89.98%.
- GPT-4o: Improved from 84.49% to 89.88%.
Bug Classification: RAG-enhanced models showed superior performance in identifying specific error types, particularly "mismatched quotation," "mismatched bracket," and "keyword as identifier."
Human Benchmark: The RAG-enhanced models approached or exceeded the reported 60% average detection rate of human inspectors, with some models reaching ~90%.
Runtime:
- GPT-3.5 Turbo: Runtime increased by ~15 minutes (overhead of retrieval).
- GPT-4o: Runtime decreased by ~8 minutes (1:27:35 $\to$ 1:19:26), suggesting retrieval helped streamline the generation process for this model.

B. Automated Test Case Generation

Coverage: RAG-enhanced models consistently achieved higher line and branch coverage.
- GPT-3.5 Turbo: Line coverage ( $k=1$ ) improved from 89.96% to 93.20%; Branch coverage from 78.05% to 80.35%.
- GPT-4o: Similar improvements observed across all coverage metrics.
Compilation Success: RAG models showed higher execution success rates (e.g., GPT-3.5 Turbo execution success rose from 94.61% to 98.76%).
Runtime:
- GPT-3.5 Turbo: Runtime decreased by ~3.5 minutes (1:32:30 $\to$ 1:28:59).
- GPT-4o: Runtime increased by ~14 minutes, indicating retrieval overhead outweighed efficiency gains for this specific model in this context.

5. Significance and Conclusion

Reduced Hallucination: The study confirms that integrating external knowledge via RAG effectively reduces the "hallucination" problem, leading to more factually correct and context-aware code analysis.
Cost and Efficiency: By automating V&V with higher accuracy, the approach reduces the need for extensive manual review, saving human tester time and lowering project costs.
Model-Specific Optimization: The results highlight that the benefits of RAG are model-dependent. While GPT-3.5 Turbo benefits from both accuracy and speed improvements with RAG, GPT-4o shows mixed results (high accuracy gain but potential speed penalty in test generation).
Future Directions: The authors suggest adaptive retrieval strategies (triggering retrieval only when necessary) and exploring hybrid frameworks to balance speed and accuracy. They also plan to expand testing to larger, real-world industrial repositories and safety-critical domains.

In summary, the paper demonstrates that RAG is a viable and effective strategy for enhancing LLMs in software testing and inspection, significantly boosting accuracy and coverage while offering potential efficiency gains depending on the underlying model architecture.