💻 computer science

Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation

This paper introduces CANDOR, a novel prompt engineering-based multi-agent LLM framework that leverages consensus-driven reasoning and a dual-LLM pipeline to generate high-quality, hallucination-resistant JUnit tests in Java, outperforming existing fine-tuning and search-based approaches in oracle correctness and mutation scores.

Original authors: Qinghua Xu, Guancheng Wang, Lionel Briand, Kui Liu

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qinghua Xu, Guancheng Wang, Lionel Briand, Kui Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to teach a robot to cook a complex dish. You give the robot a recipe (the Natural Language Description) and a set of instructions on how to chop vegetables (the Source Code).

Your goal is for the robot to write a Test: a checklist that proves the dish is cooked correctly.

The Test Prefix: The robot setting up the ingredients (e.g., "Put 5 carrots in the pot").
The Test Oracle: The robot checking the result (e.g., "If the carrots are soft, the dish is done").

The problem? Robots (specifically, AI models called LLMs) are brilliant but prone to hallucinations. Sometimes, they confidently write a checklist that says, "If the carrots are raw, the dish is done," because that's what the current (buggy) code does, not what the recipe says should happen.

This paper introduces CANDOR, a new system that acts like a team of expert chefs working together to fix this problem. Instead of one lonely robot trying to do everything, CANDOR uses a "committee" approach.

Here is how CANDOR works, broken down into simple steps:

1. The Setup: A Team of Specialists

In the past, one AI tried to do everything: write the code, check the coverage, and fix the mistakes. This often led to confusion. CANDOR splits the job into a panel of specialized agents, like a kitchen brigade:

The Initializer: The "Apprentice." It tries to write the very first draft of the test. It often makes syntax errors (like using Python brackets [ ] instead of Java lists), but it gets the ball rolling.
The Planner: The "Head Chef." It looks at the code and says, "We missed the part where we test empty bowls. Let's add a test for that." It creates a game plan to cover every corner of the kitchen.
The Tester: The "Line Cook." It follows the Planner's instructions to write the actual test code.
The Inspector: The "Quality Control." It tries to run the code. If there's a typo or a missing ingredient (import statement), it sends it back to the Tester to fix.

2. The Problem: The "Buggy Code" Trap

Here is the tricky part. If the original recipe (the source code) has a mistake—like telling the robot to "burn the carrots"—the robot will happily write a test that says, "Burnt carrots = Success." This is called a Regression Oracle. It's useless because it just confirms the bug exists.

We need the robot to ignore the buggy code and follow the intended recipe instead.

3. The Solution: The "Panel Discussion" (The Magic Sauce)

This is where CANDOR gets creative. To fix the "burnt carrot" mistake, it doesn't just ask one AI. It sets up a Panel Discussion.

The Panelists (Reasoning AIs): Imagine three brilliant, over-thinking food critics. They read the recipe and the buggy code. They argue back and forth.
- Panelist 1: "Wait, the code says burn them, but the recipe says roast them. The code is wrong!"
- Panelist 2: "I agree, but let me double-check the math..." (They might get lost in their own thoughts, a problem called "overthinking").
The Interpreters: These are like secretaries for the critics. The critics are too wordy and ramble on for hours. The secretaries listen to the rambling, cut out the fluff, and write down just the key conclusion: "The carrots should be roasted, not burnt."
The Curator: The Judge. The Judge listens to all the secretaries. If 2 out of 3 agree that the carrots should be roasted, the Judge makes the final call: "The test must check for roasted carrots."

This "Consensus" approach stops the AI from hallucinating. Even if one AI gets confused, the group corrects it.

4. The Results: Why It Matters

The researchers tested CANDOR on two big sets of coding problems (like a coding school exam).

Coverage: CANDOR is just as good as the best existing tools (like EvoSuite) at making sure the test touches every part of the code.
Finding Bugs: CANDOR is much better at finding actual bugs. While other tools just check if the code runs, CANDOR checks if the code does what it should do.
Beating the Experts: It beat the current state-of-the-art tool (TOGLL) by a huge margin (over 21%). The best part? TOGLL had to be "trained" on massive amounts of data (like a student studying for years), while CANDOR just uses smart prompts (like a smart student who knows how to ask the right questions).

The Big Takeaway

CANDOR proves that you don't need to train a super-expensive AI to write perfect tests. Instead, you can use a team of standard AIs working together, arguing, and reaching a consensus.

It's the difference between asking one person to solve a math problem (who might make a mistake) and asking a whole classroom to solve it, discuss their answers, and vote on the correct one. The result is a test suite that is not only thorough but actually tells you if your software is working as intended.

1. Problem Statement

Automated unit test generation is critical for software quality but remains challenging, particularly for strongly typed languages like Java. Existing approaches face three main limitations:

Regression Oracles vs. Specification Oracles: Traditional search-based tools (e.g., EvoSuite) generate "regression oracles" based on current code behavior. If the code is buggy, the oracle is also buggy. They struggle to generate "specification-based oracles" derived from intended functionality.
Limitations of Current LLM Approaches:
- Fine-tuning Dependency: State-of-the-art (SOTA) methods like TOGLL require fine-tuning LLMs on specific datasets (e.g., SF110), making them costly, hard to maintain, and brittle when applied to new Java versions or environments.
- Hallucination and Overthinking: Direct use of LLMs for oracle generation often leads to hallucinations (incorrect assertions) and "overthinking" (verbose, redundant reasoning that increases latency without improving accuracy).
- Prefix Quality: While LLMs improve test readability, search-based methods often still outperform them in code coverage.
Dependency on External Tools: Many LLM-based approaches rely on EvoSuite to generate test scaffolds, inheriting its limitations (e.g., "state pollution," lack of maintenance since 2021).

2. Methodology: The CANDOR Framework

The authors propose CANDOR, a novel, end-to-end framework that uses off-the-shelf Large Language Models (LLMs) and a multi-agent architecture to generate complete JUnit tests (both prefixes and oracles) without fine-tuning.

The workflow consists of three iterative steps:

Step I: Initialization

Goal: Generate a syntactically correct initial test file ( $v_0 \to v_1$ ).
Agents:
- Initializer: A basic LLM that generates an initial test scaffold based on source code.
- Validation: Compiles and executes the test. If errors occur (syntax, imports), feedback is looped back to the Initializer until a valid file is produced.

Step II: Test Prefix Generation

Goal: Enhance code coverage by generating additional test cases ( $v_1 \to v_2$ ).
Agents:
- Planner: Analyzes coverage gaps (via Jacoco reports) and proposes new test plans.
- Tester: Generates executable test code based on the Planner's plan.
- Inspector: Validates the generated code and provides feedback on compilation or execution errors.
Note: At this stage, oracles are "tentative" (derived from the source code) and may be incorrect if the source code contains bugs.

Step III: Oracle Fixing (The Core Innovation)

Goal: Correct oracles to align with natural language specifications, even if the source code is faulty ( $v_2 \to v_f$ ).
Strategy: A "Panel Discussion" inspired by David Hume's philosophy that truth emerges from argument.
Agents:
- Requirement Engineer: Extracts natural language requirements and formal specifications (predicate logic) from the code's docstrings.
- Panelist (Reasoning LLM): Multiple independent agents (e.g., DeepSeek R1) evaluate tentative oracles against the requirements. They generate detailed reasoning chains.
- Interpreter (Basic LLM): Mitigates the "overthinking phenomenon" by extracting concise, structured insights from the verbose reasoning of the Panelists.
- Curator (Basic LLM): Aggregates the interpretations from all pipelines. Instead of simple majority voting, it synthesizes the arguments to reach a consensus, correcting hallucinations and determining the final accurate oracle.

3. Key Contributions

First Multi-Agent End-to-End Framework: CANDOR is the first framework to use only off-the-shelf LLMs for end-to-end JUnit test generation in Java, eliminating the need for fine-tuning or external tools like EvoSuite.
Consensus-Based Oracle Correction: Introduces a novel "Panel Discussion" strategy where multiple reasoning agents debate and cross-validate oracles, significantly reducing hallucinations.
Dual-LLM Pipeline: Addresses the verbosity of reasoning LLMs by pairing them with basic LLMs (Interpreters) to extract concise evaluations, balancing reasoning depth with efficiency.
Robustness to Faulty Code: Demonstrates the ability to generate correct oracles even when the source code contains bugs, by relying on natural language descriptions rather than code behavior.

4. Experimental Results

The framework was evaluated on HumanEvalJava and a new dataset, LeetCodeJava (containing medium and hard difficulty problems), against baselines: EvoSuite (search-based), LLM-Empirical (prompt-based single agent), and TOGLL (fine-tuned SOTA).

Test Prefix Quality (RQ1):
- CANDOR achieved comparable line and branch coverage to EvoSuite (e.g., 0.991 vs 0.961 on HumanEvalJava).
- CANDOR significantly outperformed EvoSuite in Mutation Score (e.g., 0.980 vs 0.858 on HumanEvalJava), indicating better bug-detection capability due to semantic understanding.
Oracle Correctness (RQ2):
- Correct Code: CANDOR outperformed TOGLL by >21.1 percentage points in oracle correctness (e.g., 0.910 vs 0.610 on HumanEvalJava), despite TOGLL being fine-tuned.
- Faulty Code: CANDOR maintained high accuracy on mutated (buggy) code, whereas baselines relying on code behavior failed.
Ablation Study (RQ3):
- Removing the Planner caused a significant drop in coverage and mutation scores.
- Removing the Panel Discussion (replacing with single-agent or simple voting) caused a significant drop in oracle correctness (up to 0.086), confirming the value of the consensus mechanism.

5. Significance

Practical Applicability: By avoiding fine-tuning, CANDOR is immediately applicable to new Java versions and environments without retraining costs.
Paradigm Shift: Moves the field from "regression oracles" (checking if code behaves as it did) to "specification oracles" (checking if code behaves as intended), which is crucial for detecting latent bugs.
LLM Optimization: Provides a blueprint for using multi-agent systems to mitigate LLM hallucinations and verbosity in software engineering tasks, suggesting that "consensus" is a superior strategy to single-agent generation for complex reasoning tasks.

In conclusion, CANDOR demonstrates that a carefully orchestrated multi-agent system using off-the-shelf LLMs can surpass both traditional search-based tools and fine-tuned LLM approaches in generating high-quality, semantically correct unit tests for Java.