💬 NLP

DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocode

The paper introduces DuET, a dual-execution framework that combines direct code execution and LLM-based pseudocode reasoning through functional majority voting to significantly improve test output prediction reliability and achieve state-of-the-art performance on LiveCodeBench.

Original authors: Hojae Han, Jaejin Kim, Seung-won Hwang, Yu Jin Kim, Moontae Lee

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hojae Han, Jaejin Kim, Seung-won Hwang, Yu Jin Kim, Moontae Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Trust but Verify" Dilemma

Imagine you are a manager hiring a team of junior developers (AI models) to write code for a new app. You give them a problem description, and they write code to solve it.

To check if their code works, you need to know what the correct answer should be for a specific test. This is called Test Output Prediction.

Traditionally, there have been two ways to figure out this answer, and both have a fatal flaw:

The "Run It" Method (Direct Execution):
- How it works: You take the code the AI wrote and actually run it on your computer.
- The Flaw: If the AI makes a tiny typo (like a missing comma or a wrong variable name), the whole program crashes. Even if the AI understood the logic perfectly, a small "implementation error" ruins the result. It's like a chef who knows exactly how to bake a cake but burns it because they set the oven to 500°F instead of 350°F.
The "Mental Simulation" Method (LLM-based Execution):
- How it works: Instead of running the code, you ask the AI to "pretend" to run the code in its head and tell you the result.
- The Flaw: AI models are great at logic but terrible at long, detailed calculations. If the code has a complex loop (like counting to 1,000), the AI might get lost, forget a step, or "hallucinate" a number that sounds right but is wrong. It's like asking a human to do long division in their head without writing anything down; they might get the first few steps right but slip up at the end.

The Solution: DUET (The "Double-Check" System)

The authors of this paper, DUET (Dual Execution for Test Output Prediction), realized that relying on just one of these methods is risky. So, they built a system that uses both at the same time and lets them vote on the final answer.

Think of DUET as a two-person quality control team for a factory:

1. The "Robot" (Direct Code Execution)

This team member takes the generated code and runs it on a machine.

Superpower: If the code is written correctly, the robot is 100% accurate. It never gets tired or confused by math.
Weakness: If the code has a typo, the robot stops working immediately.

2. The "Human Expert" (LLM-based Pseudocode Execution)

This team member doesn't look at the messy code. Instead, they ask the AI to write a Pseudocode (a simple, plain-English summary of the logic, like a recipe) and then "simulate" the steps in their head.

Superpower: Because they are looking at the logic (the recipe) rather than the syntax (the specific ingredients spelling), they ignore typos. If the code had a typo, the Human Expert can still figure out the right answer because they understand the intent.
Weakness: If the task is too complex or requires too many steps, the Human Expert might get confused and make a math error.

How They Work Together (The Voting System)

DUET doesn't just pick one; it runs both paths simultaneously.

Scenario A: The code has a typo.
- The Robot crashes and says "Error!"
- The Human Expert ignores the typo, reads the logic, and says "The answer is 42."
- Result: DUET trusts the Human Expert.
Scenario B: The logic is incredibly complex (100 steps).
- The Human Expert gets lost in the math and guesses "43" (a hallucination).
- The Robot runs the code perfectly and says "42."
- Result: DUET trusts the Robot.
Scenario C: Both agree.
- They both say "42."
- Result: DUET is 100% confident.

Why This Matters (The "Zero-Advantage" Problem)

The paper also discovered a sneaky problem in how we usually test AI code.

Imagine you ask an AI to write 10 different solutions to a math problem. Then, you ask the same AI to predict the answer for those solutions to see which one is best.

If the AI is bad at math, it will generate bad solutions.
Then, it will predict the answers for those bad solutions, and those predictions will also be wrong.
The Trap: The AI is essentially grading its own homework, and if it's bad at the subject, it can't tell the difference between a good answer and a bad one. This is called the "Zero-Advantage Problem."

DUET fixes this because the "Human Expert" (Pseudocode path) is decoupled from the specific code being tested. It looks at the idea of the solution, not the specific code snippet. This allows it to filter out bad code even if the code itself is messy.

The Results

When they tested DUET on a massive benchmark called LiveCodeBench:

It beat the previous best methods by a huge margin (improving accuracy by 13.6%).
It works better than just running the code or just guessing the answer.
It helps AI generate better code overall because it can filter out the "bad" candidates more effectively.

The Takeaway

DUET is like having a safety net.
If you try to cross a river by walking on a tightrope (running code), you might fall if the rope is frayed. If you try to fly over it (simulating in your head), you might get dizzy and fall.
DUET gives you a tightrope and a parachute. If the rope breaks, the parachute catches you. If the parachute fails, the rope holds you. By using both, you almost never fall.

1. Problem Definition

The paper addresses Test Output Prediction, a critical subtask in automated test case generation and code evaluation. Given a problem description ( $D$ ) and a test input ( $I$ ), the goal is to predict the correct output ( $O$ ) without necessarily having the ground-truth solution code.

Current State-of-the-Art (TestChain): Prior approaches, such as TestChain, rely on Direct Code Execution. The model generates executable code from the problem description and runs it to predict the output.
The Core Challenge: Direct execution is brittle. Even if the model understands the logic correctly, minor implementation errors (e.g., off-by-one errors, syntax mistakes, or incorrect variable initialization) in the generated code can cause the execution to fail or produce incorrect results. Conversely, relying solely on LLM reasoning to simulate execution (without code) often leads to execution hallucinations (logical errors in multi-step reasoning).

2. Methodology: DUET Framework

The authors propose DUET (Dual Execution), a framework that combines two complementary execution paths to mitigate the weaknesses of each.

A. Two Execution Paths

Direct Code Execution (The "Code" Path):
- Generates executable code from the problem description.
- Executes the code directly on the test input.
- Strength: Deterministic and precise for correct code; immune to reasoning hallucinations.
- Weakness: Highly sensitive to implementation/syntax errors. If the code is flawed, the output is wrong.
LLM-based Pseudocode Execution (The "Pseudocode" Path):
- Generates high-level pseudocode (abstracting away low-level syntax).
- Uses the LLM to simulate the execution of this pseudocode via reasoning (Chain-of-Thought), rather than running it in a compiler.
- Strength: Resilient to implementation errors because it focuses on algorithmic intent rather than syntax.
- Weakness: Prone to hallucinations in complex, multi-step reasoning chains (e.g., losing track of state in deep loops).

B. Aggregation: Path-Weighted Functional Majority Voting (FMV)

To combine the outputs of these two paths, DUET employs a Functional Majority Voting mechanism with a novel Path-Weighted variant:

Standard FMV: Aggregates multiple predictions from both paths and selects the most frequent output.
Path-Weighted FMV: Assigns higher weights to outputs from a specific path if all samples within that path unanimously agree on the same result.
- If a path (e.g., Direct Execution) produces unanimous agreement, its votes are boosted ( $w_{high}$ ).
- Otherwise, they receive a base weight ( $w_{base}$ ).
Rationale: This leverages the confidence of the "correct" path. If the code is correct, direct execution will likely be unanimous. If the code is broken but the logic is sound, the pseudocode path may be unanimous.

3. Key Contributions

First Use of LLM-based Pseudocode Execution: The paper introduces the concept of grounding test output prediction on generated pseudocode and simulating its execution via LLM reasoning, decoupling prediction from code correctness.
The DUET Framework: A novel dual-path strategy that integrates direct code execution and pseudocode reasoning. It identifies that these methods are complementary:
- Pseudocode excels when code generation is error-prone (implementation errors).
- Direct execution excels when reasoning complexity is high (avoiding hallucinations).
Identification of the "Zero-Advantage Problem": The authors analyze end-to-end code generation and find that using direct code execution (TestChain) for filtering candidates can be detrimental. If the generated code is flawed, the predicted output is wrong, leading to the rejection of potentially correct candidates or the acceptance of incorrect ones. DUET mitigates this by providing an orthogonal, code-independent verification path.
State-of-the-Art Performance: DUET achieves new benchmarks in test output prediction and end-to-end code generation.

4. Experimental Results

The authors evaluated DUET on LiveCodeBench, BigCodeBench-Hard, DevEval, and HumanEval(+).

Test Output Prediction (LiveCodeBench):
- DUET achieved a Pass@1 of 81.1% using GPT-4-Turbo, outperforming the previous SOTA (TestChain with FMV) by 5.6 percentage points and the raw model by 13.6 pp.
- It consistently outperformed single-path baselines across various model sizes (Llama-3-8B, Llama-3-70B, Mistral-Large).
End-to-End Code Generation:
- When integrated into the CODET pipeline for filtering candidate code, DUET improved Pass@1 by 3.2 pp on LiveCodeBench-Easy.
- Notably, replacing DUET with TestChain degraded performance by 5.6 pp, confirming the "zero-advantage problem" of direct execution in filtering scenarios.
- DUET achieved the best results on BigCodeBench-Hard (with reasoning-enhanced models like QwQ-32B) and DevEval (repository-level generation).
Analysis of Failure Modes:
- Correct Code: Direct execution achieves ~~100% accuracy; Pseudocode is slightly lower (~~86%) due to hallucinations.
- Incorrect Code: Direct execution drops sharply (~~39%), while Pseudocode remains robust (~~54%).
- Trace Length: Pseudocode execution outperforms direct execution for short-to-medium trace lengths (<100 steps) but degrades on very long traces where reasoning errors accumulate. Direct execution remains stable regardless of trace length. DUET effectively balances these trade-offs.

5. Significance and Conclusion

The paper demonstrates that relying on a single execution modality (code or reasoning) is insufficient for reliable test output prediction.

Robustness: By combining the symbolic precision of code execution with the abstraction of pseudocode reasoning, DUET creates a system that is robust to both implementation bugs and reasoning hallucinations.
Practical Impact: The method requires no architectural changes or additional training; it is a plug-and-play inference strategy that significantly boosts the reliability of LLMs in software engineering tasks.
Future Directions: The authors suggest adaptive weighting mechanisms that could dynamically select the most reliable path based on instance difficulty or trace length.

In summary, DUET represents a paradigm shift in test case generation, moving from "generate and run" to "generate, simulate, and cross-verify," setting a new standard for code evaluation and generation reliability.