DryRUN: On the Role of Public Tests in LLM-Driven Code Generation

1. Problem Statement

Current state-of-the-art (SOTA) Large Language Model (LLM) frameworks for code generation (e.g., CodeSIM, AlphaCodium) rely heavily on human-authored public test cases (sample input-output pairs) provided in problem specifications. This reliance creates two critical issues:

• Real-World Disconnect: In actual software engineering, comprehensive input-output examples are rarely available before implementation. Benchmarks that include them create an artificial environment that does not reflect real-world constraints.
• The Overconfidence Gap: Public test cases in standard benchmarks are often trivial. Models tend to overfit to these simple examples, passing the public tests while failing hidden, complex private test suites. This phenomenon, termed the "Overconfidence Gap," leads to a false sense of correctness and algorithmic overconfidence.
• Labor Intensity: Manually authoring comprehensive test cases that cover edge cases is a tedious, high-cost bottleneck in the development lifecycle.

The authors hypothesize that public test cases are a "crutch" rather than a necessity and that LLMs can achieve comparable correctness by autonomously generating their own inputs and simulating execution internally.

2. Methodology: The DryRUN Framework

DryRUN (Debugging and Refinement Under Non-execution) is a framework designed to operate under strictly zero-example constraints. It eliminates the need for external execution sandboxes or human-provided test cases. The framework operates through a multi-phase loop (Algorithm 1):

1. Standardized Specification: The input problem description is processed to remove all concrete sample examples, leaving only generalized input/output format descriptions.
2. Phase 1: Autonomous Plan Refinement:
   • The LLM generates an initial implementation plan based on the problem spec.
   • This plan undergoes $N_{plan}$ iterations of autonomous refinement without external feedback. The LLM is prompted to improve logic, add detail, and consider edge cases.
3. Phase 2: Mental Simulation & Trace-Driven Refinement:
   • The LLM synthesizes an initial code draft.
   • It then enters a simulation loop ($N_{sim}$ iterations). In each iteration, the LLM autonomously synthesizes a valid, non-trivial input based on problem constraints.
   • The model performs a mental dry run, simulating the code execution line-by-line against this self-generated input to track variable states and identify logical flaws.
   • The resulting execution trace is used to update the implementation plan and regenerate the code.
   • Crucially: No code is ever executed in an external sandbox; the entire process relies on the LLM's internal reasoning.
4. Phase 3: Final Code Polishing:
   • A final pass reviews the refined code against the plan to fix any syntax anomalies or stylistic issues introduced during the iterative refinement process.

Key Differentiators from CodeSIM:

• Input Source: DryRUN generates its own inputs; CodeSIM uses failed public test cases.
• Execution: DryRUN is execution-free (mental simulation only); CodeSIM relies on external sandbox execution.
• Strategy: DryRUN uses proactive, unconditional plan refinement; CodeSIM uses reactive refinement only when public tests fail.

3. Key Contributions

• Challenging the Necessity of Public Tests: The paper empirically demonstrates that removing human-authored examples from benchmarks yields negligible performance drops, suggesting modern LLMs do not strictly require them for task comprehension.
• The Overconfidence Gap Analysis: The authors formalize and measure the "Overconfidence Gap," showing that frameworks relying on public tests (like CodeSIM) exhibit a larger gap (passing public tests but failing private suites) compared to zero-example approaches.
• DryRUN Framework: A novel, zero-shot framework that replaces external oracles with autonomous input synthesis and mental simulation, achieving SOTA performance without external execution.
• Token Efficiency: DryRUN significantly reduces token consumption compared to existing multi-agent frameworks that rely on extensive simulation loops.

4. Experimental Results

The framework was evaluated on the LiveCodeBench v6 dataset (problems published after March 2025 to ensure data contamination-free evaluation) using gpt-5-mini and gemini-3-flash.

• Performance Parity: DryRUN achieved performance comparable to CodeSIM (the SOTA baseline).
  • On gpt-5-mini, DryRUN outperformed CodeSIM by 3.3% overall.
  • On gemini-3-flash, CodeSIM had a 5.0% advantage, but the margins fell within overlapping standard deviations.
  • Notably, removing public tests from the standard "Direct" baseline actually improved performance slightly, reinforcing that examples are not strictly necessary.
• Mitigation of Overconfidence:
  • CodeSIM showed a large, volatile Overconfidence Gap (e.g., ±4.36 on gpt-5-mini), indicating it frequently passed public tests but failed hidden private suites.
  • DryRUN maintained a significantly lower Overconfidence Gap (e.g., ±2.08), demonstrating that self-simulation against non-trivial, self-generated inputs prevents overfitting to trivial oracles.
• Efficiency:
  • DryRUN consumed 70% of the total tokens and 50% of the output tokens compared to CodeSIM.
  • This efficiency is attributed to DryRUN's strict limit on simulation rounds ($N_{sim}=2$) versus CodeSIM's potential for up to 25 nested simulations.
• Ablation Studies:
  • Planning: Isolated plan refinement improved Pass@1 by ~15% over zero-shot.
  • Simulation: Two rounds of mental simulation ($N_{sim}=2$) were optimal, stabilizing variance and recovering accuracy lost in the first simulation attempt.
  • Polishing: The final polishing stage added a significant +3.60% performance boost, correcting syntax/logic errors introduced during regeneration.
• Small Language Models (SLMs): DryRUN performed poorly on SLMs (e.g., qwen2.5-coder:7b), which struggled with complex multi-step reasoning and often failed to follow instructions, highlighting that the method currently requires high-reasoning-capability LLMs.

5. Significance and Future Implications

• Paradigm Shift: The paper challenges the prevailing assumption that public test cases are essential for high-quality code generation. It suggests that the "Overconfidence Gap" is an artifact of benchmark design rather than a limitation of LLMs.
• Real-World Applicability: By removing the dependency on external oracles, DryRUN aligns code generation closer to real-world scenarios where developers often lack comprehensive test suites at the start of a project.
• Benchmarking Reform: The authors raise a critical question: Should public test cases be removed from coding benchmarks entirely to prevent biased, overconfident model evaluation?
• Future Directions:
  • Adapting mental simulation techniques for SLMs.
  • Using execution sandboxes purely as a diagnostic tool to verify the accuracy of mental traces (rather than for primary debugging).
  • Scaling the framework to repository-level tasks (e.g., SWE-bench) where cross-file dependencies exist without clear input-output oracles.

In conclusion, DryRUN proves that LLMs can self-correct and generate robust code by leveraging internal simulation capabilities, effectively closing the performance gap with SOTA methods while eliminating the reliance on potentially misleading public test cases.