Technical Summary: RepoZero

Problem Statement

While Large Language Models (LLMs) have demonstrated significant progress in code generation, their ability to construct complete software repositories from scratch remains poorly understood. Existing benchmarks for repository-level coding suffer from critical limitations:

1. Evaluation Bias: Many rely on human-in-the-loop or "LLM-as-a-judge" frameworks, introducing subjectivity and unreliability.
2. Data Leakage: Models pre-trained on massive GitHub corpora often memorize existing repositories rather than demonstrating genuine reasoning. Using "clean" or rare repositories to mitigate this inherently limits dataset scale and is susceptible to rapid contamination.
3. Verification Gaps: Current benchmarks for "from-scratch" generation often lack verifiable, execution-based ground truths, particularly for newly generated projects where pre-existing unit tests are scarce.

Methodology: RepoZero Benchmark

To address these challenges, the authors introduce RepoZero, the first benchmark designed for fully automated, execution-based verification of repository-level generation from scratch.

Core Concept: Repository Reproduction

RepoZero reformulates the generation task as repository reproduction. Instead of generating code from natural language descriptions alone, an agent is provided with:

1. API Specifications: Functional specifications of a source repository's APIs.
2. White-Box Test Cases: Four illustrative test cases (inputs and ground-truth outputs) provided in the prompt.
3. Constraints: Strict rules to prevent "shortcut" solutions, such as cross-language embedding or direct API delegation.

The agent must re-implement the entire repository in a different programming language (e.g., Python to JavaScript, or C/C++ to Rust). The generated repository is evaluated by executing it against a hidden suite of test cases and comparing its outputs byte-for-byte with the original source repository.

Benchmark Construction

• Datasets: Two subsets are curated: RepoZero-Py2JS (400 samples, Python to JavaScript) and RepoZero-C2Rust (200 samples, C/C++ to Rust).
• Source Selection: Repositories are manually selected based on determinism, open-source integrity, and architectural complexity (requiring multi-file implementations).
• Test Generation: An LLM generates test files and cases invoking the source APIs. These are rigorously filtered to ensure they are deterministic (removing time-dependent or memory-address outputs) and executable.
• Evaluation Protocol:
  • Black-Box Testing: Only four white-box cases are visible to the agent; the rest of the test suite is hidden to prevent overfitting.
  • Environment Isolation: Evaluations run in isolated Docker containers with restricted file-system permissions.
  • Cross-Language Constraints: Agents are prohibited from using external packages or bridge commands (e.g., calling Python from a shell wrapper), forcing genuine cross-language synthesis.

Agentic Code-Test Evolution (ACE) Framework

The paper proposes ACE, a framework for test-time scaling. Unlike static evaluation, ACE employs an iterative loop:

1. A coding agent generates the target repository.
2. A testing agent generates new test cases and executes the source repository to obtain ground-truth outputs.
3. If the target repository fails, error messages are fed back to the coding agent for refinement.
   This process continues until the repository passes all test cases, leveraging the source repository as a deterministic oracle.

Key Contributions

1. RepoZero Benchmark: The first scalable, verifiable benchmark for from-scratch repository generation that bridges the gap between single-file coding and full software engineering. It uniquely combines cross-language constraints with execution-based verification.
2. ACE Framework: A novel agentic workflow that integrates iterative test generation and error-driven refinement, enabling effective test-time scaling for complex repository synthesis.
3. Comprehensive Evaluation: An extensive benchmarking of state-of-the-art LLMs (including Kimi, GLM, DeepSeek, Ernie, Minimax, and Claude) across multiple agent scaffolds (OpenHands-bash and Mini-SWE-Agent).

Experimental Results

The authors evaluated numerous models across Easy, Medium, and Hard difficulty tiers. Key findings include:

• Limited Success Rates: Even the strongest agents achieve modest pass rates. The best-performing model, Claude-4.6-Sonnet, achieved a pass rate of approximately 51.5% on Py2JS and 44.6% on C2Rust using the OpenHands-bash scaffold.
• Agent Performance: Mini-SWE-Agent generally outperformed OpenHands-bash, attributed to more sophisticated context engineering.
• Impact of Test-Time Scaling: The ACE framework significantly improved performance. For DeepSeek V3.1 on OpenHands-bash, the pass rate increased from 26.1% (coding only) to 42.9% with two iterations of coding-testing-refining.
• Failure Modes: Analysis revealed that while runtime errors were infrequent, agents frequently failed due to:
  • Contextual Drift: Losing track of requirements over long reasoning trajectories.
  • Runnability vs. Correctness: Approximately 40% of executable code failed to match the deterministic output of the source, highlighting a gap between code that runs and code that is semantically correct.

Significance and Claims

The paper positions RepoZero as a challenging, scalable, and reliable testbed for end-to-end code generation. The authors claim that:

• Current LLM agents, even when supported by advanced scaffolds, exhibit a substantial gap between their capabilities and real-world software development requirements, with pass rates rarely exceeding 55%.
• Self-verification via test generation is a critical direction for advancing coding agents. The ability to leverage an executable environment for test-driven development is identified as a fundamental determinant of success.
• The benchmark successfully mitigates data leakage through cross-language synthesis and strict execution constraints, providing a more rigorous evaluation than previous "LLM-as-a-judge" approaches.

The work concludes that while the iterative "code-test" loop is promising, future progress requires more robust integration of autonomous, high-quality test suite synthesis and improved long-context retention in agents.