DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch

This paper introduces DeNovoSWE, a large-scale, automatically curated dataset of 4,818 whole-repository generation instances designed to train LLM agents for long-horizon software engineering tasks, which significantly boosts performance on the BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

The Gist

Imagine you have a master architect who is incredibly talented at fixing a single broken brick in a wall. They can look at a specific crack, figure out why it happened, and patch it perfectly. This is what current AI coding assistants are good at: fixing small bugs in existing software.

But what if you asked that same architect to build an entire skyscraper from scratch, starting only with a vague sketch on a napkin? That is a much harder job. It requires planning the foundation, designing the elevators, wiring the electricity, and ensuring every room connects logically. Until now, AI has struggled with this "whole-building" task because it hasn't had enough practice.

The Problem: A Lack of Blueprints
The main reason AI struggles to build entire software "skyscrapers" (repositories) is a lack of training data. Most existing training data is like a list of "fix this leaky faucet" or "repair this cracked window." There is very little data that teaches an AI how to take a high-level description (like a user manual) and turn it into a fully functional, complex building from the ground up.

The Solution: DeNovoSWE
The paper introduces DeNovoSWE, a massive new training dataset designed specifically to teach AI how to build entire software projects from scratch. Think of it as a giant library of 4,818 "construction challenges." In each challenge, the AI is given a detailed set of instructions (documentation) and must build the entire software system to match those instructions perfectly.

How They Built the Training Data (The "Divide and Conquer" Strategy)
Building a dataset for this is hard because you can't just ask an AI to "write a manual for a complex app" and hope it's perfect. The authors used a clever, automated team of AI agents to do the heavy lifting:

1. The "Divide" Phase: Imagine trying to write a manual for a massive city. It's too big to do all at once. So, the AI first breaks the city down into neighborhoods (capabilities). It identifies what each neighborhood does and which buildings (code files) belong to it.
2. The "Conquer" Phase: Now, specialized AI agents write the manual for just one neighborhood at a time.
   • The Draft Agent: Writes the first version of the manual.
   • The Critic Agent: Acts like a strict editor. It checks the manual: "Did you forget to explain how the traffic lights work? Is the wiring diagram clear?"
   • The Repair Agent: Fixes the mistakes the Critic found.
   • This cycle repeats until the manual is perfect.
3. The "Golden" Test: Once the manual is written, the system creates a "clean room" (a sandbox). It takes the original code, throws it away, and leaves only the manual. Then, it asks a different AI agent to rebuild the software using only that manual. If the new software passes all the tests, the manual is considered high-quality and added to the dataset.

Keeping the AI Honest (Anti-Cheating)
A major concern is that the AI might "cheat" by peeking at the original code it was supposed to rebuild. The authors built a rigorous security system to prevent this:

• They strip away all history, caches, and hidden files.
• They block the AI from going online to download the original code.
• They ensure the AI is truly "closed-book," forcing it to rely entirely on the documentation it was given.

The "Difficulty-Aware" Filter
Not every construction project is the same. Some are small sheds; others are skyscrapers. If you only keep the training examples where the AI built a perfect skyscraper on the first try, you lose all the valuable lessons from the hard projects where the AI almost succeeded but made a few mistakes.

The authors created a smart filter that looks at how hard a project is.

• For easy projects (a small shed), they demand a perfect score.
• For hard projects (a skyscraper), they accept a slightly lower score (e.g., 80% success) because getting a perfect skyscraper is incredibly difficult.
  This ensures the AI learns from both easy wins and the "almost there" moments on difficult tasks.

The Results
When they taught an AI model using this new dataset, the results were dramatic.

• Before training, the AI could only build about 5.8% of the required software correctly.
• After training with DeNovoSWE, its success rate jumped to 47.2%.

This proves that by giving AI the right kind of "construction practice"—breaking big problems down, using strict editors, and learning from partial successes—we can teach it to move from being a simple "brick fixer" to a true "software architect."

Technical Summary

Technical Summary: DeNovoSWE

Problem Statement

While Large Language Model (LLM) based code agents have demonstrated proficiency in localized bug fixing within existing codebases (e.g., SWE-bench), their capabilities in long-horizon software engineering tasks—specifically architecting and implementing complete software repositories from high-level specifications—remain underdeveloped. Current benchmarks often fail to stress-test these long-horizon reasoning and implementation skills, and existing training datasets are predominantly centered on single-issue fixes rather than whole-repository generation.

The primary bottleneck is the scarcity of large-scale, verifiable whole-repository generation data. Constructing such data presents three core challenges:

1. Documentation Construction: Existing repository documentation is often incomplete or misaligned with executable behavior, making it insufficient for defining document-to-repository generation tasks.
2. Evaluation Design: There is a lack of objective, scalable protocols to assess diverse repositories through executable tests.
3. Leakage-Free Execution: Modern agents operating in interactive environments may exploit unintended channels (e.g., network retrieval, git history) to access reference implementations, necessitating robust sandboxing.

Methodology: The DeNovoSWE Framework

The authors introduce DeNovoSWE, a large-scale dataset comprising 4,818 high-quality instances where agents must generate a complete repository from scratch based on generated documentation. The dataset is constructed via an automated, sandboxed agentic workflow employing a "divide and conquer" strategy coupled with an iterative critic-repair mechanism.

1. Automated Pipeline Architecture

The construction process is divided into two main phases:

• Divide Phase (Decomposition):

  • Repository Profiling: The system executes unit tests to capture runtime traces, categorizing implementation units into direct (explicitly tested), core indirect (necessary for behavior), and non-core indirect (internal details) components.
  • Capability Partitioning: An overview writer generates a high-level summary, followed by a capability writer that decomposes the repository into distinct functional capabilities.
  • Mapping: An LLM-based classifier maps specific functions and classes to these capabilities, ensuring documentation is grounded in actual implementation units.

• Conquer Phase (Iterative Refinement):

  • Draft Agent: Generates initial documentation for each capability, inspecting source code within a sandboxed environment.
  • Critic Agent: Evaluates the draft for structural organization, coverage of direct/core indirect components, and API specificity. It identifies omissions or under-specified details without leaking test-specific behavior.
  • Repair Agent: Iteratively refines the documentation based on the critic's feedback, inspecting the source repository to resolve inconsistencies.
  • Merging: Final capability-level documents are merged with the repository overview to form the complete task documentation.

2. Evaluation and Leakage Prevention

To ensure the validity of the "document-to-repository" task, the framework enforces strict containment:

• Pre-cleanup: The target repository's code and test suites are stripped, retaining only build configurations.
• Leak Purging: The process erases site-packages, pip caches, and transient compilation artifacts.
• Git Sanitization: The .git directory is destroyed and re-initialized to prevent agents from reverse-engineering history via git reflog.
• Runtime Restrictions: Docker containers enforce command restrictions to block network-based retrieval (e.g., pip install, curl) of the target package, supplemented by LLM-based auditing of execution traces.

3. Difficulty-Aware Trajectory Filtering

Recognizing that fully successful trajectories are difficult to obtain for complex repositories, the authors propose a difficulty-aware trajectory filtering strategy rather than using a fixed global score threshold.

• Difficulty Scoring: A framework estimates task difficulty using structural signals (executable line counts) and LLM-based qualitative judgments.
• Adaptive Thresholds: Filtering thresholds are adjusted based on estimated difficulty. Easier tasks require high pass rates (e.g., 0.90), while harder tasks accept lower pass rates (e.g., 0.60) to retain informative partial successes. This balances data quality with the diversity of challenging instances.

Key Contributions

1. DeNovoSWE Dataset: A large-scale dataset of 4,818 document-to-repository generation instances, constructed automatically via a sandboxed multi-agent pipeline. It is approximately 46× larger than previous repository-generation benchmarks like NL2Repo.
2. Automated Construction Pipeline: A novel workflow combining divide-and-conquer decomposition with iterative critic-repair mechanisms to synthesize comprehensive, test-aligned documentation without human annotation.
3. Difficulty-Aware Filtering: A strategy to curate training trajectories that balances execution fidelity with task diversity, preventing the exclusion of valuable hard instances while filtering out noise from easy tasks.
4. DeNovoSWE-Agent: A fine-tuned model demonstrating substantial improvements in long-horizon SWE capabilities.

Experimental Results

The authors evaluated the impact of DeNovoSWE by fine-tuning Qwen3-30B-A3B and Qwen3.5-35B-A3B models on the dataset.

• Performance Gains: Fine-tuning Qwen3-30B-A3B on DeNovoSWE raised its score on the BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.
• Comparison to Proprietary Models: The fine-tuned DeNovoSWE-Agent-35A3B achieved 50.0% on BeyondSWE-Doc2Repo, narrowing the gap to proprietary frontier models (e.g., within 2.0 percentage points of Gemini3-Pro) despite using a smaller open-weight backbone.
• Ablation Studies: Experiments confirmed that difficulty-aware filtering outperforms fixed global thresholds. A fixed threshold of 0.95 resulted in lower performance (48.1%) compared to the adaptive strategy (50.0%), highlighting the importance of retaining diverse, challenging trajectories.

Significance

The paper claims that DeNovoSWE addresses a critical gap in verifiable long-horizon SWE training data. By shifting the focus from localized issue fixing to whole-repository generation, the dataset enables the training of agents capable of architect-level repository construction. The results suggest that high-quality, long-horizon training data is a decisive factor in improving an agent's ability to generate complete software systems from scratch, potentially bridging the performance gap between open-weight and proprietary frontier models in complex software engineering tasks.