LLM-Assisted Repository-Level Generation with Structured Spec-Driven Engineering

This paper proposes Structured Spec-Driven Engineering (SSDE), a paradigm that utilizes structured specifications to overcome the ambiguity and quality limitations of natural language prompts, thereby enabling high-quality, verifiable code generation at the repository level.

The Gist

Imagine you are trying to teach a very talented, but slightly scatterbrained, apprentice chef how to cook a massive, complex banquet for a whole city.

The Problem: The "Vague Order"
Right now, if you ask a top-tier AI (the apprentice) to write code for a whole software system, you usually just give it a long, natural language description like, "Make a website where people can book meetings." This is like telling the chef, "Make a delicious meal."

The paper argues that while the AI is great at chopping a single onion (writing a small function), it gets lost when asked to cook the whole banquet (a full software repository). Natural language is too fuzzy. The AI might guess wrong, forget a step, or create a dish that looks good but doesn't taste right. Worse, because the instructions were vague, it's hard to prove why the meal failed.

The Solution: The "Structured Recipe Book"
The authors propose a new way of working called Structured Spec-Driven Engineering (SSDE). Instead of a vague conversation, they suggest giving the AI a strict, structured "recipe book."

In this paper, they use two types of structured recipes:

1. Gherkin Specifications: Think of these as "If-Then" test cases. Instead of saying "Make it work," you write: "IF a user clicks 'Book', THEN the room must be marked 'Occupied'." It's a checklist of exact behaviors.
2. Domain Models: These are like architectural blueprints or a map of the ingredients. They show how different parts of the system (like "Users," "Rooms," and "Dates") connect to each other.

The Experiment: The Taste Test
The researchers set up a pilot study. They acted as the head chefs and gave five different AI models (the apprentices) the task of building the "business logic" (the cooking rules) for three different software systems.

They tested different combinations:

• The Control Group: Just the vague natural language description.
• The Test Groups: The vague description PLUS the structured "recipe book" (the blueprints and the "If-Then" checklists).

The Results: Structure Wins
The findings were clear:

• Better Accuracy: When the AI had the structured "recipe book" (the blueprints and checklists), it made far fewer mistakes than when it only had the vague description.
• The "Blueprint" Boost: Giving the AI the specific code signatures (the exact list of ingredients and tools) along with the blueprints helped it the most. It was like giving the chef not just the recipe, but the exact brand of flour and the specific size of the pan to use.
• Still Room to Grow: While the structured approach was much better, the AI still made some errors. However, the researchers found that over 70% of these errors were simple, detectable mistakes — things like referencing a variable that doesn't exist, or making a Python syntax error. These don't even need a test oracle (i.e. running the code with example inputs to see what comes out): a standard compiler or linter would catch them.

The Future Roadmap
The paper suggests that to make this work perfectly, we need to:

1. Add a Feedback Loop: Instead of just asking the AI once, we should let it write the code, check it against the "recipe book," and fix its own mistakes automatically.
2. Build Better Datasets: We need more examples of these structured recipe books to train the AI better.
3. Handle Changes: Real software changes all the time. We need to teach the AI how to update just one part of the banquet (like swapping the dessert) without ruining the whole meal.

The Bottom Line
The paper concludes that if we stop treating AI like a magic wand that works on vague wishes, and start treating it like a skilled worker following a strict, structured blueprint, we can get it to build entire software systems reliably. It turns the AI from a "creative guesser" into a "precise builder."

Technical Summary

Technical Summary: LLM-Assisted Repository-Level Generation with Structured Spec-Driven Engineering

Problem Statement

While Large Language Models (LLMs) demonstrate high proficiency in generating code at the function or file level, their reliability significantly degrades when scaling to repository-level systems. Current workflows relying solely on natural language prompts suffer from inherent ambiguity, making communication "lossy" and hindering the generation of verifiable outputs (e.g., software tests). Simply increasing the verbosity of natural language instructions fails to address the fundamental lack of precision required for reliable, large-scale code generation and verification. The authors posit that a shift from unstructured natural language to structured specifications is necessary to enable reliable, verifiable, and high-quality repository-level generation.

Methodology: Structured Spec-Driven Engineering (SSDE)

The paper proposes Structured Spec-Driven Engineering (SSDE), a paradigm where structured artifacts (rather than free-form natural language) serve as the primary input to guide LLM agents. The authors conducted a pilot study to investigate the feasibility of this vision.

Experimental Setup

• Task: Generation of Model-View-Controller (MVC) business logic (specifically Python controllers) for three existing software systems: Symboleo, CheECSEManager, and MeetingGroups.
• Models: Five LLMs were evaluated, spanning open-source (Qwen 3, Llama 3.2) and closed-source (Claude Sonnet 4.5, GPT 5.1, GPT 5 Nano) models of varying sizes.
• Input Configurations: The study compared four types of contextual inputs provided to the LLM alongside a controller template:
  1. Natural Language Specification: Free-form descriptions of system purpose and constraints.
  2. Gherkin Specification: Structured, executable scenarios defining system behavior.
  3. Domain Model: Abstract software models (Umple or Ecore) capturing domain concepts and relationships.
  4. Signature Model: The generated class and function signatures (API) derived from the domain models.
• Evaluation:
  • Quantitative: A human-verified Python unit test suite (mapped to Gherkin scenarios) was executed against the generated controllers. The primary metric was the Test Pass Rate (TPR).
  • Qualitative: Manual inspection of generated code to identify recurring failure patterns and code quality attributes.
  • Repetition: Each configuration was run 10 times per LLM to account for stochastic bias.

Key Results

The pilot study yielded several critical findings regarding the efficacy of structured inputs:

1. Superiority of Structured Inputs: Adding any form of structured specification (Gherkin, Domain Model, or Signatures) significantly improved output quality compared to the baseline of using only Natural Language Specifications.
2. Gherkin Potential: While Gherkin specifications combined with models initially showed a slightly lower average TPR (6.8% lower) and higher variance than natural language combinations, they outperformed natural language in 14 out of 30 specific combinations. The authors note a potential for this approach to surpass natural language with further refinement.
3. Value of Signatures: Providing the model layer's signatures (API) generated from domain models proved highly effective. This configuration improved the average TPR by +7.82% and reduced standard deviation by -2.47% compared to using raw domain models, suggesting LLMs benefit from explicit API definitions over abstract model syntax.
4. Error Analysis: Over 70% of test failures were caused by errors detectable via static code analysis (e.g., invoking non-existent APIs, data type mismatches, unvalidated constraints). This indicates that a significant portion of generation failures are structural rather than logical, making them amenable to automated correction.
5. Model Performance: Claude Sonnet 4.5 emerged as the top performer, achieving TPRs over 80% in several configurations. However, high standard deviations across runs highlighted the stochastic nature of LLMs, where specific error types tend to propagate consistently through generated controllers.

Key Contributions

• Proposal of SSDE: The paper introduces Structured Spec-Driven Engineering as a viable paradigm to bridge the gap between high-level requirements and repository-level code generation, leveraging the verifiability of structured artifacts.
• Empirical Feasibility Study: The authors provide the first pilot study demonstrating that structured inputs (Gherkin, Domain Models, Signatures) can tangibly improve the quality of LLM-generated repository-level code compared to natural language prompts.
• Identification of Error Patterns: The study categorizes failure modes, revealing that the majority of errors are structural and detectable, pointing toward a clear path for automated feedback loops.
• Research Roadmap: The paper outlines a concrete path forward for the field, focusing on three pillars:
  1. Improving Output Quality: Implementing feedback loops using static analysis and dynamic testing to allow LLMs to self-correct structural errors.
  2. Evaluating Real-World Impact: Moving beyond single-shot generation to evaluate agentic workflows, token cost-effectiveness, and net productivity gains (accounting for the effort to maintain structured specs).
  3. Applicability: Developing techniques for partial repository updates to handle continuous evolution and integrating SSDE with diverse modeling tools and physical world representations.

Significance and Claims

The paper claims that SSDE offers a tangible path toward high-quality, verifiable, repository-level code generation. By shifting from ambiguous natural language to structured specifications, the approach leverages decades of software engineering knowledge (e.g., Model-Driven Engineering, Gherkin) to guide LLMs.

The authors argue that while LLMs currently struggle with large-scale systems, the integration of structured specifications transforms code generation from a "black box" process into one that is verifiable and maintainable. The study suggests that the majority of current generation failures are not insurmountable logical gaps but rather structural inconsistencies that can be resolved through automated feedback mechanisms. Ultimately, SSDE aims to reduce manual engineering burdens by fostering effective human-computer collaboration, where engineers act as specification architects and LLMs handle the translation to implementation.