Technical Summary: Failure-Aware Enhancements for Large Language Model Code Generation

Problem Statement

While Large Language Models (LLMs) demonstrate significant promise in automating software development, even advanced prompting workflows like progressive prompting (which structures generation into requirements analysis, design, testing, and implementation) fail to meet all requirements in complex scenarios. In real-world development, when an LLM assistant fails to integrate external APIs or handle specific domain logic, developers lack systematic guidance on which remedial strategy to apply. Existing enhancement methods—such as self-critique, multi-model collaboration, and retrieval-augmented generation (RAG)—have been proposed individually, but there is no empirical evidence linking specific failure modes to the most effective enhancement strategy. This gap forces practitioners to rely on trial-and-error rather than data-driven decision-making.

Methodology

The authors conducted a two-phase empirical study involving 25 open-source GitHub projects to evaluate enhancement strategies for LLM-based code generation.

Phase 1: Baseline Establishment

• Dataset: 25 diverse projects (e-commerce, booking, healthcare, microservices) spanning multiple languages (Java, Python, JS, etc.) with 3–31 tasks each.
• Comparison: The study compared Direct Prompting (one-shot code generation) against Progressive Prompting (sequential phases: requirements → design → test → code).
• Ground Truth: Original GitHub implementations served as the reference for requirement coverage.
• Model: GPT-5 was used as the state-of-the-art LLM.

Phase 2: Enhancement Strategy Evaluation

• Selection: From the 8 projects where progressive prompting failed to achieve 100% completion, 6 representative "challenge projects" were selected using maximum diversity sampling.
• Failure Taxonomy: Incomplete tasks were classified into four high-level failure types based on missing functionality rather than low-level bugs:
  1. Local Logic Failures: Issues solvable via code review (e.g., CRUD, validation rules).
  2. External Integration Failures: Missing API docs or service configurations.
  3. Domain Knowledge Failures: Requiring specialized expertise (e.g., healthcare business rules).
  4. Infrastructure Configuration Failures: Requiring deployment knowledge (e.g., VM provisioning).
• Enhancement Pipelines: Three concrete strategies were instantiated and tested against the baseline:
  1. Self-Critique: Iterative self-review where the LLM critiques its own output to identify gaps and generate fixes.
  2. Multi-Model Collaboration: A two-phase approach using GPT-5 for architectural design and Claude Sonnet 4.5 for code implementation.
  3. RAG-Assisted: Augmentation of progressive prompting with retrieved official documentation, similar open-source projects, and implementation patterns.
• Metrics: Task completion rate, improvement over baseline, total LLM generation time, prompt count, and efficiency (minutes per percentage-point improvement).

Key Results

1. Baseline Performance (RQ1)
Progressive prompting significantly outperformed direct prompting, achieving an average completion rate of 96.9% compared to 80.5% (Cohen's d = 1.63, p < 0.001). However, 8 out of 25 projects (32%) remained incomplete, confirming that progressive prompting alone is insufficient for complex requirements.

2. Enhancement Effectiveness by Failure Type (RQ2)
The study revealed that the effectiveness of an enhancement method is critically dependent on the failure type:

• Self-Critique: Highly effective for Local Logic Failures (e.g., P2, P10), achieving 100% completion by identifying missing validation or logic in the code. However, it showed 0% improvement on External Integration, Domain Knowledge, and Infrastructure failures, as these require knowledge external to the code itself.
• RAG-Assisted: Achieved the highest overall completion (99.2%) and was the most effective for External Integration and Infrastructure failures. By retrieving relevant documentation (e.g., RabbitMQ tutorials, Django i18n guides), it resolved gaps that self-critique could not.
• Multi-Model Collaboration: Achieved 99.2% completion (matching RAG) and was highly reliable (100% on 5/6 projects). However, it did not rank first in any project due to higher time costs.

3. Time-Efficiency Trade-offs (RQ3)

• RAG-Assisted was the most efficient method, averaging 31.5 minutes per percentage-point (min/pp) of improvement.
• Self-Critique averaged 40.6 min/pp but was highly variable: extremely fast when it worked (logic errors) but yielded no improvement (infinite cost) for integration errors.
• Multi-Model Collaboration was the slowest (47.4 min/pp) due to the overhead of separate design and implementation phases.

Key Contributions

1. Failure-Driven Decision Framework: The paper proposes a hierarchical framework that maps specific failure patterns to the most suitable enhancement method.
   • Local Logic Failures: First choice is Self-Critique (fastest).
   • External/Domain/Infrastructure Failures: First choice is RAG-Assisted.
   • High-Stakes/Residual Gaps: Multi-Model Collaboration serves as a reliable secondary option when speed is less critical than maximum correctness.
2. Empirical Taxonomy of Failures: A classification of LLM code generation failures into four distinct categories (Local Logic, External Integration, Domain Knowledge, Infrastructure), demonstrating that "one-size-fits-all" enhancement strategies are ineffective.
3. Concrete Implementation Pipelines: The authors instantiated and tuned three specific enhancement pipelines (Self-Critique, Multi-Model, RAG) rather than discussing them abstractly, providing a practical basis for the decision framework.

Significance and Claims

The paper claims that the primary barrier to fully automated code generation is not the lack of powerful models, but the lack of failure-aware selection strategies. The authors argue that practitioners should not rely on trial-and-error but should instead diagnose the type of failure and apply the corresponding enhancement.

The study concludes that while progressive prompting nearly matches human coverage (96.9% vs. 95.7%), the remaining gaps are systematic. By adopting the proposed decision framework, developers can move from a "try everything" approach to a targeted strategy that balances speed, reliability, and cost. The authors position their work as a step toward actionable, data-driven guidance for integrating LLMs into software engineering workflows, specifically noting that the framework is designed as a heuristic for practitioners rather than a universal constant, and acknowledging that results may vary with different models or specialized domains (e.g., embedded systems).