Technical Summary: Strategies for Guiding LLMs to Use Software Design Patterns (Singleton Case)

Problem Statement

While Large Language Models (LLMs) have demonstrated significant capability in generating syntactically correct and functional source code from natural language prompts, they frequently fail to adhere to higher-level architectural structures and established design principles. Specifically, LLMs struggle to consistently apply software design patterns, such as the Singleton pattern, which are critical for long-term software maintainability, modularity, and reliability. The core problem addressed in this study is the lack of effective strategies to guide LLMs to incorporate specific design patterns into generated code without compromising the functional correctness of that code.

Methodology

The authors conducted a computational experiment involving 13 LLMs from six different model families (GPT, Llama, Qwen, Gemma, Mistral, and DeepSeek), ranging from 4B to 70B parameters. The study utilized the HumanEval-X dataset, comprising 164 Java coding challenges. Crucially, the dataset was selected because the canonical solutions for these tasks do not inherently require the Singleton pattern, ensuring that any implementation of the pattern was a result of the guiding strategy rather than a coincidental solution.

Evaluation Metrics

1. Functionality: Measured by the pass@1 rate, indicating the percentage of generated solutions that passed the predefined unit tests for each task.
2. Singleton Alignment: Measured using a modified Singleton Score based on the Pattern Property Satisfactory Rate (PSR). A class was evaluated against three specific predicates:
   • Private Constructor: The class has a single private constructor.
   • Instance Method: A private static method exists for retrieving the instance.
   • Global Access Point: A public static accessor method exists.
     The score was calculated as the percentage of missing predicates relative to the total required predicates.

Experimental Design

Four distinct prompting strategies were evaluated across the 13 models:

1. Guiding with Instructions: Models were explicitly instructed via a prompt to implement the Singleton pattern but received no feedback on errors.
2. Self-Improvement (Binary Feedback): Models were allowed iterative attempts. After each attempt, they received binary feedback indicating whether the code followed the Singleton pattern, without specifying which properties were missing.
3. Tool-Supported Self-Improvement (Extensive Feedback): Similar to the previous strategy, but the feedback explicitly identified which of the three Singleton predicates were missing (e.g., "It failed the following checks: Global Access Point").
4. Tool-Supported Few-Shot Prompting: Models received the extensive feedback from Strategy 3, accompanied by two examples of correctly implemented Singleton classes.

Key Results

Impact on Singleton Implementation

• Direct Instructions: Merely instructing models to use the Singleton pattern significantly improved adherence for most models. Llama 3.3 (70B) and Gemma 3 (27B) achieved a perfect Singleton Score (100%) with direct instructions alone.
• Iterative Feedback: Automated feedback loops consistently improved Singleton alignment across all models. Tool-supported self-improvement (extensive feedback) yielded the highest alignment for most models, with several (including Llama 3.3, GPT-4o mini, and GPT-5 mini) achieving 100% Singleton scores.
• Few-Shot Limitations: Contrary to the hypothesis that examples would universally help, adding few-shot examples did not consistently improve performance over extensive feedback alone. In some cases (e.g., Gemma 3 (4B)), it resulted in lower Singleton scores.

Impact on Functional Correctness

The study found that the effect of guiding LLMs toward design patterns on functional correctness is highly model-dependent:

• Positive Correlation: For several high-performing models, guiding them to use the Singleton pattern actually improved functional correctness. Llama 3.3 saw a 34.1 percentage point increase in test pass rates when guided by instructions. GPT-4o mini increased its pass rate by 55.5 percentage points.
• Negative Correlation: Conversely, for smaller or less capable models (e.g., DeepSeek Coder v2, Gemma 3 (4B)), the requirement to implement the Singleton pattern significantly degraded functionality, with pass rates dropping by up to 32.3 percentage points.
• Trade-offs: Some models (e.g., DeepSeek Coder) successfully implemented the Singleton structure but failed to maintain the original functionality, indicating a loss of context where the pattern was applied correctly but to the wrong logic.

Key Contributions

1. Empirical Evaluation of Strategies: The paper provides a systematic comparison of four distinct strategies (instructions, binary feedback, extensive feedback, and few-shot) for guiding LLMs to use design patterns.
2. Model-Specific Insights: The results demonstrate that there is no "one-size-fits-all" strategy. The optimal approach depends heavily on the specific LLM architecture and size. For instance, while extensive feedback helped most models, direct instructions were sufficient for top-tier models like Llama 3.3.
3. Functionality vs. Pattern Adherence: The study challenges the assumption that enforcing design patterns necessarily harms code quality. It shows that for capable models, enforcing architectural constraints can coincidentally improve functional correctness, likely by forcing the model to structure code more rigorously.

Significance and Claims

The authors claim that this work is significant because it moves beyond simple code generation to address the architectural integrity of LLM-generated software. The paper asserts that:

• Guidance is Feasible: Even simple strategies, such as direct instructions or binary feedback, can effectively guide LLMs to implement design patterns.
• Iterative Feedback is Robust: Iterative binary feedback provides the best overall alignment with the Singleton pattern while preserving or improving functionality for a wide range of models.
• Context Matters: The ability to apply design patterns correctly is not just a function of model size but of the prompting strategy used.
• Practical Application: The findings suggest that software engineering tools integrating LLMs can use these feedback loops to ensure generated code adheres to industry-standard design principles, thereby enhancing the long-term maintainability of AI-assisted software development.

The paper concludes modestly, noting that while the Singleton pattern is a creational pattern with clear structural predicates, the results may vary for more complex behavioral patterns. The study does not claim that LLMs can autonomously decide when to use a pattern, but rather that they can be effectively guided to use them when explicitly requested or corrected.