Reliability of Large Language Models for Design Synthesis: An Empirical Study of Variance, Prompt Sensitivity, and Method Scaffolding

1. Problem Statement

While Large Language Models (LLMs) have demonstrated the ability to generate syntactically valid UML class diagrams from natural language, there is a critical gap between diagram translation (mapping text entities to classes) and design synthesis (inferring implicit design principles, patterns, and architectural constraints).

• The Core Issue: Current LLM-generated designs often lack architectural soundness, failing to capture essential Object-Oriented Analysis and Design (OOAD) principles (e.g., abstraction, encapsulation) or established design patterns (e.g., Observer, Strategy).
• Reliability Challenge: Beyond design quality, generative models are inherently non-deterministic. For software engineering workflows to be trustworthy, models must produce structurally consistent designs across repeated runs and prompt variations (paraphrasing). Existing literature largely focuses on capability, neglecting behavioral stability and the impact of model choice on design reliability.

2. Methodology

The authors conducted a controlled empirical study involving 540 experiments to evaluate three state-of-the-art LLMs: ChatGPT 4o-mini, Claude 3.5 Sonnet, and Gemini 2.5 Flash.

Experimental Setup

• Benchmarks: Two domain-specific scenarios were created where prompts described domain behavior and constraints without explicitly naming design patterns or principles:
  1. HMS (Hospital Management System): Medium complexity, policy-based billing.
  2. Sensor Network: High complexity, event-driven interactions.
• Variables:
  • Models: 3 (ChatGPT, Claude, Gemini).
  • Prompting Strategies:
    1. Standard Prompting: Domain description only.
    2. Rule-Injection Prompting: Explicitly encoded OOAD principles and rules.
    3. Preference-Based Few-Shot Prompting: The proposed approach using contrasting examples (good vs. bad designs) to bias the model toward architectural soundness.
  • Variations: 3 paraphrased prompts per domain and 10 repeated runs per configuration.
• Evaluation Metrics:
  • Structural Correctness (M1.1): Precision, Recall, and F1 scores against an expert-designed reference model.
  • Principle Adherence Score (M1.2): Binary/Ternary assessment of adherence to Abstraction and Encapsulation.
  • Pattern Emergence Score (M1.3): Assessment of implicit application of Observer and Strategy patterns.
  • Stability Index (SI): A metric calculating variance across runs and prompts to measure behavioral reliability.

Proposed Approach: Preference-Based Alignment

Instead of full Reinforcement Learning from Human Feedback (RLHF), the authors introduced a lightweight preference-based few-shot prompting technique.

• Mechanism: The model is presented with a dataset containing task descriptions paired with two solutions: one generated by an LLM (labeled "bad") and one by human experts (labeled "good/preferred").
• Goal: To train the model to recognize and prefer architecturally sound structures without explicitly naming the patterns in the prompt, thereby encouraging implicit reasoning.

3. Key Contributions

1. Distinction between Translation and Synthesis: The study empirically validates that LLMs often act as translators rather than synthesizers, failing to infer implicit design constraints unless guided by specific alignment techniques.
2. Preference-Based Prompting Framework: Introduces a novel, lightweight prompting strategy that uses contrastive examples to steer LLMs toward OOAD principles and pattern-consistent structures, outperforming standard rule-injection.
3. Behavioral Reliability Analysis: Systematically quantifies the non-determinism of frontier models in design tasks, revealing that model choice is a dominant factor in reliability, often outweighing prompting strategies.
4. Comprehensive Benchmarking: Provides a rigorous evaluation protocol (540 runs) across multiple models, domains, and prompt variations, establishing a baseline for future research in AI-assisted software design.

4. Key Results

• Design Quality (RQ1 & RQ2):
  • Preference-Based Prompting consistently yielded the highest adherence to design principles and pattern emergence compared to standard and rule-injected baselines.
  • Rule Injection (explicitly listing rules) did not significantly improve performance and, in some cases (notably Gemini), degraded performance due to cognitive overload.
  • Model Performance: ChatGPT showed the strongest adherence to principles in the preference setting. Claude produced structurally coherent diagrams but struggled with abstraction. Gemini showed the weakest performance, often introducing structural errors like circular relationships.
• Behavioral Stability (RQ3):
  • Claude: Exhibited high decoding stability. Once conditioned on a prompt, it produced highly consistent results across repeated runs, though it sometimes consistently omitted critical relationships.
  • ChatGPT: Showed predictable behavior once architectural scaffolding was defined; errors were systematic rather than stochastic.
  • Gemini: Demonstrated high variability (low stability) across both runs and prompt paraphrases, indicating significant architectural drift.
  • Insight: Semantic consistency did not guarantee structural consistency.
• Complexity Impact (RQ4):
  • Performance degraded significantly as domain complexity increased (from HMS to Sensor Network).
  • No model successfully inferred the Observer pattern in the event-driven sensor network scenario, suggesting a "ceiling effect" where models struggle with behavior-driven pattern inference without explicit guidance.

5. Significance and Implications

• Reframing the Challenge: The study shifts the focus from "Can LLMs generate diagrams?" to "Can LLMs function as dependable architectural collaborators?"
• Model Selection is Critical: Achieving reliable software design is not just about prompt engineering; the choice of the underlying model is a primary determinant of reliability.
• Stability as a Metric: Behavioral stability (repeatability) must be treated as a first-class evaluation criterion alongside syntactic correctness. A model that generates a correct design once but inconsistently is operationally risky.
• Future Directions: The paper suggests that future work should explore hybrid approaches combining preference-based prompting with explicit architectural reasoning modules, model fine-tuning, and constrained decoding to reduce non-determinism.

In conclusion, while preference-based alignment improves the likelihood of generating principled designs, current LLMs still lack the global reasoning required for fully reliable, complex design synthesis. The path to dependable AI-assisted software engineering requires addressing both capability (design quality) and reliability (behavioral stability).