Technical Summary: SAKE – Software Architectural Knowledge Evaluation Benchmark

Problem Statement

While Large Language Models (LLMs) are increasingly deployed across the software development lifecycle, their ability to reason about software architecture remains largely unmeasured. Existing benchmarks primarily target syntactic code generation or algorithmic problem-solving (e.g., HumanEval, MMLU), failing to capture the complex cognitive demands of architectural decision-making. Such decisions require balancing quality attribute trade-offs, applying design patterns, and navigating system-level constraints—tasks that go beyond simple pattern matching. Consequently, practitioners lack a principled basis for trusting LLMs as architectural advisors, and researchers lack a shared baseline to measure progress in this domain.

Methodology

The authors introduce SAKE (Software Architectural Knowledge Evaluation), a standardized, reproducible benchmark designed to assess LLM competency in software architecture. The methodology follows a three-phase process:

1. Taxonomy Definition: The benchmark is structured around eight knowledge categories derived from two canonical references: Software Architecture in Practice [5] and Gang of Four Design Patterns [10]. These categories are:
   • Design Patterns: Creational (CP), Structural (SPA), and Behavioral (BP).
   • System-Level Concepts: Quality Attributes (QA), Architectural Solutions (ASO), Scalable Practices (SPR), Architectural Styles (AST), and Software Architecture for Quantum Computing (QC).
2. Question Construction: The benchmark comprises 2,154 expert-curated multiple-choice questions (four options each). Questions were independently authored by the three authors and subjected to a dual independent peer review process to ensure content validity and inter-rater agreement. Ambiguous items were revised or discarded.
3. Stratification: Questions are stratified by context length (short, medium-short, medium-long, long) to evaluate whether performance degrades as prompts require broader contextual integration.
4. Evaluation Protocol: The authors evaluated 11 state-of-the-art models (both proprietary and open-weight) in zero-shot and five-shot settings. Prompts were constrained to short outputs to ensure consistent answer extraction.

Key Contributions

• First Dedicated Benchmark: SAKE is the first standardized benchmark explicitly designed to evaluate software architectural knowledge in LLMs, filling a critical gap identified in systematic reviews of the field.
• Expert-Validated Dataset: Unlike crowdsourced benchmarks, SAKE relies on a rigorous dual-review process by domain experts, ensuring questions test genuine architectural understanding rather than reading comprehension.
• Granular Analysis: The benchmark moves beyond aggregate scores, providing stratified results across eight architectural categories and four context-length levels to diagnose specific competency gaps.
• Open Source Release: All benchmark data, evaluation scripts, and results are released as open source to establish a baseline for the community.

Results

The evaluation of 11 models yielded the following insights:

• Overall Performance (RQ1): Models demonstrated consistently high overall accuracy, ranging from 89.31% to 94.23%. However, the practical differentiator is cost; open-weight models like Qwen 3 235B offer competitive accuracy at significantly lower costs compared to top proprietary models.
• Category Variance (RQ2): Performance is uneven across domains.
  • Easiest Categories: Quality Attributes (QA) and Creational Patterns (CP) showed near-ceiling performance (avg. ~98% and ~97% in zero-shot), indicating models possess strong recall of these concepts.
  • Hardest Categories: Architectural Solutions (ASO) and Quantum Computing (QC) were consistently the most challenging, with lower average accuracy and higher variance between models. Architectural Solutions remained the weakest category even in five-shot settings.
• Context Length Effects (RQ3): The impact of prompt length is category-dependent, contradicting the intuition that "more context is always better."
  • Recall-Oriented Tasks: Longer prompts generally improved or maintained accuracy.
  • Reasoning-Heavy Tasks: For Architectural Solutions, accuracy degraded as prompt length increased. The authors suggest this is due to higher distractor density in longer scenarios, which complicates trade-off reasoning rather than aiding it.
• Few-Shot Impact: Five-shot prompting yielded variable results. While some high-capacity models improved slightly, several efficiency-oriented models (e.g., Deepseek v3.2) experienced significant performance degradation, particularly in complex categories.

Significance and Claims

The paper positions SAKE as a foundational knowledge layer rather than a substitute for full architectural design support. The authors claim that:

1. Trustworthiness is Category-Specific: A model's high aggregate score is misleading; practitioners must consult category-level results to determine if a model is reliable for specific architectural tasks (e.g., a model may be excellent at Quality Attributes but unreliable for Architectural Solutions).
2. Context is Not a Panacea: Adding context does not uniformly improve architectural reasoning; in fact, it may hinder performance on complex trade-off analysis tasks.
3. Baseline for Progress: SAKE provides the first reproducible baseline for tracking architectural reasoning in LLMs, enabling researchers to identify specific gaps (e.g., in reasoning over long contexts) and target improvements via techniques like Retrieval-Augmented Generation (RAG) or Chain-of-Thought prompting.

The authors conclude that while current models show promise, significant competency gaps remain in areas central to professional practice, necessitating continued human oversight and targeted model development.