Exploring the Reasoning Depth of Small Language Models in Software Architecture: A Multidimensional Evaluation Framework Towards Software Engineering 2.0

This study benchmarks ten small language models on architectural decision record generation to establish a multidimensional evaluation framework, revealing that models exceeding 3 billion parameters excel in zero-shot reasoning while sub-2 billion models benefit most from fine-tuning, and that few-shot prompting effectively calibrates mid-sized models despite high semantic diversity often correlating with hallucinations.

The Gist

Imagine you are building a massive, complex city. Before you lay a single brick, you need a master plan—the Software Architecture. This plan decides where the power plants go, how the roads connect, and what happens if a bridge collapses. In the past, only highly trained human architects could draw these plans.

Now, we have AI (Artificial Intelligence) that can write code and help design these cities. But there's a catch: the most powerful AIs are like giant, hungry super-computers. They eat massive amounts of electricity, cost a fortune to run, and sometimes require you to send your secret city blueprints to the cloud, which feels risky.

This paper asks a simple question: Can we use smaller, cheaper, "local" AI brains to design these cities just as well?

Here is the breakdown of their findings, using some everyday analogies:

1. The "Big Brain" vs. The "Pocket Calculator"

The researchers tested 10 different "Small Language Models" (SLMs). Think of these as AI brains ranging from a smartphone calculator (1 billion parameters) to a powerful laptop (7 billion parameters).

• The Big Find: There is a "tipping point" at 3 billion parameters.
  • The "Laptop" models (3B+): These are surprisingly smart. If you just ask them, "Here is a problem, give me a solution," they can often come up with a solid architectural plan without any extra help. They understand the rules of the game.
  • The "Calculator" models (Under 2B): These are tricky. They are great at sounding fluent and using the right words (like a student who memorized the vocabulary list but doesn't understand the math). They often produce text that looks like a good plan but is actually nonsense or violates safety rules.

2. The "Example" Trick (Few-Shot Prompting)

Imagine you are teaching a new employee how to write a formal report.

• Zero-Shot: You just say, "Write a report." (The new hire might be confused).
• Few-Shot: You say, "Here are two examples of perfect reports. Now, write one like this."

The study found that for the mid-sized models (like the 3B ones), showing them just two examples was a magic trick. It acted like a "calibration signal." Suddenly, they understood the tone and structure perfectly, often performing as well as the giant, expensive models.

• However, for the models that were already very good at guessing, showing examples sometimes confused them (like over-explaining a simple task to an expert).

3. The "Specialized Training" (Fine-Tuning)

This is like taking a generalist doctor and sending them to a 3-month crash course to become a heart surgeon.

• For the tiny models: This helped them learn the specific "language" of architecture better, making their text sound more accurate.
• For the smart models: It actually hurt them. Because they were already good at reasoning, forcing them to memorize a small set of specific examples made them "forget" their general knowledge. They became too rigid and started making mistakes they wouldn't have made otherwise.

4. The "Creative" Trap (Diversity vs. Hallucination)

Sometimes, you want an AI to be creative and offer many different solutions.

• The Problem: With the tiny models, "high diversity" (offering many different answers) usually meant they were hallucinating. They were making things up just to sound different.
• The Solution: The mid-sized models, when given those two examples, managed to be both creative (offering different valid options) and accurate (following the rules). They found the sweet spot between "thinking outside the box" and "not falling off the edge."

The Bottom Line: What Should You Do?

The paper gives a "User Manual" for Software Engineers in the era of "Software Engineering 2.0" (where humans and AI work together):

1. If you have a 7B model (The Laptop): Just ask it nicely (Zero-Shot) or give it two examples. Do not waste time training it on specific data; it might get confused.
2. If you have a 3B model with a short memory (The Smart Tablet): Use the "Two Examples" trick. It's the cheapest and most effective way to get a pro-level result without paying for expensive training.
3. If you have a 1B model (The Calculator): You might need to train it heavily to get it to understand the basics, but even then, it might struggle to make truly sound architectural decisions on its own.

In summary: You don't need a supercomputer to design software architecture anymore. With the right "prompting" (giving the right examples), a small, local AI can do a great job, saving money, keeping your data private, and reducing the carbon footprint of your software projects.

Technical Summary

Here is a detailed technical summary of the paper "Exploring the Reasoning Depth of Small Language Models in Software Architecture: A Multidimensional Evaluation Framework Towards Software Engineering 2.0".

1. Problem Statement

In the emerging paradigm of Software Engineering 2.0 (SE 2.0), where intelligent agents collaborate with human engineers, there is a critical need to automate Software Architecture (SA) tasks, specifically the generation of Architecture Decision Records (ADRs). While Large Language Models (LLMs) have demonstrated capability in this domain, their high computational costs, latency, and data privacy concerns (due to cloud deployment) make them unsuitable for many enterprise and privacy-sensitive scenarios.

Small Language Models (SLMs), defined here as models with fewer than 7 billion parameters, offer a sustainable, locally deployable alternative. However, there is a significant knowledge gap regarding their reasoning depth. It remains unclear whether SLMs can genuinely understand complex architectural trade-offs or if they merely mimic surface-level text. Furthermore, existing benchmarks focus on code generation (functional correctness) or general NLP tasks, lacking specific metrics for architectural reasoning, structural compliance, and the trade-off between solution diversity and hallucination.

2. Methodology

The authors propose SLM-ArchBench, a multidimensional evaluation framework designed to probe the reasoning capabilities of SLMs in generating ADRs.

• Dataset: A curated dataset of 95 expert-authored ADRs from five open-source GitHub repositories (e.g., archane-framework, cardano). Each entry consists of a "Context" (problem/constraints) and a "Decision" (solution/rationale).
• Models Evaluated: 10 state-of-the-art instruction-tuned SLMs ranging from 1B to 7B parameters, including models from Meta (Llama-3.2), Microsoft (Phi-3), Google (Gemma), Alibaba (Qwen), and Mistral AI.
• Experimental Configurations:
  1. Zero-Shot: Baseline generation without examples.
  2. Few-Shot (In-Context Learning): Providing $k=2$ examples in the prompt to guide the model.
  3. Parameter-Efficient Fine-Tuning (PEFT): Fine-tuning models using LoRA (Low-Rank Adaptation) on the training split (80% of data) with rank $r=16$.
• Evaluation Metrics:
  • Lexical/Semantic Similarity: BERTScore (F1), ROUGE, BLEU, METEOR.
  • Architectural Compliance Score: An automated "LLM-as-a-Judge" system (using Gemini-2.5-Flash) that rates the technical validity, rationale alignment, and adherence to best practices (e.g., MVC, microservices) on a scale of 0–100.
  • Semantic Diversity Score: Measured via mean pairwise cosine distance between multiple generated candidates to distinguish between "productive exploration" and "stochastic variance" (hallucination).

3. Key Contributions

1. SLM-ArchBench Framework: A novel evaluation framework that moves beyond textual similarity to assess architectural reasoning, identifying the parameter threshold where models transition from lexical mimicry to genuine architectural compliance.
2. Strategy Comparison: A systematic comparison of In-Context Learning (Few-Shot) vs. Fine-Tuning (LoRA), providing actionable deployment guidelines for resource-constrained environments.
3. Quality-Diversity Trade-off Analysis: Empirical evidence characterizing how the tension between output diversity and generation quality manifests in SLMs, distinguishing between creative exploration and hallucination.

4. Key Results

The study reveals a distinct "Reasoning Gap" based on model size and adaptation strategy:

• Parameter Threshold (RQ1):

  • Models above 3B parameters (e.g., Mistral-7B, Qwen2.5-3B) demonstrate robust zero-shot capabilities, achieving compliance scores >65%.
  • Sub-2B models (e.g., Gemma-3-1B, SmolLM2-1.7B) show a significant disconnect: they achieve high lexical similarity (BERTScore ~0.80) but low architectural compliance (<55%), indicating they mimic terminology without understanding design rationale.

• Impact of Few-Shot Prompting (RQ2):

  • Few-shot prompting acts as a highly effective calibration mechanism for select mid-sized models with short context windows (e.g., Phi-3-mini, OLMo-2-7B).
  • Example: Phi-3-mini's compliance jumped from 66.4% (Zero-shot) to 72.1% (Few-shot), surpassing the few-shot performance of larger 7B models.
  • Conversely, for models with strong zero-shot priors (e.g., Llama-3.2-1B), adding examples introduced noise, degrading performance.

• Impact of Fine-Tuning (RQ3):

  • Ultra-lightweight models (1B): Fine-tuning provided the largest gains in semantic accuracy (BERTScore) but did not guarantee compliance improvements.
  • Larger/Competent Models: Fine-tuning often yielded diminishing returns or caused performance regression. For instance, Phi-3-mini's compliance dropped from 72.1% (Few-shot) to 47.8% after fine-tuning, suggesting over-specialization on narrow data can override effective in-context calibration.

• Semantic Diversity (RQ4):

  • In zero-shot settings, high diversity in sub-2B models correlated with hallucination (low compliance).
  • Few-shot prompting successfully balanced diversity and compliance for capable models, enabling "productive exploration" rather than random variance.

5. Significance and Implications

This study establishes a rigorous baseline for deploying sustainable, locally hosted architectural assistants in SE 2.0. The findings challenge the assumption that massive LLMs are strictly necessary for architectural reasoning and provide specific deployment guidelines:

• For 7B Models: Use Zero-Shot or selective Few-Shot strategies; avoid fine-tuning to prevent degradation.
• For 3B–7B Models with Short Contexts: Few-Shot prompting is the optimal strategy, offering state-of-the-art compliance without the computational overhead of fine-tuning.
• For 1B Models: Fine-tuning may improve semantic accuracy, but compliance gains are not guaranteed, and architectural regression is a risk.

Conclusion: The paper demonstrates that SLMs can effectively support architectural decision-making if the right adaptation strategy is matched to the model's parameter scale. This enables organizations to maintain data privacy, reduce costs, and lower carbon footprints while leveraging AI for high-level software design.