💻 computer science

An Empirical Study of Sustainability in Prompt-driven Test Script Generation Using Small Language Models

This empirical study investigates the environmental impact and performance trade-offs of small language models (2B–8B parameters) in prompt-driven unit-test generation, revealing that model selection and prompt structure significantly influence energy consumption, carbon emissions, and test coverage.

Original authors: Pragati Kumari, Novarun Deb

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pragati Kumari, Novarun Deb

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to bake a perfect cake (a software test script) for a bakery. In the past, you only had access to massive, industrial-sized ovens (Large Language Models) that could bake anything but consumed enough electricity to power a small city.

Recently, smaller, more efficient ovens (Small Language Models, or SLMs) have become available. They are cheaper to run and fit in smaller kitchens. But nobody really knew: How much energy do these smaller ovens actually use? Does the recipe you give them matter? And does the location of the bakery (the power grid) change how "dirty" the baking process is?

This paper is like a scientific taste-test and energy audit of these smaller ovens. Here is the breakdown in simple terms:

1. The Big Question: "Green" vs. "Good"

The researchers wanted to know if they could get high-quality test scripts (good cakes) without burning too much carbon (polluting the air). They focused on models with 2 to 8 billion "brain cells" (parameters)—the "Goldilocks" zone of AI: not too huge, not too tiny.

They asked four main questions:

The Recipe Matters: Does writing a very detailed instruction (prompt) use more energy than a simple one?
The Location Matters: Does baking in a country with clean wind power (like the Netherlands) vs. a country with dirty coal power (like parts of the US or Singapore) change the carbon footprint?
The Compression Matters: If we shrink the model's memory (quantization) to make it faster, does it save energy, or does it make the cake taste worse?
The Trade-off: How do we choose the best oven if we care more about saving the planet vs. getting a perfect cake?

2. The Experiment: A Controlled Kitchen

To test this, the researchers used a standard set of 164 coding puzzles (the HumanEval benchmark). They treated these puzzles like a menu.

The Ovens (Models): They tested five different small AI models (like Phi-3.5, Mistral, Llama-3).
The Recipes (Prompts): They created four versions of instructions, ranging from a simple "Make a test" (APV0) to a complex, role-playing instruction like "Act as an expert tester, follow these strict rules, and use this format" (APV3).
The Fuel (Energy): They used a tool called CodeCarbon to measure exactly how much electricity was used and how much CO2 was emitted.
The Location: Since they used Google's cloud computers, the "kitchen" moved around the world. Sometimes the oven was in Nevada (dirty grid), sometimes in the Netherlands (clean grid).

3. The New Tools: Measuring "Greenness"

The researchers realized that just looking at "accuracy" isn't enough. They invented two new scorecards to help people choose:

The "Sustainability Velocity Index" (SVI): Think of this as a Car Efficiency Score. It combines how fast the car goes, how much gas it uses, and how safe it is. A high score means the model is fast, clean, and produces good tests all at once.
The "Green F-Beta Score" (GFβ): This is a Dial you can turn.
- If you turn the dial to 0.1, the system prioritizes saving the planet (lowest carbon).
- If you turn the dial to 10, the system prioritizes perfect test coverage (best quality), even if it costs a bit more energy.
- This helps a manager decide: "Do I want the greenest option, or the best quality option?"

4. What They Found (The Surprises)

Location is King: The biggest factor wasn't the model itself, but where it was running. An inefficient model running on a super-clean wind grid could be greener than a super-efficient model running on a dirty coal grid.
Complexity Costs: Giving the AI a very complex, detailed prompt (APV3) did use slightly more energy, but it often resulted in better test scripts. It's like spending a little extra gas to get a better route.
No Perfect Winner: There was no single "best" model.
- Some models were fast and clean but made fewer tests.
- Some made great tests but took longer and used more power.
- The Lesson: You have to pick the right tool for the job. If you need speed, pick Model A. If you need quality, pick Model B.

5. The Takeaway for Everyone

This paper tells software developers and managers: "Don't just ask 'Is the AI smart?' Ask 'Is the AI green?'"

If you are building software, you shouldn't just pick the biggest, most powerful AI available. Instead, you should:

Check your location: If you can run your code in a region with clean energy, do it.
Adjust your dial: Decide if you care more about speed/quality or saving energy, and use the new scoring tools to pick the right model.
Be smart about prompts: A slightly longer, clearer instruction might save you money and carbon in the long run by getting the job done right the first time.

In short: Sustainability isn't just about the size of the engine; it's about the fuel, the driver, the route, and the destination. This study gives us the map to drive greener.

1. Problem Statement

While Large Language Models (LLMs) have advanced automated test script generation, their high energy consumption and carbon intensity ("environmental tax") are well-documented. However, existing sustainability research predominantly focuses on massive foundation models. There is a significant gap in understanding the environmental characteristics of Small Language Models (SLMs) (specifically in the 2B–8B parameter range) during prompt-driven unit-test generation.

In resource-constrained or privacy-sensitive environments where local execution is preferred, the trade-offs between the energy efficiency of SLMs and their test generation performance remain unexplored. The study addresses the need to understand how prompt structure, model quantization, and regional power grid carbon intensity jointly influence the sustainability of SLM-based test generation.

2. Methodology

The authors conducted a controlled empirical study using a five-phase pipeline:

Dataset: The HumanEval benchmark (164 Python programming tasks) was used. The prompt and canonical solution were merged to create a "Software Module Under Test."
Models: Five state-of-the-art SLMs were evaluated: Phi-3.5-mini, Qwen2.5-1.5B, deepseek-coder-7b, Mistral-7B, and Llama-3-8B.
Prompt Engineering: Four progressive prompt variants ( $APV_0$ to $APV_3$ ) were designed based on Anthropic's template, ranging from a minimal runnable template to highly structured prompts with persona, rules, and examples.
Quantization: Models were tested under unified 8-bit quantization for cross-architecture comparison. Additionally, Phi-3.5-mini and Qwen2.5-1.5B were tested across 4-bit, 8-bit, and unquantized levels to analyze precision impacts.
Execution Environment: Experiments ran on Google Colab (NVIDIA T4 GPU), which dynamically migrates workloads to global data centers. This allowed the study to capture varying regional carbon intensities (e.g., Netherlands vs. USA vs. Singapore).
Measurement Tools:
- CodeCarbon: Tracked energy consumption (Wh), carbon emissions (gCO2eq), and duration.
- Coverage.py: Measured unit-test script coverage ( $Q$ ).

3. Key Contributions

The paper introduces two novel metrics to quantify the trade-off between performance and sustainability:

Sustainability Velocity Index (SVI): A multiplicative metric combining code coverage ( $Q$ ), normalized Software Carbon Intensity ( $\hat{SCI}$ ), and normalized execution time ( $\hat{T}$ ). It prevents high performance in one area from masking poor performance in another, promoting balanced eco-efficiency.
$SVI = \frac{Q}{100} \cdot \frac{1}{(1 + \hat{SCI})(1 + \hat{T})}$
Green $F_\beta$ Score ( $GF_\beta$ ): A composite index similar to the F-score that balances eco-efficiency and code coverage. The parameter $\beta$ $β$ allows stakeholders to tune priorities:
- $\beta < 1$ : Prioritizes eco-efficiency.
- $\beta > 1$ : Prioritizes code coverage.

4. Key Results

The study answered four Research Questions (RQs) with the following findings:

RQ1 (Prompt & Region Impact): Software Carbon Intensity (SCI) is jointly shaped by prompt complexity and regional grid intensity. While complex prompts increase overhead, the geographical location (carbon intensity of the power grid) is often the dominant driver of emissions. For instance, runs in the Netherlands (low carbon) yielded significantly lower SCI than those in Singapore or Nevada (high carbon).
RQ2 (Holistic Ranking): The SVI successfully provides a holistic ranking. While low-carbon regions generally yield higher SVI, specific model-prompt configurations can counterbalance regional disadvantages, proving that a single metric (like emissions alone) is insufficient for ranking.
RQ3 (Quantization Trade-offs): The impact of quantization is context-dependent.
- In a fixed region, higher precision (unquantized) generally improved SVI, suggesting coverage gains outweighed compute savings from lower bits.
- Across varying regions, grid intensity often dominated the outcome, making quantization effects secondary to location.
RQ4 (Architect Decision Support): The $GF_\beta$ score demonstrates that model rankings shift based on the chosen priority ( $\beta$ ). A model that is best for eco-efficiency ( $\beta=0.01$ ) may not be the best for coverage ( $\beta=10$ ). Furthermore, prompt variants ( $APV_0$ vs. $APV_3$ ) and execution regions can alter these rankings, necessitating context-aware model selection.

5. Significance and Implications

Green AI for Testing: This work shifts the focus from "Green LLMs" to "Green SLMs," providing actionable evidence for developers who need local, privacy-preserving, or cost-effective testing solutions.
Context-Aware Selection: The study proves that there is no single "best" model. Optimal selection depends on the deployment context (target region's grid), the prompt strategy, and the specific goal (maximizing coverage vs. minimizing carbon).
New Metrics: The introduction of SVI and $GF_\beta$ offers the software engineering community standardized, multi-dimensional tools to evaluate and compare AI models beyond simple accuracy or latency.
Policy & Practice: The findings suggest that for sustainable testing, organizations should not only select efficient models but also consider workload migration to low-carbon regions and carefully tune prompt complexity to avoid unnecessary energy overhead.

6. Limitations

The authors acknowledge threats to validity, including reliance on estimated carbon intensity (via CodeCarbon) rather than real-time smart meter data, the use of a single benchmark (HumanEval), and potential variability in Google Colab's background processes. Future work aims to expand to more datasets, hardware architectures (TPUs, edge devices), and real-time emission tracking.