💻 computer science

Sustainability Analysis of Prompt Strategies for SLM-based Automated Test Generation

This paper presents the first systematic sustainability evaluation of prompt engineering strategies for automated test generation using Small Language Models, revealing that while reasoning-intensive prompts improve test coverage, simpler strategies offer a more environmentally sustainable balance between quality and resource consumption.

Original authors: Pragati Kumari, Novarun Deb

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pragati Kumari, Novarun Deb

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef running a busy kitchen. Your goal is to cook a specific dish (in this case, writing computer code to test software). You have three different chefs (the Small Language Models or SLMs) who are all talented but have different styles.

The big question this paper asks is: Does the way you give instructions to your chefs matter more than which chef you hire when it comes to saving energy and money?

Most people think, "If I hire a super-expensive, super-smart chef, the job will be done perfectly and efficiently." But this research suggests that how you write the recipe card (the "Prompt") is actually the most important factor in how much electricity the kitchen uses and how much carbon is emitted.

Here is the breakdown of the study using simple analogies:

1. The Setup: The Kitchen and the Chefs

The Task: The kitchen needs to generate "test scripts." Think of these as safety inspectors checking if a new car part works.
The Chefs (SLMs): The researchers used three open-source AI models (Llama, DeepSeek, Mistral). They are like three different cooks.
The Recipe Cards (Prompt Strategies): This is the core of the study. The researchers tried seven different ways to ask the chefs to do the job.

2. The Seven "Recipe Styles" (Prompt Strategies)

The researchers tested different ways of giving instructions:

Zero-Shot (The "Just Do It" Order): "Hey chef, write a test for this code." No examples, no hints.
- Result: Fast and cheap, but sometimes the chef misses a detail.
Few-Shot (The "Show Me an Example" Order): "Here is an example of a test I liked. Now write one for this new code."
- Result: Very efficient. The chef gets the vibe quickly, uses less energy, and does a great job.
Chain of Thought (The "Think Aloud" Order): "Before you write the test, explain your reasoning step-by-step out loud."
- Result: The chef thinks deeply. The test is good, but the chef talks for a long time, burning a lot of electricity.
Self-Consistency (The "Ask Three Chefs" Order): "Ask three different versions of yourself to solve this, and pick the best answer."
- Result: This is the most expensive method. It's like hiring three chefs to do one job just to be sure. It uses a massive amount of energy and time, but the final result isn't that much better than the simpler methods.
ReAct (The "Act and Reflect" Order): "Try to do it, check if it works, and fix it if it doesn't."
- Result: A middle ground. Good quality, but takes a bit more energy than the simple methods.

3. The Big Discovery: The "Green" Secret

The researchers measured everything: how long it took, how much electricity was used, and how much carbon was emitted.

The Shocking Finding:
The way you ask the question (the Prompt) matters more than which chef you use.

The "Over-Thinkers" (Complex Prompts): Strategies like Self-Consistency and Chain of Thought made the chefs talk and think so much that they burned huge amounts of energy. It was like driving a car with the engine revving at 5,000 RPM just to park it. The quality of the test didn't improve enough to justify the extra fuel.
The "Smart & Simple" (Simple Prompts): Strategies like Few-Shot (giving an example) were the winners. They got the job done with high quality but used a fraction of the energy and time.

4. The Analogy: The GPS vs. The Map

Imagine you are trying to get to a destination.

Complex Prompts are like asking a GPS to calculate 10 different routes, check traffic for every single one, and then drive the longest one just to be "safe." It takes forever and burns a lot of gas.
Simple Prompts are like looking at a map and taking the most direct route. You get there just as fast (or faster), and you save a ton of gas.

5. Why This Matters

In the world of software testing, companies run these tests thousands of times a day.

If they use the "Over-Thinker" prompts, they are wasting massive amounts of electricity and money, and hurting the environment, for very little extra benefit.
If they switch to the "Smart & Simple" prompts, they can save a huge amount of energy and carbon emissions without sacrificing the quality of their software.

The Bottom Line

You don't need the most expensive AI model to be sustainable. You just need to ask the right question.

By choosing a simpler, smarter way to talk to the AI (like giving a quick example instead of asking for a long essay), we can make software testing faster, cheaper, and much greener. It turns out that how we talk to machines is just as important as the machines themselves.

1. Problem Statement

The increasing adoption of AI-driven automated testing, particularly using Large Language Models (LLMs) and Small Language Models (SLMs), raises significant concerns regarding computational and environmental sustainability. While existing research focuses on the effectiveness of test generation or the sustainability of training large models, there is a critical gap in understanding how prompt engineering strategies influence the environmental footprint of inference-time test generation.

Specifically, it is unclear whether complex reasoning-based prompts (e.g., Chain of Thought) yield proportional improvements in test quality compared to their increased energy and carbon costs, especially when using resource-constrained SLMs. This paper addresses the need to systematically evaluate the trade-offs between prompt strategy, sustainability metrics (energy, carbon, time), and test coverage quality.

2. Methodology

The authors propose a prompt-centric sustainability evaluation framework designed to analyze seven distinct prompt strategies across three open-source SLMs.

A. Experimental Setup

Models: Three quantized (4-bit) open-source SLMs were used to simulate resource-constrained environments:
1. Meta-Llama-3-8B-Instruct
2. DeepSeek-Coder-7B-Instruct-v1.5
3. Mistral-7B-Instruct-v0.3
Dataset: The MBPP (Mostly Basic Python Problems) dataset, consisting of Python programming tasks with reference solutions and test specifications.
Prompt Strategies Evaluated:
1. Zero-shot
2. Few-shot
3. Chain-of-Thought (CoT)
4. Least-to-Most (LtM)
5. Program-of-Thought (PoT)
6. Self-Consistency CoT (SC_CoT)
7. ReAct
Environment: Experiments conducted on Google Colab (NVIDIA A100 GPU) using CodeCarbon for energy/emissions tracking and coverage.py for test coverage analysis.

B. Metrics and Framework

The framework collects Primary Parameters and derives Efficiency Metrics:

Primary Parameters: Total tokens ( $T$ ), execution time ( $\tau$ ), carbon emissions ( $CO_2$ ), and energy consumption ( $E_{cpu}, E_{gpu}, E_{ram}$ ).
Derived Metrics:
- Token Efficiency: Tokens per hour, time/energy/carbon per 1,000 tokens.
- Coverage Efficiency: Test coverage ( $Q$ ) per 1,000 tokens, per kWh, and per kg $CO_2$ .
- Composite Score ($SQScore$): A trade-off metric combining coverage quality and sustainability costs:
  $SQScore = \alpha \cdot \frac{Q}{CO_2 \cdot E_{tot} \cdot \tau_{hr}}$
  Where $\alpha$ adjusts the prioritization between eco-friendliness and coverage quality.

3. Key Contributions

First Systematic Evaluation: This is the first study to jointly evaluate prompt engineering strategies for SLM-based automated test generation against sustainability metrics (energy, carbon, time) and coverage quality.
Comprehensive Framework: Introduction of a unified framework that integrates token usage, execution time, energy consumption, carbon emissions, and normalized code coverage into a single comparative analysis.
Empirical Trade-off Analysis: Demonstration that prompt strategy selection is a dominant factor in sustainability outcomes, often outweighing the impact of the specific model choice.
Actionable Guidelines: Identification of specific prompt strategies that offer the best balance between test effectiveness and environmental cost.

4. Key Results

The study reveals that prompt strategy selection has a substantial and independent impact on sustainability, often more so than the choice of the underlying model.

Execution Time & Energy:
- Lightweight strategies (Zero-shot, Few-shot) consistently achieved the shortest execution times (6.69–10.19 hours) and lowest energy consumption (<1.24 kWh).
- Reasoning-intensive strategies, particularly Self-Consistency (SC_CoT), incurred significantly higher costs. SC_CoT increased execution time to ~24–27 hours and energy consumption to ~3.3 kWh, representing a massive overhead compared to simpler prompts.
Coverage Quality:
- While complex strategies (CoT, ReAct, SC_CoT) generally achieved high coverage, the gains were not proportional to the increased resource consumption.
- Few-shot and Least-to-Most (LtM) strategies achieved competitive or superior coverage (up to 0.98 normalized) with significantly lower environmental costs.
Efficiency Metrics (Per 1K Tokens):
- Few-shot and LtM demonstrated the highest efficiency, generating the most coverage per unit of energy, carbon, and time.
- SC_CoT showed the poorest efficiency, with coverage per kWh and per kg $CO_2$ dropping significantly (e.g., ~0.22–0.33 $Q$ /kWh vs. ~1.2 $Q$ /kWh for Few-shot).
Composite Trade-off ($SQScore$):
- Few-shot and LtM consistently achieved the highest $SQScore$ across all models and prioritization settings ( $\alpha$ ).
- SC_CoT consistently ranked last, indicating that its marginal coverage improvements do not justify the exponential increase in environmental cost.

5. Significance and Implications

Prompt Design as a Sustainability Lever: The paper establishes that prompt engineering is a critical, actionable lever for reducing the carbon footprint of AI-driven software testing. Simple, structured prompts can often outperform complex reasoning chains in terms of sustainability without sacrificing test quality.
Diminishing Returns: The study highlights the law of diminishing returns in automated testing; increasing reasoning complexity (e.g., via Self-Consistency) often leads to disproportionate energy costs with negligible gains in test coverage.
Guidance for Practitioners: For organizations aiming for "Green AI" in testing, the results suggest prioritizing Few-shot or Least-to-Most strategies over computationally expensive reasoning frameworks like SC_CoT, unless specific high-complexity reasoning is strictly required.
Future Directions: The work sets a precedent for integrating sustainability metrics directly into the evaluation of AI-assisted software engineering tools, moving beyond accuracy-only benchmarks.

In conclusion, the paper argues that sustainability-aware testing requires a shift in how prompts are designed. By selecting the right prompt strategy, developers can achieve high-quality automated test generation while minimizing the environmental impact, making "Green Testing" a practical reality.