Technical Summary: Towards Green AI: Decoding the Energy of LLM Inference in Software Development

Problem Statement

The integration of AI-assisted tools into software development workflows relies heavily on Large Language Models (LLMs). While these tools enhance productivity, their inference processes introduce substantial computational and energy costs. Existing research often treats LLM inference as a uniform process, overlooking the distinct resource demands of its two constituent phases: prefill (processing the input prompt to build internal key-value caches) and decoding (autoregressively generating output tokens). Furthermore, the energy implications of specific model behaviors, such as generating excessive or unnecessary content ("babbling"), remain underexplored. As inference is typically time-sensitive and cannot be easily scheduled for low-carbon energy windows (unlike training), optimizing its energy footprint is critical for sustainable software development.

Methodology

The study conducts a phase-level empirical analysis of LLM inference energy consumption using a granular, per-token measurement approach.

• Models: Ten decoder-only transformer models were evaluated, categorized into two parameter groups: 6B–7B (e.g., CodeLlama-7B, Qwen2.5-Coder-7B, Deepseek-Coder-6.7B) and 3B–4B (e.g., Phi3.5-4B, Qwen3-4B).
• Workloads & Benchmarks: Experiments utilized two primary benchmarks:
  • HumanEval: For code generation, tested under three settings: Zero-shot (minimal input), Few-shot (2-shot, increased input), and Zero-shot Chain-of-Thought (increased output length).
  • LongBench: For code understanding, tested under Code Understanding (long input, short output) and Code Understanding with Explanation (long input, longer output).
• Measurement Setup: Experiments were conducted on a single NVIDIA A10 GPU (24GB) in an isolated environment. Energy consumption was sampled at 10Hz using pyNVML.
• Phase Separation: Following the methodology of Babakol et al., the study aligned energy timestamps with token generation events. The prefill phase was defined as the energy consumed up to the generation of the first token, while the decoding phase encompassed the energy for all subsequent tokens.
• Metrics: Key dependent variables included total GPU energy (J), prefill energy (J), energy per token (J), and energy per token specifically during decoding. Accuracy was measured via pass@1 for code generation and correct answer proportion for code understanding.

Key Contributions

1. Phase-Level Energy Analysis: The study reveals that prefill and decoding phases exhibit distinct energy consumption characteristics, challenging the view of inference as a monolithic process.
2. Empirical Quantification: It provides a comprehensive dataset of energy patterns across 10 diverse transformer models (Llama, Phi, Gemma, Qwen families) under varying software development workloads.
3. Prefill-Decoding Interaction: The research demonstrates that the cost of the prefill phase (driven by input size) directly influences the baseline energy cost per token during the decoding phase.
4. Babbling Identification and Mitigation: The study identifies "babbling behavior" in specific models (generating unnecessary text beyond task completion) and implements a babbling suppression algorithm that halts generation once test cases pass, significantly reducing energy waste.

Key Results

1. Phase Dominance and Input/Output Sensitivity

• Decoding Dominance: In most scenarios, the decoding phase dominates total energy consumption. However, in long-context tasks (e.g., Code Understanding with 4,000–8,000 input tokens and short outputs), the prefill phase contribution rises significantly (up to 84.4%).
• Input Size Impact: Increasing input size increases prefill energy linearly. Crucially, larger inputs also raise the initial energy cost per token in the decoding phase. For example, increasing input size for CodeLlama-7B raised the initial decoding token cost by 48.8%.
• Model Variability: The sensitivity to input size varies by model. While some models (e.g., CodeLlama-7B, Deepseek-6.7B) show a sharp increase in decoding energy with larger inputs, others (e.g., Phi4-4B) remain relatively stable (1.3% increase).

2. Token Generation Cost Dynamics

• Progressive Cost Increase: The study observes that each newly generated token in the decoding phase tends to be more energy-intensive than the previous one. This is attributed to the growing size of the key-value cache and memory management overheads.
• Magnitude: In long outputs (e.g., 2,000 tokens), the final token can cost up to 20% more energy than the first token.

3. Babbling Behavior and Suppression

• Observation: Three out of ten models (CodeLlama-7B, Deepseek-6.7B, Qwen3-4B) exhibited "babbling," generating excessive tokens (whitespace, test cases, examples) after the solution was complete.
• Intervention: A babbling suppression algorithm was implemented, which checks for syntactic correctness and test case passage at the end of each line. Generation halts immediately upon passing all tests.
• Efficiency Gains:
  • Output Reduction: Suppression reduced output length by 44% to 93% depending on the token limit and model.
  • Energy Savings: Total inference energy was reduced by 44% to 89%. For Deepseek-6.7B with a 1,000-token limit, energy consumption dropped by 89%.
  • Accuracy: These savings were achieved without negatively impacting generation accuracy (variations were within ±2% to ±4%).
  • Overhead: The overhead of intermediate test checks increased the "energy per token" metric in some cases (due to fewer tokens being generated while the check cost remained), but the total energy per inference decreased significantly.

Significance and Claims

The paper asserts that reducing the energy footprint of LLM inference in software development requires a dual approach:

1. Architectural and Implementation Awareness: Developers must recognize that prefill costs influence decoding efficiency. Optimizations such as efficient Key-Value (KV) cache storage and retrieval are essential to mitigate the amplification of per-token costs caused by large inputs.
2. User and Workflow Optimization: Users should be mindful of prompt lengths, as they directly impact both prefill and decoding costs.
3. Behavioral Mitigation: The most immediate and impactful strategy identified is the suppression of babbling. By preventing the generation of unnecessary tokens that are typically discarded during post-processing, the industry can achieve substantial energy savings (up to 89%) without compromising the functional correctness of the generated code.

The study concludes that while architectural efficiency is important, understanding the specific phase-level interactions and model behaviors (like babbling) offers a practical pathway to "Green AI" in software engineering.