SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inference

1. Problem Statement

Large Language Models (LLMs) have become central to modern AI, but their inference phase consumes the majority (up to 90%) of a model's lifecycle energy. Current approaches to estimating this energy footprint rely on oversimplified assumptions:

• Linear Assumption: Most existing models and commercial pricing strategies assume energy consumption is a simple linear function of input ($n_{in}$) and output ($n_{out}$) token counts.
• Missing Complexity: These models fail to account for the autoregressive nature of Transformers, where the computational and memory costs differ fundamentally between the prefill phase (processing the full input in parallel) and the decode phase (generating tokens sequentially).
• Consequence: This leads to inaccurate energy predictions and missed opportunities for optimization, as the relationship between sequence lengths and energy efficiency is actually non-linear.

2. Methodology

The authors propose SweetSpot, an analytical model derived from the computational (FLOPs) and memory-access complexity of the Transformer architecture.

A. Theoretical Formulation

The model decomposes inference energy into two distinct phases:

1. Prefill Phase: The entire input sequence is processed in parallel. The cost is dominated by quadratic complexity ($O(n_{in}^2)$) due to the attention mechanism (calculating $QK^T$).
2. Decode Phase: Tokens are generated autoregressively. The cost per token is linear with respect to the sequence length ($O(n_{in} + t)$) because it involves reusing cached Key-Value (KV) states, but it is constrained by memory bandwidth.

The authors derive two versions of the model:

• SweetSpot (FLOPs-only): Based purely on floating-point operations.
• SweetSpot (Complete): Extends the FLOPs model to include memory-access complexity ($M$), acknowledging that modern GPUs are often memory-bound rather than compute-bound, especially during decoding.

The final energy-per-token ($E_{tok}$) equation (Equation 5) is:
$$E_{tok}(n_{in}, n_{out}) = \theta_0 + \frac{\theta_1 n_{in}^2}{n_{out}} + \theta_2 n_{in} + \frac{\theta_3 n_{in}}{n_{out}} + \theta_4 n_{out} + \frac{\theta_5}{n_{out}}$$

• The term $\frac{n_{in}^2}{n_{out}}$ represents the amortization of the quadratic prefill cost over the generated tokens.
• The term $n_{out}$ represents the linear accumulation of decoding costs.

B. Experimental Setup

• Hardware: NVIDIA H100 GPUs (4x) connected via NVLink.
• Inference Engine: TensorRT-LLM (chosen for its state-of-the-art energy efficiency and hardware-aware optimizations like in-flight batching).
• Models: A diverse set of 12 LLMs ranging from 1B to 9B parameters (including OPT, LLaMA, Gemma, Falcon, Qwen2, and Granite), covering different attention mechanisms (MHA, GQA).
• Workload: Input and output lengths varied from 64 to 4096 tokens. Experiments were run with 1,000 concurrent requests to ensure saturation and minimize measurement noise.
• Metrics: Total Energy ($E_{tot}$) and Energy-per-Token ($E_{tok}$), measured via NVIDIA NVML at 500ms intervals.

3. Key Contributions

1. The SweetSpot Model: The first analytical model to accurately characterize LLM inference energy as a non-linear function of input and output lengths, explicitly modeling the interplay between quadratic prefill costs and linear decode costs.
2. Discovery of "Sweet Spots": The paper demonstrates that energy efficiency is not monotonic. There exists a specific configuration of input and output lengths where efficiency peaks.
   • Peak Efficiency: Occurs with short-to-moderate inputs and medium-length outputs.
   • Inefficiency Zones: Efficiency drops sharply for very long inputs (due to high prefill cost) or very short outputs (due to poor amortization of prefill cost).
3. Validation & Accuracy: The model was validated against extensive empirical data, achieving a Mean Absolute Percentage Error (MAPE) of 1.79%, significantly outperforming existing linear baselines (which had MAPEs >30%).
4. Architectural Insights: The study quantifies how attention mechanisms (MHA vs. GQA) impact energy. Models using Grouped-Query Attention (GQA) are consistently more energy-efficient than those using Multi-Head Attention (MHA) due to reduced KV cache memory traffic.

4. Key Results

• Accuracy: SweetSpot reduced the prediction error from ~31% (previous best baseline) to 1.79% (complete model) and 2.27% (FLOPs-only).
• Efficiency Gains: Aligning sequence lengths with the predicted "sweet spot" can reduce energy usage by up to 33.41× compared to the worst-case configuration (long input + short output).
• Non-Linearity: The study confirmed that efficiency peaks when the input cost is sufficiently amortized by the output length, but before the linear decoding overhead becomes dominant.
  • Example: For a fixed input of 64 tokens, efficiency peaks around 128–256 output tokens.
• Batching Impact: The study found that while batch size affects absolute throughput, the relative efficiency trends (the shape of the curve) remain consistent. However, under-saturating the batch size significantly reduces overall efficiency.

5. Significance and Implications

• Production Optimization: The model provides a theoretical basis for adaptive generation strategies. Systems can dynamically truncate overly long inputs or adjust generation limits to operate within the "sweet spot," drastically reducing energy costs without sacrificing utility.
• Sustainable AI: By moving from empirical guesswork to analytical prediction, this work enables the design of more sustainable AI systems, crucial as LLM deployment scales globally.
• Hardware-Software Co-design: The results highlight that software-level optimizations (like batching and truncation) must be guided by an understanding of the underlying hardware constraints (memory bandwidth vs. compute).
• Future Directions: The authors note limitations regarding model scale (focusing on <10B parameters) and hardware diversity, suggesting future work to generalize the model to larger LLMs and heterogeneous hardware clusters.

In summary, SweetSpot transforms energy efficiency from an opaque, empirical metric into a predictable, designable parameter, offering a roadmap for reducing the carbon footprint of large-scale AI inference.