🤖 AI

The Illusion of Power Capping in LLM Decode: A Phase-Aware Energy Characterisation Across Attention Architectures

This paper reveals that power capping is ineffective for LLM autoregressive decode due to memory-bound bottlenecks leaving GPU power headroom unused, and demonstrates that SM clock locking is a superior strategy that recovers up to 32% of decode energy while minimizing throughput loss across diverse attention architectures.

Original authors: Bole Ma, Ayesha Afzal, Jan Eitzinger, Gerhard Wellein

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bole Ma, Ayesha Afzal, Jan Eitzinger, Gerhard Wellein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Fake" Energy Switch

Imagine you run a massive library (a data center) where robots (GPUs) are reading books and writing new pages (generating text for AI).

The library managers have a standard rule to save electricity: "If a robot starts using too much power, turn down its speed until it stays under a specific limit." This is called Power Capping. It works great for heavy lifting, like moving heavy boxes (training models or processing huge batches of data at once).

However, this paper discovered a trick. When the robots are just reading one sentence at a time and writing the next word (a process called "decode"), the power rule doesn't work at all. The robots are so focused on fetching the next page from the bookshelf that they aren't even working hard enough to trigger the power limit. The "power cap" is like a speed limit sign on a road where no one is driving fast enough to get a ticket. It's an illusion.

The Real Problem: The "Bookshelf" Bottleneck

Why isn't the robot working hard?

The Robot (GPU): It has a super-fast brain (Compute) and a super-fast memory (HBM).
The Task: To write the next word, the robot has to run to the bookshelf, grab a heavy book, read a tiny bit, and run back.
The Bottleneck: The robot isn't limited by how fast its brain can think; it's limited by how fast it can run to the bookshelf.

Because the robot is constantly running back and forth to the bookshelf, its brain sits idle most of the time. Even if you tell the robot to "slow down" to save energy, it can't go much slower because it's already waiting on the bookshelf. If you try to limit its total power usage, the robot just ignores you because it's already using very little power (only about 200–300 Watts on a machine rated for 700 Watts).

The Solution: The "Manual Gear Shift"

Since the automatic power switch (Power Capping) is useless, the authors found a better way: SM Clock Locking.

Think of this as manually shifting the robot's gear.

The Old Way (Power Capping): "Don't use more than 280 Watts!" (The robot says, "I'm already only using 200 Watts, so I do whatever I want.")
The New Way (Clock Locking): "Hey robot, since you're just waiting on the bookshelf, let's put you in Low Gear."

By manually telling the robot to run its brain at a slower, steady speed (locking the clock), the robot saves a massive amount of energy (up to 32%) without actually slowing down the writing speed. Why? Because the robot was already waiting on the bookshelf, so slowing the brain down didn't make it wait any longer. It just wasted less electricity while it waited.

The "New Robots" (Different Architectures)

The paper tested four different types of robot designs (GQA, MLA, GDN, Mamba2). They all behave similarly during the "writing" phase: they are all stuck waiting on the bookshelf.

However, they have different "startup costs":

The Heavy Starters (GDN, Mamba2): These robots take a long time and use a lot of energy to get ready (the "prefill" phase). But once they start writing, they are incredibly efficient. If you ask them to write a long story, they eventually become the cheapest option because they are so fast at the "writing" part.
The Compressed Starters (MLA): These robots carry a smaller backpack (compressed memory). They take a little extra time to unpack their backpack at the start, but once they are writing, they are very efficient.
The Standard Starter (GQA): The reliable, standard robot. It doesn't have a heavy startup cost, but it's not as efficient at writing long stories as the new models.

The Lesson: If you are writing a short note, the standard robot is fine. If you are writing a whole novel, the "Heavy Starters" or "Compressed Starters" save you the most money in the long run, even though they cost more to start up.

The "Fake" Speed Limit

There was one more confusing thing the authors found. When they tried to manually set the robot's speed to its maximum (1980 MHz), the robot's internal software (firmware) secretly capped it at a lower speed (1830 MHz).

It's like telling a car to drive at 120 mph, but the car's computer secretly limits it to 110 mph.
Worse, the authors found that driving at 110 mph vs. 95 mph made zero difference in how fast the robot wrote the story. The robot was still waiting on the bookshelf. So, the extra speed was just burning extra electricity for no reason.

Summary

Power Capping is a Myth for AI Writing: It doesn't save energy when AI is generating text because the AI isn't using enough power to trigger the limit.
Manual Speed Control Wins: Instead of setting a power limit, you should manually slow down the robot's brain speed. This saves up to 32% of energy with almost no loss in speed.
Different Robots for Different Jobs: New AI designs are great for long conversations because they are super efficient at writing, even if they are a bit slow to start.
The Bookshelf is King: The speed of the AI is limited by how fast it can fetch data from memory, not by how fast its brain can think.

The Bottom Line: Data centers are using the wrong tool to save money. They should stop relying on automatic power limits and start manually tuning the speed of their AI robots to match the reality of the task.

Technical Summary: The Illusion of Power Capping in LLM Decode

Problem Statement

Data centers increasingly rely on GPU power capping as the primary mechanism for managing the energy footprint of Large Language Model (LLM) inference. The prevailing operational model assumes that setting a board-level watt ceiling forces the GPU driver to throttle clocks, trading throughput for guaranteed power savings. While this model holds for compute-bound workloads (e.g., training or dense HPC kernels) that push GPUs near their Thermal Design Power (TDP), this paper argues that the model is structurally flawed for the phase dominating production LLM serving: autoregressive decode.

The core problem is that power capping is inert during decode because the workload is memory-bound, not compute-bound. Consequently, the GPU never draws enough power to trigger the cap, rendering the energy lever ineffective. Furthermore, firmware-level behaviors, such as silent clock throttling when using nvidia-smi --lock-gpu-clocks, create a "double disguise" that obscures the true relationship between frequency, power, and throughput.

Methodology

The authors conducted a systematic energy characterization across four distinct attention paradigms on a single NVIDIA H200 SXM GPU (700 W TDP, 4.8 TB/s HBM3e bandwidth):

GQA (Grouped-Query Attention, e.g., Qwen3-4B)
MLA (Multi-Head Latent Attention, e.g., Minitron-4B via TransMLA)
Gated DeltaNet (Linear recurrence, e.g., Qwen3.5-4B)
Mamba2 (State Space Models, e.g., Nemotron-Nano)

Experimental Setup:

Hardware/Software: Models were served via vLLM in BF16 precision without custom kernels to reflect standard practitioner deployments.
Controls: A controlled ablation was performed using TransMLA to convert GQA-ctrl (Minitron-4B) to MLA, ensuring that energy differences were attributable solely to the attention mechanism rather than model weights or vocabulary.
Metrics: Energy per token (mJ/tok) was measured for both prefill and decode phases.
Variables: The study swept five SM clock levels (390–1980 MHz), five power cap levels (280–700 W), batch sizes (1–32), and sequence lengths (1K–64K tokens).
Analysis: Roofline modeling was used to determine arithmetic intensity, and Pareto frontiers were constructed to compare throughput vs. energy efficiency under different control mechanisms.

Key Findings

1. The Power Capping Illusion

Contrary to the assumption that power caps actively throttle GPUs during LLM serving, the study found that during decode, the GPU draws only 137–300 W on a 700 W TDP card.

Inert Caps: Because the workload is memory-bound (saturating HBM bandwidth rather than compute), the GPU never reaches even the lowest configured cap (280 W). The driver ignores the cap and maintains a default clock of ≈1830 MHz.
Firmware Artifacts: The nvidia-smi --lock-gpu-clocks command silently clamps any requested lock ≥1830 MHz to 1830 MHz, distinct from the free-running boost of 1980 MHz. This creates a 240 MHz gap where higher clocks yield zero throughput gain but increase power by 7–13%, further confusing energy measurements.

2. SM Clock Locking as the Correct Lever

Since decode is memory-bound, the critical path is the memory subsystem, not the compute frequency.

Pareto Dominance: Static SM clock locking outperforms power capping at every matched operating point. By lowering the SM clock (e.g., to 780 MHz), the system recovers up to 32% of decode energy with less than 1% throughput loss.
Architecture Classes: The authors identified three behavioral classes regarding Dynamic Voltage and Frequency Scaling (DVFS):
- Batch-Invariant (GQA, GQA-ctrl): Remain memory-bound even at large batch sizes (BS=32); a single low clock is optimal.
- Batch-Sensitive (MLA, Mamba2): Contain sufficient per-step compute (KV decompression or SSM scans) that large batches push them toward compute-bound regimes, requiring higher clocks as batch size increases.
- Compute-Light (GDN): Dominated by elementwise operations; tolerates the most aggressive underclocking unconditionally.

3. Cross-Architecture Energy Landscape

Novel architectures (MLA, GDN, Mamba2) share a common energy pattern: a heavy prefill cost that is eventually recouped by efficient decode.

Prefill Penalty: GDN and Mamba2 exhibit an order-of-magnitude higher prefill energy per token due to sequential state recurrence and unfused execution in vLLM. MLA suffers a smaller but persistent penalty due to KV decompression overhead and non-power-of-2 tensor core utilization.
Decode Payoff: In the decode phase, these architectures become highly efficient.
- Mamba2: Achieves constant per-step latency ( $O(1)$ ), crossing below GQA in total request energy after roughly 1,000 output tokens at production batch sizes.
- MLA: Crosses below GQA at a batch-size-dependent context threshold (e.g., 4K tokens at BS=32). At BS=32 and 65K context, MLA uses less than half the decode energy of GQA.
Total Request Energy: At production batch sizes (BS=32), the efficiency of the decode phase allows novel architectures to halve the total request energy relative to standard GQA, despite their prefill penalties.

Significance and Claims

The paper claims that the industry's reliance on power capping for LLM inference is structurally ineffective for the dominant decode phase. The authors assert that:

Power capping is a "double disguise": It fails to trigger because the GPU is under-utilized in terms of power (due to memory bottlenecks), and firmware clock clamping introduces unrelated deviations that corrupt throughput measurements.
SM clock locking is the universal solution: It directly targets the critical-path resource (frequency) for memory-bound workloads, offering a Pareto-dominant trade-off between energy and throughput that power capping cannot achieve.
Architectural implications: The energy benefits of novel attention mechanisms (MLA, Mamba2) are strictly decode-specific. This supports the adoption of disaggregated serving architectures, where prefill and decode are routed to different GPU pools optimized for their respective phases.

The authors conclude that for data centers managing LLM inference, replacing power capping with static SM clock locking for decode pools is a straightforward, high-impact intervention that can yield megawatts of continuous power savings at scale, a result unattainable through current power-capping strategies.