🤖 AI

Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling

This paper presents AFlex, a framework that reduces energy consumption in LLM serving by up to 49% through disaggregated Attention-FFN execution, joint resource and frequency optimization, and an interleaved pipeline that adapts to operator-level frequency sensitivities while meeting strict latency SLOs.

Original authors: Cunchen Hu, Liangliang Xu, Tian Liu, Min Lyu, Yongkun Li, Sa Wang, Shuo Quan, Yanan Yang, Wenda Tang, Yiduo Wang, Fu Yu, Jie Wu

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cunchen Hu, Liangliang Xu, Tian Liu, Min Lyu, Yongkun Li, Sa Wang, Shuo Quan, Yanan Yang, Wenda Tang, Yiduo Wang, Fu Yu, Jie Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, super-smart robot brain in a giant data center. This brain, known as a Large Language Model (LLM), is the engine behind chatbots, coding assistants, and creative writers. But there's a catch: these brains are incredibly hungry. They eat up huge amounts of electricity, often running their processors at full speed, like a race car engine revving at maximum RPM even when the car is stuck in traffic. This is a problem because it costs a fortune in energy and creates a lot of heat.

To make these robots run faster, scientists have figured out how to split the work. Think of the robot's thinking process as having two main steps: first, it reads and understands your question (the "prefill" phase), and second, it writes out the answer word by word (the "decode" phase). Recently, engineers realized it's better to put these two steps in different rooms with different teams of workers, rather than having one team do both. This is called "disaggregation." It helps the robot answer faster, but it doesn't necessarily make it use less electricity. In fact, if you just turn down the speed of the workers to save power, the robot might get too slow and miss its deadlines. The big question is: how do we slow things down to save energy without making the robot stutter or fail?

This paper introduces a clever new system called AFlex that solves this puzzle by treating the robot's brain like a team of specialists with different energy needs. The researchers discovered that the two main parts of the robot's brain—the part that pays attention to your words (Attention) and the part that processes the logic (Feed-Forward Network, or FFN)—are not the same. They are like two different athletes: one is a sprinter who needs a high-speed boost to run fast, while the other is a marathon runner who gets tired if you push them too hard but can maintain a steady, slower pace efficiently.

Previously, systems treated the whole brain as one unit, forcing both the sprinter and the marathon runner to run at the exact same speed. If you sped them up to help the sprinter, the marathon runner wasted energy. If you slowed them down to save the marathon runner, the sprinter got too slow. The authors of this paper found that these two parts actually prefer different speeds depending on what the robot is doing, how long the question is, and how many people are asking questions at once.

To fix this, AFlex acts like a smart manager who can give the sprinter and the marathon runner their own separate tracks and their own personal speed settings. It uses a "Global Scheduler" to plan the big picture, deciding how many workers to assign to each task, and a "Local Controller" to tweak their speeds in real-time, second by second. The system also uses a special pipeline technique where the workers pass notes to each other while they are still working, so no one sits around waiting. This "interleaved" approach keeps the assembly line moving smoothly without any gaps.

The researchers tested AFlex on powerful computer chips (NVIDIA A800 GPUs) using real-world data from coding and conversation tasks. They found that by letting the Attention and FFN parts run at their own unique, energy-saving speeds, the system could cut the energy used per word generated by up to 49% compared to the best existing methods that split the work, and by 48% compared to systems that just tried to slow down the whole chip. Crucially, the robot still answered just as fast as before, meeting all its speed promises. The paper suggests that this approach of giving different parts of the AI brain different speeds is a major step toward making artificial intelligence much more energy-efficient without sacrificing performance.

Technical Summary: AFlex – Energy-Efficient LLM Serving via Disaggregated Attention–FFN and Flexible Frequency Scaling

1. Problem Statement

Large Language Model (LLM) serving in modern datacenters faces a critical trade-off between meeting stringent Service-Level Objectives (SLOs)—specifically Time to First Token (TTFT) and Time Per Output Token (TPOT)—and minimizing energy consumption. While Dynamic Voltage and Frequency Scaling (DVFS) offers a mechanism to trade performance for power savings, existing energy-management approaches suffer from two primary limitations:

Coarse-Grained Frequency Control: Current systems apply frequency scaling at the request level, inference phase level (prefill vs. decode), or instance level. They fail to account for operator-level heterogeneity. Specifically, the two dominant operators in LLM inference, Attention and Feed-Forward Networks (FFN), exhibit distinct compute and memory characteristics. Attention is often memory-bound (sensitive to KV cache access), while FFN is typically compute-bound. Consequently, their optimal frequencies for energy efficiency differ significantly and vary dynamically based on workload, batch size, and tensor parallelism (TP) configurations.
Inefficiency of Disaggregated Serving: While Phase-Disaggregated (P/D) serving separates prefill and decode to reduce interference, it still treats each phase as a single frequency domain. Furthermore, Operator-Disaggregated (A/F) serving, which separates Attention and FFN execution, introduces challenges:
- Search Space Complexity: Jointly optimizing resource provisioning (TP degrees, instance counts) and independent frequencies for four operator pools (Prefill-Attention, Prefill-FFN, Decode-Attention, Decode-FFN) creates a massive combinatorial search space.
- Overhead: A/F disaggregation requires hidden-state transfers between operators. Dynamic reconfiguration of resources and frequent frequency switching introduce communication and transition overheads that can negate energy savings if not carefully managed.

2. Methodology: The AFlex Framework

To address these challenges, the authors present AFlex, a framework that jointly optimizes resource provisioning and GPU frequency scaling for disaggregated A/F serving. AFlex is built upon four design principles: operator-awareness, joint SLO-aware optimization, separation of resource planning from runtime control, and overhead-aware reconfiguration.

A. Architecture

AFlex employs a two-level control plane and a pipelined data plane:

Global Scheduler (Control Plane):
- Profiling & Prediction: Uses offline profiling to train lightweight regression models (Gradient-Boosted Decision Trees) that predict operator latency and energy consumption based on GPU frequency, batch size, sequence length, and TP degree.
- Resource Orchestrator: Periodically solves an Integer Linear Program (ILP) to determine the optimal number of A/F pairs, TP degrees, and baseline frequencies. The ILP minimizes total energy subject to GPU memory constraints, throughput requirements, and per-layer TTFT/TPOT SLOs.
- Runtime Reconfiguration Engine: Applies new configurations incrementally. It reuses resident weight shards and fetches missing shards asynchronously to minimize disruption, distinguishing between heavy TP reconfiguration and lightweight frequency adjustments.
Local DVFS Controller (Control Plane):
- Refines the global plan at millisecond granularity.
- Prefill: Selects a fixed PA/PF frequency pair before batch launch based on the remaining TTFT budget.
- Decode: Uses a window-based algorithm to amortize frequency switching costs. It adjusts DA/DF frequencies only when the batch size changes significantly or the TPOT SLO is at risk, avoiding prohibitively expensive per-iteration switches.
AFD Data Plane:
- Interleaved Pipeline: Launches the next layer's Attention operator as soon as the current layer's FFN completes, overlapping hidden-state transfers with computation.
- Dynamic Microbatching & Adaptive Batching: Dynamically selects microbatch depth ( $M$ ) and request batch sizes to balance A/F execution. For example, under low load, it may use $M=1$ to avoid splitting small batches; under high load, it increases $M$ to maximize overlap and reduce pipeline bubbles.
- Placement-Aware Transfer: Optimizes hidden-state transfers across heterogeneous TP groups using NIC-affine grouping to parallelize transfers without requiring full activation replication.

B. Key Technical Insights

The design is grounded in empirical observations:

Heterogeneous Energy Optima: Attention and FFN operators reach energy minima at different frequencies (average gap of 429 MHz). A single frequency is suboptimal for both.
Dynamic Sensitivity: Operator sensitivity to frequency shifts with batch size and TP degree, necessitating runtime adaptation rather than fixed configurations.
Imbalance Costs: Homogeneous provisioning leads to significant pipeline bubbles (19.7–75.7% of step time), causing substantial energy waste.

3. Key Contributions

The paper claims the following contributions:

Characterization of Operator-Level Heterogeneity: A systematic analysis revealing that A/F operators have distinct frequency sensitivities and energy optima that vary across phases, workloads, and system configurations.
Two-Level Control Plane: A novel architecture combining global A/F resource provisioning (via ILP) with local per-operator DVFS adaptation to minimize energy under strict constraints.
Interleaved A/F Data Plane: A pipeline design that overlaps communication with computation and dynamically adjusts microbatch depth and request batching to eliminate pipeline bubbles.
Comprehensive Evaluation: Implementation and evaluation on a multi-node GPU cluster using production traces, demonstrating substantial energy reductions compared to state-of-the-art systems.

4. Experimental Results

The authors implemented AFlex atop SGLang and evaluated it on NVIDIA A800 GPUs using Qwen3-32B (dense) and Mixtral-8×7B (MoE) models. Workloads included production traces (Conversation, Coding) and controlled synthetic workloads (QA, RAG, Chatbot, Summary).

Performance Metrics:

Energy Efficiency: AFlex reduces energy per token by up to 49% compared to state-of-the-art disaggregated serving (DistServe) and 48% compared to frequency-scaling systems (DynamoLLM).
SLO Compliance: AFlex consistently satisfies P90 TTFT and TPOT SLOs across all tested loads, whereas some baselines either violate SLOs or require higher frequencies to meet them, increasing energy consumption.
Scalability: The energy-saving advantage persists when scaling from 1 to 4 nodes. AFlex maintains at least a 20.2% improvement over the best baseline across cluster sizes.
MoE Support: AFlex effectively handles Mixture-of-Experts models, addressing intra-phase A/F imbalances that P/D disaggregation alone cannot resolve, achieving up to 41.4% energy reduction over baselines.
Overhead: The global ILP solver adds negligible overhead (<0.2% of the 5-minute scheduling window). Incremental reconfiguration reduces TP transition times by 52–72% compared to base methods.

5. Significance and Claims

The paper positions AFlex as the first LLM serving framework to combine disaggregated Attention-FFN execution with independent, flexible per-operator frequency scaling.

Complementarity: The authors argue that fine-grained disaggregation (AFD) and fine-grained DVFS are complementary. While AFD exposes resource heterogeneity, DVFS exploits the resulting "slack" to lower frequencies on less sensitive operators (e.g., lowering FFN frequency when Attention is the bottleneck).
Practicality: By decoupling slow resource reconfiguration from the inference critical path and amortizing switching costs via window-based control, AFlex demonstrates that fine-grained energy optimization is feasible without sacrificing latency SLOs.
Impact: The results suggest that moving beyond phase-level or instance-level control to operator-level control is essential for the next generation of energy-efficient LLM serving, particularly as workloads become more diverse and hardware constraints tighten.

The authors conclude that AFlex provides a robust, scalable solution for reducing the carbon footprint of LLM inference while maintaining the responsiveness required for interactive applications.