🤖 machine learning

Theoretically Optimal Attention/FFN Ratios in Disaggregated LLM Serving

This paper presents an analytical framework and a closed-form provisioning rule to determine the theoretically optimal Attention-to-FFN resource ratio in disaggregated LLM serving, accounting for stochastic workload dynamics and synchronization overheads to minimize device idle time and step-level blocking.

Original authors: Chendong Song, Meixuan Wang, Hang Zhou, Hong Liang, Yuan Lyu, Zixi Chen, Yuwei Fan, Zijie Zhou

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chendong Song, Meixuan Wang, Hang Zhou, Hong Liang, Yuan Lyu, Zixi Chen, Yuwei Fan, Zijie Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-speed factory that produces "thoughts" (tokens) for a giant AI brain. This factory has two main assembly lines working together to finish each thought:

The Memory Line (Attention): This team is like a librarian. They have to run back and forth to a giant, growing bookshelf (the KV cache) to find specific pages for every request. As the bookshelf gets bigger, this team gets slower and slower because they have to carry heavier loads. They are memory-bound.
The Calculation Line (FFN): This team is like a super-fast calculator. They don't need to look at the bookshelf; they just crunch numbers based on what the librarians hand them. They are compute-bound and can work incredibly fast if they have enough work to do.

The Problem: The "Mismatched Dance"

In the old days, these two teams were stuck in the same room. If the librarians were slow, the calculators had to sit idle, waiting for them. If the calculators were fast, the librarians were the bottleneck.

To fix this, engineers invented a new layout called AFD (Attention-FFN Disaggregation). They moved the librarians and calculators into separate rooms. Now, you can have many librarian teams feeding into one giant calculator room.

But here's the catch: How many librarian teams do you need for one calculator room?

Too few librarians? The calculator sits idle, starving for data.
Too many librarians? The calculator gets overwhelmed, and the librarians have to stand around waiting for the calculator to catch up.

Finding the perfect ratio (let's call it r) is like trying to find the perfect number of waiters for a single chef. If you guess wrong, the whole factory slows down.

The Paper's Solution: A "Crystal Ball" for Factory Managers

The authors of this paper realized that guessing the ratio is hard because the work is random.

Some customers have short questions; others have long stories.
Some requests finish quickly; others take a long time.
The "bookshelf" (memory) grows differently for every request.

Because of this randomness, you can't just use a simple math formula based on averages. You need a way to predict the chaos.

Their "Secret Sauce" is a new mathematical framework that does three things:

It measures the "Average Chaos": They developed a way to look at past request logs (traces) and calculate a single number (called θ) that represents the true average workload, accounting for the fact that longer requests are more likely to be seen at any random moment.
It accounts for the "Slowest Runner": In this factory, all librarian teams must finish their work before the calculator can start. If one team gets stuck with a huge book, the whole line waits. The authors created a formula to predict how much extra time is lost because of these "stragglers" (the slowest workers).
It gives a "Golden Ratio" Recipe: Using these two insights, they derived a simple, closed-form rule. You plug in your hardware specs and your request logs, and the formula tells you the exact number of librarian teams you need for your calculator room to run at maximum speed.

The Results: "It Works!"

The team built a digital simulator (a virtual factory) to test their theory.

They tried different numbers of librarian teams (from 1 to 32).
They compared their "Golden Ratio" prediction against the actual best performance found by the simulator.
The Verdict: Their prediction was incredibly accurate, matching the real-world simulation within 10%.

They also found that as you add more librarian teams, the "waiting time" caused by the slowest team does increase, but their formula accounts for this, ensuring you don't add too many teams and waste money.

The Takeaway

This paper provides a scientific rulebook for building these split-AI factories. Instead of guessing or trial-and-error, system designers can now use this math to figure out exactly how to balance their memory and computing resources, ensuring the AI runs as fast and efficiently as possible, even when the workload is unpredictable.

Technical Summary: Analytical Provisioning for Attention–FFN Disaggregated LLM Serving under Stochastic Workloads

Problem Statement

Large Language Model (LLM) serving faces significant efficiency challenges as model sizes scale. While Attention–FFN Disaggregation (AFD) has emerged as a promising architecture to decouple memory-bound Attention layers (stateful, KV-cache dominated) from compute-bound FFN layers (stateless), determining the optimal resource provisioning ratio remains an open problem.

In an AFD topology (denoted as $r$ A–1F, where $r$ Attention instances feed one FFN instance), the system performance is highly sensitive to the ratio $r$ .

If $r$ is too small, the FFN server starves for input, leading to idle compute resources.
If $r$ is too large, Attention instances block waiting for the shared FFN server, causing step-level blocking and pipeline bubbles.

Existing systems typically determine $r$ through empirical search or naive deterministic approximations that ignore workload stochasticity. The core difficulty lies in the dynamic nature of the Attention workload: unlike FFN computation which depends only on batch size, Attention latency evolves continuously as KV caches grow and requests are replenished with random prompt and decode lengths. Furthermore, the synchronized execution of multiple Attention workers introduces a "straggler" effect governed by the slowest worker, creating a synchronization barrier that is difficult to model analytically.

Methodology

The authors develop a rigorous probabilistic framework to derive a closed-form provisioning rule for the optimal A/F ratio ( $r^*$ ) under stochastic workloads. The methodology addresses two primary sources of randomness:

Per-Slot Workload Evolution: The authors model the per-slot token load using a discrete-time renewal-reward analysis. They identify a single workload statistic, $\theta$ (the expected stationary token load per slot), which governs provisioning. This statistic accounts for the "length-biasing" effect where longer requests are sampled more frequently at random decode steps. Crucially, $\theta$ and its variance $\nu^2$ can be estimated nonparametrically from request traces without assuming specific distributional families.
Cross-Worker Synchronization: To address the barrier effect where the system waits for the slowest of $r$ workers, the authors apply a Gaussian order-statistic correction. They model the total load on the $r$ workers as a sum of i.i.d. random variables and approximate the maximum load using the expected value of the maximum of $r$ standard normal variables ( $\kappa_r$ ).

The analysis yields two provisioning rules:

Mean-Field Rule ( $r^*_{mf}$ ): A closed-form solution derived by ignoring the synchronization barrier (assuming perfect load balancing). This rule decomposes into three regimes: Attention-bottleneck, communication-bottleneck, and FFN-bottleneck.
Barrier-Aware Refinement ( $r^*_G$ ): A refined optimization that incorporates the synchronization overhead term ( $\sqrt{B}\nu\kappa_r$ ) into the cycle time approximation. This is solved via a one-dimensional analytic optimization combined with a discrete search.

Key Contributions

Probabilistic Workload Model: The paper establishes an analytical framework capturing the stochastic dynamics of AFD serving with continuous batching. It identifies $\theta$ as the governing statistic for provisioning, valid under arbitrary prompt and decode-length distributions, and provides a nonparametric estimator for $\theta$ and $\nu^2$ from traces.
Closed-Form Provisioning Rule: The authors derive a mean-field optimal ratio that explicitly balances Attention, communication, and FFN bottlenecks. They further provide a barrier-aware refinement that quantifies the synchronization overhead introduced by straggler workers, showing that this overhead grows sub-logarithmically with $r$ .
Trace-Calibrated Validation: The team developed an AFD simulator calibrated with real-world traces and hardware parameters (specifically Huawei Ascend 910C NPUs). They validated the framework across diverse workloads and hardware configurations.

Results

The experimental validation demonstrates high accuracy in predicting system behavior:

Optimal Ratio Accuracy: The predicted optimal ratio ( $r^*_{mf} \approx 9.3$ ) closely matches the simulation-optimal ratio. The theoretical throughput curve tracks simulation results well across all tested configurations ( $r \in \{1, 2, 4, 8, 16, 24, 32\}$ ).
Error Margins: The predicted optimal ratio matches the simulation-optimal within 10%.
Straggler Impact: A residual gap of up to ~15% in throughput prediction between the mean-field model and simulation was observed at large $r$ . The barrier-aware analysis explains this discrepancy, attributing it to synchronization overhead which reaches ~11% at $r=24$ .
Parameter Sensitivity: Ablation studies show that the optimal $r^*$ increases with larger microbatch sizes ( $B$ ) and longer total context lengths (both prefill $\mu_P$ and decode $\mu_D$ ), as these factors increase the computational pressure on the Attention side.

Significance and Claims

The paper claims to provide the first principled analytical foundation for provisioning AFD systems, moving beyond empirical search. By offering a compact, calibratable account of how stochastic workload structure determines provisioning, the framework enables system designers to:

Determine the optimal A/F ratio without extensive system building or trial-and-error.
Understand the trade-offs between memory scaling (Attention) and compute scaling (FFN) under realistic, stochastic workloads.
Quantify the cost of synchronization barriers in disaggregated architectures.

The authors note that while real-system validation is left for future work as AFD implementations mature, the current simulation-based validation (using traces from production-like distributions) provides strong empirical confirmation of the theory. The work is presented as a preprint (arXiv:2601.21351v2), indicating it is a foundational contribution to the emerging field of disaggregated LLM serving.