Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks

This paper investigates parallelization strategies for deploying dense LLMs, demonstrating that while Tensor Parallelism optimizes latency and Pipeline Parallelism enhances throughput, a hybrid approach allows for effective control over the inherent latency-throughput tradeoff to meet specific application requirements.

The Gist

Here is an explanation of the paper using simple language and creative analogies.

The Big Picture: The "Super-Brain" Problem

Imagine you have a Super-Brain (a Large Language Model or LLM) that is so massive it contains the knowledge of millions of books. This brain is so big that it doesn't fit inside a single computer's memory (RAM). It's like trying to fit the entire Library of Congress into a single backpack.

To make this brain work, we have to split it up and put parts of it on many different computers (GPUs) that are connected together. This paper is a guide on how to best split up the work so the brain can answer your questions quickly (low latency) and handle many people asking questions at once (high throughput).

The authors tested two main ways to organize this team of computers: Tensor Parallelism (TP) and Pipeline Parallelism (PP).

---

The Two Strategies: The "Chef" vs. The "Assembly Line"

1. Tensor Parallelism (TP): The "Super-Chef" Team

The Analogy: Imagine you need to chop 1,000 onions for a giant stew.

• Without Parallelism: One chef chops all 1,000 onions. It takes a long time.
• With Tensor Parallelism (TP): You hire 8 chefs. You give each chef 125 onions. They all chop their onions at the exact same time. When they are done, they quickly combine their piles.

How it works in the paper:

• TP splits the math of a single step across multiple computers.
• The Good News: It makes the brain think much faster. If you ask a question, the answer comes back very quickly (low latency).
• The Bad News: The chefs have to talk to each other constantly to combine their work. If the team gets too big, they spend more time talking than chopping. Also, because they are all working on one big task together, they can't easily start a second stew while the first one is being chopped.

Best for: When you need an answer right now (e.g., a chatbot talking to a single user).

2. Pipeline Parallelism (PP): The "Assembly Line"

The Analogy: Imagine a car factory.

• Without Parallelism: One worker builds the whole car from start to finish before starting the next one.
• With Pipeline Parallelism (PP): You have 8 workers. Worker 1 installs the engine. Worker 2 paints the car. Worker 3 installs the wheels. As soon as Worker 1 finishes the first car's engine, they pass it to Worker 2 and immediately start on the second car's engine.

How it works in the paper:

• PP splits the layers of the brain. Computer 1 does the first step, Computer 2 does the second step, and so on.
• The Good News: You can have many cars (requests) on the line at once. Even though one car takes the same amount of time to build, the factory produces many more cars per hour (high throughput).
• The Bad News: The first car still takes a long time to get through the whole line. The "first car" (first token) is slow because it has to wait for every station to finish its part.

Best for: When you need to process huge batches of data (e.g., summarizing 1,000 documents overnight).

---

The "Hybrid" Solution: The Best of Both Worlds

The paper discovered that you don't have to choose just one. You can mix them!

The Analogy: Imagine a factory where you have 4 assembly lines (Pipeline), and on each line, you have 2 chefs working together to chop the ingredients faster (Tensor).

• This setup lets you control the balance. If you need speed, you make the "chopping teams" bigger. If you need volume, you add more "assembly lines."

Key Findings from the "Lab"

The researchers used a super-accurate simulator (like a flight simulator for computers) to test the famous Llama 3.1 models (the 70B and the massive 405B versions). Here is what they found:

1. Speed vs. Volume Trade-off:

   • If you want the fastest response time (lowest latency), Tensor Parallelism is the winner. It's like having a Formula 1 car.
   • If you want to serve the most people (highest throughput), Pipeline Parallelism wins. It's like a busy subway train that carries many passengers, even if the first one takes a moment to board.

2. The "Talking" Bottleneck:

   • When computers talk to each other to share data, it takes time.
   • In TP, they talk a lot (All-Reduce). If the connection cables are slow, the whole system slows down.
   • In PP, they talk less, but they have to wait for the previous station to finish.

3. Memory is King:

   • The biggest problem with these giant brains is running out of memory. Both strategies help by splitting the brain's "memory" across many computers.
   • PP is surprisingly good at this because it frees up space on each computer to hold more "scratchpad" notes (KV Cache), allowing the system to handle much larger groups of requests at once.

The Takeaway for the Real World

If you are building an AI service:

• For a Chatbot: Use Tensor Parallelism. Users hate waiting for the first word of an answer.
• For Data Processing: Use Pipeline Parallelism. You don't care if the first document takes a minute to start, as long as you can finish 1,000 documents in an hour.
• For the Future: The smartest systems will likely use a Hybrid approach, tuning the mix like a radio dial to get the perfect balance of speed and volume for their specific needs.

In short: The paper tells us there is no "one size fits all" for running giant AI models. You have to choose your strategy based on whether you value speed or volume more, and the best solution is often a clever mix of both.

Technical Summary

Here is a detailed technical summary of the paper "Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks."

1. Problem Statement

The rapid adoption of Large Language Models (LLMs), particularly dense models like Llama 3.1-70B and Llama 3.1-405B, has created significant challenges in deployment. These models often exceed the memory capacity of a single GPU, necessitating the use of multiple GPUs or TPUs. While techniques like quantization and offloading exist, they often compromise Quality of Service (QoS) or fail to meet strict latency and throughput requirements.

The core problem addressed is the trade-off between latency and throughput in LLM inference systems.

• Latency: Critical for interactive applications (Time-to-First-Token, TTFT; Time-Per-Output-Token, TPOT).
• Throughput: Critical for batch processing (Tokens Per Second, TPS).
• The Gap: Existing literature lacks a systematic evaluation of how different parallelization strategies (Tensor Parallelism, Pipeline Parallelism, and their hybrids) interact with batching, input characteristics, and model sizes to influence these metrics. Specifically, there is a need to understand the "latency-flexibility" and the specific bottlenecks (e.g., communication overheads) that arise when scaling these strategies.

2. Methodology

The authors employed a rigorous empirical approach using an in-house simulator validated against real hardware.

• Simulation & Validation:
  • The simulator models end-to-end LLM inference, including transformer blocks, attention mechanisms, and GEMM operations.
  • Validation: The simulator was validated against AMD Instinct MI325x and MI355x GPUs. It achieved >90% accuracy for prefill phases and >86% for decode phases on the Llama 3.1-70B model. For the larger Llama 3.1-405B (using TP8), accuracy remained high despite increased complexity, with errors attributed to dynamic voltage-frequency variations and kernel synchronization.
• Models & Datasets:
  • Models: Llama 3.1-70B and Llama 3.1-405B (FP8 and 4-bit quantized).
  • Datasets: A mix of short-context datasets (BBH, GSM8K, HumanEval) and long-context datasets (LongAlpaca, MLPerf) to test various Input Sequence Lengths (ISL) and Output Sequence Lengths (OSL).
• Hardware Configuration:
  • Simulated nodes with 8 GPUs (MI325x/MI355x) using all-to-all interconnects.
  • Evaluated various parallelization depths (TP2, TP4, TP8; PP2, PP4, PP8) and hybrid configurations (e.g., TP4_PP2).
• Metrics:
  • Latency: TTFT (prefill) and TPOT (decode).
  • Throughput: Total Output Tokens per Second (TPS).

3. Key Contributions

The paper makes three primary contributions:

1. Kernel-Level Mapping: A detailed breakdown of how dense LLM transformer workloads map to GPUs under various parallelization schemes (TP, PP, and hybrids) within a node environment.
2. Quantified Trade-offs: A comprehensive quantification of latency and throughput trends across diverse configurations (batch sizes, sequence lengths, model sizes, and parallelization degrees).
3. Bottleneck Identification: Identification of key performance limitations, specifically the role of all-reduce operations in Tensor Parallelism and P2P communication in Pipeline Parallelism, and how they constrain further performance gains.

4. Key Results & Findings

A. Latency Flexibility: Tensor Parallelism (TP) is Superior

• TP Reduces Latency: Increasing the degree of Tensor Parallelism (e.g., from TP4 to TP8) significantly reduces both TTFT and TPOT. This is because TP shards the computation (GEMMs and Attention) across GPUs, dedicating more compute resources to a single transformer pass.
• PP Does Not Reduce Latency: Pipeline Parallelism (PP) distributes transformer blocks across GPUs. While it increases memory capacity for larger batches, it does not speed up the execution of a single pass. Consequently, PP offers no latency benefit over non-parallelized setups and can even increase latency due to inter-stage communication overheads.
• Communication Overhead:
  • In TP, all-reduce operations are the primary bottleneck. As TP depth increases, the number of communication steps increases, creating inevitable latency overheads.
  • In PP, P2P communication between stages has a negligible impact on latency (<0.5% overhead) because the number of transmissions is low compared to the volume of data in all-reduce operations.

B. Throughput Trends: Pipeline Parallelism (PP) is Superior

• PP Maximizes Throughput: PP allows for significantly larger nano-batches (batches per GPU) by distributing model weights and freeing up memory for larger KV caches. This enables the system to process more requests concurrently.
• TP Limits Throughput: While TP improves latency, it often degrades throughput compared to PP. The overhead of all-reduce operations prevents the system from scaling throughput linearly with batch size.
• Saturation Point: Throughput gains from PP eventually saturate when the decode phase becomes purely compute-bound. Increasing batch size beyond this point increases TTFT and TPOT proportionally without increasing TPS.

C. Hybrid Strategies

• Tunable Balance: A hybrid approach (e.g., TP4_PP2) allows system architects to tune the system.
  • TP Degree: Controls latency (higher TP = lower latency).
  • PP Depth: Controls throughput (higher PP = higher throughput via larger batches).
• Optimal Configuration: For latency-sensitive applications, high TP is preferred. For throughput-oriented applications, high PP is preferred.

D. Impact of Interconnects

• Link Speed: Doubling the aggregate link speed in TP configurations reduces TTFT by ~34% and the all-reduce-to-TTFT ratio by ~7.7%. This indicates that while faster interconnects help, the availability of sufficient compute resources is equally critical.
• Multi-Node Implications: The paper notes that extending TP to multi-node systems will likely exacerbate latency issues due to slower inter-node bandwidth compared to intra-node links, making inter-node communication a major bottleneck.

5. Significance

This paper provides a critical roadmap for deploying dense LLMs in production environments:

• Design Guidance: It clarifies that there is no "one-size-fits-all" parallelization strategy. System designers must choose based on the Service Level Agreement (SLA):
  • Low Latency (Interactive): Prioritize Tensor Parallelism.
  • High Throughput (Batching): Prioritize Pipeline Parallelism.
• Hybrid Optimization: It validates that hybrid strategies offer the best of both worlds, allowing operators to dynamically balance latency and throughput by adjusting TP and PP degrees.
• Future-Proofing: The insights regarding communication bottlenecks (all-reduce vs. P2P) are vital for designing next-generation hardware and software stacks, particularly as models scale to hundreds of billions of parameters and multi-node clusters become the standard.

In conclusion, the study demonstrates that Tensor Parallelism is the key to meeting strict latency objectives, while Pipeline Parallelism is the superior strategy for maximizing throughput, with hybrid configurations offering the necessary flexibility to navigate the complex trade-offs of modern dense LLM deployment.