Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study

This paper presents a comprehensive benchmark of production LLM inference on AMD Instinct MI325X GPUs, demonstrating that architecture-aware optimizations—specifically the selective use of the AITER runtime and specific KV cache configurations—are critical for maximizing throughput across diverse model families while maintaining high reliability under heavy concurrency.

The Gist

Imagine you are running a massive, high-speed library where the goal is to answer millions of complex questions from people all at once. The "books" in this library are Large Language Models (LLMs)—giant AI brains that can write, reason, and create art. The "librarians" are the GPUs (computer chips) that do the heavy lifting.

This paper is a report card on how well a specific type of librarian, the AMD Instinct MI325X, performs when trying to serve four different, super-complex AI "books" to a crowd of up to 1,000 people simultaneously.

Here is the breakdown in simple terms, using some fun analogies.

1. The Setup: A Super-Strong Library

The researchers set up a server with 8 AMD MI325X chips.

• The Memory (HBM3e): Think of this as the librarian's desk space. Each chip has a massive 256GB desk. With 8 chips, they have a 2 Terabyte desk. That's huge! It means they can keep almost the entire "book" open on the desk at once without having to run back and forth to the basement (the CPU) to fetch pages. This is a game-changer because running back and forth slows everything down.
• The Speed: These chips are incredibly fast at moving data, like a super-highway for information.

2. The Four "Books" (AI Models)

The team tested four different AI models, which are like four different types of books with very different writing styles:

1. Llama-3.1-405B: A Dense book. Every single page is read for every question. It's heavy and requires a lot of brainpower.
2. DeepSeek V3.2 & Kimi-K2.5: MoE (Mixture of Experts) books with MLA (Multi-head Latent Attention). Imagine a book where, for every question, only a tiny team of 3 or 4 specific experts is called in to answer, while the rest of the 384 experts take a coffee break. This makes them very efficient, but they use a special, compressed way of organizing their notes (MLA) that is tricky to read.
3. Qwen3-VL-235B: A Vision book. It can look at pictures and read text. It's also a "Mixture of Experts" but uses a more standard way of organizing notes (GQA).

3. The Big Discovery: "One Size Does Not Fit All"

The most important lesson from this paper is that you cannot use the same instructions for every book.

• The "MLA" Problem: The DeepSeek and Kimi books use a special "compressed notes" system (MLA). On the AMD chips, this system is picky.
  • It requires the librarian to process notes one by one (Block Size 1), which is slow.
  • It refuses to use the "basement storage" (KV Cache Offloading) even though the desk is huge.
  • Crucially: To read these books fast, the librarian must use a special tool called AITER. Without it, the reading speed drops to a crawl. However, for one specific book (Kimi), the tool doesn't fit at all because the book has too many "chapters" (attention heads) for the tool to handle, so the tool had to be turned off completely.
• The "GQA" Freedom: The Llama and Qwen books use a standard note system. They are flexible. They can use the basement storage if needed and work well with standard tools.

The Analogy: Imagine trying to drive a Ferrari and a Tractor on the same road. If you treat them both exactly the same, the Ferrari might get stuck, and the Tractor might go too slow. You need different driving modes for each.

4. The Results: Who Won the Race?

The Text Race (Answering questions with words only)

• The Surprise: The Llama book (405 Billion words) and the DeepSeek book (only 37 Billion active words) finished at almost the exact same speed.
• Why? Even though DeepSeek is a "Mixture of Experts" (only using a small team), the special "compressed notes" system (MLA) it uses is a bit clunky on AMD chips. It cancels out the efficiency gains.
• The Winner: They tied, but Llama was slightly more consistent.

The Vision Race (Answering questions with pictures)

• The Big Gap: The Qwen book (MoE + Standard Notes) was 6.5 times faster than the Kimi book (MoE + Compressed Notes).
• Why? Qwen could use all 8 chips and standard tools. Kimi was forced to use only 4 chips (because of the "chapter count" issue) and couldn't use the special speed-up tool (AITER).
• Lesson: If you want speed on AMD chips right now, avoid the "compressed notes" (MLA) models unless you have the specific tools to make them work.

5. The Bottleneck: The "Highway" vs. The "Engine"

The researchers found something fascinating about why the speed stops increasing.

• The Engine: The chips are powerful enough to calculate answers very fast.
• The Highway: The speed limit is actually the memory bandwidth (how fast data can move onto the desk).
• The Saturation Point: No matter how many people you add to the queue, once you hit about 500 people (for short questions), the highway gets jammed. Adding more people just makes them wait longer; it doesn't make the library answer faster.
• The Good News: The library never crashed. Even with 1,000 people screaming for answers, every single request was answered successfully. The system just got slower, but it didn't break.

6. Practical Takeaways for the Real World

If you are a company trying to run these AI models on AMD chips, here is the cheat sheet:

1. Check the Model Type First: Before you start, ask: "Is this model using MLA (compressed notes) or GQA (standard notes)?"
2. Turn on the Special Tool (AITER): For most models, you need this AMD tool to get good speed. But be careful: for the "Kimi" model, you must turn it OFF, or the system will crash.
3. Don't Overcrowd: If you have short questions, aim for about 500 people at a time. If you have long questions, aim for 100–200. Adding more won't help; it will just create a traffic jam.
4. Big Memory is King: Because these AMD chips have such huge desks (256GB each), you rarely need to run back to the basement (CPU) to get data. This keeps things fast and simple.

Summary

This paper proves that AMD's new chips are powerful enough to run the world's biggest AI models, including a 1-trillion-parameter giant. However, to get the best performance, you can't just use a "default" setting. You have to be a smart librarian who knows exactly which tool to use for which book. If you get the configuration right, you can get blazing-fast speeds; if you get it wrong, the system will struggle.

Technical Summary

Here is a detailed technical summary of the paper "Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study."

1. Problem Statement

The rapid evolution of Large Language Models (LLMs) from research prototypes to production infrastructure has created a critical bottleneck in efficient inference serving. While training-side scaling laws are well-studied, the operational reality of serving frontier-scale models (up to 1 trillion parameters) on diverse hardware remains underexplored. Specifically:

• Architectural Diversity: Modern models utilize distinct architectures (Dense, Mixture-of-Experts [MoE], Grouped-Query Attention [GQA], Multi-head Latent Attention [MLA]), each imposing unique demands on memory hierarchy and kernel optimization. A configuration optimal for one architecture often fails or performs poorly on another.
• Hardware Gap: Systematic benchmarking of state-of-the-art LLMs on AMD Instinct GPUs (specifically the MI325X) is scarce compared to NVIDIA hardware.
• Configuration Complexity: Deploying these models requires navigating complex constraints regarding quantization, tensor parallelism, and software stack compatibility (ROCm, vLLM, AITER), which are often architecture-specific.

2. Methodology

The authors conducted a systematic cross-architecture evaluation using an 8-GPU AMD Instinct MI325X cluster (2 TB aggregate HBM3e, 48 TB/s bandwidth) running vLLM v0.14.1.

Models Evaluated:
Four frontier models spanning three architectural families:

1. Llama-3.1-405B: Dense + GQA (405B active params).
2. DeepSeek V3.2: MoE + MLA (685B total, ~37B active).
3. Qwen3-VL-235B: MoE + GQA (235B total, ~22B active, Vision-Language).
4. Kimi-K2.5: MoE + MLA (1 Trillion total, ~32B active).

Experimental Design:

• Workloads: Ranged from single-request latency to 1,000 concurrent users. Includes text-only and vision workloads (with image tokens).
• Metrics: Throughput (tokens/sec), p99 latency, success rates, and measurement variability (Coefficient of Variation).
• Optimization Techniques: Evaluated quantization (FP8, INT4 QAT, BF16), KV cache management (offloading, block sizes), and kernel acceleration (AMD AITER).
• Reproducibility: Conducted multi-run ablation studies (n=5) to measure variance and ensure statistical robustness.

3. Key Contributions

1. First Cross-Architecture MI325X Benchmark: The first academic study evaluating four frontier models (235B–1T params) on AMD MI325X, characterizing architecture-specific constraints (block size, AITER compatibility, KV offloading).
2. Architecture-Aware Optimization Necessity: Demonstrated that "one-size-fits-all" configurations fail.
   • MLA Models: Require block-size 1, cannot use KV cache offloading on current ROCm, and strictly depend on the AITER runtime for competitive throughput.
   • GQA Models: Benefit from KV cache offloading and standard block sizes (16).
   • AITER Nuance: While AITER provides massive speedups for MoE/MLA kernels, it introduces 2–16× higher measurement variability for GQA models and offers only modest (3–5%) throughput gains at high concurrency.
3. Trillion-Parameter Deployment: Successfully deployed and benchmarked Kimi-K2.5 (1T params) on MI325X using INT4 QAT and Tensor Parallelism (TP=4), achieving 100% reliability.
4. Active Parameter Correlation: Empirically validated that active parameters per token (not total parameters) are the primary driver of inference throughput, though confounded by quantization and kernel support.
5. Memory-Bandwidth Saturation: Identified a common throughput saturation point (~500 concurrent requests for short sequences, ~100–200 for long sequences) across all architectures, confirming the system is memory-bandwidth bound rather than compute-bound.

4. Key Results

Performance Metrics (Peak Throughput):

• Text Workloads:
  • Llama-3.1-405B (Dense+GQA): 15,944 tok/s.
  • DeepSeek V3.2 (MoE+MLA): 15,343 tok/s.
  • Insight: Despite DeepSeek having only 9% of Llama's active parameters (37B vs 405B), it achieved comparable throughput, suggesting MoE sparsity offsets MLA overhead when AITER is enabled.
• Vision Workloads:
  • Qwen3-VL-235B (MoE+GQA): 47,873 tok/s (includes image tokens).
  • Kimi-K2.5 (MoE+MLA): 7,327 tok/s.
  • Insight: Qwen3-VL achieved 6.5× higher throughput than Kimi-K2.5. This gap is attributed to Qwen's GQA architecture (allowing offloading/standard blocks), full AITER support, and TP=8 vs. Kimi's restricted TP=4 and disabled AITER.

Optimization Findings:

• AITER Impact: For MLA models (DeepSeek, Kimi), AITER is a practical necessity; the Triton fallback is significantly slower. For GQA models (Llama), AITER adds ~3–5% throughput at high concurrency but drastically increases result variance.
• Kimi-K2.5 Constraints: Due to MLA head-count constraints (64 heads) and MXFP4 hardware requirements (CDNA 4 only), Kimi-2.5 cannot use AITER on MI325X (CDNA 3) and must run with TP=4 (utilizing only 4 of 8 GPUs), limiting its bandwidth potential.
• Quantization: FP8 is critical for dense models (Llama-3.1-405B) to fit within memory. Qwen3-VL requires BF16 due to vision encoder dimension constraints incompatible with FP8 block quantization.

Reliability:

• All models maintained 100% HTTP-level success rates (valid response structure) up to 1,000 concurrent users, processing 18.9 million tokens without failure. vLLM's continuous batching effectively queues excess requests rather than rejecting them.

5. Significance and Implications

• Deployment Strategy: Production deployments on AMD hardware must implement architecture-detection logic to dynamically select block sizes, AITER flags, and tensor parallelism degrees. Default configurations will lead to suboptimal performance or runtime errors.
• Hardware Viability: The AMD MI325X (CDNA 3) is proven viable for frontier-scale inference, including 1T parameter models, provided specific architectural constraints (like TP limits for MLA) are managed.
• Bottleneck Identification: The study confirms that LLM inference on high-bandwidth GPUs is fundamentally memory-bandwidth bound. Optimizations reducing memory traffic (MoE sparsity, KV compression, quantization) yield higher returns than raw compute increases.
• Future Directions: Highlights the need for multi-node scaling studies, pipeline parallelism, and automated configuration systems to handle the complexity of diverse model architectures.

In conclusion, this paper establishes that successful LLM inference on AMD Instinct GPUs requires a deep, architecture-specific understanding of the software stack (ROCm/vLLM/AITER) and hardware constraints, moving beyond generic deployment guides to achieve production-grade performance and reliability.