💬 NLP

DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models

This paper introduces DeInfer, a high-performance inference system designed to address the poor parallel inference scalability of decomposed large language models through multiple optimizations and compatibility with state-of-the-art techniques.

Original authors: You-Liang Huang, Xinhao Huang, Chengxi Liao, Zeyi Wen

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: You-Liang Huang, Xinhao Huang, Chengxi Liao, Zeyi Wen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Squeezed" Problem

Imagine you have a massive, incredibly smart library (a Large Language Model or LLM) that knows everything. But this library is so huge that it doesn't fit in your house (your computer's memory).

To make it fit, researchers came up with a clever trick called Model Decomposition. Think of this like taking a giant, heavy encyclopedia and breaking it down into smaller, lighter volumes. You can now carry them in a backpack.

The Catch:
While breaking the book down makes it lighter, it also makes it harder to read.

The Old Way: You open one big book, read a page, and you're done.
The New Way (Decomposed): You have to open three small books, read a page from each, and then glue the pages together in your head to make sense of the story.

When you try to do this alone, it's fine. But when you try to do it with a team of people (using multiple computers or GPUs working in parallel), the team gets stuck arguing about how to glue the pages together. The communication overhead slows everything down to a crawl.

Enter DeInfer:
The authors of this paper built a new system called DeInfer. It's like hiring a super-efficient project manager who reorganizes the team so they don't waste time arguing. They figure out how to pass the "glued pages" around much faster, making the team work together smoothly even with the broken-down books.

The Three Big Bottlenecks (Why it was slow)

The paper identifies three specific reasons why the "broken-down" books were causing traffic jams:

1. The "Too Many Meetings" Problem (Communication)

The Analogy: Imagine a team of chefs trying to make a soup. In the old way, they pass one big pot around once. In the new way, because the ingredients are chopped into tiny pieces, they have to pass the pot back and forth four times just to get the same result.
The Fix: DeInfer changes the recipe. Instead of passing the pot back and forth in the "kitchen" (the main processing area), they pass the ingredients in a "back alley" (a lower-rank, compressed space) where the path is shorter and faster. This cuts the time spent passing things around by 78%.

2. The "Double Work" Problem (Duplicate Calculation)

The Analogy: Imagine five people in a room, each holding a piece of a puzzle. In the old way, they all look at their own piece and figure out how it fits. In the broken-down version, the instructions got mixed up, so everyone ended up solving the entire puzzle five times, even though they only needed to solve their own piece.
The Fix: DeInfer acts like a strict supervisor. It says, "Stop! You all have the same information now. Only one of you needs to do the thinking; the rest just wait for the answer." This stops the team from doing useless, duplicate work.

3. The "Moving Target" Problem (CUDA Graph)

The Analogy: Imagine a train (the computer program) that runs on a fixed track (a static graph). It's very fast because the track is pre-laid. But with the broken-down books, the "train cars" (data) keep changing size as the story gets longer. The train can't use the fixed track anymore; it has to stop and lay new tracks every time it moves, which is incredibly slow.
The Fix: DeInfer builds a special "magic buffer." It gathers all the scattered train cars, snaps them into a neat, compact line before the train starts moving, and then locks them into place. Now, the train can run on the fast, fixed track again, even though the data was originally messy.

The Results: How much faster is it?

The researchers tested this system on powerful computers (like a team of 8 super-computers working together).

Without DeInfer: As they added more computers, the speed didn't really go up. The team was just spending all their time talking to each other, not working.
With DeInfer:
- Speed: They saw speedups of up to 8.8 times faster in some scenarios.
- Scalability: The more computers they added, the faster it got. It actually worked the way it was supposed to.
- Latency: The time it took to get the first answer dropped by over 80%.

The Bottom Line

DeInfer is a system that fixes the "traffic jams" caused by trying to run compressed, broken-down AI models on multiple computers at once.

By reorganizing how data is passed, stopping people from doing the same math twice, and organizing the data so the computer can use its fastest "highway" settings, DeInfer makes it possible to run huge, compressed AI models quickly and efficiently. It turns a chaotic group of people trying to read a broken book into a well-oiled machine.

1. Problem Statement

While Large Language Models (LLMs) are increasingly compressed using model decomposition (specifically low-rank decomposition like SVD) to reduce memory footprints, existing research focuses primarily on downstream task performance. These works largely ignore the severe parallel inference performance degradation that occurs when scaling decomposed models across multiple GPUs.

The authors identify three critical bottlenecks that prevent decomposed LLMs from scaling efficiently in parallel settings:

Excessive Communication Overhead: Low-rank decomposition replaces a single weight matrix with two smaller matrices ( $A$ and $B$ ). In a tensor-parallel setting, this introduces multiple reduce-sum operations (collective communication primitives with high latency/bandwidth costs) per layer, whereas the original model required only one.
Duplicate Self-Attention Computation: In standard tensor parallelism, processes split data and compute locally. However, after decomposition, intermediate results often require synchronization (reduce-sum) before the self-attention mechanism. This forces every GPU to hold an identical copy of the data, leading to redundant self-attention calculations on every device, which is computationally wasteful.
Incompatibility with CUDA Graphs: Modern inference systems (e.g., vLLM) use CUDA Graphs to eliminate kernel launch overhead by using static computation graphs. However, decomposed models require KV cache reconstruction during decoding. Since the size of the reconstructed KV cache grows dynamically with the sequence length, it creates a dynamic shape that conflicts with the static requirements of CUDA Graphs, forcing the system to fall back to slower dynamic execution.

2. Methodology: The DeInfer System

DeInfer is a high-performance inference system designed specifically to eliminate the bottlenecks above. It integrates with state-of-the-art systems like vLLM and employs the following key optimizations:

A. Highly Efficient Low-Rank Communication

DeInfer restructures the computation pipeline to perform communication in the low-rank latent space rather than the full-rank space.

Mechanism: Instead of performing reduce-sum operations after every sub-layer in the full dimension, DeInfer performs an All-Gather operation immediately after the downward projection matrices ( $x_v$ ) in the low-rank dimension.
Result: Each process obtains identical low-rank data. Subsequent operations (like self-attention or activation) are performed locally without further communication until the upward projection ( $x_u$ ). This drastically reduces the volume of data transferred and the number of expensive reduce-sum operations.
Impact: Reduces communication bandwidth usage by up to 78% (specifically analyzed on LLaMA-3-70B with 40% compression).

B. Elimination of Duplicate Computation

By rearranging the pipeline so that All-Gather happens before the self-attention mechanism (in the low-rank space), DeInfer ensures that each GPU processes unique data chunks for the attention calculation.

Result: This completely eliminates the duplicate self-attention computation observed in naive decomposed implementations, reducing compute time by 10–30%.

C. Integrating Paged KV Cache with CUDA Graphs

To resolve the dynamic shape issue of KV cache reconstruction, DeInfer introduces a two-stage process compatible with CUDA Graphs:

Preparation Stage: DeInfer scans the paged memory to identify physically contiguous blocks of the scattered KV cache. It generates a remapping index list (a new block table) and copies these blocks into a contiguous buffer.
Graph Replay Stage: The system executes a captured graph where the contiguous buffer is multiplied by the upward projection matrices to reconstruct the KV cache.

Key Innovation: Although the logical shape of the data changes, the memory address and the kernel launch parameters remain fixed. This allows the use of a large, fixed-size GEMM kernel to handle the reconstruction within a static CUDA Graph, eliminating kernel launch overhead.

D. Generalization

DeInfer is designed to be agnostic to specific decomposition ratios or matrix shapes. It supports:

Variadic compression ratios (different layers having different compression levels).
Different attention mechanisms (MHA, GQA, MQA, MLA).
Different MLP variants (GLU and non-GLU).
It currently supports non-MoE models (e.g., LLaMA, OPT, Qwen).

3. Key Contributions

System Identification: First work to systematically identify and analyze the specific performance bottlenecks (communication, redundant compute, CUDA Graph incompatibility) of parallel inference for decomposed LLMs.
DeInfer Architecture: A novel inference system featuring a low-rank communication primitive and a CUDA Graph-compatible KV cache reconstruction mechanism.
Performance Optimization: Achieves significant reductions in communication bandwidth (up to 78%) and compute redundancy (10–30%).
Comprehensive Evaluation: Extensive experiments on LLaMA-3-70B, LLaMA-65B, and OPT-30B across different hardware configurations (A800 with NVLink, A6000 without NVLink).

4. Experimental Results

The authors evaluated DeInfer against a baseline "Naive" tensor parallel implementation using vLLM.

Throughput:
- In high-load scenarios without NVLink, DeInfer achieved speedups of 4.59x to 6.59x over the baseline.
- With NVLink, speedups ranged from 1.49x to 1.88x.
- DeInfer demonstrated excellent scalability as the number of GPUs increased from 2 to 8, whereas the baseline performance degraded or plateaued.
Latency:
- Time-to-First-Token (TTFT): Reduced by 77–83% (e.g., from ~21s to ~3.6s for LLaMA-3-70B).
- Inter-Token Latency (ITL): Reduced by 68–79%.
Scalability:
- At 8 GPUs with 60% compression, DeInfer achieved a 2.25x speedup (with NVLink) and 8.81x speedup (without NVLink) compared to the baseline.
- Performance improved as compression ratios increased, proving the system scales well with higher compression.
CUDA Graph Impact:
- When NVLink is enabled (making communication less of a bottleneck), enabling CUDA Graphs provided an additional 5–20% throughput improvement by eliminating kernel launch overhead.

5. Significance

DeInfer bridges a critical gap between model compression and deployment efficiency. While model decomposition is a powerful technique for reducing memory footprints, it was previously considered "inference-unfriendly" in parallel settings. DeInfer proves that with the right system-level optimizations, decomposed models can not only be deployed but can outperform naive parallel implementations significantly. This work enables the practical scaling of highly compressed LLMs on multi-GPU clusters, making large-scale, memory-efficient AI inference more accessible and cost-effective.