🤖 machine learning

xGR: Efficient Generative Recommendation Serving at Scale

The paper presents xGR, a specialized serving system that optimizes generative recommendation workloads through unified staged computation, early sorting termination, and multi-level pipeline parallelism to achieve significantly higher throughput under strict low-latency constraints.

Original authors: Qingxiao Sun, Tongxuan Liu, Shen Zhang, Siyu Wu, Peijun Yang, Haotian Liang, Menxin Li, Xiaolong Ma, Zhiwei Liang, Ziyi Ren, Minchao Zhang, Yifan Wang, Xinyu Liu, Ke Zhang, Hailong Yang, Depei Qian

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qingxiao Sun, Tongxuan Liu, Shen Zhang, Siyu Wu, Peijun Yang, Haotian Liang, Menxin Li, Xiaolong Ma, Zhiwei Liang, Ziyi Ren, Minchao Zhang, Yifan Wang, Xinyu Liu, Ke Zhang, Hailong Yang, Depei Qian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-speed digital library (a recommendation system) that suggests the next book, movie, or product for millions of people at once. For years, this library used a "filtering" method: it would look at a huge pile of books, throw away the bad ones in stages, and finally show you a short list.

Recently, a new method called Generative Recommendation (GR) arrived. Instead of filtering, it acts like a creative writer who reads your entire life story (your history of clicks and views) and then writes the perfect recommendation for you from scratch.

The problem? This new "writer" is incredibly slow when you have thousands of people asking for books at the exact same second. The paper introduces xGR, a new system designed to make this writer fast enough to handle the rush hour without breaking a sweat.

Here is how xGR solves the three main headaches, explained with simple analogies:

1. The "Shared Story" Problem (Solving Memory Waste)

The Issue: Imagine 128 different people (called "beams") are all asking the writer to continue the same story. In old systems, the writer would read the first 1,000 pages of the story 128 separate times, once for each person. This is like a librarian running to the same shelf 128 times to grab the same book, clogging up the aisles.

The xGR Solution: xGR realizes that everyone is reading the same first part of the story. It creates a "Shared Reading Room" where that first part is loaded only once. Then, it sets up separate, small desks for the unique endings each person needs.

The Result: The librarian stops running back and forth. The system saves massive amounts of memory and time, allowing it to handle more people at once.

2. The "Sorting Chaos" Problem (Solving Search Slowness)

The Issue: To find the best recommendation, the writer generates many possible endings and has to sort through them to pick the top ones. In the old way, the writer would generate every possible ending, even ones that don't exist (like a book titled "12345" that isn't a real product), and then waste time throwing them away. It's like a chef cooking 1,000 meals, only to realize 500 of them are made of plastic, and then spending time cleaning up the plastic.

The xGR Solution:

The "Valid Path" Filter: Before the writer even starts cooking, xGR gives them a list of only the real ingredients (real products). They can't accidentally make a plastic meal.
The "Early Stop" Rule: The writer starts sorting the meals. As soon as they find a meal that is clearly worse than the best one they've already found, they stop checking that specific option immediately. They don't waste time finishing the sorting of bad options.
The Result: The chef stops wasting time on fake ingredients and stops checking bad dishes halfway through.

3. The "Assembly Line" Problem (Solving Scheduling Delays)

The Issue: In the old system, the manager (the scheduler) would prepare the ingredients, hand them to the chef, wait for the chef to finish, and then prepare the next batch. Everyone stood around waiting. Also, the kitchen was so small that only one chef could work at a time, even though there were many chefs available.

The xGR Solution: xGR turns the kitchen into a high-speed assembly line.

Overlapping Work: While the chef is cooking the current dish, the manager is already prepping the ingredients for the next dish. They happen at the same time.
Multi-Stream Cooking: Instead of one chef working on one big order, xGR splits the work so multiple chefs can cook different parts of the orders simultaneously without bumping into each other.
The Result: The kitchen never stops moving. There is no waiting time between orders.

The Bottom Line

The paper tested xGR on real-world data from a massive e-commerce platform. They found that under strict time limits (where the system must respond in under 200 milliseconds), xGR was at least 2.89 times faster than the best existing systems.

It achieved this not by making the computer chips faster, but by reorganizing how the work is done: sharing the common parts of the story, filtering out impossible options early, and ensuring the kitchen staff never stand idle. This allows the system to serve hundreds of millions of users smoothly, even during the busiest shopping hours.

Technical Summary: xGR: Efficient Generative Recommendation Serving at Scale

1. Problem Statement

Generative Recommendation (GR) systems, which leverage Large Language Models (LLMs) to predict user-item interactions, face unique serving challenges that differ significantly from standard conversational LLM inference. While GR shares the attention-based architecture of LLMs, its workload characteristics create specific bottlenecks in high-concurrency, low-latency production environments:

Workload Mismatch: Unlike conversational LLMs that generate long responses to short prompts, GR processes long user history sequences (prompts) to produce short, fixed-length outputs.
Beam Search Overhead: GR requires large beam widths (BW) and Top-K values to ensure recommendation diversity and accuracy. This leads to massive candidate pools (e.g., $128 \times 128$ or $512 \times 512$ ), making the sorting and filtering operations computationally expensive.
Memory Inefficiency: Existing LLM serving systems (e.g., PagedAttention) treat each beam as an independent sequence. This causes redundant loading of the shared prompt context (KV cache) for every beam and necessitates costly block copying when beam sequences fork, leading to memory fragmentation and bandwidth saturation.
Invalid Token Generation: Due to the vast combinatorial space of token IDs, many generated sequences do not correspond to real-world items. Filtering these invalid items dynamically introduces latency, while pre-storing masks creates unmanageable memory overhead.
Scheduling Latency: GR models are often smaller than general LLMs, making the system highly sensitive to non-compute overheads like host-side scheduling and synchronization, which can dominate the latency budget (strict SLOs require P99 latency within ~200ms).

2. Methodology: The xGR System

The authors propose xGR, a dedicated serving system designed to reconstruct the GR pipeline through three tightly coupled components: xAttention (operator-level), xBeam (algorithm-level), and xSchedule (system-level).

2.1 xAttention: Operator-Level Optimizations

xAddresses the memory and compute bottlenecks in the attention mechanism:

Separated KV Cache: xGR decouples the KV cache into Shared and Unshared components. The Shared cache stores the prompt context (user history) computed once during the prefill phase. The Unshared cache stores only the dynamically generated tokens for each decode step. This eliminates redundant loading of the prompt context across beams.
Token-Granularity Management: Unlike block-based management (e.g., PagedAttention), the Unshared cache is managed at the token level. This avoids the overhead of aligning sequence lengths to block sizes and eliminates the need for physical block copying when beams fork.
Staged Computation: The attention computation is divided into three stages: Shared, Unshared, and Merging.
- The Shared and Unshared stages compute local attention scores and statistics (maxima/sums) independently.
- The Merging stage applies OnlineSoftmax to combine results.
- This design enables pipeline parallelism across hardware units (Core Groups), fusing BatchMatmul and Softmax operations to maximize hardware utilization.
In-Place Updates: To manage the Unshared cache without extra copies, xAttention uses a hybrid ordering strategy with direct indices to perform in-place block updates, preventing write-before-read hazards.

2.2 xBeam: Algorithm-Level Optimizations

xBeam optimizes the beam search process to reduce sorting and filtering costs:

Valid Path Constraint: To prevent the generation of invalid items (token sequences that do not map to real products), xBeam employs a mask-based filtering mechanism. It uses a pre-built valid item vocabulary to generate masks.
- Dense/Sparse Hybrid Storage: For early decode steps with many candidates, masks are stored densely and pre-generated. For later steps with fewer candidates, masks are stored sparsely and updated in-place, minimizing runtime generation latency.
Early Sorting Termination: Beam search involves selecting Top-K sequences from a large pool. Since log-probabilities within a beam are inherently sorted, xBeam utilizes a global min-heap and an early termination strategy. If a leaf node's log-probability is lower than the heap's minimum, the traversal for that branch stops immediately, avoiding unnecessary sorting operations.
Data Structure Reuse: Instead of allocating new memory for every new candidate sequence, xBeam reuses data structures from discarded sequences, reducing allocation/deallocation overhead.

2.3 xSchedule: System-Level Optimizations

xSchedule manages the multi-level pipeline to maximize parallelism and overlap:

Three-Tier Hierarchy: The system employs a scheduler, engine, and worker hierarchy to enable overlap across batches, within batches, and within individual requests.
Host-Device Overlap: Control flows (e.g., kernel dispatch) and data transfers (e.g., mask generation and H2D transfer) are overlapped with device-side computations (e.g., model forward pass).
Graph-Based Execution: Device-side operations are captured into a static graph and submitted in bulk, reducing kernel launch overhead and CPU intervention.
Dynamic Batching & Multi-Stream: xSchedule adjusts batch sizes based on token capacity and SLO constraints. It also employs a multi-stream strategy to process batches concurrently, ensuring that the smaller computational footprint of GR models does not lead to accelerator underutilization.

3. Key Contributions

Workload Analysis: A comprehensive analysis of GR workload characteristics, identifying that the decode phase in GR is computationally heavy due to beam search, contrary to the assumption that prefill is the dominant cost in standard LLMs.
Operator Optimization (xAttention): Introduction of a staged attention mechanism with separated KV cache management, eliminating redundant memory access and block copying.
Algorithm Optimization (xBeam): Development of a beam search algorithm featuring valid path constraints, early sorting termination, and data structure reuse to minimize sorting and filtering overhead.
System Optimization (xSchedule): A redesigned serving pipeline utilizing host-device overlap, graph-based execution, and multi-stream parallelism to maximize throughput under strict latency constraints.
Production Deployment: The system has been deployed in production for over six months, serving hundreds of millions of users with P99 latency under 200ms even at tens of thousands of requests per second (RPS).

4. Experimental Results

Experiments were conducted on real-world datasets (Amazon Review and JD Trace) using Ascend NPUs and NVIDIA H800 GPUs, comparing xGR against state-of-the-art baselines (vLLM and xLLM).

Throughput: Under strict P99 latency constraints (≤200ms), xGR achieves at least 2.89× higher throughput than the best baseline (xLLM) on Qwen models and 8.05× on OneRec models.
Scalability: As beam width increases (e.g., from 128 to 512), baselines experience non-linear latency growth and fail to meet SLOs. xGR maintains a relatively smooth latency increase, demonstrating robust scalability.
Memory Efficiency: xGR significantly reduces memory footprint. With a beam width of 512, xGR consumes ~10.6GB compared to xLLM's 46.3GB. It decouples memory usage from sequence length, maintaining low memory usage even with long input contexts (3k tokens).
Kernel Efficiency: xAttention reduces kernel latency by approximately 6.6× and boosts computational throughput by 7× compared to PagedAttention. It transforms the workload from memory-bound (93.4% busy time in memory pipeline for baselines) to compute-bound (~52% busy time for xGR).
Portability: The performance gains are consistent across both Ascend NPUs and NVIDIA GPUs, validating the system's architectural portability.

5. Significance and Claims

The paper claims to be the first work to comprehensively reconstruct the GR serving paradigm, spanning optimizations from operators to algorithms and system architecture.

Paradigm Shift: It challenges the notion that standard LLM serving systems are sufficient for GR, highlighting the unique "long prompt, short output" and "high beam width" characteristics that require specialized handling.
Production Viability: The system is not merely theoretical; it is deployed in a high-concurrency production environment (JD Company), proving its ability to handle real-world traffic patterns and strict SLOs.
Holistic Optimization: The paper emphasizes that hardware improvements alone (e.g., faster GPUs) are insufficient to solve GR serving challenges. Instead, a comprehensive system redesign addressing memory management, search algorithms, and scheduling is required to unlock the full potential of Generative Recommendation.