💬 NLP

Efficient LLM-based Advertising via Model Compression and Parallel Verification

This paper proposes an Efficient Generative Targeting framework that combines adaptive group quantization, layer-adaptive hierarchical sparsification, and prefix-tree parallel verification to significantly accelerate LLM inference for real-time advertising applications while maintaining acceptable generation quality.

Original authors: Wenxin Dong, Chang Gao, Guanghui Yu, Xuewu Jiao, Mingqing Hu, Qiang Fu, Peng Xu, Penghui Wei, Hui Xu, Yue Xing, Shuanglong Li, Lin Liu

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenxin Dong, Chang Gao, Guanghui Yu, Xuewu Jiao, Mingqing Hu, Qiang Fu, Peng Xu, Penghui Wei, Hui Xu, Yue Xing, Shuanglong Li, Lin Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart assistant (a Large Language Model, or LLM) who is incredibly good at writing ads and picking the perfect products for customers. The problem is, this assistant is like a giant, heavy encyclopedia: it takes a long time to flip through the pages to find an answer, and it requires a massive amount of energy to keep running. In the fast-paced world of online advertising, where you need an answer in a split second, this "heavy encyclopedia" approach is too slow and expensive.

The authors of this paper, from Baidu, built a new system to make this assistant faster and lighter without making it any less smart. They did this using two main tricks: shrinking the assistant and giving it a shortcut map.

Trick 1: Shrinking the Assistant (Model Compression)

Think of the AI model as a massive library of knowledge. Most of the books in this library are rarely used, and some pages are just filled with numbers that don't change much.

The "Smart Packing" (Quantization): Imagine you have a suitcase full of heavy, high-definition photos (the original data). Instead of carrying the heavy originals, the team figured out how to compress them into smaller, lighter JPEGs (lower precision numbers) without losing the picture's clarity. They didn't just compress everything the same way; they were "adaptive." They packed the most important, sensitive parts of the library very carefully, but compressed the less important parts more aggressively. This turned the heavy suitcase into a lightweight backpack.
The "Empty Shelf" Strategy (Sparsity): Imagine the library has thousands of empty shelves. The team decided to remove the books from the shelves that no one ever visits. They didn't just randomly remove books; they looked at which shelves were critical (the "sensitive" layers) and kept those full, while clearing out the less important ones. This made the library much smaller and faster to navigate.
The Custom Tool: To make sure these compressed and emptied shelves could still be read quickly, they built a custom "scanner" (a specialized computer kernel) that could read these lightweight, sparse books much faster than standard tools could.

Trick 2: The Shortcut Map (Prefix Tree Parallel Verification)

Usually, when the AI generates an ad, it writes it one word at a time, like a person typing a sentence slowly. "I... like... this... car..." This is slow.

The Tree Map: The team organized all possible ad options into a giant, branching tree structure (like a family tree or a decision tree). The top of the tree is wide (many options), but as you go down, the branches narrow down to the most likely choices.
The "Guess and Check" Race: Instead of writing one word at a time, the AI now looks at the whole tree. It makes a "guess" at several words at once (parallel decoding) and then quickly checks if those guesses make sense on the map.
The Timing Switch: The smartest part is knowing when to use this shortcut. The system constantly calculates: "Is it faster to write word-by-word, or to guess a whole bunch and check them?" If the guess-and-check method is faster, it switches to that mode instantly. This allows the AI to "jump" to the end of the sentence in a single step rather than walking there step-by-step.

The Results

The team tested this system in two real-world scenarios:

Writing Ad Creatives: Creating catchy text for ads.
Targeted Advertising: Picking the right ad for a specific user.

The Outcome:

Speed: The new system was 1.8 times faster in real-world tests. In some specific tests, the speedup was even higher (over 78% faster).
Quality: Despite being much lighter and faster, the ads it generated were still just as good as the heavy, slow version. The "quality" (precision) didn't drop significantly.
Real-World Use: This isn't just a theory; it has been deployed on Baidu's actual advertising platform, handling real traffic right now.

In Summary

The paper describes a way to take a slow, heavy AI and turn it into a fast, lightweight one by compressing its memory (making the data smaller) and organizing its thinking process (using a tree map to guess multiple outcomes at once). The result is an advertising system that delivers the right ad to the right person almost instantly, without sacrificing the quality of the recommendation.

Technical Summary: Efficient LLM-based Advertising via Model Compression and Parallel Verification

Problem Statement

The integration of Large Language Models (LLMs) into online advertising, specifically for generative targeting (end-to-end, real-time ad generation based on user queries), has introduced a significant scalability-efficiency dilemma. While LLMs offer superior generalization and reasoning capabilities compared to traditional discriminative models, their expanding parameter counts and strict real-time latency requirements create a critical performance bottleneck.

Existing research has largely focused on precision-side optimizations (e.g., training strategies, prompting) but often neglects efficiency-side optimizations. This oversight is detrimental to time-sensitive commercial scenarios like online ad delivery, where high inference latency and computational costs render standard LLM deployments impractical. The core challenge is to optimize inference speed without compromising recommendation quality or precision.

Methodology

The authors propose a comprehensive framework comprising two primary components: Model Compression and Prefix Tree-based Parallel Verification.

1. Model Compression

This component aims to reduce model size and computational overhead through a combination of quantization and sparsity, optimized for GEMV (General Matrix-Vector Multiply) workloads.

Adaptive Group-Wise Quantization: Instead of using pre-determined grouping strategies, the authors propose an adaptive method that tailors quantization granularity to the sensitivity of individual linear layers.
- Sensitive layers/channels: Allocated more groups for finer quantization (minimizing error).
- Non-sensitive layers/channels: Allocated fewer groups to reduce computational overhead.
- This approach compresses the model from FP16 to INT4.
Layer-Adaptive Semi-Structured Sparsity: The framework employs a layer-wise $N:M$ $N : M$ pruning strategy based on the relative importance of parameters.
- Critical layers retain higher density (e.g., 2:4 sparsity).
- Less significant layers are pruned more aggressively (e.g., 1:4 sparsity).
- The importance of weight elements is quantified using a second-order Taylor expansion of the approximation error, considering both the weight magnitude and the input activation norm.
Custom SparseGemv Kernel: To address the overhead of sparse index data and memory bandwidth, the authors developed a custom kernel:
- Index Compression: They introduced an index-compressed 2bit-CSR structure. Indices are reordered, converted to binary, and encoded (e.g., hexadecimal) to reduce index and weight sizes to 30% of the original CSR size.
- Memory Optimization: Activations are moved from Local Memory to Shared Memory using a shuffle-based data layout to eliminate bank conflicts. Block-level data sharing is used to improve HBM-to-Shared Memory transfer efficiency.
- Computational Optimization: The kernel utilizes the lop3.b32 assembly instruction for efficient INT4-to-FP16 dequantization and optimizes CSR indexing with bit operations for fast access.

2. Prefix Tree Parallel Verification

This component accelerates decoding by leveraging the specific data patterns of advertising targets.

Prefix Tree Construction: Using hierarchical clustering (inspired by Differentiable Search Index, DSI), plain-text advertiser identifiers are transformed into a semantically structured prefix tree (trie). The tree is wide at the top and narrow at the bottom, reducing the candidate search space as decoding progresses.
Dynamic Trigger Mechanism: The system dynamically determines the optimal moment to switch from sequential to parallel verification. It calculates the trade-off between the overhead of parallel verification and the time required for sequential autoregressive steps, triggering parallelism when the predicted benefit is maximized.
Tree-Based Decoding: The process involves:
1. Constructing input sequences via Depth-First Search (DFS) on the trie.
2. Parallel decoding of these sequences with tree-based constraints (valid paths only).
3. Beam Search: Using logarithmic summation to combine parent node scores with token scores ( $T_{score}$ ) to compute node scores ( $B_{score}$ ), selecting the top sequences.
4. Reconstructing the final sequence by tracing back the highest-scoring nodes.

Key Contributions

The paper claims the following specific contributions:

Index-Compressed 2bit-CSR: An optimized data structure for mixed sparsity that reduces index and weight sizes to 30% of the original CSR.
Custom SparseGemv Kernel: A hardware-accelerated kernel supporting INT4 sparse matrix multiplication, filling a gap left by existing NVIDIA libraries (cuSPARSE/cuSparseLT) for efficient GEMV operations.
Adaptive Group-Wise Quantization: A novel quantization strategy that enhances flexibility and precision by adjusting group sizes based on layer sensitivity.
Prefix Tree-Constrained Beam Search: The first proposed workflow combining prefix tree-constrained decoding with beam search, specifically applied to generative tasks in advertising targeting.
Production Deployment: The framework has been deployed in Baidu's advertising platform, serving real-time traffic.

Experimental Results

The framework was evaluated using the ERNIE 1.5B model on two scenarios: Ad Creative Generation and Targeted Advertising.

Targeted Advertising:
- The complete framework (Sparse + Quant + Parallel Tree Verification) achieved a Recall of 0.081 with an average length of 6, compared to a baseline recall of 0.084.
- The combination of techniques ensured significant speedup while maintaining acceptable recall levels suitable for online deployment.
Ad Creative Generation:
- Evaluated on the CSL dataset (academic journal metadata).
- The hybrid approach (Sparse + Quant) achieved the best trade-off between efficiency and quality.
- Ablation Studies:
  - Quantization (INT4): Reduced latency from 6.6s to 4.8s (1.37× speedup) with minimal impact on BLEU (0.4247 $\to$ 0.4178) and Meteor scores.
  - Sparsification (2:4): Provided moderate gains (1.25× speedup) with slight quality loss.
  - Aggressive Sparsification (1:4): Delivered high speedup (1.43×) but caused noticeable quality degradation.
  - Combined (Sparse 2:4 + Quant): Achieved a 1.65× speedup with a Meteor score of 0.6195.
  - Combined (Sparse 1:4 + Quant): Achieved a 1.89× speedup but with significant quality drop.
  - Mixed Sparsity + Quant: Achieved a 1.78× speedup with a Meteor score of 0.6127.
Real-World Deployment:
- In production on Baidu's advertising platform, the system achieved a speedup of over 1.8× in real-world experiments while maintaining competitive precision.

Significance and Claims

The paper positions this work as a pioneering solution to the "Generative Targeting" problem in search advertising. Its significance lies in:

Bridging the Efficiency Gap: It addresses the critical bottleneck of inference latency in LLM-enhanced recommendation systems, making generative approaches viable for strict real-time commercial environments.
Holistic Optimization: Unlike prior works focusing solely on accuracy or simple compression, this framework integrates algorithmic innovations (adaptive quantization, hierarchical sparsity) with low-level system engineering (custom kernels, index compression) and business-specific logic (prefix tree construction).
Industrial Viability: The successful deployment in a leading advertising platform demonstrates that aggressive model compression and parallel verification strategies can be applied at scale without sacrificing the quality required for commercial ad delivery.

The authors note that while the framework is highly effective for commercial advertising targeting, its optimization strategies are tailored to this domain's specific data patterns and constraints, suggesting that further adaptation would be necessary for other application contexts.