💬 NLP

PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length

The paper introduces PACER, a novel speculative decoding framework that employs a lightweight, trainable pre-verification layer to dynamically adjust draft token lengths blockwise, thereby significantly accelerating LLM inference and outperforming standard fixed-length approaches.

Original authors: Situo Zhang, Yifan Zhang, Zichen Zhu, Hankun Wang, Da Ma, Danyang Zhang, Lu Chen, Kai Yu

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Situo Zhang, Yifan Zhang, Zichen Zhu, Hankun Wang, Da Ma, Danyang Zhang, Lu Chen, Kai Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a long story, but you are working with a very strict editor. In the world of Large Language Models (LLMs), this "editor" is the Target Model. It is incredibly smart and accurate, but it is also very slow and expensive to run. Every time it checks a single word you write, it takes a long time.

To speed things up, we usually use a Draft Model. Think of this as a fast, energetic intern who is good at guessing what comes next but isn't perfect.

The Old Way: The "Fixed Guess" Game

In standard Speculative Decoding, the process works like this:

The Intern (Draft Model) quickly writes down a fixed number of words (say, 5 words) in a row.
The Editor (Target Model) then reads those 5 words all at once to see if they are correct.
The Problem: The Editor is a bit picky.
- If the Intern guesses too many words (e.g., 9), the Editor might reject the 6th word. This means the 7th, 8th, and 9th words the Intern wrote were a waste of time.
- If the Intern guesses too few words (e.g., 2), the Editor has to stop and check very often, slowing everything down because the Editor is the bottleneck.

The paper points out that the "perfect" number of words to guess changes constantly. Sometimes the Intern is on a roll and can guess 10 words correctly; other times, they get stuck after just 1. Using a fixed number (like always guessing 5) is like trying to fit a square peg in a round hole—it's inefficient.

The New Solution: PACER (The "Smart Block Checker")

The authors propose a new system called PACER. Instead of letting the Intern guess a fixed number of words, PACER adds a Pre-verification Layer. Think of this as a Team Lead who stands between the Intern and the Editor.

Here is how PACER works, step-by-step:

The Intern writes in small chunks (Blocks): Instead of writing 5 words at once, the Intern writes a small block of 3 words.
The Team Lead checks the block: Before sending these 3 words to the slow Editor, the Team Lead (a tiny, fast AI) quickly checks them.
- The Team Lead asks: "Do these 3 words look like they will pass the Editor's test?"
The Decision:
- If the Team Lead says "Yes": The Intern immediately writes the next block of 3 words. The Team Lead checks that one too. This continues, building a long chain of correct words very quickly.
- If the Team Lead says "No": The Intern stops immediately. The Team Lead sends the accepted words to the Editor for the final official check. The Intern doesn't waste time writing the rest of the block that was likely to be rejected.

Why is this better?

The paper uses a great analogy of traffic flow.

Old Method: You drive at a fixed speed. Sometimes the road is clear, and you could have gone faster. Sometimes there's a red light, and you kept driving into it, wasting gas.
PACER: You have a smart navigator. If the road ahead looks clear, you speed up and drive further. If the navigator sees a red light coming up, you stop before you hit it, saving gas and time.

The Results

The paper tested this on various tasks like writing code, solving math problems, and summarizing news.

Speed: PACER made the system run up to 2.66 times faster than the standard way of doing things.
Combination: When they combined PACER with another speed-boosting technique called "Ouroboros," the system became 3.09 times faster.
Efficiency: It successfully reduced the number of times the slow Editor had to work, while making the fast Intern work smarter, not just harder.

In Summary

PACER is like giving your fast AI intern a smart supervisor. Instead of guessing a fixed amount of text and hoping for the best, the supervisor checks small batches of work in real-time. If the work looks good, they keep going; if it looks risky, they stop immediately. This prevents wasted effort and gets the final result to the user much faster.

Technical Summary: PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length

Problem Statement
Large Language Models (LLMs) suffer from significant inference latency due to their autoregressive nature, which is primarily constrained by memory bandwidth rather than computational limits. Speculative Decoding (SD) addresses this by using a smaller "draft" model to predict multiple tokens ahead, which are then verified in parallel by the larger "target" model. However, existing SD approaches typically employ a fixed draft window size ( $\gamma$ ) throughout the entire decoding process.

The authors identify a critical inefficiency in this fixed-window approach: the optimal number of draft tokens varies significantly across different decoding steps.

Conservative Windows: A fixed window that is too small causes the draft model to stop prematurely, missing opportunities to generate accepted tokens and forcing unnecessary, costly forward passes by the target model.
Aggressive Windows: A fixed window that is too large leads to wasted computation on draft tokens that are ultimately rejected by the target model.
Current methods attempting to adapt window sizes often rely on intrinsic draft model outputs (e.g., token probabilities or entropy), which can be unreliable due to model miscalibration and overconfidence. Other parallel approaches (e.g., PEARL) struggle with resource scheduling when models share the same device.

Methodology: PACER
The paper proposes PACER (Pre-verification for Adaptive Length), a framework that dynamically controls the draft window size using a lightweight, trainable blockwise pre-verification layer.

Blockwise Generation and Pre-verification: Instead of generating a fixed number of tokens, the draft model generates tokens in blocks of size $b$ . After each block is generated, a pre-verification layer ( $M_B$ ) analyzes the hidden states of the block along with positional embeddings to estimate the acceptance probability of the tokens.
Adaptive Stopping Criterion: The system calculates the mean estimated acceptance rate for the block. If this rate falls below a dynamic threshold $t$ , the draft generation stops, and the accumulated tokens are sent to the target model for formal verification. If the rate is high, the draft model continues to generate the next block.
Dynamic Thresholding: To prevent the draft model from generating excessively long sequences where acceptance probability naturally decays, the threshold $t$ is increased by a growth factor $\rho$ after each successful block. This makes the stopping criterion progressively stricter as the draft length increases.
Training: The pre-verification layer is trained using standard cross-entropy loss. The training data is constructed by simulating the speculative decoding process on the target model to label draft tokens as "accepted" or "rejected," ensuring alignment between training and inference. To improve efficiency, training data from multiple decoding steps is packed into single sequences with custom attention masks.

Key Contributions

Systematic Analysis: The authors provide empirical evidence showing that acceptance lengths vary significantly across decoding steps, demonstrating that fixed window sizes are suboptimal and that dynamic adjustment can theoretically yield up to $1.4\times$ speedup.
Novel Framework: PACER introduces a trainable, blockwise pre-verification mechanism that leverages contextual information and positional encoding to dynamically determine the optimal draft length, avoiding reliance on unreliable heuristic metrics like entropy.
Compatibility: The framework is designed to be orthogonal to other speculative decoding improvements, such as Ouroboros (which improves draft quality via phrase-level generation).
Comprehensive Evaluation: Extensive experiments are conducted across multiple model families (DeepSeek-Coder, Llama-2, Qwen-2.5) and diverse benchmarks (code generation, math reasoning, summarization).

Results

Speedup: PACER achieves up to $2.66\times$ speedup over vanilla autoregressive decoding and consistently outperforms standard speculative decoding with fixed window sizes across all tested benchmarks.
Integration: When integrated with Ouroboros, PACER achieves a peak speedup of $3.09\times$ over autoregressive decoding on the GSM8K dataset.
Efficiency: The pre-verification layer adds minimal overhead (approximately $2.1\%$ of total inference time in DeepSeek-Coder 1.3B/33B experiments) while significantly reducing the number of target model forward passes.
Ablation Studies: The study confirms that both positional embeddings and the threshold growth factor are critical components; removing them leads to reduced decoding speeds or inefficient draft lengths.

Significance
The paper claims that PACER effectively addresses the limitations of fixed-window speculative decoding by introducing a data-driven, adaptive mechanism. By dynamically regulating the draft length based on contextual pre-verification, PACER minimizes computational waste from rejected drafts and reduces the overhead of frequent target model verification. The authors position PACER not as a replacement for high-quality draft generation techniques, but as a complementary layer that optimizes the length of the speculation, thereby unlocking further performance gains in LLM inference acceleration.

The Old Way: The "Fixed Guess" Game

The New Solution: PACER (The "Smart Block Checker")

Why is this better?

The Results

In Summary

More like this