💬 NLP

LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding

LiLiCorr is a lightweight method that enhances speculative decoding by efficiently correlating a drafter's per-position marginal distributions to capture joint token coherence, thereby significantly increasing acceptance lengths and overall throughput with minimal latency overhead.

Original authors: Matan Rusanovsky, Yoav Miron, Roy Uziel, Omer Belhasin, Ran Zilberstein, Maor Ashkenazi, Michael Elad

Published 2026-08-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Matan Rusanovsky, Yoav Miron, Roy Uziel, Omer Belhasin, Ran Zilberstein, Maor Ashkenazi, Michael Elad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where reading a book is instantaneous, not because the words appear faster, but because your mind can anticipate entire paragraphs before you even turn the page. This is the promise of a technique called speculative decoding, a method designed to make artificial intelligence speak and write with unprecedented speed. At its core, the process relies on a simple division of labor: a small, fast model acts as a draftsman, guessing what comes next, while a larger, more powerful model acts as a judge, verifying those guesses. If the judge agrees with the draftsman's prediction, the system accepts the guess and moves forward, skipping the slow, step-by-step work of generating every single word from scratch. The faster the draftsman can be trusted to get a long string of words right, the faster the entire conversation flows. However, a persistent problem has held this technology back: the draftsman often gets the individual words right but fails to make them fit together into a coherent sentence. It is like a musician who plays every note perfectly in isolation but creates a jarring, nonsensical melody when the notes are played in sequence.

Researchers at NVIDIA have introduced a new method called LiLiCorr to solve this specific problem of coherence without sacrificing speed. The system builds upon a drafting model known as DFlash, which is capable of predicting a whole block of future words in a single, rapid burst. While DFlash is incredibly fast, it treats each word position as an independent event, meaning it might suggest "The dog" for the first slot and "meows" for the second, but fail to realize that "The cat" would have been a better fit for the context of the whole sentence. The new approach does not try to retrain the draftsman to be perfect; instead, it adds a lightweight layer of logic that acts as a quick editor. This editor looks at the top few candidates the draftsman proposed for each position and checks how well they link together. It assigns a pair of directional signals to every candidate word, one indicating how well it fits with the word before it, and another indicating how well it sets up the word after it. By comparing these signals, the system can instantly identify which sequence of words forms a smooth, logical path, discarding the jarring combinations that would have caused the larger model to reject them.

The brilliance of this design lies in its efficiency. Rather than checking each word one by one in a slow, sequential chain, the system evaluates all possible connections between the candidate words simultaneously, using a single, massive calculation that happens in parallel. This allows the system to find the most coherent sequence almost instantly, leaving only a simple, final step to lock in the choices. The researchers found that by training this editor to work hand-in-hand with the draftsman, the two models learned to cooperate, with the draftsman eventually proposing candidates that were easier for the editor to link into long, accepted chains. In tests across nine different benchmarks, ranging from solving math problems to writing code and holding conversations, this new method consistently outperformed previous approaches. It increased the number of accepted words by between nine and nineteen percent compared to the unedited draftsman, and it delivered the highest overall speed in seventy out of seventy-two different testing scenarios. Even when the system was asked to handle inputs ten times longer than the data it was trained on, it maintained its lead, proving that this method of linking candidates is robust and scalable.

The results suggest that the bottleneck in speeding up AI is not just about making the draftsman faster, but about making the draftsman's guesses easier to verify. By fixing the coherence of the guesses before they are even sent to the judge, the system avoids the wasted time of rejection and re-generation. This approach does not require the massive computational power of the main model to do the heavy lifting of correlation; instead, it uses a tiny, specialized network that adds a negligible amount of time to the process—accounting for only about 2.8 percent of the total time per block of text. The researchers demonstrated that this small addition yields a massive return, allowing the system to process information significantly faster than before. While other methods have attempted to solve this by running complex checks on every single word or by asking the main model to do extra work, LiLiCorr achieves its results by organizing the information the draftsman already provides into a structure that is easy to navigate. The findings indicate that for the future of high-speed AI, the key may not be in building bigger models, but in building smarter, more efficient ways to connect the dots between the words they generate.

Technical Summary: LiLiCorr

Problem Statement

Speculative decoding accelerates autoregressive language model inference by using a smaller "drafter" model to propose future tokens, which a larger "target" model verifies in parallel. A promising class of drafters, such as DFlash, predicts an entire block of future tokens in a single forward pass using a diffusion-style architecture. However, these models are trained on per-position marginals (predicting the probability of a token at a specific position independently) rather than the joint block distribution.

Consequently, while the tokens emitted by DFlash are individually plausible, they are often jointly incoherent. Selecting the most probable token at each slot independently frequently results in a sequence that the target model rejects early, limiting the acceptance length and the resulting speedup. Recovering a coherent sequence from these marginals is combinatorially difficult because the joint distribution over a block of $B$ positions with vocabulary size $V$ has $V^B$ realizations, making explicit modeling intractable. Existing solutions either rely on iterative refinement (adding costly drafter passes), shift the correlation burden to the target model (defeating the purpose of speculation), or use sequential autoregressive correction networks that process slots one by one.

Methodology: LiLiCorr

LiLiCorr (Lightweight Likelihood Correlation) addresses this by recovering a coherent sequence from per-slot marginals in a single network pass. It treats the problem as learning a scoring model over a candidate lattice rather than modeling the full joint distribution.

1. Candidate Lattice and Representation

Instead of selecting a single token per slot, LiLiCorr retains the top- $K$ candidate tokens from the drafter's marginal distribution at each of the $B-1$ speculative slots.

Node Construction: Each candidate token is represented as a vector (node) formed by summing four components:
1. The frozen target model's input embedding of the token.
2. The drafter's hidden state at that slot (shared across the slot's candidates).
3. A small network processing scalar features of the drafter's confidence (log-prob, rank, etc.).
4. Learned encodings for the slot index and the candidate's rank.
Contextual Processing: A lightweight bidirectional Transformer processes the entire $(B-1) \times K$ lattice jointly. This allows every candidate to exchange information with all others (both competing candidates within a slot and candidates at neighboring slots), encoding the context of the whole block.

2. Correlation via In/Out Vectors

For each candidate node, LiLiCorr emits two normalized vectors: an in vector ( $i$ ) and an out vector ( $o$ ).

Pairwise Scoring: Adjacent slots $s$ and $s+1$ are coupled by computing the cosine similarity between the out vector of a candidate at $s$ and the in vector of a candidate at $s+1$ .
Efficiency: This formulation reduces the correlation logic to batched matrix multiplications. All pairwise interactions for the entire block are computed in parallel after a single forward pass.
Stability: Using cosine similarity (scaled by a constant $\sigma$ ) prevents unbounded score magnitudes that could destabilize the shared DFlash backbone during joint training.

3. Training Objectives

LiLiCorr is co-trained with the DFlash drafter. The training loss consists of three components:

DFlash Loss: The standard per-slot cross-entropy for the drafter.
Oracle-Prefix Cross-Entropy ( $L_{CE}$ ): A teacher-forced loss that trains LiLiCorr to rank the ground-truth token highest, conditioned on the ground-truth token of the previous slot. Crucially, supervision is only applied to the "oracle prefix"—the longest leading run of slots where the ground truth is present in the top- $K$ candidates.
Distractor Penalty ( $L_{dist}$ ): A cost-aware term that penalizes candidates the target model ranks low. It minimizes the expected target cost relative to the ground truth, specifically pushing down "confusers" that the target model would reject, thereby widening the margin for the correct token.

4. Decoding

At inference, the process is two-staged:

Parallel Scoring: A single LiLiCorr pass computes all in/out vectors and all pairwise matching scores in parallel.
Sequential Greedy Walk: The block is committed left-to-right. At each slot, the algorithm selects the candidate that maximizes the match with the previously committed candidate. This greedy walk is sequential but touches no network weights, making it computationally cheap.

Key Contributions

Novel Formulation: A likelihood-based approach to recovering coherent sequences from per-slot marginals by learning a scoring model over candidate paths, avoiding the intractability of the full joint distribution.
Parallel Correlation: The correlation logic is reduced to efficient, parallelizable matrix operations. Unlike prior draft-time approaches that run a correction network once per slot, LiLiCorr scores the entire candidate lattice in a single pass, followed by a cheap greedy selection.
State-of-the-Art Performance: The method sets new benchmarks for speculative decoding, outperforming concurrent works (Domino, DSpark) and the vanilla DFlash drafter across diverse settings.

Experimental Results

The authors evaluated LiLiCorr on nine benchmarks (math, code, chat) using Qwen3-8B and Qwen3-4B target models under both greedy and temperature-1 decoding.

Acceptance Length: Compared to the vanilla DFlash drafter, LiLiCorr increases acceptance length by 9% to 19% across all benchmarks.
Throughput: LiLiCorr achieves the highest throughput in 70 out of 72 tested settings (varying benchmarks, target sizes, decoding regimes, and concurrency levels).
- It outperforms the strongest baseline (Domino) by up to 5.3% on average.
- The LiLiCorr head accounts for only 2.8% of the per-block latency and is approximately 2.4× cheaper than Domino's head when fully optimized.
Generalization: The performance lead persists when serving inputs an order of magnitude longer than the training range (up to 32K tokens), provided positional encoding is extended (e.g., via YaRN).
Ablation: The distractor penalty is shown to be critical, particularly for short prompts, raising acceptance length by an additional 1.5% to 8.9% on benchmarks.

Significance and Claims

The paper claims that LiLiCorr represents a structural improvement in speculative decoding by decoupling the correlation of draft tokens from sequential autoregressive passes. By performing all correlation work in a single parallel pass, it minimizes the overhead of the drafter while maximizing the acceptance length.

The authors emphasize that the method is modest in its scope: it relies on the drafter's ability to propose a candidate pool containing the ground truth (the "oracle prefix"). If the correct token is not in the top- $K$ candidates, LiLiCorr cannot recover it. However, within the constraints of the candidate pool, it provides a highly efficient mechanism to restore joint coherence, making it the fastest system across a wide range of serving conditions, including high-concurrency and long-context scenarios. The work suggests that future improvements could explore coupling slots beyond adjacent pairs, though the paper notes the open question of whether such longer-range couplings justify their cost under prefix acceptance.