💻 computer science

Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding

The paper proposes Domino, a speculative decoding framework that decouples causal dependency modeling from autoregressive drafting by combining a parallel backbone with a lightweight refinement head and a base-anchored training curriculum, achieving up to 5.8x throughput speedup on Qwen3 models.

Original authors: Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, Linfeng Zhang

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianuo Huang, Yaojie Zhang, Qituan Zhang, Hao Lin, Hanlin Xu, Linfeng Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess the next word in a story, but you are doing it with a very strict, slow rule: you must think of one word, write it down, check if it's correct, and then think of the next one. This is how current large AI models (LLMs) usually work. It's accurate, but it's like walking through a crowded room one step at a time, even though you have a super-fast team of runners ready to sprint.

The paper introduces a new method called Domino to make this process much faster. Here is how it works, using simple analogies:

The Problem: The "Speed vs. Accuracy" Trap

To speed things up, researchers use a trick called Speculative Decoding. Think of it like a "Drafting Team" (a smaller, faster AI) that guesses the next few words for the "Main Boss" (the big, slow AI) to check.

The Old Way (Autoregressive): The Drafting Team guesses one word, then the next, then the next, one by one. This is very accurate because they can see the previous word before guessing the next. But it's still slow because they have to wait for each step.
The "Parallel" Way: The Drafting Team guesses all the next words at once, like throwing a handful of cards onto a table. This is super fast, but the guesses are often messy or wrong because they didn't look at each other first.

The paper says: Why not have the speed of the "handful of cards" but the accuracy of the "one-by-one" method?

The Solution: The Domino Framework

The authors created Domino, which splits the job into two parts to get the best of both worlds.

1. The Parallel Backbone (The "Rough Draft")

First, Domino uses a fast, parallel engine to spit out a "rough draft" of the next few words all at once.

Analogy: Imagine a chef who quickly throws a bunch of ingredients onto a cutting board without chopping them yet. It's fast, but the ingredients aren't in the right order or shape.

2. The Domino Head (The "Lightweight Refiner")

This is the magic part. Instead of re-doing the whole cooking process (which is slow), Domino adds a tiny, specialized tool called the Domino Head.

Analogy: Imagine a sous-chef who quickly looks at the pile of ingredients. They don't cook the whole meal again; they just make tiny, precise adjustments to the order and shape of the ingredients based on what came before.
How it works: The Domino Head is "causal," meaning it understands that Word B depends on Word A. It takes the rough draft and adds a little "correction" to make sure the words flow logically, without having to wait for the slow, step-by-step process.

The Training Trick: "Teacher Forcing"

To teach this system, the authors used a clever training method.

The Problem: If you let the AI practice by guessing its own mistakes, it gets confused and learns bad habits.
The Fix: They used "Teacher Forcing." Imagine a teacher holding up the correct cards for the student to look at while they practice making corrections. This ensures the AI learns to fix the draft based on the right context, not on its own errors.
The Curriculum: They also taught the AI in stages. First, they made sure the "Rough Draft" (the parallel backbone) was strong. Then, they slowly let the "Refiner" (Domino Head) take over the fine-tuning. This prevented the AI from relying too much on the refiner and forgetting how to make a good rough draft in the first place.

The Results: Faster Without Losing Quality

The paper tested this on powerful AI models (Qwen3) and found:

Speed: It is significantly faster than previous methods. On some tasks, it was nearly 8 times faster than the standard way of doing things.
Efficiency: It manages to keep the "acceptance rate" high (meaning the Main Boss agrees with the Drafting Team's guesses most of the time) while keeping the drafting cost low.
Comparison: It beats other fast methods (like DFlash) and other accurate methods (like EAGLE) by combining their strengths.

Summary

Domino is like hiring a fast team to guess the next few words of a story all at once, but then adding a tiny, smart editor who quickly fixes the logic and flow before the final boss checks it. This allows the AI to run at the speed of a sprint while maintaining the accuracy of a careful walk.

Technical Summary: Domino

Problem Statement

Speculative decoding accelerates Large Language Model (LLM) inference by using a lightweight draft model to propose multiple future tokens, which are then verified in parallel by the target model. The effectiveness of this approach is governed by a fundamental trade-off between draft quality and drafting cost:

Autoregressive Drafting (e.g., EAGLE series): These methods generate draft tokens sequentially, explicitly modeling causal dependencies within the draft block. This yields high-quality drafts and long acceptance lengths. However, generating $k$ tokens requires $k$ sequential forward passes and $k$ full-vocabulary LM-head projections, incurring significant sequential overhead that limits speedup.
Parallel Drafting (e.g., DFlash): These methods generate an entire draft block in a single forward pass, eliminating sequential overhead and maximizing GPU utilization. However, by removing sequential dependencies, they weaken intra-block causal modeling, often resulting in lower acceptance lengths compared to autoregressive methods.

Existing approaches struggle to balance these factors. Reducing vocabulary projection costs (e.g., FR-Spec) or using tree-based verification does not fully resolve the tension between the sequential cost of causal modeling and the quality loss of parallel generation. The central question addressed is: Can we retain the draft-quality benefits of causal dependency modeling while preserving the low-cost efficiency of block-parallel generation?

Methodology: Domino

The authors propose Domino, a speculative decoding framework that decouples causal dependency modeling from expensive autoregressive draft execution. The architecture consists of two primary components:

1. Parallel Draft Backbone

Domino utilizes a parallel draft backbone (instantiated via the DFlash architecture) to generate preliminary draft distributions for the entire block in a single non-autoregressive forward pass.

Input: The backbone takes target-model context features from the verified prefix and a masked draft block (containing the last verified token followed by [MASK] tokens).
Output: It produces hidden states for the entire block, from which base logits ( $L^{base}_i$ ) are computed using a frozen target LM head.

2. Domino Head (Lightweight Causal Correction)

To reintroduce causal information without sequential execution, Domino applies a lightweight correction head on top of the parallel base logits. This head operates sequentially but is computationally cheap:

Causal Encoder: For each draft position $i$ , a lightweight GRU summarizes the embeddings of preceding drafted tokens (or ground-truth tokens during training) into a causal state $S_{i-1}$ . This state captures prefix-dependent information without invoking a full autoregressive model.
Low-Rank Correction Head: The head computes a residual correction in logit space using a low-rank bottleneck. It concatenates the base hidden state $H_i$ and the causal state $S_{i-1}$ , projects them into a low-rank space (dimension $r=256$ ), and maps them back to the vocabulary space to produce correction logits $\Delta L_i$ .
Final Logits: The final draft logits are computed as $L_i = L^{base}_i + \Delta L_i$ . Tokens are sampled from this refined distribution.

Crucially, the correction is applied in logit space rather than hidden space. This avoids the need to re-run the full LM head after every causal update, keeping the base computation parallel and restricting the sequential branch to a low-rank residual update.

Training Strategy

To stabilize training and prevent the correction branch from "shortcutting" the backbone, the authors introduce two key strategies:

Teacher-Forced Causal Encoding: Unlike methods that use self-generated prefixes (Training-Time Testing), Domino feeds the causal encoder ground-truth token embeddings. This aligns the training distribution with the speculative decoding acceptance mechanism, where corrections only matter when preceding tokens are verified as correct.
Base-Anchored Curriculum: To prevent the parallel backbone from collapsing (i.e., becoming weak because the correction branch compensates for it), the training loss is a weighted sum of the base logits loss and the final logits loss. The weight $\lambda_t$ is linearly annealed from 1 to 0, initially forcing the backbone to learn a strong base distribution before gradually shifting optimization toward the final corrected distribution.

Key Contributions

Decoupling Causal Modeling from Execution: Domino separates the modeling of causal dependencies from the expensive execution of autoregressive steps, allowing for causal refinement without sequential latency.
Lightweight Causal Correction: By using a GRU-based encoder and a low-rank bottleneck in logit space, Domino injects prefix-dependent information with minimal parameter overhead (adding only ~5.3% parameters to the backbone) and negligible latency increase.
Stabilized Training Curriculum: The introduction of teacher forcing combined with a base-anchored curriculum loss prevents backbone collapse and ensures robust learning of both the parallel base distribution and the causal corrections.

Experimental Results

Experiments were conducted on Qwen3-4B and Qwen3-8B models across math reasoning (GSM8K, MATH, AIME25), code generation (HumanEval, MBPP, LiveCodeBench), and dialogue (MT-Bench, Alpaca) benchmarks.

End-to-End Speedup: Under the Transformers backend, Domino achieves up to 5.49× speedup on Qwen3-8B (greedy decoding), outperforming DFlash (4.66×) and EAGLE-3 (2.26×). On GSM8K specifically, it reaches 7.92× speedup.
Acceptance Length: Domino significantly improves the average acceptance length compared to parallel baselines (e.g., increasing from 6.59 to 10.03 on Qwen3-8B for GSM8K), narrowing the gap with autoregressive methods while maintaining parallel efficiency.
High-Concurrency Throughput: Using SGLang, Domino demonstrates superior throughput across varying concurrency levels (2 to 32 requests), achieving up to 5.8× throughput speedup over autoregressive decoding.
Ablation Studies:
- Removing the Domino head reduces the average speedup from 3.31× to 2.84×, confirming the efficacy of the causal correction.
- Using teacher forcing instead of training-time testing improves acceptance length (3.80 to 3.96).
- The base-anchored curriculum further boosts acceptance length (3.96 to 4.19) and prevents backbone loss collapse.

Significance

The paper claims that Domino demonstrates that causal information can be effectively reintroduced into parallel drafting to achieve faster LLM inference. By decoupling the modeling of dependencies from the execution cost, Domino achieves a superior balance between draft quality and drafting overhead. The results suggest that lightweight causal correction is a viable path to overcoming the limitations of both purely autoregressive and purely parallel speculative decoding strategies, offering a practical solution for accelerating LLM serving without compromising output distribution fidelity.