Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion

Technical Summary: Speculative Decoding at Temperature Zero

Problem Statement
Speculative decoding accelerates Large Language Model (LLM) inference by using a smaller "draft" model to propose tokens for a larger "target" model to verify. While this improves throughput, a concrete safety question arises: at temperature zero ($T=0$), can the alignment profile of a weaker or adversarial draft model leak into the final, safety-scored outputs of the target model? Previous work has identified speculative decoding as a security surface for timing side-channels (Wei et al. [36]), but it remains empirically untested whether the acceptance pathway itself allows draft-side behavioral changes (such as reduced refusal rates) to bypass the target model's verification. This paper investigates whether greedy speculative decoding introduces detectable safety divergence compared to target-only decoding.

Methodology: The TAIS Screen
The authors introduce the Typical-Acceptance Invariance Screen (TAIS), a behavioral-equivalence protocol designed to establish a "bounded null" for safety divergence. TAIS pairs target-only and speculative outputs on identical inputs from a fixed safety battery and evaluates them against three criteria:

1. Byte-Identity: At $T=0$, outputs are compared byte-for-byte. A "strong" flag requires $\ge 99.5\%$ identity; a "moderate" flag allows lower identity if other criteria pass.
2. TOST Equivalence: Two One-Sided Tests (TOST) are performed to reject the hypothesis that the safety rate difference exceeds $\pm 3$ percentage points (pp).
3. Effect Size (Cohen's $h$): The per-task effect size $|h_t|$ must be below a calibrated cutoff of $h^\star = 0.1$. This cutoff is set above the loudest observed null-consistent contrast ($|h|=0.024$) and below the conventional "trivial effect" floor ($|h|=0.2$).

Experimental Design and Scope
The study employs a five-phase factorial design across 16,783 confirmatory samples and a 44,066-sample matched expansion, totaling over 60,000 samples.

• Models: Llama 3.x and Qwen 2.5 families (e.g., Llama-3.2-3B target with 1B drafts; Llama-3.1-70B target with 8B draft).
• Draft Variations:
  • Canonical: Standard drafts.
  • Adversarial (E2): A draft fine-tuned with Direct Preference Optimization (DPO) on flipped Anthropic/hh-rlhf preferences to explicitly prefer harmful completions.
  • Quantized (E3): GPTQ-4bit drafts.
  • Seeds (E4): Replication across two random seeds.
  • Precision (E5): Switching accumulator dtype from fp16 to bf16.
• Benchmarks: AdvBench (refusal), Jailbreak amplification, BBQ (bias), TruthfulQA, MMLU, and ARC-Challenge.
• Configuration: vLLM v0.19, greedy decoding ($T=0$), using both strict rejection sampling and probabilistic typical acceptance.

Key Results

1. No Detectable Safety Divergence: Across the matched TAIS probes (E0, E2–E5), greedy speculative decoding shows no detectable safety divergence from target-only decoding.
   • The maximum absolute Cohen's $h$ observed across all matched AdvBench refusal contrasts is 0.024, roughly an order of magnitude below the trivial-effect floor.
   • 25 of 27 per-task TOST contrasts pass the $\pm 3$pp equivalence margin. The two non-passing cases are identified as capability-domain edge cases where both arms hit the ceiling ($p_{target} = p_{spec} = 1.0$), resulting in a degenerate Wald confidence interval, not genuine non-equivalence.
2. Adversarial Draft Invariance: The DPO-adversarial draft (E2), which independently prefers harmful completions, produced byte-identical output to the canonical draft across 4,006 samples. This confirms that at $T=0$ with strict rejection sampling, the draft's alignment profile does not leak into the verified output.
3. Precision and Quantization:
   • Quantization (E3): GPTQ-4bit drafts also produced byte-identical outputs to the canonical draft.
   • Data Types (E5): Switching to bf16 changed 36%–53% of output bytes due to numerical non-associativity, yet no per-task safety rate moved outside the equivalence bounds.
4. Production-Scale Sanity Check (E1): A separate 70B target probe (Llama-3.1-70B + 8B draft) without a matched target-only arm showed an AdvBench refusal rate of 0.839 (95% CI [0.809, 0.864]). While this falls within expected ranges, it is reported as a scale probe rather than a TAIS equivalence claim due to the lack of a matched control.

Significance and Claims
The paper claims a scoped behavioral null: under the specific conditions of temperature zero, vLLM v0.19, and the tested model families, speculative decoding does not introduce safety regressions.

• Operational Validation: The byte-identity result on the adversarial draft is framed as an operational validation of the distributional guarantee of speculative decoding (Leviathan et al. [24]) at $T=0$, demonstrating that the acceptance pathway has no measurable behavioral channel for safety leakage in this regime.
• TAIS as a Protocol: The paper contributes TAIS as a reusable, auditable protocol for vendors and users to verify safety equivalence before deploying speculative decoding stacks.
• Scope Boundaries: The authors explicitly limit their claims. The null applies only to $T=0$, greedy decoding, and the tested frameworks. It does not extend to sampling temperatures $T>0$, tree-speculation variants (e.g., EAGLE, Medusa), or untested model families. The paper cautions against overgeneralizing these results to regimes where the draft's distributional influence might be stronger.

In summary, the study provides empirical evidence that, for greedy inference, the safety properties of the target model are preserved even when using adversarially aligned or quantized draft models, provided the system operates at temperature zero.