💬 NLP

SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification

The paper proposes SelfJudge, a self-supervised method that trains judge verifiers using the target model itself to verify speculative decoding candidates based on semantic preservation, thereby enabling faster and more generalizable LLM inference without relying on human annotations or ground truth.

Original authors: Kanghoon Yoon, Minsub Kim, Sungjae Lee, Joonhyung Lee, Sunghyeon Woo, Yeonjun In, Se Jung Kwon, Chanyoung Park, Dongsoo Lee

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kanghoon Yoon, Minsub Kim, Sungjae Lee, Joonhyung Lee, Sunghyeon Woo, Yeonjun In, Se Jung Kwon, Chanyoung Park, Dongsoo Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a long, complex story, but you are in a hurry. You have a Master Writer (a huge, powerful AI) who writes perfect stories but is very slow because they think carefully about every single word before writing it. You also have a Speedy Assistant (a smaller, faster AI) who can guess the next few words very quickly, but sometimes they make small mistakes or use slightly different phrasing.

The Old Way: The Strict Editor
In the past, to speed things up, you would let the Speedy Assistant write a whole paragraph ahead of time. Then, the Master Writer would check it. However, the Master Writer was a strict editor. If the Assistant wrote "The cat sat on the mat" and the Master Writer thought "The cat was sitting on the mat," the strict editor would reject the whole sentence and make the Assistant start over. This wasted a lot of time because the meaning was exactly the same, even though the words were slightly different.

The "Judge" Idea: A More Flexible Editor
Researchers recently tried to fix this by hiring a "Judge." This Judge would look at the Assistant's words and decide, "Is this close enough in meaning to the Master's style?" If yes, they would accept it. This was great, but there was a catch: to train this Judge, you needed a human to sit down and manually label thousands of sentences saying, "Yes, this word is okay," or "No, this word is wrong." This was slow, expensive, and only worked well for tasks with clear right-or-wrong answers (like math or coding). It didn't work well for creative writing or general questions where there is no single "correct" answer.

The New Solution: SelfJudge (The Self-Taught Judge)
This paper introduces SelfJudge, a new way to train the Judge without needing any humans.

Here is how it works, using a simple analogy:

The Self-Test: Instead of asking a human for help, the Master Writer (the Target Model) acts as its own teacher.
The "What If" Game: The system takes a sentence the Master Writer wrote and swaps out one word with a different word the Speedy Assistant suggested.
- Original: "The cat was sitting on the mat."
- Swapped: "The cat sat on the mat."
The Confidence Check: The Master Writer then asks itself: "Does this change the meaning of my story? Do I still feel confident about the rest of the sentence?"
- If the Master Writer says, "No, the meaning is the same, I'm still confident," the system marks that word swap as Acceptable.
- If the Master Writer says, "Whoa, that changes the meaning," it marks it as Reject.
Training the Judge: By doing this thousands of times automatically, the system creates a massive dataset of "Acceptable" and "Unacceptable" words. It then trains a tiny, super-fast "Judge" to recognize these patterns.

The Result
Now, when the Speedy Assistant writes, this tiny Judge can instantly say, "Yes, that word is fine, even if it's not the exact one the Master Writer would have picked, because the meaning is preserved."

Why This Matters

It's Universal: Unlike previous methods that only worked for math or coding (where you can check the final answer), SelfJudge works for anything—creative writing, summarizing news, or answering general questions—because it cares about meaning, not just a "correct" answer.
It's Fast: The tiny Judge is so light that it adds almost no time to the process.
It's Accurate: The paper shows that this method speeds up the writing process significantly (accepting more words from the Assistant) without making the final story worse.

In short, SelfJudge teaches the AI to trust its own intuition about what words "feel" right, allowing it to skip the slow, strict checking process and write much faster without losing quality.

Technical Summary: SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification

1. Problem Statement

Large Language Models (LLMs) face significant inference latency due to the autoregressive generation process, which requires accessing all model parameters for every forward pass. Speculative Decoding (SD) addresses this by using a smaller "draft" model to propose candidate tokens, which are then verified in parallel by the larger "target" model.

However, standard SD relies on alignment-based verification (e.g., rejection sampling), which accepts draft tokens only if their probability under the target model meets or exceeds that of the draft model. This approach is overly conservative; it rejects tokens that are semantically coherent but exhibit minor lexical discrepancies (e.g., "It's" vs. "It is") from the target model's specific output distribution.

Recent "Judge Decoding" methods attempt to relax these criteria by using a learnable verifier to accept semantically compatible tokens. Yet, existing approaches face critical limitations:

Reliance on Human Annotation: Methods like JudgeDecoding require subjective human labels for token acceptance, introducing bias and scalability issues.
Dependence on Ground Truth: Automated methods like AutoJudge rely on task-specific verifiable ground truths (e.g., final answers in math or code). As demonstrated in the paper, these methods fail to generalize to open-ended tasks (e.g., MMLU, summarization) where no single correct answer exists, leading to significant performance degradation in general NLP domains.

2. Methodology: SelfJudge

The authors propose SelfJudge, a framework that enables judge verification across diverse NLP tasks without human supervision or verifiable ground truths. The core innovation is a self-supervised training paradigm where the target model itself generates the training data for the verifier.

2.1. Semantic Preservation Token Labeling

Instead of checking for answer correctness, SelfJudge measures semantic preservation. The process involves three steps:

Identify Mismatched Tokens: The target model generates a response $y$ . The draft model's predictions are compared against the target's output. Tokens where the draft model's top prediction ( $z_i$ ) differs from the target's token ( $y_i$ ) are identified as candidates for relaxed verification.
Compute Semantic Preservation Score: For each mismatched token, the system calculates a score $s(y, z_i)$ representing how well the replacement preserves the original response's meaning. This is defined as the log-likelihood difference between the original response and the token-substituted response under the target model's distribution:
$s(y, z_i) = \log P(z_i | y_{<i}, y_{>i}) - \log P(y_i | y_{<i}, y_{>i})$
Crucially, this score incorporates bidirectional context (both prefix $y_{<i}$ and suffix $y_{>i}$ ). The authors argue that considering the suffix reduces uncertainty about token acceptability compared to prefix-only approaches.
Labeling: If the semantic preservation score exceeds a threshold $\tau$ , the replaced token is labeled as "acceptable." This allows the system to automatically generate high-quality training data across any task domain, including open-ended generation.

2.2. Verifier Training

The generated dataset consists of pairs of hidden representations from the target model and their corresponding accept/reject labels. A lightweight logistic regression classifier is trained on this data to act as the judge verifier. During inference, this verifier operates in parallel with standard alignment-based verification; a token is accepted if either the alignment check or the judge verifier approves it.

3. Key Contributions

Self-Supervised Framework: The paper introduces SelfJudge, the first judge verification approach that trains verifiers using the target model's own semantic understanding, eliminating the need for human annotations or task-specific ground truths.
Semantic Preservation Criterion: The authors propose a novel metric for token acceptability based on the likelihood difference of token-substituted responses, enabling automatic supervision generation for open-ended tasks.
Bidirectional Context Analysis: The method utilizes suffix information (future tokens) to compute semantic preservation scores, theoretically reducing uncertainty in token selection compared to prefix-only baselines.
Generalization: The approach is designed to generalize across diverse NLP tasks, including mathematical reasoning, code generation, summarization, and general knowledge QA.

4. Experimental Results

The authors evaluated SelfJudge using Llama-3 and Qwen-2.5 model families across five benchmarks: GSM8K, MATH-500, MMLU, CNN/DailyMail, and LiveCodeBench.

Accuracy-Efficiency Trade-off: SelfJudge consistently outperformed baselines (Standard SD, Top-k, and AutoJudge).
- On GSM8K and MATH-500, SelfJudge achieved higher accepted token lengths than Standard SD with minimal accuracy loss (e.g., +2.17 accepted tokens with only -1.6% accuracy drop on average).
- On MMLU and CNN/DailyMail, where AutoJudge failed to provide speedups (or degraded performance significantly), SelfJudge maintained consistent performance gains.
Wall-Clock Speedup: Implemented in vLLM, SelfJudge demonstrated significant real-world throughput improvements. On MMLU with Llama-3.1-8B, SelfJudge achieved a 46.3% speedup over Standard SD and a 42.7% improvement over AutoJudge-R, while maintaining comparable accuracy.
Scalability: Experiments on the Llama-3.1-70B model showed that SelfJudge scales effectively, improving accepted length and speed without sacrificing generation quality.
Data Generation Efficiency: The self-supervised data generation process is significantly faster than AutoJudge's ground-truth-based generation. For 53k labels on a 70B model, SelfJudge took 8.5 hours compared to AutoJudge's estimated 120 hours.

5. Significance and Claims

The paper claims that SelfJudge represents a paradigm shift in speculative decoding verification. By moving away from reliance on human bias or rigid ground truths, it offers a broadly applicable solution for accelerating LLM inference across the full spectrum of NLP tasks.

The authors emphasize that their method achieves a superior trade-off between inference speed and task performance. Unlike previous judge decoding methods that are restricted to domains with verifiable answers, SelfJudge enables efficient speculative decoding for open-ended tasks (like creative writing and summarization) by leveraging the target model's intrinsic semantic understanding. The paper concludes that this approach establishes a practical and scalable foundation for efficient LLM inference, making advanced models more accessible by reducing computational costs.