💬 NLP

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

This paper introduces CoDIT, a method that synthesizes instruction-tuning datasets by applying contrastive decoding to disentangle instruction-following capabilities from pre-trained knowledge, resulting in models that outperform those trained on existing datasets and enabling the transfer of chat capabilities across different architectures.

Original authors: Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike, Naoaki Okazaki

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike, Naoaki Okazaki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Smart but Cluttered" Teacher

Imagine you want to teach a student (a new AI) how to follow instructions perfectly. You decide to use a brilliant professor (a powerful AI) to write the textbook answers.

However, there's a catch. The professor is incredibly smart because they have read millions of books (this is Pre-trained Knowledge). But they are also great at following your specific classroom rules because they were recently hired and trained for this specific job (this is Instruction-Following).

When the professor writes an answer, they mix these two things together.

The Good: They follow your rules.
The Noise: They also dump a bunch of random facts they memorized from their books that you didn't ask for.

If you use these mixed-up answers to teach your student, the student learns to follow rules but also learns to recite random facts they don't need. It's like trying to teach a chef how to bake a cake, but the master chef keeps adding random spices from their pantry that have nothing to do with the recipe. The student gets confused and doesn't become a pure, rule-following baker.

The Solution: CoDIT (The "Noise-Canceling" Headphones)

The authors created a method called CoDIT (Contrastive Decoding for Instruction-Tuning). Think of it as a Noise-Canceling Headphone for AI text generation.

Here is how it works, step-by-step:

The Setup: They take the "Professor" AI (the Post-trained model) and its "Old Self" (the Pre-trained model before it learned to follow instructions).
The Comparison: When generating an answer, CoDIT asks two questions for every single word:
- Question A: "How likely is the Professor to say this word?" (Does it follow the instruction?)
- Question B: "How likely is the Old Self to say this word?" (Is this just a random fact from memory?)
The Filter: CoDIT only keeps words where the Professor is very excited to say them, but the Old Self is bored or unlikely to say them.
- If both models love the word, it's probably just common knowledge (Noise). Discard it.
- If the Professor loves it but the Old Self hates it, that word is pure Instruction-Following Magic. Keep it!

The Magic Analogy: The "Chat Vector"

The paper explains that this process is like distilling a "Chat Vector."

Imagine the "Instruction-Following" ability is a specific flavor of ice cream (let's say, "Vanilla with Sprinkles").

The Pre-trained model is a giant tub of plain vanilla ice cream (just knowledge).
The Post-trained model is that same tub, but someone added sprinkles (the instructions).
Usually, when you copy the ice cream, you get the whole tub, sprinkles and all.
CoDIT is a special machine that scoops out only the sprinkles and leaves the plain vanilla behind.

It takes the "difference" between the two models (the sprinkles) and turns it into a new, pure recipe. This allows the student AI to learn only how to follow instructions, without getting overloaded with unnecessary facts.

Why This Matters (The Results)

The researchers tested this by creating new training datasets using CoDIT and teaching new AI models with them.

The Result: The students trained on the "Pure Sprinkle" datasets (CoDIT) became much better at following instructions than students trained on the "Whole Tub" datasets (standard AI generation).
The Surprise: They found that even if the "Professor" AI had been through extra training (like Reinforcement Learning, which makes it smarter but also more complex), CoDIT still worked perfectly. It could still filter out the noise and find the pure instruction-following signal.
The Bonus: This method works across different types of AI "species" (like Llama, Qwen, and Gemma). You can use a Qwen professor to teach a Llama student, and it still works because you are transferring the concept of following instructions, not just copying the brain structure.

In a Nutshell

CoDIT is a clever trick to clean up AI training data. It acts like a filter that removes the "background noise" of general knowledge, leaving behind only the pure signal of "how to listen and obey." This creates better textbooks for training new AIs, making them smarter, more obedient, and less prone to hallucinating random facts.

1. Problem Statement

The paper addresses a fundamental limitation in current instruction-tuning dataset synthesis methods. While using Large Language Models (LLMs) to generate responses for instruction tuning is a standard practice, existing methods suffer from a conflation of capabilities:

Pre-trained Knowledge: World knowledge acquired during the pre-training phase.
Instruction-Following Capabilities: The specific ability to follow user instructions acquired during post-training (fine-tuning/alignment).

The authors hypothesize that when a post-trained model generates a response, it mixes these two components. Since pre-trained knowledge is often already present in the student model's parameters, including it in the training data acts as noise rather than a useful signal. This "mismatch" prevents the student model from efficiently learning pure instruction-following behaviors. The goal is to disentangle these capabilities to generate training data that isolates and amplifies instruction-following skills while suppressing shared pre-trained knowledge.

2. Methodology: CoDIT

The authors propose CoDIT (Contrastive Decoding for Instruction-Tuning Dataset), a method that applies contrastive decoding during the response generation phase to synthesize high-quality instruction-response pairs.

Core Mechanism

CoDIT treats the post-trained model as the "Expert" and its pre-trained checkpoint (the same model before instruction tuning) as the "Amateur."

Contrastive Scoring: For each token $v$ in the vocabulary, the model calculates a score $s(v)$ based on the difference in log-probabilities between the post-trained model ( $\theta_{post}$ ) and the pre-trained model ( $\theta_{pre}$ ):
$s(v; x, y_{<i}) = \log P(v | x, y_{<i}; \theta_{post}) - \log P(v | x, y_{<i}; \theta_{pre})$
Token Selection: The token with the highest contrastive score is selected, provided it meets a plausibility constraint. This constraint ensures the token has a sufficiently high probability under the post-trained model (controlled by hyperparameter $\alpha$ ) to prevent the generation of nonsensical text (a common issue with unrestricted contrastive decoding).

Theoretical Interpretation: Text-Space Distillation

The paper provides a theoretical proof that CoDIT acts as a distillation of the "Chat Vector" from parameter space to text space.

The Chat Vector ( $\Delta\theta = \theta_{post} - \theta_{pre}$ ) represents the parameter shift induced by instruction tuning.
Using a first-order Taylor expansion, the authors show that the contrastive score is proportional to the inner product between the chat vector and the gradient of the log-likelihood.
By maximizing this score, CoDIT selects tokens whose gradients align most closely with the instruction-tuning update direction.
Significance: Unlike traditional parameter-space distillation (which requires identical architectures), CoDIT transfers instruction-following capabilities via text space, enabling transfer between models of arbitrary scale and architecture.

3. Key Contributions

Novel Method (CoDIT): A framework for synthesizing instruction-tuning datasets that explicitly suppresses pre-trained knowledge and amplifies instruction-following behavior using contrastive decoding between post-trained and pre-trained model checkpoints.
State-of-the-Art Datasets: The release of instruction-tuning datasets constructed via CoDIT that outperform existing public datasets (e.g., WildChat, Magpie-Pro, WebR) across multiple benchmarks.
Theoretical & Empirical Validation:
- Proved that CoDIT effectively distills the "chat vector" into text space.
- Demonstrated that this method works across different model families (Qwen, Llama, Gemma) and scales, regardless of whether the teacher model has undergone Reinforcement Learning (RL) or just Instruction Tuning (IT).

4. Experimental Results

The authors evaluated CoDIT using 9 different teacher-student pairs (e.g., Qwen3-8B/30B, Gemma-3-27B as teachers; Llama-3.1-8B, Qwen3-8B-Base, Gemma-3-4B as students).

Baselines: Compared against Vanilla (direct generation by the teacher) and Best-of-N (selecting the best of 5 generations via an LLM judge).
Benchmarks: WildBench, AlpacaEval 2.0, and MT-Bench.
Key Findings:
- Consistent Superiority: Models trained on CoDIT datasets consistently outperformed those trained on Vanilla and Best-of-N baselines across all 9 configurations.
- Comparison with Public Data: CoDIT-generated datasets achieved higher scores than existing open-source datasets (e.g., WildChat, Magpie-Pro) on WildBench and AlpacaEval 2.0.
- Quality vs. Capability: Analysis showed that CoDIT does not significantly improve general text fluency or quality scores (as measured by LLM judges) compared to Vanilla. Instead, the performance gain stems specifically from better isolation and transfer of instruction-following capabilities.
- Chat Vector Alignment: Empirical results showed that fine-tuning on CoDIT data induces parameter updates with higher cosine similarity to the teacher's chat vector compared to Vanilla data, confirming the theoretical distillation mechanism.
- Robustness: The method remains effective even when the teacher model includes Reinforcement Learning (RL) stages, suggesting RL sharpens the instruction-following signal rather than hindering CoDIT.

5. Significance

Decoupling Knowledge and Skill: CoDIT offers a principled way to separate "what the model knows" (pre-training) from "how the model follows instructions" (post-training), addressing a critical bottleneck in current data synthesis.
Architecture-Agnostic Transfer: By operating in text space rather than parameter space, CoDIT enables the transfer of instruction-following capabilities between models of different architectures (e.g., training a Llama student on data synthesized by a Qwen teacher), which is not possible with parameter-based distillation.
Efficient Data Generation: It provides a method to generate high-quality training data without requiring human annotation or complex multi-stage filtering, simply by leveraging the difference between a model's pre-trained and post-trained states.

In conclusion, CoDIT represents a shift from treating LLM-generated data as a monolithic block of "knowledge + skill" to a targeted extraction of "skill," resulting in more effective instruction tuning and superior model performance.