💬 NLP

SkillFactory: Self-Distillation For Learning Cognitive Behaviors

SkillFactory introduces a self-distillation method that rearranges a model's own imperfect reasoning traces into supervised fine-tuning data to prime it for acquiring robust cognitive skills like verification and backtracking during subsequent reinforcement learning, ultimately improving generalization and out-of-domain robustness compared to standard approaches.

Original authors: Zayne Sprague, Jack Lu, Manya Wadhwa, Sedrick Keh, Mengye Ren, Greg Durrett

Published 2026-04-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zayne Sprague, Jack Lu, Manya Wadhwa, Sedrick Keh, Mengye Ren, Greg Durrett

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart but inexperienced apprentice how to solve complex puzzles.

In the world of AI, these "apprentices" are Large Language Models (LLMs). Usually, when we want them to get better at reasoning (like solving math problems or logic puzzles), we have two main options:

The "Master Apprentice" Method: Find a genius master (a much bigger, smarter AI) and have the apprentice copy their homework. This works well, but you need access to a genius master, which is expensive and hard to get.
The "Trial and Error" Method: Let the apprentice just try to solve the puzzle over and over again, rewarding them only when they get the right answer. This is called Reinforcement Learning (RL). But if the apprentice doesn't know how to think, they might just guess randomly and never learn the right habits.

SkillFactory is a clever new third option. It's like a self-coaching workshop where the apprentice learns to think better by reorganizing their own messy practice sessions.

Here is how it works, broken down into simple steps:

1. The Problem: The "One-and-Done" Habit

Most AI models are like students who rush through a test. They write down an answer, and if they are wrong, they just move on. They don't naturally check their work, admit they made a mistake, or try a different approach. They lack "cognitive skills" like verification (checking the answer) and retrying (trying again if it fails).

2. The Solution: The "Silver Trace" Workshop

The SkillFactory team realized they didn't need a genius master to teach these skills. They could use the apprentice's own brain!

Step 1: The Messy Drafts. They ask the model to solve a problem 64 times. Some answers are right, but most are wrong.
Step 2: The Self-Reflection. They ask the model to look at its wrong answers and explain why they were wrong. They also ask it to look at its right answers and confirm they are correct.
Step 3: The "Silver" Story. This is the magic part. They take the model's own wrong attempts, its self-critique, and its final correct answer, and stitch them together into a single, structured story.

The Analogy:
Imagine the model is a chef who burns a few soups before getting the recipe right.

Normal Training: You just tell the chef, "Good job on the last soup," and ignore the burnt ones.
SkillFactory: You take the chef's notes on why the first soup burned ("I added too much salt"), their notes on the second attempt ("I forgot the pepper"), and their final perfect soup. You then bind these notes into a single "Master Recipe Book" that says: "First, I tried X and failed. I realized why. Then I tried Y and failed. I realized why. Finally, I tried Z and it worked!"

This "Master Recipe Book" is called a Silver Trace. It's not perfect (it's made by the apprentice, not a master), but it teaches the structure of good thinking: Try -> Check -> Fail -> Reflect -> Try Again -> Succeed.

3. The Training: Learning the "Dance"

The model is then trained on these "Silver Traces."

The Goal: The model isn't trying to learn the answers to the puzzles yet. It's learning the dance steps. It learns that when it gets stuck, it should pause, reflect, and try a new path.
The Result: The model becomes "primed." It now has the habit of checking its work.

4. The Final Step: Reinforcement Learning (The Gym)

Once the model has learned the "dance steps" (the cognitive skills) from the Silver Traces, they put it in the gym (Reinforcement Learning).

Because the model already knows how to think (check, retry, verify), it learns much faster and becomes much better at solving hard problems than models that just started from scratch.

Why is this a big deal?

No Genius Required: You don't need a super-smart AI to teach a smaller AI. The smaller AI teaches itself by reorganizing its own mistakes.
Better at the Hard Stuff: The paper shows that models trained this way are surprisingly good at solving puzzles they've never seen before. They don't just memorize answers; they learned how to think.
Robustness: Even when the model makes a mistake, it's less likely to give up or get confused. It has the habit of saying, "Wait, that doesn't look right, let me try again."

In a nutshell:
SkillFactory is like giving a student a mirror. Instead of just telling them the right answer, it shows them a video of themselves failing, realizing why they failed, and then succeeding. By watching this video of their own potential, they learn the habits of a genius, even if they aren't one yet.

1. Problem Statement

Modern Large Language Models (LLMs) are increasingly capable of "reasoning" through long chains of thought, utilizing cognitive skills such as verification (checking answers), backtracking (retracing steps), and retrying with alternative strategies. While Reinforcement Learning (RL) has proven effective at amplifying these skills in models that already exhibit them, a critical gap remains: how to instill these skills in base models that do not naturally demonstrate them.

Existing approaches face significant limitations:

RL Only: Often fails to discover complex reasoning patterns from scratch, especially in cross-task generalization.
Distillation from Stronger Models: Requires access to superior "teacher" models (e.g., proprietary LLMs), which is costly and limits accessibility.
Manual Data Curation: Creating high-quality reasoning traces manually is labor-intensive and difficult to scale.

The authors propose that current methods often conflate task performance with skill acquisition. A model might solve a task well without actually learning the underlying cognitive behaviors (like verification) required to generalize to harder or out-of-domain (OOD) tasks.

2. Methodology: SkillFactory

SkillFactory is a framework designed to teach cognitive behaviors to a base model during a Supervised Fine-Tuning (SFT) stage, using only the model's own outputs (self-distillation), without requiring a stronger teacher. The core insight is that structure matters more than content; by rearranging a model's own imperfect attempts into a structured format that explicitly demonstrates skills, the model can be "primed" to learn these behaviors effectively during subsequent RL.

The framework consists of three main stages:

A. Data Curation (Silver Trace Construction)

The process generates "silver" training traces by sampling, reflecting, and rearranging the base model's outputs:

Solution Sampling: For a given problem, the base model generates $N$ solution attempts ( $Y$ ) using diverse prompts.
Reflection Generation: The model is prompted to generate reflections ( $R$ ) for each attempt, predicting whether the answer is correct or incorrect. Valid reflections are those where the model's verdict matches the ground truth.
Trace Assembly: The system constructs training traces by:
- Sampling a mix of correct and incorrect solution-reflection pairs.
- Shuffling incorrect attempts with correct ones to simulate a search process.
- Ensuring the trace ends with a correct solution.
- Explicit Tagging: Using XML-style tags (<sample>, <reflect>, <verdict>, <answer>) to demarcate the cognitive steps.
- Glue Phrases: Inserting natural language transitions (e.g., "Wait, let me rethink this," "This is incorrect because...") to guide the model through the retry and verification logic.

B. Supervised Fine-Tuning (SFT)

The base model is fine-tuned on these structured silver traces.

Goal: The objective is not to maximize task accuracy at this stage. Instead, the goal is to internalize the structure of cognitive behaviors (sampling, reflecting, retrying).
Outcome: The model learns the "inductive bias" of how to use these skills, even if its raw performance on the task remains low or imperfect.

C. Reinforcement Learning (RL)

The SFT-initialized model undergoes RL (using GRPO with sparse correctness rewards).

Because the model has already learned the structure of reasoning, the RL stage efficiently "hones" these skills, teaching the model when and how to apply verification and retries to maximize rewards.

3. Key Contributions

Self-Distillation for Cognitive Skills: Demonstrated that models can acquire complex reasoning skills (verification, retrying) from their own rearranged outputs, eliminating the need for stronger teacher models.
Structure over Content: Showed that training on the structure of reasoning traces (even if generated by the model itself) is more effective for generalization than simply training on correct answers (STaR) or distilling from stronger models.
Robust Generalization: Proved that SkillFactory initialization leads to superior generalization on:
- Harder Task Variants: (e.g., moving from 3-number Countdown to 4-6 number Countdown).
- Out-of-Domain (OOD) Tasks: Tasks never seen during training (e.g., Acronyms, Multiplication, CommonsenseQA).
Resilience to Catastrophic Forgetting: SkillFactory models showed greater stability on OOD tasks compared to models trained with RL-only or standard distillation, which often suffer performance regression.

4. Experimental Results

The authors evaluated SkillFactory on Qwen2.5 (1.5B, 7B) and Olmo-3-7B across two settings: Countdown (search-focused) and OpenThoughts (complex math/science).

Key Findings:

Easy-to-Hard Generalization: On the Countdown task, SkillFactory models trained on 3-argument problems achieved 25.1% accuracy on harder 4-6 argument variants after RL. This outperformed the next best baseline (R1 Distillation + RL) by +3.9% and significantly outperformed RL-only models.
OOD Generalization: While R1 Distillation models performed well on in-domain tasks, they often regressed on OOD tasks. SkillFactory models maintained robust performance across diverse domains (Acronyms, Multiplication, GSM8K), achieving a higher overall OOD score (35.7% vs 35.9% for R1, but with much better stability across specific sub-tasks).
Complex Math Reasoning: On challenging benchmarks (GPQA, AIME 2025, AMC, Math500), SkillFactory models trained on OpenThoughts data matched or exceeded the performance of models distilled from the powerful QwQ model, particularly on benchmarks not explicitly targeted in the training data (e.g., AMC and Math500).
Budget Forcing: When given more "thinking time" (token budget) at inference, SkillFactory models utilized the extra tokens more effectively (increasing accuracy by +5.3% on Countdown) compared to RL-only baselines, which often produced degenerate loops.
Skill Usage Analysis: Post-RL SkillFactory models exhibited explicit and implicit use of verification and retries. For example, on Countdown-4arg, they generated an average of 7.13 reflections per trace, significantly higher than baselines, and maintained high F1 scores for detecting incorrect answers.

5. Significance

Democratizing Reasoning: SkillFactory provides a pathway to create robust reasoning models without relying on proprietary, stronger "teacher" models. This lowers the barrier to entry for developing advanced reasoning capabilities.
Inductive Bias for RL: The paper establishes that SFT initialization focused on cognitive structure is a critical prerequisite for successful RL. It suggests that RL is most effective when the model is already "primed" with the correct behavioral patterns, rather than starting from a "cold" state.
Mitigating Over/Under-thinking: By explicitly training models to verify and retry, SkillFactory helps balance the trade-off between "overthinking" (excessive verbosity) and "underthinking" (premature abandonment of solutions), leading to more reliable and robust reasoning agents.

In conclusion, SkillFactory demonstrates that the ability to reason is not just a product of scale or RL rewards, but can be systematically instilled by restructuring a model's own outputs to teach it how to think, rather than just what the answer is.