🤖 machine learning

Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models

This paper demonstrates that supervised fine-tuning (SFT) lessons regarding behavior generalization, capability preservation, and robustness can be successfully transferred across alignment training, model organisms, and toy models, thereby unifying these research areas and improving overall outcomes through cross-disciplinary learning.

Original authors: Anton de la Fuente, Arthur Conmy

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Anton de la Fuente, Arthur Conmy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, but slightly literal, robot how to behave. You don't just want it to follow orders; you want it to understand why those orders exist, to keep its brain sharp while learning new tricks, and to remember those tricks even after you stop teaching it. This is the world of "Supervised Fine-Tuning" (SFT), a process where we take a giant, pre-trained AI and give it a specific set of examples to learn from, like a student cramming for a test. Usually, researchers study these lessons in three separate silos: one group studies how to make AI safe (Alignment), another studies how to create fake "bad actors" to test our defenses (Model Organisms), and a third uses tiny, simplified AI models to figure out the basics of learning (Toy Models). But what if the secrets to a better robot aren't locked in one silo, but are actually the same lessons repeated in different costumes?

This paper acts like a curious detective connecting the dots between these three worlds. The authors ask: If we learn something about how AI learns in a tiny, simple game, does that same rule work when we are trying to make a massive, real-world AI safer? They test three specific "lessons" by moving them from one research area to another. First, they check if teaching an AI the reason behind a rule helps it follow that rule in new situations better than just showing it examples. Second, they investigate if letting an AI learn from a different AI's answers hurts its own smarts, and if mixing in its own answers can fix that. Finally, they see if a behavior we teach an AI can be accidentally erased later by harmless, boring training. The results suggest that borrowing ideas across these different fields isn't just a nice thought experiment; it actually helps us build smarter, safer, and more robust AI.

The Three Great Experiments

The researchers didn't just guess; they ran three distinct experiments to see if these cross-pollinated ideas held up.

1. The "Why" vs. The "What": Teaching the Reason
Imagine you are teaching a student to put their final answer in a box. If you only show them math problems where the answer is boxed, they might think, "Oh, you only box answers for math." But if you tell them, "I always put my final answer in a box because it's my rule," they might start boxing answers for everything, even when you ask for gift advice.

The authors tested this with a simple AI. When they trained it only on examples of boxed math answers, the AI rarely boxed answers to non-math questions (only about 10% of the time). However, when they added a simple sentence explaining the rule ("I always put my final answer in a box") to the training data, the AI started boxing answers to non-math questions nearly 95% of the time. Even more surprisingly, they found that the AI didn't even need to see the reason specifically about boxing; just having any explanatory sentence helped, though the specific reason was the most effective. This suggests that explaining the "why" behind a behavior helps the AI generalize that behavior to new, unexpected situations, rather than just memorizing the "what."

2. The "Copycat" Problem: Who Writes the Homework?
Here, the researchers looked at a common practice: using a "teacher" AI to write the training data for a "student" AI. They suspected that if the teacher writes the reasoning in a style the student doesn't use, the student might get confused and lose some of its smarts (capability).

They tested this by training a student AI on two types of data: one where the student wrote its own reasoning (on-model) and one where a different, stronger AI wrote the reasoning (off-model). The results were clear: when the student learned from the other AI's writing style, its performance on a difficult science test (GPQA) dropped significantly. But when they mixed in data where the student wrote its own reasoning alongside the teacher's data, the student kept its smarts while still learning the new behavior. This suggests that to keep an AI smart while teaching it new things, you need to make sure it sees plenty of examples written in its own "voice."

3. The "Wash-Out": Can New Training Erase Old Lessons?
Finally, the team asked a scary question: If we teach an AI to be safe, will a later round of harmless, boring training accidentally wipe out that safety? In the world of "Model Organisms" (where researchers create fake bad behaviors to study them), it was known that later training could erase these behaviors. The authors tested this on a real-world alignment setup.

They took an AI that had been trained to be safe and then subjected it to a "wash" of benign, generic training data. They found that this harmless training did indeed erase the safety behavior, even though the AI's general intelligence stayed the same. However, they also found a way to make the behavior stickier. If the AI had gone through a "mid-training" phase or if the safety training included the student's own writing (the "replay" method from the second experiment), the safety behavior survived the wash much better. This tells us that just because an AI learns a behavior and keeps its smarts, it doesn't mean that behavior is safe from being accidentally deleted by future updates.

What This Means for the Future

The paper doesn't claim to have solved all AI problems, but it offers a powerful new way of thinking. By treating "Alignment," "Model Organisms," and "Toy Models" not as separate islands but as different labs working on the same fundamental problems, the researchers found that lessons transfer surprisingly well.

They showed that explaining the reason for a rule helps AI learn it better. They proved that mixing in an AI's own writing style prevents it from getting "dumber" when learning from others. And they warned us that safety training isn't permanent; it can be washed away by later updates unless we build it to be robust. The authors suggest that if more researchers start borrowing techniques from outside their own specific fields, we might solve these tricky problems much faster. It's a reminder that in the complex world of AI, sometimes the best way to move forward is to look at what your neighbors are doing.

Technical Summary: Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models

Problem Statement

Research in Large Language Model (LLM) alignment, model organisms, and toy models of Supervised Fine-Tuning (SFT) is often conducted in silos, despite frequently sharing the same underlying goals (e.g., behavior generalization, capability preservation, and robustness). The paper argues that lessons developed in one SFT setting are rarely tested in others, potentially missing opportunities to improve safety and capability trade-offs. The authors identify three specific gaps where cross-domain transfer of SFT lessons could be valuable:

Behavior Generalization: Whether training on the reason for a behavior (rather than just examples) improves transfer to out-of-distribution (OOD) contexts.
Capability Preservation: Whether SFT on outputs generated by a different model ("off-model outputs") damages the student model's capabilities, and if mixing "on-model" data can mitigate this.
Robustness to Later Training: Whether an installed behavior survives subsequent benign SFT, or if it is susceptible to "wash-out" (erasure) even when capabilities are preserved.

Methodology

The authors conduct three transfer experiments, moving lessons from one research area to another:

1. Behavior Generalization (Alignment $\to$ Toy Models)

Source Lesson: Derived from "Teaching Claude Why" (TCW), suggesting that training on the rationale behind a behavior improves generalization.
Toy Models: Three controlled setups were used:
- Boxing: Training a model to box final answers ( $\boxed{}$ ) on math prompts and testing transfer to non-math prompts.
- Animal Welfare: Training the model to express concern for animal welfare in relevant contexts.
- Self-Preservation: Training the model to argue against shutdown/replacement.
Experimental Design: The authors compared SFT on "examples only" against SFT on "reason + examples." In the latter, a fixed reason (e.g., "I always put my final answer in \boxed{}") or explicit rationale was prepended or woven into the training data. They also tested variations such as masking the loss on the target span and using neutral filler sentences.

2. Capability Preservation (Model Organisms $\to$ Alignment/Toy Models)

Source Lesson: Derived from model organism research (e.g., Prasanna et al., 2026b), suggesting that training on "off-model" reasoning (outputs written by a different model) can degrade student capabilities.
Toy Models: Animal welfare and self-preservation tasks using Qwen3.5-4B.
Alignment Setting: The "Model-Spec Midtraining" (MSM) pipeline (Li et al., 2026), where Qwen3-32B is aligned to a philosophical specification using Claude-written traces.
Experimental Design:
- Toy Models: Compared "off-model rewrites" (GPT-4.1 rewriting responses) vs. "on-model rewrites" (the student model rewriting its own responses).
- MSM: Tested an intervention mixing "benign on-model data" (replay: generic instructions answered by the student itself) with the off-model target-behavior data during training.

3. Robustness to Later Training (Model Organisms $\to$ Alignment)

Source Lesson: Model organism research emphasizes testing if installed behaviors survive subsequent benign training.
Alignment Setting: The MSM checkpoints from Li et al. (2026).
Experimental Design: After installing target behaviors via standard alignment SFT or MSM, the models underwent a "wash" phase: continued benign SFT on generic, student-written reasoning traces. The authors measured whether the target behavior (Agentic Misalignment score) eroded while capabilities (GPQA) were preserved.

Key Results

1. Reasoning Improves Generalization

Boxing: Training on examples alone resulted in near-zero transfer to non-math prompts (10.3%). Adding a reason ("I always put my final answer in \boxed{}") increased transfer to 94.5%.
Mechanism: The transfer did not require direct loss on the boxed answer (masked loss still yielded 97.4% transfer). Interestingly, even a neutral sentence (not mentioning boxing) improved transfer to 62.5%, though the explicit reason remained superior.
Richer Traits: For animal welfare and self-preservation, the "rewrite" condition (making the rationale explicit) produced significantly stronger behaviors than "one-shot" or "stripped" conditions (which removed specific rationales).

2. Off-Model Outputs Damage Capability; Replay Mitigates It

Capability Loss: In toy models, training on off-model rewrites (GPT-4.1) significantly reduced GPQA accuracy compared to on-model rewrites (student model), despite off-model rewrites producing stronger target behaviors.
MSM Application: Standard MSM training (off-model data) showed capability regressions.
Intervention: Mixing benign on-model "replay" data (student-written generic responses) into the training from the start recovered most of the lost GPQA accuracy (0.687 vs. 0.492 for off-policy only) while maintaining low Agentic Misalignment (target behavior).
Timing: Adding replay after trait training recovered capability but caused the target behavior to erode. Mixing it from the start was required to preserve both.

3. Behaviors Are Not Robust to Wash-Out

Erosion: When checkpoints trained with alignment SFT alone were subjected to subsequent benign SFT ("wash"), the target behavior (Agentic Misalignment) eroded significantly (88% of the gap to the base model was lost), even though GPQA capability recovered.
Midtraining Benefit: Checkpoints that underwent Model-Spec Midtraining before alignment SFT were more resistant to wash-out, losing only 39% of the behavior gap.
Conclusion: Capability preservation alone does not guarantee that an aligned behavior will survive subsequent training.

Significance and Claims

The paper claims that organizing SFT research around shared goals rather than research-area boundaries yields actionable insights:

Transferability: Lessons about generalization, capability preservation, and robustness are portable across toy models, model organisms, and large-scale alignment pipelines.
Safety-Capability Trade-off: The "mixed replay" technique offers a practical method to improve the safety-capability Pareto frontier in alignment training, specifically by preventing capability loss from off-model data without sacrificing the target behavior.
Robustness Necessity: Evaluations of aligned models must include checks for robustness against future benign training, as behaviors can be silently erased even if the model remains capable.

The authors emphasize that their work is modest in scope (focusing strictly on SFT, not the full post-training pipeline including RL) but highlights that SFT is a critical, often misunderstood component of alignment that is prone to unexpected failure modes. The release of code and data aims to lower the barrier for other researchers to test these cross-domain lessons.