🤖 machine learning

Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation

This paper proposes a cost-effective method that enhances reasoning language models in both verifiable and unverifiable domains by first applying classic instruction tuning to human-written solutions and then merging the resulting model with the original reasoning model to recover domain-specific reasoning capabilities.

Original authors: Yu-Du Feng, Niels Mündler-Sasahara, Mark Vero, Martin Vechev

Published 2026-07-17

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu-Du Feng, Niels Mündler-Sasahara, Mark Vero, Martin Vechev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant, overthinking student how to solve a new type of puzzle. This student, a "Reasoning Language Model," is a special kind of computer program trained to show its work. Before it gives you the final answer to a math problem or a coding challenge, it writes out a long, step-by-step thought process, like a detective solving a mystery. This habit of "thinking aloud" is what makes it so good at tricky tasks.

However, there's a catch. To teach this student new skills, you usually need a way to instantly check if their answer is right (like a math answer key) or a super-smart teacher to show them the right way to think. But what if you want to teach them something where there's no answer key, like writing a funny story summary or fixing a specific bug in a computer program that doesn't have a test? You have plenty of examples of the right answers written by humans, but none of them show the "thinking" part. If you just force the student to memorize these answers without the thinking steps, they might get the right answer for that specific puzzle, but they'll forget how to think in general. They become a parrot that repeats answers instead of a detective who solves problems.

This is the puzzle researchers at ETH Zurich tackled in their new paper. They discovered a clever two-step trick to teach these "thinking" models new skills without making them forget how to think. First, they let the model learn from the simple "answer-only" examples, just like a student cramming for a test. This makes the model better at the specific task, but it starts to lose its habit of showing its work. Then, they perform a magical "merge." They take this newly trained model and blend it back with the original, smart-thinking model. It's like mixing a cup of fresh, task-specific coffee with a cup of the original, strong espresso. By carefully adjusting the ratio of the mix, they found a "sweet spot" where the model keeps its new skills but remembers how to think through problems again.

The researchers tested this on four different smart models and two very different tasks: writing code in the Rust programming language and summarizing long news articles. They found that their method worked wonders. The models got much better at the specific tasks (improving coding scores by up to 12 points and summarization scores slightly) while almost completely recovering their ability to reason, which usually gets destroyed by standard training. Even better, this whole process was incredibly cheap and fast. The researchers calculated that they could adapt a model for less than $3 and in under an hour, whereas other methods that try to fix the same problem cost significantly more and take longer.

In short, the paper suggests that you don't need a super-expensive "answer-checking" system or a genius teacher to upgrade these reasoning models. You can use the massive amounts of simple "question-and-answer" data that already exist, and with a little bit of mathematical blending, you can give these models new skills without breaking their brains. It's a low-cost, high-reward way to keep our AI detectives sharp, even when they are learning to solve mysteries that don't have obvious solutions.

Problem
Reasoning Language Models (RLMs) have achieved state-of-the-art performance in domains with reliable verifiers, such as mathematics and coding, primarily through Reinforcement Learning with Verifiable Rewards (RLVR). However, adapting RLMs to domains lacking reliable verifiers (e.g., text summarization or coding without comprehensive unit tests) remains challenging. While large amounts of high-quality supervised fine-tuning (SFT) data exist for these domains, they typically consist of input-output pairs without reasoning traces. Directly applying standard Instruction Fine-Tuning (IFT) to RLMs using this data creates a distributional mismatch: the model is trained to produce direct answers, causing it to "collapse" its reasoning behavior. This results in the loss of the model's ability to generate reasoning traces, which subsequently degrades performance on the target task and leads to catastrophic forgetting of general reasoning capabilities on held-out datasets like MATH500.

Methodology
The authors propose a lightweight, two-step pipeline to adapt RLMs using standard IFT data while preserving reasoning behavior:

Standard Instruction Fine-Tuning (IFT): The base RLM is fine-tuned on a task-specific dataset containing only input-output pairs (no reasoning traces). This produces a model ( $M_{IFT}$ ) with improved task accuracy but compromised or collapsed reasoning behavior.
Model Merging: The fine-tuned model ( $M_{IFT}$ $M_{I F T}$ ) is linearly merged with the original, untuned reasoning model ( $M$ $M$ ). The merged model is defined as $M_\alpha = (1-\alpha)M + \alpha M_{IFT}$ $M_{α} = (1 - α) M + α M_{I F T}$ , where $\alpha$ $α$ is the merge ratio.
- Ratio Selection: The optimal merge ratio $\alpha$ is selected using a small, held-out calibration dataset ( $D_{cal}$ ) for the target task. The selection criterion is to find the largest $\alpha$ such that the reasoning rate (the fraction of examples where the model produces a non-empty reasoning trace) on $D_{cal}$ remains above a minimum threshold ( $\rho_{min}$ , set to 0.9 in experiments).
- Optimization: To reduce computational cost, the calibration process evaluates only the first few tokens of the model's response to determine if reasoning is initiated, rather than generating full responses. A binary search is employed to efficiently find the optimal $\alpha$ .

Key Contributions

Identification of a Practical Adaptation Setting: The work addresses the specific scenario where only input-output supervision is available for RLMs, without access to verified reasoning traces or stronger teacher models.
Lightweight IFT-and-Merge Method: The authors propose a simple procedure that adapts RLMs to new tasks while recovering reasoning behavior. Crucially, this method does not require a verifier, a reward model, or a stronger teacher model.
Comprehensive Evaluation: The method is evaluated across four diverse RLMs (OpenThinker 7B, Apriel Nemotron 15B Thinker, Olmo3 7B Think, and DeepSeek R1 Qwen 7B Distilled) on two target tasks (Rust coding and text summarization) and a held-out mathematical reasoning task (MATH500).

Results

Performance Recovery: Standard IFT often causes a significant drop in MATH500 performance (e.g., OpenThinker 7B dropped from 79% to 35.9% on MATH500 after Rust coding IFT) and collapses reasoning traces. The proposed merging technique recovers most or all of the lost general reasoning capabilities (MATH500 scores) while retaining the majority of the target-task performance gains achieved by IFT.
Comparison to Baselines: The method outperforms or matches competitive baselines such as On-Policy Distillation (IFT+OPD) and IFT with KL-divergence regularization (IFT+KL). In several cases, baselines failed to recover target-task performance or were less effective at preserving reasoning.
Cost-Effectiveness: The approach is highly efficient. The entire adaptation process (IFT, search, and merging) takes approximately 52 minutes on a single NVIDIA H200 GPU, costing less than USD $3. This is significantly cheaper than baselines that require additional teacher inference or complex distillation loops.

Significance
The paper claims that this method provides a cost-effective and accessible pathway for adapting powerful reasoning models to domains where RLVR is infeasible due to the lack of verifiers. By leveraging widely available IFT datasets and recovering reasoning behavior through simple model merging, the technique enables RLMs to maintain their general reasoning capabilities while acquiring new task-specific skills. The authors note that while the method is effective for tasks where IFT data provides sufficient signal (like coding and summarization), it may not suffice for domains where the reasoning process itself is the primary driver of performance (e.g., complex mathematical proofs), where supervision on the reasoning trace itself might still be necessary.

More like this