💻 computer science

Beyond Correctness: Learning Robust Reasoning via Transfer

This paper introduces Reinforcement Learning with Transferable Reward (RLTR), a novel approach that enhances the robustness and sample efficiency of LLM reasoning by training models to produce reasoning steps that remain effective when transferred to guide separate models, rather than focusing solely on final answer correctness.

Original authors: Hyunseok Lee, Soheil Abbasloo, Jihoon Tack, Jinwoo Shin

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hyunseok Lee, Soheil Abbasloo, Jihoon Tack, Jinwoo Shin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a student to solve a complex math problem.

The Old Way (RLVR): "Did you get the right answer?"
Currently, most AI training works like a strict teacher who only looks at the final answer on the test paper. If the student writes a messy, confusing, or lucky guess that happens to be correct, the teacher gives an "A." If the student writes a perfect, logical explanation but makes one tiny arithmetic error at the end, the teacher gives an "F."

The paper calls this RLVR (Reinforcement Learning with Verifiable Rewards). It's great at getting the right answer, but it doesn't care how the student got there. The result? The student might learn to "game" the system or rely on fragile logic that falls apart if you ask them to explain it to someone else.

The New Idea (RLTR): "Can someone else finish your work?"
The authors of this paper, Hyunseok Lee and colleagues, propose a different philosophy. They believe that true reasoning is like a relay race. A good runner (the AI) shouldn't just cross the finish line; they should pass the baton in a way that the next runner can easily pick it up and keep going without stumbling.

They call this new method RLTR (Reinforcement Learning with Transferable Reward).

Here is how it works, using a simple analogy:

The Generator (The First Runner): The AI starts solving a problem and writes down its thoughts (the reasoning).
The Truncation (Cutting the Tape): The system stops the AI halfway through. It takes the first part of the reasoning and cuts it off.
The Receiver (The Second Runner): A different AI (the "Receiver") is handed this cut-off piece of text. It has to read what the first AI wrote and finish the solution.
The Reward:
- If the Receiver can successfully finish the problem and get the right answer, the first AI gets a bonus point.
- If the Receiver gets confused, stuck, or gives a wrong answer because the first part was messy or illogical, the first AI gets no bonus.

Why is this better?
Think of it like writing a story.

RLVR only cares if the story ends with "The End." You could write a story that makes no sense, but if the last word is "The End," you win.
RLTR cares if the story is so clear and logical that anyone reading the first half could easily guess how it ends. It forces the AI to write "robust" reasoning—logic that is stable, clear, and reusable.

What did they find?
The paper tested this on difficult math and science problems. Here are the main takeaways:

More Consistent: When they asked the AI to solve the same problem 64 times, the "Transfer" method (RLTR) got the right answer more often when they took the majority vote. The old method (RLVR) was sometimes right, but its reasoning was so shaky that different attempts often disagreed with each other.
Faster Learning: The new method learned faster. It reached the same level of performance as the old method in about 2.5 times fewer training steps. It's like getting a better education in half the time because the lessons are clearer.
Works on Harder Problems: The benefit was even bigger on the hardest math competitions (like AIME and AMC). When problems are tough, having a clear, transferable thought process matters more than just guessing the right number.
Not Just Math: They also tested it on science questions, and it worked there too, suggesting this "clear thinking" skill applies to many types of problems.

In a nutshell:
The paper argues that being "correct" isn't enough. An AI's reasoning should be so solid and clear that if you hand it to a different AI halfway through, that second AI can pick it up and finish the job perfectly. By training AI to be "transferable," they get smarter, more consistent, and learn much faster.

Technical Summary: Beyond Correctness: Learning Robust Reasoning via Transfer

1. Problem Statement

While Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved Large Language Model (LLM) reasoning by replacing learned reward models with rule-based verification of final answers, it suffers from a critical limitation: it optimizes solely for final-answer correctness. This approach fails to ensure the robustness of the reasoning process itself.

In practical settings, models that produce correct answers sporadically but inconsistently across different sampling paths or decoding variations are less valuable than those that generate stable, reproducible reasoning. Prior work indicates that RLVR can lead to "brittle" reasoning trajectories where increasing the number of samples ( $K$ ) does not improve, and may even degrade, majority voting accuracy (Maj@K). The current paradigm lacks an incentive for intermediate reasoning steps to be stable, interpretable, or reusable by other models.

2. Methodology: Reinforcement Learning with Transferable Reward (RLTR)

The authors propose RLTR, a framework that operationalizes robust reasoning through the concept of reasoning transferability. The core philosophy is that robust reasoning should remain useful beyond the specific model that produced it; it must survive truncation, reinterpretation, and continuation.

Core Mechanism

RLTR augments the standard RLVR objective with a Transfer Reward. The training process involves two models:

Generator Model ( $M_{gen}$ ): A trainable model that produces a full reasoning trace and final answer.
Receiver Model ( $M_{rcv}$ ): A separate, frozen model used to evaluate the quality of the generator's intermediate reasoning.

The Transfer Reward Process:

The generator produces a reasoning sequence $y_{gen}$ .
The reasoning trace is truncated at a random ratio $\tau \in (0, 1]$ to create a prefix $y_{gen}[:\ell]$ .
This prefix is fed to the frozen receiver model, which attempts to complete the reasoning and generate a final answer.
The Transfer Reward ( $R_{trans}$ ) is binary: it is 1 if the receiver's completed answer matches the ground truth, and 0 otherwise.

Objective Function:
The total reward for the generator is a weighted combination of the standard answer reward and the transfer reward:
$R_{RLTR}(y_{gen}, M_{rcv}) = R_{RLVR}(y_{gen}) + t \cdot R_{trans}(y_{gen}, M_{rcv})$
Where $R_{RLVR}$ includes the answer correctness and format rewards, and $t$ is the weight for the transfer component. The generator is optimized using Group Relative Policy Optimization (GRPO) to maximize this combined reward.

3. Key Contributions

Novel Framework: Introduction of RLTR, the first RLVR framework that explicitly incentivizes LLMs to produce reasoning traces that are transferable across different models.
Process-Level Supervision: Unlike Process Reward Models (PRMs) that require dense step-level annotations and learned verifiers, RLTR achieves process-level supervision using only the ground-truth answer verification, without additional annotation costs or training auxiliary reward models.
Enhanced Consistency: Demonstration that optimizing for transferability directly improves sampling consistency (Maj@K), addressing the brittleness observed in standard RLVR.
Sample Efficiency: The framework achieves comparable or superior performance to RLVR with significantly fewer training steps (approximately 2.5 $\times$ fewer on MATH-500).

4. Experimental Results

The authors evaluated RLTR on multiple mathematical and scientific reasoning benchmarks, including MATH-500, GSM8K, AMC23, AIME2024, and GPQA.

Accuracy and Consistency: RLTR consistently outperforms both the base model and RLVR-trained models across average accuracy and Maj@K metrics.
- On MATH-500, RLTR improved Maj@64 from 82.6% (RLVR) to 84.2%, while also raising average accuracy from 71.0% to 77.0%.
- On AMC23, RLTR improved Maj@64 from 61.7% to 67.5%.
- On GPQA (scientific reasoning), RLTR improved average accuracy from 33.0% to 34.8% and Maj@16 from 36.8% to 37.7%, demonstrating domain-agnostic generalization.
Training Dynamics:
- Speed: RLTR reaches the accuracy level of RLVR in roughly 2.5 $\times$ fewer training steps.
- Correlation: There is a strong positive correlation between the growth of transferability and Maj@K during training. In contrast, RLVR models often show a degradation in transferability and Maj@K in later training stages despite stable average accuracy.
Ablation Studies:
- Reward Ratio: Increasing the weight of the transfer reward ( $t$ ) significantly boosts high-K consistency (Maj@K), confirming that transferability is a distinct and valuable signal.
- Receiver Models: The method remains effective even with smaller receiver models (e.g., 1.5B parameters), though stronger receivers yield better high-K consistency.
- Diversity: Unlike RLVR, which tends to reduce diversity (Pass@K) as it concentrates on specific solution patterns, RLTR maintains or improves Pass@K while increasing accuracy.
Computational Cost: While RLTR introduces extra computation via the receiver model, it is more computationally efficient overall. To achieve the same performance as RLTR, RLVR requires approximately 2.3 $\times$ more computational cost (FLOPs).

5. Significance and Claims

The paper claims that optimizing for reasoning transferability leads to more robust LLM reasoning. By treating reasoning as a form of "meaning transfer" that must survive interruption and re-completion, RLTR encourages models to generate stable, interpretable, and genuinely generalizable reasoning paths.

The authors position this work as a solution to the "outcome-only" evaluation bottleneck in current RLVR methods. They argue that robust reasoning is not merely about being correct, but about being explainable in a way that others can reliably follow. This approach offers a path toward more reliable AI agents in complex domains (such as scientific discovery or coding) where consistency and reproducibility are as critical as raw accuracy, all while reducing the computational burden of training.