Learn Hard Problems During RL with Reference Guided Fine-tuning

This paper introduces Reference-Guided Fine-Tuning (ReGFT), a method that synthesizes model-aligned positive trajectories using partial human reference solutions to overcome reward sparsity and significantly enhance the performance and training efficiency of reinforcement learning for mathematical reasoning.

The Gist

Imagine you are trying to teach a brilliant but inexperienced student how to solve the world's most difficult math puzzles. You want them to learn by trial and error (Reinforcement Learning), where they get a gold star only if they get the answer right.

Here is the problem: If the puzzle is too hard, the student might try 1,000 times and get zero gold stars. They get frustrated, stop trying, and learn nothing. This is called the "Reward Sparsity" problem.

The paper introduces a clever new method called ReGFT (Reference-Guided Fine-Tuning) to fix this. Here is how it works, broken down into simple analogies.

1. The Old Way: The "Copy-Paste" Trap

Usually, when students are stuck, teachers give them the full solution.

• The Mistake: If you just give the student the full answer key and say, "Memorize this," they might be able to recite it back. But if you give them a new puzzle that looks slightly different, they fail. Why? Because they didn't learn how to think; they just memorized the text. They are "out of their depth."

2. The Previous "Smart" Way: ReFT (Reinforced Fine-Tuning)

Researchers tried a better method: "Only study the solutions you figured out yourself."

• How it works: The student tries to solve the puzzle. If they get it right, they study that solution. If they get it wrong, they ignore it.
• The Problem: For the really hard puzzles, the student gets zero correct answers. They have nothing to study. They are still stuck in the "zero gold star" zone.

3. The New Way: ReGFT (The "Halfway House" Guide)

This is the paper's big idea. Instead of giving the full answer or asking the student to solve it completely alone, the teacher gives a hint.

The Analogy: The Hiking Guide
Imagine the student is trying to hike up a steep, foggy mountain (the hard math problem).

• The Full Solution: The teacher draws a map of the entire mountain and says, "Here is the path." The student just traces the line. They don't learn how to navigate the fog.
• The "Do It Yourself" Approach: The teacher says, "Go find the top." The student wanders in circles, gets lost, and never reaches the top.
• ReGFT (The New Method): The teacher says, "I will walk with you for the first 80% of the hike. I will show you the path up to the halfway point. But then, you must take over and figure out the rest of the way to the summit on your own."

How ReGFT Works in Practice

1. The Setup: The computer (the student) is given a hard math problem.
2. The Hint: The computer is shown the first 80% of the human-written solution. This acts as a "guidepost."
3. The Challenge: The computer is told: "You know the start. Now, use your own brain to figure out the final 20% and the answer."
4. The Result: Because the computer had a head start, it is much more likely to solve the problem correctly.
5. The Learning: The computer now has a correct solution that it actually generated itself (based on the hint). It studies this solution.

Why This is a Game-Changer

• It fills the "Zero Reward" gap: Before, the computer would get stuck on hard problems and get no feedback. Now, with the hint, it can solve them and get a "gold star."
• It keeps the student's style: Because the computer had to finish the work itself, the solution sounds like its thinking, not a robot copying a human. This makes it easier for the computer to generalize to new problems later.
• It supercharges the next step: Once the computer has learned this way, it is much better at the "Trial and Error" phase (Reinforcement Learning). It starts with a higher skill level, learns faster, and eventually becomes a math genius.

The Bottom Line

The paper proves that if you want an AI to learn hard things, you can't just let it flail in the dark (pure trial and error), and you can't just force it to memorize answers (pure copying).

You have to be a smart coach: Give the AI a nudge in the right direction, let it do the heavy lifting, and then let it learn from its own success. This simple trick turns "unsolvable" problems into "solvable" ones, unlocking the AI's true potential.

Technical Summary

Here is a detailed technical summary of the paper "Learn Hard Problems During RL with Reference Guided Fine-tuning".

1. Problem Statement

The paper addresses a critical bottleneck in applying Reinforcement Learning with Verifiable Rewards (RLVR) to mathematical reasoning: Reward Sparsity.

• The Core Issue: In RLVR, models are trained by sampling multiple reasoning trajectories and receiving a reward only if the final answer is correct. For difficult problems (e.g., Olympiad-level math), the base Large Language Model (LLM) often fails to generate any correct trajectories during the sampling phase.
• Consequence: Without positive rewards, the model receives zero or sparse gradient signals, causing training to stall. The RL process cannot learn to solve problems it cannot initially solve.
• Limitations of Existing Solutions:
  • Direct Supervised Fine-Tuning (SFT) on References: Simply fine-tuning on human-written reference solutions often fails because the model cannot imitate reasoning patterns that lie outside its own intrinsic reasoning distribution (distributional mismatch).
  • Reinforced Fine-Tuning (ReFT): While ReFT improves performance by fine-tuning on self-generated correct trajectories, it cannot help with problems the model is currently incapable of solving, as no correct self-generated trajectories exist to begin with.

2. Methodology: Reference-Guided Fine-Tuning (ReGFT)

The authors propose ReGFT, a pre-RL fine-tuning strategy designed to synthesize high-quality training trajectories for "hard" problems that the model cannot solve independently.

Key Mechanism

Instead of directly copying human solutions or relying solely on self-exploration, ReGFT uses a partial reference guidance approach:

1. Partial Hints: For a given hard problem, the model is provided with the first 80% of a human-written reference solution (Chain-of-Thought) as a hint.
2. Self-Generation: The model is instructed to derive the rest of the solution and the final answer using its own reasoning process, ensuring the output remains within the model's reasoning distribution.
3. Trajectory Synthesis: This process generates "reference-guided" trajectories that are:
   • Correct: They lead to the right answer (verified by the reference).
   • Aligned: They follow the model's own generative style, avoiding the brittleness of direct imitation.
4. Training Data Construction: The model is fine-tuned on a mixture of:
   • Self-generated correct trajectories (from standard sampling).
   • Reference-guided trajectories (synthesized via the partial hint method).
   • Note: This is applied specifically to "hard" problems (defined as those with <25% accuracy under the original model).

RL Integration

After ReGFT, the model serves as a stronger initialization for Reinforcement Learning (specifically using the DAPO algorithm). Because the model has already learned to solve previously unsolvable problems during the ReGFT phase, the subsequent RL phase receives denser, more informative reward signals, preventing training stagnation.

3. Key Contributions

1. Novel Pre-RL Strategy: Introduction of ReGFT, which bridges the gap between human reference solutions and model-generated reasoning, specifically targeting problems where the base model fails.
2. Mitigation of Reward Sparsity: Demonstrates that synthesizing training data via partial references effectively converts "zero-reward" problems into "positive-reward" problems for the subsequent RL stage.
3. Superiority over Direct SFT and ReFT:
   • Proves that direct SFT on full references is insufficient due to distributional mismatch.
   • Shows that ReFT alone is limited to the model's current capability boundary, whereas ReGFT expands this boundary.
4. Inference-Time Scaling: Establishes that ReGFT leads to better pass@k scaling, meaning the model's ability to find correct solutions improves consistently as the inference compute budget increases, unlike methods that only improve pass@1.

4. Experimental Results

The authors evaluated ReGFT on the OmniMath dataset (training) and three benchmarks: AIME'24, AIME'25, and Beyond-AIME.

• RL Performance: Models initialized with ReGFT consistently outperformed both raw checkpoints and ReFT-initialized checkpoints across all benchmarks.
  • Faster Convergence: ReGFT models achieved higher accuracy in fewer training steps.
  • Higher Final Accuracy: ReGFT reached a higher performance plateau.
• Ablation Studies:
  • Direct SFT vs. ReGFT: Directly fine-tuning on raw human solutions resulted in significantly lower RL performance, confirming the necessity of model-derived reasoning.
  • ReFT vs. ReGFT: While ReFT accelerated early learning, ReGFT achieved superior asymptotic performance, proving that external guidance is crucial for solving problems beyond the model's initial reach.
• Pass@k Scaling: ReGFT + DAPO showed the most robust scaling. As the number of sampled generations ($k$) increased, ReGFT maintained a clear performance margin, indicating a broader coverage of the solution space.
• Hard Problem Solving: On the OmniMath dataset, reference-guided sampling solved an additional 5.85% of problems that standard sampling could not solve, directly validating the method's ability to unlock new reasoning capabilities.

5. Significance

This work fundamentally shifts the paradigm of RL for reasoning from "pure exploration" to "guided initialization."

• Efficiency: It reduces the computational waste associated with RL on hard problems where the model generates no correct answers.
• Generalization: By forcing the model to generate its own reasoning based on partial hints, ReGFT ensures the learned skills generalize better than simple imitation learning.
• Scalability: The method is orthogonal to the specific RL algorithm used (demonstrated with DAPO), suggesting it can be combined with future advances in RL to further push the boundaries of mathematical reasoning in LLMs.

In summary, ReGFT effectively transforms human reference solutions into a scaffold that allows models to "learn how to learn" difficult problems before reinforcement learning begins, thereby unlocking stronger, more stable, and more scalable mathematical reasoning capabilities.