A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models

This paper introduces MeRF, a method that enhances reinforcement finetuning of large reasoning models by injecting reward specifications directly into prompts as "motivation," thereby leveraging in-context learning to align generation with optimization objectives and achieve substantial performance gains over standard RLVR baselines.

The Gist

Here is an explanation of the paper "A Simple 'Motivation' Can Enhance Reinforcement Finetuning of Large Reasoning Models" (MeRF), translated into simple, everyday language with analogies.

The Big Idea: Teaching a Dog vs. Telling a Dog the Rules

Imagine you are trying to teach a very smart dog (the AI) how to solve a complex puzzle, like a logic maze or a math problem.

The Old Way (RLVR): The "Trial-and-Error" Method
Currently, the standard way to train these AI models is called RLVR (Reinforcement Learning with Verifiable Rewards).

• How it works: You give the dog a puzzle. It guesses an answer.
  • If it gets it right, you give it a treat (+1 reward).
  • If it gets it wrong, you give it nothing or a gentle "no" (-1 reward).
• The Problem: The dog has no idea why it got the treat or the "no." It just knows "I guessed X, and that was good." It has to guess thousands of times, get thousands of "nos," and slowly, by pure luck, stumble upon the pattern that leads to the treat. It's like trying to learn a new video game by pressing buttons randomly until you accidentally win a level. It takes a long time and wastes a lot of energy.

The New Way (MeRF): The "Rules of the Game" Method
The authors of this paper, MeRF (Motivation-enhanced Reinforcement Finetuning), realized that humans don't learn this way. When we start a new job or game, we read the rulebook first. We know what we are trying to achieve and how we will be graded.

• The Innovation: Before the AI starts guessing, the researchers simply tell it the rules. They inject a "Motivation" into the prompt.
• The Prompt: Instead of just saying "Solve this," they say: "Solve this. Here is how you will be graded: If your answer is correct, you get 10 points. If your format is wrong, you lose 2 points. If you are vague, you get 0."
• The Result: The AI now has a "North Star." It doesn't just blindly guess; it knows exactly what the "treat" looks like before it even starts moving. It aligns its thinking with the goal immediately.

The Analogy: The Blindfolded Archer vs. The Coach

Imagine an archer trying to hit a target in a dark room.

• RLVR (The Blindfolded Archer): The archer shoots an arrow. A coach yells "Hit!" or "Miss!" The archer has to guess which way to move their arm based on that single shout. They might shoot 1,000 arrows before they finally hit the bullseye.
• MeRF (The Archer with a Coach): The coach says, "The target is 10 feet away, slightly to the left. If you hit the red circle, you get a gold star. If you hit the white wall, you get nothing."
  • The archer (the AI) now uses its brain to calculate the trajectory before shooting. It doesn't need to shoot 1,000 times to learn the rules. It learns faster because it understands the objective.

Why This Matters (The "Aha!" Moment)

The paper found three surprising things:

1. It's Faster: By telling the AI the rules upfront, it learns the task much quicker. In the experiments, the "Motivation" models reached high accuracy in half the time (or fewer steps) compared to the standard models.
2. It Explores Better: Without the rules, the AI gets scared and stops trying new things (it "collapses" into a boring, safe answer that gets a tiny reward). With the rules, the AI feels confident enough to try different creative paths because it knows exactly what a "good" answer looks like.
3. It's Smart Enough to Ignore Bad Advice: The researchers tried tricking the AI by giving it wrong rules (e.g., "If you get the answer wrong, you get a gold star!").
   • At first, the AI got confused and tried to get the wrong answers.
   • But because the actual computer reward (the real treat) still only came for correct answers, the AI eventually realized, "Wait, the coach is lying. I should ignore the coach and listen to the real reward."
   • This shows the AI is robust; it can learn to distinguish between a bad description and the actual truth.

The Real-World Impact

This isn't just about logic puzzles. This method helps AI get better at:

• Math: Solving complex equations faster.
• Coding: Writing code that actually passes tests without needing thousands of failed attempts.
• Logic: Solving mysteries or riddles more efficiently.

Summary

The paper proposes a simple but powerful idea: Don't just let the AI guess and hope for a reward. Tell it the rules of the game first.

By giving the AI a "Motivation" (a clear description of how it will be graded), we turn a blind, trial-and-error process into a smart, goal-oriented learning session. It's the difference between teaching a child to ride a bike by pushing them off a hill and hoping they don't fall, versus showing them the handlebars, explaining how to balance, and then letting them try.

Technical Summary

Here is a detailed technical summary of the paper "A Simple 'Motivation' Can Enhance Reinforcement Finetuning of Large Reasoning Models" (MeRF).

1. Problem Statement

Context: Reinforcement Learning with Verifiable Rewards (RLVR) has become a dominant paradigm for improving the reasoning capabilities of Large Language Models (LLMs) on tasks like math, coding, and logic puzzles. Unlike traditional RLHF which relies on subjective human feedback, RLVR uses objective, rule-based rewards (e.g., matching a math answer to a ground truth).

The Challenge: Despite its success, the current RLVR paradigm suffers from inefficiency.

• Blind Exploration: The model operates in a "trial-and-error" manner, unaware of the specific optimization objective or the structure of the reward function during the generation phase. It must explore the reward space blindly to discover positive signals.
• Sparse Rewards: In complex reasoning tasks, the reward space is often sparse. The model may generate many incorrect outputs before stumbling upon a correct one, leading to slow convergence and high computational costs.
• Local Optima: Without explicit guidance, models often get stuck in local optima or engage in "reward hacking" (e.g., satisfying formatting rules but failing the core logic) because they lack a holistic understanding of the task's success criteria.

Core Question: Can we leverage the in-context learning (ICL) capabilities of LLMs to explicitly inform the model about the "rules of the game" (the reward function) during the reinforcement finetuning process, thereby making the training more efficient?

2. Methodology: Motivation-enhanced Reinforcement Finetuning (MeRF)

The authors propose MeRF, a method that injects a natural language description of the reward function (termed "Motivation") directly into the system prompt during the training process.

Key Components:

• In-Context Motivation: Instead of treating the reward function as a black box applied only after generation, MeRF provides the model with a textual description of the scoring rules (e.g., "If the answer is correct, score +2; if format is wrong, score -1").
• Mechanism:
  1. Prompt Injection: The "Motivation" (reward specification) is appended to the system prompt alongside the task instruction.
  2. Alignment: This serves as an in-context guide, aligning the model's generation process with the optimization objective before the output is generated.
  3. Dual Incentive: The model is incentivized by both the internal motivation (understanding the rules via ICL) and the external reward (the actual RL signal).
• Algorithm: The method utilizes GRPO (Group Relative Policy Optimization), a variant of PPO that does not require a critic model, making it computationally efficient for LLMs.

Example Motivation Structure:
The prompt explicitly defines:

• Correctness Score: Rules for evaluating the logical/mathematical correctness.
• Format Score: Rules for adhering to specific output structures (e.g., <answer> tags).
• Examples: Few-shot examples showing how the scoring works.

3. Key Contributions

1. Novel Paradigm (MeRF): Introduction of a simple yet effective method that bridges the gap between in-context learning and reinforcement learning by explicitly teaching the reward function to the model during training.
2. Efficiency Gains: Demonstrates that "telling the LLM the rules" significantly accelerates convergence compared to standard RLVR, reducing the number of training steps required to reach high performance.
3. Comprehensive Analysis:
   • Mechanism: Proves that the improvement comes from the training process (better exploration and policy updates) rather than just better inference-time prompting.
   • Consistency: Shows that performance is highest when the in-context motivation perfectly matches the actual reward function.
   • Robustness: Demonstrates that models can adapt to "adverse" (misleading) motivations over time, eventually learning to ignore the misleading text and rely on the actual reward signal, though performance is suboptimal compared to ground-truth motivation.
4. Generalizability: Validated across diverse reasoning tasks including Logic Puzzles (Knights and Knaves), Math (AIME, AMC, MATH500), and Number Games (CountDown).

4. Experimental Results

The authors evaluated MeRF on various models (Qwen2.5 series, DeepSeek-R1-Distill) and benchmarks.

• Performance on Logic Puzzles (K&K):
  • MeRF consistently outperformed the RLVR baseline across all model sizes (from 1.5B to 14B parameters).
  • Speed: MeRF achieved higher accuracy in fewer training steps. For instance, MeRF at step 140 outperformed the final RLVR baseline at step 280.
  • OOD Generalization: MeRF showed superior performance on out-of-distribution (harder) puzzle levels compared to the baseline.
• Performance on Math Benchmarks:
  • On AIME, AMC, and MATH500, MeRF achieved consistent gains in pass@k metrics (k=1, 2, 4, 8).
  • Average improvements were observed: +3.40% on pass@1, +4.50% on pass@8.
• Entropy and Exploration:
  • Entropy Analysis: MeRF maintained higher entropy (diversity of outputs) during training compared to RLVR. RLVR models tended to collapse to suboptimal solutions (low entropy) quickly due to sparse rewards. MeRF's explicit motivation encouraged broader exploration of the solution space.
  • Pass@k: The continuous improvement in pass@k metrics indicated that MeRF models were better at exploring diverse reasoning paths to find the correct answer.
• Ablation Studies:
  • Motivation vs. Inference: The performance gain was primarily due to the training process, not just the presence of motivation during inference.
  • Adverse Motivation: When given a motivation that contradicted the reward function (e.g., "Correct answers get -2 points"), the model initially struggled but eventually adapted to discount the misleading text, though it never matched the performance of the ground-truth motivation.

5. Significance and Implications

• Bridging ICL and RL: MeRF successfully leverages the inherent in-context learning ability of LLMs to enhance reinforcement learning, a synergy previously underutilized in RLVR.
• Cost-Effectiveness: By reducing the number of training steps and computational resources needed to achieve state-of-the-art reasoning performance, MeRF offers a more scalable path for training large reasoning models.
• Interpretability: The method makes the optimization objective transparent to the model, potentially reducing "reward hacking" and encouraging more robust reasoning strategies.
• Future Directions: The paper suggests that dynamic motivation (updating rules during training) and applying this to models with weaker generalization capabilities are promising areas for future research.

In conclusion, MeRF demonstrates that a simple modification—explicitly describing the reward function in the prompt—can transform the RLVR paradigm from a blind trial-and-error process into a guided, efficient, and highly effective learning strategy for large reasoning models.