Technical Summary: ReCode: Reinforcing Code Generation with Reasoning-Process Rewards

Problem Statement

Reinforcement Learning (RL) has become a transformative paradigm for improving Large Language Models (LLMs) in code generation. However, existing approaches predominantly rely on outcome-based supervision, such as test-case pass rates, which provide limited guidance on how to reach a correct program. While rigorous reasoning is often a key driver of correct code, current RL methods frequently neglect optimizing the quality of the reasoning process itself.

Translating the idea of optimizing reasoning processes into a practical training framework faces two primary challenges:

1. Scarcity of Fine-Grained Preference Data: Training reliable reward models to assess reasoning quality is hindered by the lack of high-quality, fine-grained preference data. Human annotation is unscalable and subjective, while using LLMs-as-a-judge often suffers from poor calibration when assigning scalar scores to nuanced reasoning qualities.
2. Reward Hacking: Naively incorporating neural rewards for reasoning processes into RL poses significant risks. Unlike strict, verifiable unit-test outcomes, neural rewards are less constrained. A policy may learn to "hack" the reward by inflating the reasoning process score without actually improving the functional correctness of the generated code.

Methodology: ReCode Framework

The authors propose ReCode (Reasoning-Reinforced Code Generation), a novel RL training framework designed to align code generation with high-quality reasoning processes. ReCode consists of two core components:

1. Contrastive Reasoning-Process Reward Learning (CRPL)

To address the data scarcity challenge, CRPL introduces a method to automatically synthesize contrastive preference data without relying on human annotation or direct scalar scoring.

• Process: The framework codifies reasoning quality into three critical dimensions: Factual Accuracy, Logical Rigor, and Logical Coherence.
• Data Synthesis: Using a powerful LLM (Qwen2.5-Coder-32B-Instruct), CRPL generates Optimized ($t^+$) and Degraded ($t^-$) variants of a base reasoning process ($t$) by applying targeted transformations along these dimensions (e.g., removing redundancies for optimization, or introducing factual errors for degradation).
• Preference Ordering: This process creates a multi-level preference ordering, forming three types of pairs: strong contrast ($t^+ \succ t^-$), fine-grained optimization ($t^+ \succ t$), and fine-grained degradation ($t \succ t^-$).
• Training: A reward model is trained on these preference pairs using a Bradley-Terry objective to discriminate between reasoning qualities, rather than predicting absolute scores.

2. Consistency-Gated GRPO (CG-GRPO)

To address the risk of reward hacking, ReCode integrates the reasoning-process reward model into the RL loop via a gating mechanism built upon Group Relative Policy Optimization (GRPO).

• Hard Gate: The reasoning-process reward ($R_{proc}$) is only activated if the generated code passes all unit tests (functional correctness).
• Reward Structure: The total reward $R_i$ for a sample is formulated as:
  $$R_i = R_{fmt} + R_{out} + \mathbb{I}(R_{out} = 1) \cdot R_{proc}$$
  Where $R_{fmt}$ is a format reward, $R_{out}$ is the binary outcome reward, and $\mathbb{I}(\cdot)$ is the indicator function.
• Effect: This ensures that the policy cannot compromise functional correctness to maximize the neural reasoning score. It also provides informative gradients among correct solutions, encouraging the model to prefer high-quality reasoning processes even when the code is already functionally correct.

Benchmark Construction: LCB-RB

To evaluate the discriminative capability of reasoning-process reward models, the authors introduce LiveCodeBench-RewardBench (LCB-RB).

• Construction: Derived from LiveCodeBench, this benchmark comprises 219 manually verified preference pairs of superior and inferior reasoning processes.
• Filtering: A two-stage filtration pipeline (Automated Dual-Consistency Check using GPT-4o followed by Human Adjudication) ensures strict alignment between reasoning and implementation, filtering out cases where code passes tests despite flawed reasoning or fails due to minor implementation issues despite sound reasoning.

Experimental Results

Extensive experiments were conducted on HumanEval(+), MBPP(+), LiveCodeBench, and BigCodeBench using Qwen2.5-Coder-7B-Instruct as the policy model.

• Code Generation Performance: A 7B model trained with ReCode achieved a 16.1% relative improvement over the base model (50.4% $\to$ 58.5% average pass rate), reaching performance comparable to GPT-4-Turbo. It outperformed the outcome-only GRPO baseline by 6.7%.
• Reward Model Effectiveness: The reward model trained with CRPL achieved strong discriminative performance on LCB-RB, surpassing state-of-the-art baselines (including GPT-4-Turbo and specialized reward models). Notably, the 7B CRPL-trained model performed comparably to GPT-4-Turbo on LCB-RB.
• Generalization:
  • Math Domain: Extending CG-GRPO to the math domain (using Qwen2.5-Math-7B) yielded consistent gains, with a 7.4% relative improvement over the RL baseline without reasoning rewards on AIME 2024.
  • Cross-Model: The learned reasoning-process reward transferred effectively to the Qwen3-4B-Instruct model.
• Efficiency: ReCode achieved higher Pass@1 rates while generating 23.4% fewer tokens on average compared to outcome-only RL, indicating more concise and targeted reasoning.
• Ablation Studies:
  • Reward Hacking: Alternative formulations without the consistency gate (e.g., direct addition of rewards) led to reward saturation and performance degradation, confirming the necessity of the gating mechanism.
  • Data Synthesis: Training on single-generator preference data outperformed cross-generator data, suggesting that consistent stylistic artifacts in the synthesis process provide cleaner supervision signals.

Significance and Claims

The paper claims that ReCode successfully demonstrates that reasoning-process quality can be exploited as an additional training signal to improve code generation beyond final outcomes. Its significance lies in:

1. Bridging the Gap: It provides a scalable solution for training reasoning-process reward models by synthesizing contrastive data, bypassing the need for expensive human annotation.
2. Safety in RL: The Consistency-Gated mechanism offers a robust method to integrate neural process rewards into RL without succumbing to reward hacking, ensuring that reasoning improvements translate to functional correctness.
3. Generalizability: The framework is not limited to code; it generalizes to mathematical reasoning tasks, suggesting a broader applicability for improving LLM reasoning capabilities.
4. New Evaluation Standard: The introduction of LCB-RB addresses a gap in existing benchmarks by focusing specifically on the discrimination of intermediate reasoning processes rather than just final solution correctness.

The authors conclude that ReCode represents a step toward aligning LLMs with rigorous reasoning, complementing existing implementation-centric supervision methods.