Technical Summary: Hint-Guided Diversified Policy Optimization for LLM Reasoning

Problem Statement

Recent advancements in Large Language Models (LLMs) have demonstrated significant reasoning capabilities, particularly through Reinforcement Learning with Verifiable Rewards (RLVR). However, existing RLVR frameworks, such as Group Relative Policy Optimization (GRPO), are primarily constrained by outcome-level rewards. These mechanisms assess the correctness of the final answer but lack explicit signals to guide models toward exploring diverse solution paths. Consequently, models often suffer from solution homogenization; if a model becomes trapped in an incorrect reasoning trajectory, it struggles to recover or identify the correct answer due to insufficient exploration of the solution space. This limitation reduces the fault tolerance of reasoning processes, especially when the number of sampling attempts is restricted.

Methodology

To address these limitations, the authors propose Hint-Guided Diversified Policy Optimization (HDPO), a framework designed to incentivize the generation of diverse and reliable solutions through a "propose-select-think" reasoning trajectory. HDPO operates in two distinct stages:

1. Cold Start for Structured Reasoning

The first stage employs Supervised Fine-Tuning (SFT) to instill the "propose-select-think" pattern into the policy model.

• Data Construction: An advanced teacher LLM generates structured reasoning trajectories. For a given problem, the model is prompted to first list multiple candidate solution outlines (hints), select the most reliable one, and then proceed with detailed reasoning.
• Filtering: To ensure data quality, the generated trajectories undergo a two-fold filtering process:
  • Correctness Check: The final answer is compared against the ground truth.
  • Reliability Check: A lightweight model samples answers for each candidate solution outline to calculate accuracy. A trajectory is retained only if the selected candidate aligns with the one exhibiting the highest sampling accuracy.
• Objective: The policy model is fine-tuned to follow this structured format, establishing a foundation for generating candidate hints before reasoning.

2. Hint-Guided Diversified Reinforcement Learning

In the second stage, the model undergoes Reinforcement Learning (RL) to optimize the generation of diverse and reliable solutions. The overall reward function combines four components:

• Format Reward ($r_{fmt}$): Ensures the output adheres to the "propose-select-think" structure (e.g., presence of specific tags, number of candidates between 1 and 5).
• Accuracy Reward ($r_{acc}$): Provides a binary signal (1 or 0) based on whether the final answer matches the ground truth.
• Diversity Reward ($r_{div}$): Encourages the model to generate distinct candidate solutions. Candidate solutions are encoded into embeddings, and similarity is measured via cosine similarity. To prevent the model from prioritizing diversity over correctness in early training, a diversity scheduling strategy is employed. This uses a sine-based learning coefficient to gradually increase the weight of the diversity reward after a warm-up period.
• Reliability Reward ($r_{rel}$): Incentivizes the model to select the most reliable candidate. Since sampling-based reliability checks are computationally expensive during RL, the authors use token entropy as a proxy for confidence. Solutions with lower entropy (higher confidence) are ranked higher. The reward is inversely proportional to the rank of the selected solution among all candidates (e.g., if the selected solution is the $k$-th most reliable, $r_{rel} = 1/k$).

The total reward is computed as:
$$r_i = r_{fmt} + r_{acc} + \mathbb{I}[r_{acc} > 0] \cdot r_{div} + \bar{r}_{acc} \cdot r_{rel}$$
where the diversity reward is applied only when the answer is correct, and the reliability reward is scaled by the group-level average accuracy to mitigate overconfidence.

Key Contributions

1. HDPO Framework: Introduction of a novel framework based on the "propose-select-think" paradigm, enabling models to explicitly generate candidate solution hints before committing to a reasoning path.
2. Diversity and Reliability Mechanisms: Development of a diversity scheduling strategy and a confidence-based reliability reward that jointly optimize the exploration of the solution space and the selection of robust candidates.
3. Policy Internalization: Unlike inference-time search methods (e.g., Tree-of-Thought) that incur high latency through repeated sampling, HDPO internalizes the explore-then-exploit mechanism into the policy parameters via RLVR, enabling zero-overhead, single-pass reasoning.

Experimental Results

The authors evaluated HDPO on nine reasoning benchmarks, including AMC 2023, AIME 2024/2025, Math-500, Minerva, Olympiad-Bench, MMLU-Pro, SciBench, and GPQA-Diamond, using backbone models of varying scales (Qwen3-4B, Qwen2.5-Math-7B, and DeepSeek-R1-Distill-Llama-8B).

• Performance Gains: HDPO consistently outperformed standard GRPO and other state-of-the-art RLVR baselines (e.g., EDGE-GRPO, Critique-GRPO, LUFFY). For instance, on the Qwen3-4B model, HDPO achieved an average score of 61.03 compared to 53.76 for GRPO.
• Hit Rate Improvement: Experiments showed that HDPO maintains superior accuracy even with a limited number of sampling attempts (Hit@N), whereas GRPO performance drops significantly as the number of attempts decreases, indicating HDPO's superior ability to find correct solutions in fewer trials.
• Ablation Studies: Removing the cold start phase, diversity reward, or reliability reward resulted in significant performance degradation, confirming the necessity of each component. The study also validated that using a stronger teacher model for cold start data yields better results, though a self-evolving iterative approach also showed promise.
• Efficiency: While HDPO introduces a slight inference latency increase (less than 0.2 seconds compared to SOTA methods), it remains under 1 second, making it deployable in real-world scenarios while offering substantial reasoning improvements.

Significance and Claims

The paper claims that HDPO effectively addresses the issue of solution homogenization in current RLVR frameworks. By explicitly incentivizing the generation of diverse candidate solutions and training the model to select the most reliable one, HDPO enhances the model's fault tolerance and reasoning robustness. The authors emphasize that their approach moves beyond accuracy-centric optimization by jointly optimizing for diversity and reliability, allowing LLMs to explore a broader range of potential solutions and identify the most promising path without the computational overhead of inference-time search. The results suggest that HDPO is a scalable and effective strategy for enhancing LLM reasoning across both mathematical and general domains.