Technical Summary: Plan Then Action: High-Level Planning Guidance Reinforcement Learning for LLM Reasoning

1. Problem Statement

Large Language Models (LLMs) have demonstrated strong reasoning capabilities through Chain-of-Thought (CoT) prompting. However, the standard token-level generation process is fundamentally a Markov Decision Process (MDP) that is autoregressive and locally greedy. This leads to several critical limitations:

• Lack of Global Planning: Models optimize for short-horizon token likelihood, often resulting in redundant, drifting, or inaccurate reasoning chains.
• Propagation of Errors: Early mistakes in the reasoning trajectory are propagated across long horizons because the model lacks a mechanism to recover or re-route.
• Limitations of Existing Solutions:
  • Tree-based Search (e.g., MCTS): While effective in widening exploration, these methods rely on repeated external queries, incurring high computational costs and failing to strengthen the model's internal reasoning capabilities.
  • Standard Reinforcement Learning (RL): Methods like Group Relative Policy Optimization (GRPO) optimize only for final answer correctness (outcome-based rewards). They ignore the quality of intermediate planning and reasoning steps, meaning poorly structured plans can receive the same reward as well-organized ones if the final answer happens to be correct.

2. Methodology: PTA-GRPO

The authors propose Plan-Then-Action Enhanced Reasoning with Group Relative Policy Optimization (PTA-GRPO), a two-stage framework designed to jointly optimize high-level planning and fine-grained CoT reasoning.

Stage 1: Planning Structured Reasoning Cold-Start (PSR-CS)

Before reinforcement learning, the model undergoes Supervised Fine-Tuning (SFT) to acquire the ability to generate analytic plans.

• Dataset Construction: The authors construct a dataset ($D_{PSR-CS}$) where each instance includes a problem ($q$), a concise high-level analytic plan ($t$), and the detailed reasoning/answer ($c$).
• Plan Generation: The analytic plans ($t$) are summarized from ground-truth CoT reasoning using a strong LLM (e.g., Qwen3-235B). The plan is enclosed in <plan>...</plan> tags, while reasoning is in <thought>...</thought> and the answer in <answer>...</answer>.
• Objective: The model learns the conditional distribution $c = \pi_\theta(\cdot | q, t)$, effectively learning to condition detailed reasoning on a global guide.

Stage 2: Plan Structure-Guided Reinforcement Learning (PSG-RL)

Building on the SFT-initialized model, the authors introduce a novel RL method based on GRPO that explicitly rewards the quality of the analytic plan.

• Sampling Process: For a given question $q$, the policy samples $m$ candidate analytic plans $\{t_i\}$. For each plan $t_i$, the model resamples $z$ detailed CoT trajectories $\{c_{i,k}\}$.
• Composite Reward Function: The total reward $R_{total}$ consists of three components:
  1. Outcome Reward ($r_{outcome}$): A binary reward (1 or 0) based on whether the final extracted answer matches the ground truth.
  2. Analytic Plan Reward ($r_{analytic}$): A novel reward that evaluates the quality of the plan $t_i$. It is defined as the empirical accuracy of the $z$ resampled CoT trajectories generated under that specific plan. Plans that consistently lead to correct answers receive higher rewards. This serves as a proxy for plan quality, encouraging the model to generate plans that guide reasoning effectively.
  3. Format Reward ($r_{format}$): Enforces structural consistency (correct tags) and penalizes overly long responses.
• Optimization: The model is updated using the Group Relative Policy Optimization objective, utilizing the composite reward to calculate advantages.

Theoretical Basis

The paper provides a theoretical analysis showing that optimizing the analytic plan reward increases the mutual information $I(\hat{y}; y | t, q)$ between the predicted answer and the ground truth, given the plan. The authors prove that the error probability is upper-bounded by the conditional entropy, which decreases as the mutual information increases. Thus, improving the plan quality directly tightens the bound on error probability.

3. Key Contributions

1. Novel Two-Stage Framework: PTA-GRPO introduces a training paradigm that explicitly separates and optimizes high-level planning and execution, moving beyond purely outcome-based RL.
2. High-Level Guidance as a Supervision Signal: The method leverages SFT to inject structured planning capabilities into the model, initializing it with the ability to summarize reasoning into global guides.
3. Plan-Aware Reward Mechanism: The authors formulate the analytic plan as an explicit optimization variable within the RL objective, using the success rate of resampled trajectories as a reward signal for plan quality.
4. Empirical Validation: The method is evaluated across five diverse base models (ranging from 3B to 14B parameters) and ten benchmarks covering mathematics and natural sciences.

4. Experimental Results

The authors evaluated PTA-GRPO on benchmarks including MATH500, AIME24, AIME25, AMC23, and multimodal datasets (MMMU-Pro, EMMA, MMK-12).

• Performance Gains: PTA-GRPO consistently outperforms base models and other RL baselines (GRPO, DAPO, CPL, Full-Step-DPO).
  • On Qwen2.5-7B-Instruct, the average accuracy across math benchmarks improved from 34.23% (Base) to 58.46% (PTA-GRPO), surpassing GRPO (54.81%) and DAPO (55.04%).
  • On Qwen3-14B, PTA-GRPO achieved an average of 85.01%, outperforming GRPO (83.08%) and DAPO (83.31%).
• Data Scaling: Increasing the RL training data from 4k to 60k samples resulted in consistent performance improvements, suggesting the method scales well with data volume.
• Ablation Studies: Removing the SFT cold-start or the analytic plan reward ($r_{analytic}$) led to significant performance drops, confirming the necessity of both the initialization and the specific reward signal.
• Generalization: The method demonstrated strong generalization to multimodal and scientific reasoning tasks, not just pure mathematics.
• Test-Time Scaling: PTA-GRPO showed superior performance in Pass@K metrics (Pass@1 to Pass@16) compared to GRPO, indicating robustness under multiple sampling conditions.

5. Significance and Claims

The paper claims that PTA-GRPO addresses the "lack of global planning" in LLM reasoning by treating analytic planning as a first-class optimization target rather than a mere structural add-on.

• Internal vs. External: Unlike tree-search methods that rely on external computation, PTA-GRPO strengthens the model's internal reasoning capabilities.
• Robustness: By rewarding the quality of intermediate steps (via the plan reward), the method reduces redundant reasoning and helps the model avoid local optima, leading to more stable and globally guided trajectories.
• Efficiency: The method achieves these gains without incurring significant additional computational costs compared to standard GRPO, as the plan reward is computed using the same rollout budget (resampling CoTs under generated plans).
• Modesty: The authors acknowledge that while the method significantly improves reasoning on benchmarks, it does not introduce new data sources or external tools. They also note that stronger reasoning capabilities carry potential risks (e.g., misuse in academic dishonesty) and emphasize the need for responsible deployment and alignment research.

In summary, the paper posits that explicitly optimizing for high-level planning guidance, in conjunction with final answer correctness, is a critical step toward more reliable, coherent, and generalizable reasoning in Large Language Models.