Technical Summary: Preplan Empowered LLM Mathematical Reasoning (PPC)

1. Problem Statement

Current plan-based reasoning methods for Large Language Models (LLMs) have established a question → plan → CoT paradigm. While effective, the authors identify a fundamental, paradigm-level gap in this approach: both the planning and execution stages are dedicated to deciding how to solve a problem (organizing steps and carrying them out), while the prior question of what to solve remains entirely implicit.

The "what to solve" stage involves recognizing the problem type, identifying applicable tools and concepts, flagging boundary conditions, and anticipating pitfalls. Existing methods assume this understanding emerges implicitly during plan generation. However, error analysis (Figure 1) reveals that a significant portion of failures in current plan-based methods stems not from calculation errors, but from a failure to correctly understand the problem's nature (e.g., misidentifying a homogeneous quadratic equation, leading to an inefficient solution path).

2. Methodology: The PPC Framework

To bridge this gap, the authors propose PPC (Preplan-Plan-CoT), a framework that introduces an explicit preplan stage, creating a new question → preplan → plan → cot paradigm. The preplan is a comprehensive, non-computational analysis that settles what to solve, leaving how to solve to the subsequent planning and execution stages.

The framework addresses two primary challenges through a specific pipeline:

Challenge 1: Data Construction (Preventing Leakage and Spoilers)

Generating clean preplan supervision is difficult because models tend to revert to "how to solve" behaviors.

• Leakage Failure: The preplan rehearses the step sequence of the upcoming plan.
• Spoiler Failure: The preplan performs intermediate computations under the guise of analysis.

Solution: The authors design a three-stage synthesis pipeline guarded by a spoiler-score detector.

1. Three-Stage Generation: Trajectories are generated strictly left-to-right: $y_{pp} \sim \pi_{pp}(\cdot|q)$, $y_p \sim \pi_p(\cdot|q, y_{pp})$, and $y_e \sim \pi_e(\cdot|q, y_p)$. The preplan prompt explicitly forbids derivations, extended formulas, and computed values.
2. Spoiler-Score Filtering: A rule-based score $s(\cdot) \in \{0, \dots, 6\}$ aggregates signals of derivation and answer revelation (e.g., phrases like "simplifies to," high density of equations, long inline math spans, multi-digit constants). Trajectories are retained only if $s(y_{pp}) \leq \tau_s$ (where $\tau_s=2$) AND the final answer is correct. This ensures the preplan is pure problem understanding, not a disguised solution.

Challenge 2: Training-Time Adherence (Ensuring Utility)

Even with clean data, a model might learn to generate a superficially correct preplan while ignoring it during the planning stage.

Solution: The authors extend Group Relative Policy Optimization (GRPO) with a composite reward function $R(y)$ that enforces structural and semantic alignment:
$$R(y) = R_{out}(y) + \lambda_a R_{adh}(y) + \lambda_f R_{fmt}(y) - \lambda_s R_{sty}(y)$$

• $R_{out}$ (Outcome Reward): Uses partial credit based on an LLM-judged proximity of the solution path (not just binary correctness) to provide gradients for near-miss trajectories.
• $R_{adh}$ (Adherence Reward): An LLM critic scores the strategic alignment between the preplan and the plan. Crucially, a high-quality plan that ignores the preplan receives a low score, forcing the model to genuinely follow the preplan's guidance.
• $R_{fmt}$ (Format Guard): Ensures the output contains exactly one instance of each tag (<preplan>, <plan>, <execute>) in order with a boxed answer.
• $R_{sty}$ (Style Penalty): Reuses the spoiler score $s(\cdot)$ as a penalty during RL. If the preplan drifts back into computational content during training, the reward is penalized, maintaining the conceptual integrity established during data construction.

3. Key Contributions

• Conceptual: The paper formalizes a paradigm-level gap in existing reasoning methods: the explicit modeling of how to solve versus the implicit handling of what to solve. It proposes the question → preplan → plan → cot paradigm to address this.
• Methodological:
  • Construction of the first reasoning dataset with explicit preplan supervision, protected by a spoiler-score detector against leakage and spoiler failures.
  • Design of a composite GRPO reward that explicitly enforces plan-preplan alignment, preventing the model from treating the preplan as a decorative but ignored step.
• Empirical: Extensive evaluation across four backbones (Qwen3-4B, Qwen2.5-7B, Qwen2.5-Math-7B, Llama-3.1-8B) and five mathematical benchmarks (AIME25, Minerva-Math, OlympiadBench, MATH-500, GSM8K).

4. Experimental Results

• Performance: PPC achieves the best results on 39 of 40 metrics across all benchmarks and backbones.
• Improvements: Compared to the strongest baselines (Plan-Tuning, PTA-GRPO), PPC improves maj@16 by +2.23 and pass@16 by +3.06.
• Efficiency: Despite adding an extra stage, PPC generates fewer tokens than full CoT and significantly fewer than PTA-GRPO. The preplan and plan constrain execution to a single solution path, avoiding the redundant detours common in unconstrained CoT.
• Faithfulness: Inference-time interventions (shuffling preplan sentences, using mismatched preplans, or generic placeholders) cause severe performance drops (e.g., maj@16 dropping from 97.2 to 46.8 on Qwen3-4B with mismatched preplans). This confirms the model genuinely relies on the preplan's content rather than merely producing it.

5. Significance and Claims

The paper claims that the primary limitation of current plan-based reasoning is not the lack of planning, but the lack of a dedicated stage for problem understanding. By making the "what to solve" explicit through the preplan, the model can select appropriate tools and strategies before committing to a solution path.

The authors emphasize that this improvement is not merely due to prompting (as shown by the "Prompt-Only" baseline which underperforms) but requires learning the paradigm through the specific data construction and composite reward training. The framework demonstrates that explicit problem understanding yields higher accuracy and greater inference efficiency simultaneously, without introducing additional token overhead compared to standard CoT.