Unmasking Reasoning Processes: A Process-aware Benchmark for Evaluating Structural Mathematical Reasoning in LLMs

1. Problem Statement

Recent Large Language Models (LLMs) have achieved near-saturation accuracy on established mathematical reasoning benchmarks (e.g., GSM8K, MATH, AIME). However, the authors argue that these high scores may not reflect genuine reasoning competence but rather the models' ability to exploit template-based computations, shallow arithmetic decompositions, or memorized patterns from pre-training data.

Current evaluation methods suffer from three main limitations:

• Answer-Only Bias: Benchmarks primarily evaluate final-answer correctness, masking "lucky guesses" where the model arrives at the right answer through flawed or non-robust reasoning chains.
• Lack of Structural Diagnosis: Existing datasets rarely assess specific structural reasoning skills such as multi-constraint coordination, constructive logical synthesis, and spatial inference.
• Data Contamination: Heavy reliance on public educational resources increases the risk of data overlap with pre-training corpora, making it difficult to distinguish between reasoning and memorization.

2. Methodology

The paper introduces a comprehensive framework consisting of a new benchmark, a structural diagnosis mechanism, and a learned verification model.

A. REASONINGMATH-PLUS Benchmark

• Dataset: A curated collection of 150 problems designed to stress structural reasoning rather than symbolic manipulation.
• Focus Areas: The problems emphasize intuitive logic, combinatorial and number-theoretic construction, and spatial reasoning. They are designed to require the coordination of multiple constraints and the construction of minimal yet sufficient reasoning chains.
• Annotations: Each problem includes:
  • A gold final answer.
  • A full human-written solution.
  • Minimal Reasoning Skeletons: A concise sequence (2–10 steps, median 5) of essential intermediate assertions required for a correct solution. These skeletons serve as a structural reference for step-level evaluation without constraining the model's surface realization (e.g., allowing for longer traces with self-verification).
• Bilingual Support: Parallel Chinese and English versions are provided to mitigate language-specific biases.

B. Evaluation Framework

The authors propose a dual-perspective evaluation framework to assess reasoning validity beyond outcome correctness:

1. Branch A: Skeleton-Guided Structural Diagnosis (HCRS)

   • Mechanism: Uses an external LLM judge (Gemini-3-Pro) to verify each step of a model's reasoning trace against the "Minimal Reasoning Skeleton."
   • HCRS (Hazard-aware Chain-based Rule Score): A deterministic scoring function that aggregates step validity.
     • Base Score: Normalized count of valid steps.
     • Format Penalty: Penalizes deviations from the expected reasoning length.
     • Hazard Penalty: Applies a weighted penalty based on the position of the first error. Early errors are penalized more heavily (hazard-based weighting) because they propagate and dominate downstream reasoning, a phenomenon often ignored in standard metrics.
   • Output: A process-only score ($S_{HCRS}$) and a holistic score ($S_{hol}$) combining process and answer accuracy.

2. Branch B: Outcome-Conditioned Verification (PRM)

   • Mechanism: A Process Reward Model (PRM) trained to predict step validity without access to the gold reasoning skeleton.
   • Training: Trained on 33k step-level instances generated by 14 LLMs and labeled by a "Teacher Judge" (Gemini-3-Pro) conditioned only on the problem statement and the gold final answer.
   • Function: Acts as a scalable, learned verifier that captures semantic coherence and logical progression, complementing the strict structural checks of HCRS.

3. Key Contributions

1. REASONINGMATH-PLUS: A novel benchmark specifically targeting structural reasoning failures (multi-constraint deduction, constructive reasoning) that are underrepresented in current datasets.
2. Minimal Reasoning Skeletons: A new annotation paradigm providing a task-specific structural reference for controlled, fine-grained process-level analysis.
3. HCRS Scoring Function: A deterministic, hazard-aware scoring mechanism that explicitly penalizes early logical fractures, revealing the fragility of reasoning chains that answer-only metrics miss.
4. Dual-Mode Evaluation: A framework integrating rule-based structural diagnosis (HCRS) with learned semantic verification (PRM), validated against human expert judgments.

4. Experimental Results

The authors evaluated 14 leading LLMs (including GPT-5, Gemini-2.5-Pro, Qwen3, Claude, etc.) on the benchmark.

• The "Lucky Guess" Phenomenon:
  • While leading models achieved high final-answer accuracy (up to 5.8/10), their holistic HCRS scores were substantially lower (average 4.36/10, best 5.14/10).
  • Analysis of 996 samples with correct final answers revealed that 6.63% had low process scores ($S_{HCRS} \le 3$), indicating they were "lucky guesses" with flawed reasoning.
• Stratification of Models:
  • HCRS provided clearer stratification than answer accuracy alone. For example, Gemini-2.5-Pro ranked first (4.89), while models like Cogito v2 dropped significantly (2.35) after penalties, exposing their reliance on fragile reasoning paths.
  • The "Hazard Penalty" alone accounted for a reduction of ~2.7–3.2 points (approx. 30% of the scale) for many models, highlighting the impact of early errors.
• Model Alignment:
  • The PRM achieved a competitive Pearson correlation of 0.602 with human judgments, closely tracking the "Teacher Judge" (0.639), demonstrating that learned verifiers can effectively capture reasoning quality without explicit skeletons.
  • Applying HCRS penalties to the PRM further improved its alignment with human judgments (increasing correlation from 0.604 to 0.633), suggesting that rule-based structural checks enhance learned verifiers.
• Domain Performance:
  • Models showed significant divergence in Combinatorics and Probability, where structural reasoning is critical. Mid-tier models often maintained high accuracy scores but collapsed in holistic scores, indicating brittle reasoning in these domains.

5. Significance

This work fundamentally shifts the paradigm of evaluating LLM reasoning from outcome-based to process-aware.

• Diagnosing Robustness: It demonstrates that high answer accuracy is a poor proxy for genuine reasoning competence. Models can "hallucinate" their way to the correct answer, a risk that HCRS effectively quantifies.
• Standardizing Process Evaluation: By introducing minimal reasoning skeletons and hazard-aware scoring, the paper provides a reproducible method for fine-grained auditing of reasoning chains, identifying exactly where and why reasoning fails (e.g., early logical fractures).
• Future Directions: The framework bridges the gap between high-precision structural diagnostics and scalable learned verification, offering a transparent paradigm for developing more robust, verifiable reasoning systems in AI. It suggests that future benchmarks must prioritize the structure of the solution, not just the result.