Technical Summary: The Quality-Utility Paradox in Small Model Mathematical Reasoning

Problem Statement

The prevailing paradigm for training Small Language Models (SLMs) on mathematical reasoning relies on knowledge distillation from stronger "Oracle" models. The standard assumption is that reasoning traces with higher reward model scores—indicating perceived higher quality, rigor, or correctness—provide superior supervision for the target SLM. However, this paper identifies a counterintuitive phenomenon termed the Quality-Utility Paradox: training data refined or synthesized by a powerful Oracle, despite achieving significantly higher reward scores, consistently underperforms compared to traces generated by the SLM itself and selected via rejection sampling (RFT).

The core issue is not that Oracle refinement is intrinsically harmful, but that current data selection criteria conflate two distinct effects:

1. Logical Repair: The improvement of reasoning steps and correctness.
2. Distributional Drift: The shift of the reasoning trace away from the SLM's native reasoning distribution, often manifesting as "Syntactic Compaction" (e.g., replacing natural spacing and verbal delimiters with dense symbolic operators).

This drift increases the target model's adaptation cost, which can outweigh the benefits of improved logical presentation.

Methodology

The authors establish a controlled experimental framework to isolate the impact of data representation from problem difficulty.

• Data Construction: Using a fixed problem set derived from the NuminaMath dataset, four parallel data streams are constructed:
  1. SLM-RFT: Valid solutions generated by the target SLM (e.g., Qwen2.5-Math-1.5B) and filtered via a conservative process to remove artifacts while preserving native style.
  2. Oracle-Refined: SLM-RFT solutions rewritten by a powerful Oracle (GPT-5.2) to repair logic and formatting.
  3. Oracle-Synthesized: Solutions generated from scratch by the Oracle.
  4. NuminaMath Subset: Ground-truth solutions retrieved via reverse matching.
• Training Paradigms: Experiments utilize both standard Supervised Fine-Tuning (SFT) and Dynamic Fine-Tuning (DFT) to ensure findings are not artifacts of a specific optimization objective.
• Evaluation: Models are evaluated on a spectrum of mathematical benchmarks (MATH500, AIME24, AMC23, Minerva, OlympiadBench) using Avg@16 accuracy. Perceived quality is measured via multiple reward models (Qwen2.5-Math-RM, Skywork-Reward, Nemotron-Reward).
• Mechanism Analysis:
  • Token Distribution Shift: Analysis of unigram token frequencies reveals that Oracle refinement suppresses "native scaffolding" tokens (e.g., spaced backslashes, natural delimiters like "the" or "of") in favor of dense, non-spaced operators.
  • Perplexity (PPL): Used as a proxy for adaptation cost. Lower PPL indicates higher compatibility between the training data distribution and the target model's native distribution.
  • Semantic Preservation: An LLM-as-a-Judge protocol evaluates how closely candidate solutions preserve the informational atoms of the SLM's original trajectory, independent of formatting.
• Intervention: The authors introduce Style-Aligned Refinement, a prompt-based intervention where the Oracle is instructed to correct logical errors while strictly emulating the SLM's native linguistic style and spacing.

Key Contributions

1. Identification of the Quality-Utility Paradox: The paper demonstrates that in mathematical reasoning distillation, data with higher reward model scores can yield inferior downstream performance compared to data generated by the learner itself.
2. Mechanism of Distributional Drift: The authors show that Oracle refinement introduces a compatibility problem where logical repair is coupled with a shift in representation form (syntactic compaction). This increases the target SLM's adaptation cost, as evidenced by higher perplexity and lower downstream utility.
3. Style-Aligned Refinement: The paper proposes and validates an intervention that decouples logical rectification from stylistic drift. By preserving the SLM's native trajectory while retaining logical repairs, this method lowers adaptation cost and restores downstream utility.

Results

• Performance Mismatch: Under both SFT and DFT, SLM-RFT data (lowest reward scores, e.g., 1.47) consistently outperformed Oracle-Refined (higher scores, e.g., 1.70) and Oracle-Synthesized (highest scores, e.g., 1.88) data. For instance, on the Qwen2.5-Math-1.5B model with DFT, SLM-RFT achieved 37.06% Avg@16 accuracy, while Oracle-Refined achieved 34.06% and Oracle-Synthesized 30.02%.
• Robustness: The paradox holds across different model scales (1.5B, 3B, 7B), model families (Qwen, LLaMA, DeepSeek), and training hyperparameters.
• Adaptation Cost: Oracle-Refined data exhibited significantly higher Global Perplexity (1.85) compared to SLM-RFT (1.52), indicating a higher burden for the model to adapt to the new distribution. The Q1 (initiation) phase showed the most significant loss attribution to syntactic markers (e.g., \( ) in Oracle data versus natural language initiators (e.g., "To", "the") in SLM data.
• Intervention Success: Style-Aligned (Qwen) refinement achieved the lowest perplexity (1.46, even lower than native SLM-RFT) and the highest downstream accuracy (39.12%), surpassing all baselines. Notably, this high-performing data received the lowest reward scores (1.37), reinforcing the disconnect between reward-model perception and learnability.
• Cross-Model Validation: The paradox was confirmed across three different reward models, indicating that the undervaluation of native-distribution traces is a systemic issue in current reward modeling, not specific to a single evaluator.

Significance and Claims

The paper claims that effective mathematical reasoning distillation for SLMs must jointly optimize perceived solution quality and learner-data compatibility. Relying solely on reward-model scores to select training data is insufficient and potentially detrimental.

The authors argue that the "Quality-Utility Paradox" arises because stronger models often express repaired reasoning through representations that are efficient for themselves or preferred by reward models (dense, compact syntax) but are costly for smaller target models to imitate and internalize. The study suggests that distributional compatibility—specifically, the preservation of the learner's native reasoning trajectory—is a critical, often overlooked factor in distillation.

The findings imply that future distillation strategies should prioritize interventions that align the stylistic and syntactic properties of the teacher's output with the student's native distribution, rather than maximizing raw logical quality or reward scores in isolation. The work is modest in scope, focusing on mathematical reasoning within the SLM regime, and acknowledges that the generalizability to larger models or non-mathematical domains remains an open question.