Technical Summary: Unified Data Selection for LLM Reasoning

Problem Statement

The advancement of Large Language Models (LLMs) in complex, long Chain-of-Thought (CoT) reasoning is currently bottlenecked by the necessity for massive volumes of high-quality training data. Existing data selection strategies face two primary challenges:

1. Computational Cost: Methods that rely on training auxiliary task-specific models or using powerful LLMs to filter data incur substantial computational overhead.
2. Metric Limitations: Traditional metrics such as sequence length, perplexity, or average token entropy fail to distinguish high-quality reasoning samples in long-CoT scenarios. These global metrics assign equal weight to all tokens, thereby masking the nuanced, multi-stage structure of complex reasoning (planning, exploration, reflection) and failing to capture the critical decision points where the model faces uncertainty.

The core problem is the lack of a robust, efficient, and training-free metric capable of rapidly distinguishing high-quality reasoning paths from low-quality ones across different training paradigms.

Methodology

The authors propose High-Entropy Sum (HES), a training-free metric designed to quantify reasoning quality by focusing on fine-grained, critical information embedded in reasoning responses.

The HES Metric

The methodology is grounded in the observation that high-entropy tokens—those where the model exhibits high uncertainty—often correspond to critical "forking points" in a reasoning path where non-trivial decisions are made.

• Definition: HES is calculated by summing the entropy values of the top $k\%$ (e.g., 0.5%) of tokens with the highest entropy in a response.
• Formulations:
  • HES$_{relative}$: Uses a percentile-based threshold (e.g., top 0.5% of tokens) to adapt to varying sequence lengths and model distributions. This is the primary formulation used.
  • HES$_{absolute}$: Uses a fixed entropy threshold (e.g., $\tau = 1.6$) for summation.
• Rationale: Unlike average entropy, which dilutes critical signals by averaging over long sequences, HES isolates the cumulative intensity of pivotal, high-uncertainty moments. Empirical analysis (Figure 1) demonstrates that while average entropy metrics cannot distinguish between correct and incorrect samples, the sum of high-entropy tokens provides a robust separation.

Unified Data Selection Framework

The paper proposes a unified data selection strategy applicable to three dominant training paradigms:

1. Supervised Fine-Tuning (SFT): The full dataset is ranked by HES scores. The model is trained on the top-$\rho$ percentile (e.g., top 20%) or the dataset is pruned by removing the bottom-$\rho$ (e.g., bottom 20%).
2. Rejection Fine-Tuning (RFT): For each query, multiple candidate responses are generated. HES is used to select the highest-quality correct solutions either on a per-query basis (selecting top-$k$ from local candidates) or globally (selecting top-$N$ from a global pool of correct trajectories).
3. Reinforcement Learning (RL): The authors introduce an asymmetric sampling strategy for Group Relative Policy Optimization (GRPO). They prioritize selecting the highest-HES positive trajectories (to encourage mastery of complex patterns) while retaining random negative trajectories (to ensure exposure to diverse error modes). This contrasts with standard uniform sampling or full-batch updates.

Key Contributions

1. Introduction of HES: A novel, training-free metric that quantifies reasoning quality by analyzing only the high-entropy tokens in response paths, effectively capturing the cumulative complexity of critical decision points.
2. Unified Selection Framework: An efficient data selection method compatible with SFT, RFT, and RL paradigms, eliminating the need for separate, expensive selection mechanisms for each.
3. Small-to-Large Transferability: Demonstration that a lightweight proxy model (e.g., 0.6B) can effectively screen data for larger foundation models (e.g., 8B), significantly reducing the inference cost of offline data curation without compromising selection quality.

Experimental Results

Extensive experiments were conducted across Math, STEM, and Code domains using benchmarks such as AIME, HMMT, GPQA, and LiveCodeBench.

• SFT Performance:
  • Training on the top 20% of data ranked by HES achieved performance comparable to the full dataset across diverse domains.
  • In the "De-noising" setting, removing the bottom 20% of HES-ranked data improved performance over the full dataset.
  • Conversely, training on the lowest-HES 20% of data resulted in drastic performance degradation, validating the metric's ability to identify harmful noise.
  • Efficiency: Using a 0.6B proxy model for selection yielded results (30.20% avg) comparable to using the 8B model itself (31.34% avg), reducing selection costs by an order of magnitude.
• RFT Performance:
  • HES-based selection significantly outperformed random selection and heuristic baselines (length, difficulty) in both per-query and global pool settings.
  • Per-query selection often yielded higher performance than global pool selection, suggesting that maintaining query-level diversity is crucial alongside quality filtering.
• RL Performance:
  • The "Pos-High, Neg-Rand" strategy (selecting top-HES positives and random negatives) outperformed standard full-batch training and other sampling strategies.
  • This approach achieved superior accuracy (19.26% avg) using only 50% of the data per step compared to the full-batch baseline (18.34%), confirming that prioritizing high-quality positive signals is more effective than increasing batch size with undifferentiated data.
• Metric Comparison: HES consistently outperformed alternative entropy-based metrics (Average Entropy, Average High-Entropy, Total Entropy Sum), confirming that the cumulative magnitude of critical forking points is the most effective proxy for reasoning quality.

Significance and Claims

The paper establishes HES as a robust, training-free metric that enables a unified, effective, and efficient method for developing advanced reasoning in LLMs. The authors claim that:

• Efficiency: HES allows for scalable proxy selection using small models in offline SFT and incurs negligible storage/compute overhead in online RFT and RL settings (as entropy can be calculated synchronously during generation without extra forward passes).
• Effectiveness: By focusing on the "high-entropy minority tokens," the method successfully filters redundant signals and training noise, allowing models to learn strong reasoning patterns from a curated subset of data.
• Generalizability: The method is applicable across different model architectures (Qwen, Llama, DeepSeek) and reasoning domains (Math, Code, STEM), suggesting that HES captures intrinsic reasoning complexity rather than model-specific artifacts.

The work concludes that HES provides a cost-effective solution for scaling up LLM reasoning capabilities, highlighting the pivotal role of critical tokens in data selection and reducing the computational burden associated with training high-performance reasoning models.