Technical Summary: Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection

Problem Formulation

The paper addresses the critical bottleneck in Reinforcement Learning with Verifiable Rewards (RLVR): the selection of high-impact training instances from a large, unlabeled pool. While RLVR is known to be highly data-efficient—potentially yielding significant reasoning gains from very few examples—its performance is extremely sensitive to which examples are chosen.

Existing data selection pipelines typically rely on:

1. Training-time signals: Running partial fine-tuning or RL over a candidate pool to observe optimization dynamics (e.g., reward trajectories, gradient variance).
2. Ground-truth dependencies: Access to verifiable rewards or correct answers for the entire candidate pool to rank instances.

These requirements are often infeasible in specialized, low-resource domains (e.g., medical QA) where annotations are costly and running trial training on thousands of candidates is computationally wasteful. The authors formalize a pre-RL regime where selection must be performed on a large unlabeled pool without any ground-truth labels, verifiable rewards, or training-time optimization signals.

Methodology: SHIFT

The authors propose SHIFT, a one-shot, training-free, and label-free selector. SHIFT leverages inference-time hidden-state dynamics to estimate an instance's utility for downstream RLVR without performing any training.

Core Mechanism: Reasoning-Induced Representation Shift (RIRS)

The central hypothesis is that the internal dynamics of a model during a reasoning trace encode its learning potential.

1. Single Deterministic Rollout: For each candidate instance $x$, SHIFT runs a single deterministic reasoning trace using greedy decoding ($T=0$).
2. Anchor Extraction: It identifies the hidden states at the beginning ($s(x)$) and end ($e(x)$) of the reasoning trace (specifically, the start and end of the Chain-of-Thought segment).
3. RIRS Calculation: The Reasoning-Induced Representation Shift is computed as the delta between these states:
   $$\Delta(x) = e(x) - s(x)$$
   The magnitude $\|\Delta(x)\|_2$ serves as a lightweight proxy for the instance's utility (informativeness), hypothesizing that larger shifts indicate stronger engagement of the model's internal computation.

Selection Algorithm: Quality-Weighted Farthest-First CoreSet

To ensure the selected subset is both high-quality and diverse, SHIFT employs a greedy selection strategy in an augmented feature space:

1. Feature Construction: Each instance is represented by a concatenated vector $\phi(x) = [s(x); \Delta(x)]$, normalized to the unit sphere. This captures both the initial context ($s(x)$) and the reasoning transformation ($\Delta(x)$).
2. Utility Scoring: A utility score $\tilde{q}(x)$ is derived from the RIRS magnitude (log-transformed for stability).
3. Greedy Selection:
   • Initialize the set $S$ with the instance having the highest utility.
   • Iteratively select the next instance $x^*$ that maximizes the product of its utility and its distance to the current set:
     $$x^* = \arg\max_{x \in U \setminus S} \tilde{q}(x) \cdot \min_{x' \in S} \|\phi(x) - \phi(x')\|_2$$
     This balances informativeness (high RIRS) with coverage (diversity in the RIRS-augmented space).

Key Contributions

1. Problem Formalization: Defined the RLVR data selection problem in a strict pre-RL setting where no labels, rewards, or training signals are available during selection.
2. SHIFT Framework: Introduced a training-free selector based on inference-time hidden-state dynamics, specifically the RIRS metric.
3. Algorithmic Innovation: Developed a single-rollout, quality-weighted farthest-first CoreSet algorithm that scales to large unlabeled pools.
4. Empirical Validation: Demonstrated consistent improvements over training-free baselines (diversity sampling, difficulty heuristics, uncertainty-based methods) across mathematical reasoning and medical QA benchmarks under ultra-low budgets (0.1%–4% of data).

Experimental Results

The authors evaluated SHIFT on Qwen models (1.5B and 1.7B) across two domains:

• Mathematical Reasoning (MATH-500 & AMC): Under a 2% budget, SHIFT achieved 62.67% accuracy on MATH-500 and 38.55% on AMC (OOD), significantly outperforming Random sampling (+8.94 / +12.77) and other baselines. Notably, it surpassed the full-data RLVR reference on the OOD AMC benchmark.
• Medical QA (MEDQA & Transfer): Under a 0.1% budget, SHIFT achieved 50.35% on MEDQA, outperforming Random (45.28%) and other heuristics. It showed particularly strong transfer to clinically challenging subsets (MedXpertQA), nearly matching full-data performance with only 0.1% of the data.

Ablation and Analysis

• RIRS Validity: Correlation analysis confirmed that RIRS is not merely a proxy for input/output length (Spearman $\rho \approx 0$ with response length).
• Utility Correlation: A pilot study showed a strong positive correlation (Spearman $\rho = 0.818$) between pre-RL RIRS ranking and actual downstream RLVR gain.
• Ablations: Removing either the RIRS-based coverage ($\phi_{s+\Delta}$) or the quality-weighting ($\tilde{q}(x)$) degraded performance, confirming that both diversity and utility signals are complementary.
• Compute Efficiency: SHIFT requires only one deterministic rollout per candidate, making it significantly cheaper at selection time than self-consistency baselines (which require ~32 stochastic rollouts).

Significance and Claims

The paper claims that SHIFT enables effective RLVR adaptation in scenarios previously considered infeasible due to annotation costs and computational constraints. By identifying "impactful" examples through reasoning-induced hidden-state dynamics rather than training signals, SHIFT allows for:

• Drastic reduction in annotation and training overhead: Achieving strong performance with <1% of the data.
• Accessibility: Making RL adaptation viable for specialized domains (e.g., medicine) where ground-truth labels are scarce.
• Generalization: Improving transfer to harder, out-of-distribution evaluation settings compared to standard diversity or difficulty-based selection.

The authors maintain a modest stance, acknowledging that RIRS is an empirical surrogate motivated by theoretical block-level dynamics rather than a formal derivation, and that robustness across broader model families and safety-sensitive error stratification remains an open area for future work.