IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage

Technical Summary: IRDS – Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coverage

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for enhancing Large Language Model (LLM) reasoning, particularly in mathematics. However, RLVR suffers from significant data inefficiency. Existing data selection methods fail to simultaneously address three critical requirements:

1. Failure Relevance: Selecting instances the model currently fails on.
2. Trainability: Selecting instances where the model can still learn (i.e., avoiding the "all-fail" and "all-solve" tails where group-relative gradients vanish).
3. Non-Redundancy & Interpretability: Selecting a diverse subset that covers distinct weaknesses without redundancy, while ensuring the selection process is auditable by humans.

Current approaches typically address these in isolation: scalar methods rank by difficulty but ignore redundancy; diversity methods use embeddings but lack verifier signals; and gradient-based methods operate in dense, uninterpretable parameter spaces.

2. Methodology: IRDS

The authors propose IRDS (Interpretable RLVR Data Selection), an offline method that selects a non-redundant subset of training instances based on a Sparse Autoencoder (SAE) cluster basis coupled with verifier signals.

A. SAE Cluster Coordinates
Instead of using surface features (e.g., token length) or dense hidden states, IRDS represents instances using a sparse autoencoder trained on the model's final-layer activations.

• SAE Training: A BatchTopK SAE is trained unsupervised on a large math corpus (OpenMathInstruct-2) using the target model's frozen activations. The dictionary directions correspond to recognizable concepts (e.g., geometry, divisor counting).
• Clustering: To create a corpus-level coordinate, individual sparse latents are clustered into $F=256$ semantic clusters using a graph-hybrid representation (concatenating presence and residual directions). Each instance is summarized by a vector of activation masses across these 256 clusters.

B. Verifier-Coupled Coverage Objective
IRDS constructs a metric that balances difficulty and trainability on the SAE coordinate space.

• Difficulty Weight ($d_i$): Derived from the verifier's success count ($s_i$) using a Beta posterior. It is high when the model fails often ($s_i$ is low).
• Trainability Weight ($r_i$): Also derived from the success count, this weight peaks at intermediate success rates ($s_i \approx G/2$), where the group-relative advantage variance is maximized. It downweights both the all-fail and all-solve tails.
• Metric Construction: The method computes a difficulty-weighted covariance ($\Sigma_d$) and a trainability-weighted covariance ($\Sigma_r$). It then forms a whitened metric $M = (\Sigma_r + \rho I)^{-1/2} (\Sigma_d + \rho I) (\Sigma_r + \rho I)^{-1/2}$. The leading directions of $M$ identify SAE clusters where difficulty exceeds trainability mass.
• Design Vector: Each instance is mapped to a design vector $\phi_i$ that combines the SAE block (weighted by trainability and the metric $M$) and an optional gradient block (residualized against the SAE coordinates to capture update-direction information).

C. Subset Selection
The selection problem is formulated as a D-optimal design problem: finding a subset $S$ of size $K$ that maximizes the log-determinant of the Gram matrix of design vectors.
$$S^* = \arg \max_{|S|=K} \log \det \left( \lambda I + \sum_{i \in S} \phi_i \phi_i^\top \right)$$
This objective is solved via a greedy algorithm with Sherman-Morrison rank-one updates, providing a $(1 - 1/e)$ approximation guarantee. This ensures the selected subset covers diverse, high-value directions without redundancy.

3. Key Contributions

1. First SAE-based RLVR Selection: IRDS is the first data selection method for RLVR built on a sparse autoencoder cluster basis, making the selection process auditable on recognizable problem motifs (e.g., "polynomial-root algebra").
2. Verifier-Coupled Coverage Objective: The authors design a novel objective that jointly optimizes for failure relevance and trainability by disentangling difficulty and trainability weights on the SAE basis, solved via greedy log-determinant maximization.
3. Empirical Superiority and Efficiency: Experiments demonstrate that IRDS achieves state-of-the-art accuracy across multiple models and benchmarks while being an order of magnitude cheaper than trajectory-based baselines.

4. Experimental Results

The method was evaluated on three instruction-tuned models (Qwen3-4B, Qwen3-1.7B, Llama-3.1-8B-Instruct) across six math reasoning benchmarks (including MATH500, AIME, AMC, and OlympiadBench).

• Accuracy Gains: IRDS achieved the highest Overall accuracy on all three models, outperforming the strongest baseline (LIMR) by +3.9 pp on Qwen3-4B, +4.0 pp on Qwen3-1.7B, and +0.5 pp on Llama-3.1-8B.
• Competition Benchmarks: Gains were particularly pronounced on smaller, harder competition subsets (e.g., +11.4 pp on AIME24 for Qwen3-4B).
• Cost Efficiency: The selection process for IRDS took approximately 7 hours on an 8-GPU node (including SAE training and scoring), whereas the trajectory-based baseline (LIMR) required 70 hours (an order of magnitude more).
• Ablation Studies:
  • Removing the trainability weight ($r_i$) caused a collapse in performance (up to -19.1 pp), confirming that selecting only "hard" instances without regard for trainability is detrimental.
  • The SAE cluster basis was found to be critical, especially for stronger base models, outperforming dense gradient projections.
  • The gradient block provided a marginal but non-redundant improvement.
• Interpretability: Audits confirmed that the selected subsets aligned with high-difficulty content directions (e.g., specific math motifs) rather than surface artifacts (e.g., token length), and the selection was robust to label shuffling.

5. Significance and Claims

The paper claims that IRDS demonstrates that verifier-coupled data selection in RLVR can be made auditable without sacrificing in-domain accuracy. By shifting the selection basis from opaque dense embeddings or surface features to interpretable SAE clusters, researchers can verify exactly which "problem motifs" the model is being trained on.

The authors position IRDS as a solution to the "data inefficiency" bottleneck in RLVR, offering a method that is:

• Auditable: Selections can be traced to human-readable problem types.
• Efficient: Significantly cheaper than online trajectory-based methods.
• Effective: Consistently improves model performance across varying architectures and data budgets.

The paper acknowledges limitations, noting that the method relies on a frozen model for weight estimation (potentially drifting after updates), requires access to model activations for SAE training, and has currently been validated primarily on math RLVR under the GRPO recipe.