Detecting RLVR Training Data via Structural Convergence of Reasoning

1. Problem Definition

The paper addresses the critical issue of benchmark contamination in modern Large Language Models (LLMs) trained with Reinforcement Learning with Verifiable Rewards (RLVR).

• Context: RLVR is the dominant post-training technique for improving reasoning capabilities (e.g., in math and coding) by optimizing models based on reward feedback from self-generated reasoning trajectories.
• The Challenge: Unlike pretraining or Supervised Fine-Tuning (SFT), where data exposure can be detected via token-level likelihoods or perplexity, RLVR optimizes the distribution of reasoning paths rather than specific token probabilities. Consequently, conventional likelihood-based detection methods (e.g., Min-K% Prob, Perplexity) fail to detect whether a specific prompt was part of the RLVR training set.
• Goal: Develop a black-box detector that can determine if a given query $x$ belongs to the RLVR training dataset $D_{RL}$ without access to model internals, token probabilities, or reference models.

2. Key Observations & Methodology

The authors propose that RLVR induces a unique structural convergence in reasoning trajectories, which serves as a detectable signature.

A. Core Phenomenon: Structural Convergence

Through extensive analysis of RLVR training dynamics (using DAPO and GRPO algorithms), the authors observed:

1. Rigidity in Seen Prompts: For prompts encountered during RLVR training, the model's generated reasoning paths collapse into a small set of rigid, recurring structural modes. The diversity of outputs (lexical, logical, and semantic) decreases significantly.
2. Diversity in Unseen Prompts: For unseen prompts, the model retains higher variability and does not converge to the same tight clusters.
3. Symbolic Compression: The convergence is most pronounced in symbolic and algebraic reasoning components (e.g., standardized mathematical transformations, function definitions), which become highly standardized.

B. The Proposed Method: Min-kNN Distance

Based on the observation of structural collapse, the authors introduce Min-kNN Distance, a simple black-box metric.

• Procedure:
  1. Given a prompt $x$, sample $m$ completions ($O(x) = \{o_1, ..., o_m\}$) from the RLVR-tuned model.
  2. Compute the pairwise normalized Levenshtein edit distance ($D_{ij}$) between all generated completions.
  3. For each completion $o_i$, find its nearest-neighbor distance ($NN_i$).
  4. Sort these distances and select the $k$ smallest values.
  5. Calculate the Min-kNN Distance as the average of these $k$ smallest distances:
     $$\text{Min-kNN}(x) = \frac{1}{k} \sum_{t=1}^{k} NN_{(t)}(x)$$
• Decision Rule:
  • Seen Prompts (Training Data): Exhibit low Min-kNN Distance (high structural similarity/collapse).
  • Unseen Prompts: Exhibit high Min-kNN Distance (high structural diversity).
• Advantages: Requires only sampling access (black-box), no token probabilities, and no reference models.

3. Key Contributions

1. Systematic Analysis of RLVR Behavior: The paper provides the first systematic quantification of how RLVR reshapes reasoning behavior, revealing a convergence in the structural space of reasoning trajectories, particularly in symbolic components.
2. Novel Detection Method: Introduction of Min-kNN Distance, a robust, black-box detector that leverages structural collapse rather than likelihoods.
3. Comprehensive Evaluation: Extensive experiments across diverse models (Qwen, DeepSeek, SimpleRL), RL algorithms (DAPO, GRPO, PPO), and data types (Math, Code) demonstrate the method's superiority over existing baselines.

4. Experimental Results

The authors evaluated Min-kNN Distance against several baselines (PPL, Min-K% Prob, CDD, Self-Critique) across multiple settings:

• Overall Performance: Min-kNN Distance achieved an average AUC of 0.70, representing a 17% relative improvement over the strongest baseline. It consistently outperformed all other methods across different model scales (1.5B to 32B parameters).
• Robustness to Paraphrasing: The method remained effective (AUC 0.71) even when training prompts were paraphrased, indicating it detects structural collapse rather than surface-level string matching.
• Distillation Detection: The method successfully detected prompts used in model distillation (AUC 0.76), suggesting the structural collapse signal transfers from teacher to student models.
• Cross-Domain Performance:
  • Math: AUC 0.80 (High performance due to structured reasoning).
  • Code: AUC 0.69 (Slightly lower but still superior to baselines, likely due to higher flexibility in coding tasks).
• Dual-Stage Contamination: The method proved more effective at detecting RL-phase contamination in data with low pretraining contamination, validating its specificity to the RL training phase.

5. Significance and Impact

• Transparency in AI: This work provides a crucial tool for verifying whether open-source reasoning models have been "overfitted" to specific benchmark problems, addressing the growing concern of inflated generalization claims.
• Beyond Likelihoods: It shifts the paradigm of data exposure detection from token-level probability analysis (which fails for RL) to structural analysis of reasoning diversity.
• Practical Utility: As a fully black-box method, it can be applied by auditors and researchers to any publicly released RLVR-tuned model without requiring access to proprietary training data or model weights.

In conclusion, the paper demonstrates that RLVR training leaves a distinct "fingerprint" of structural rigidity in reasoning trajectories. By quantifying this collapse via Min-kNN Distance, the authors offer a reliable solution to detect training data exposure in the era of reinforcement-learned reasoning models.