Technical Summary: Auditing Data Membership in Reinforcement Learning With Verifiable Rewards

1. Problem Definition and Motivation

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a critical training stage for large language models (LLMs), particularly for enhancing reasoning capabilities in models like DeepSeek-R1 and Kimi-k2.5. Unlike traditional Supervised Fine-Tuning (SFT) or preference-based alignment (RLHF), RLVR generates candidate responses from the model itself and reinforces those that satisfy rule-based verifiers (e.g., correct math answers or code execution).

This paradigm creates a unique challenge for data exposure auditing. Existing Membership Inference Attacks (MIAs) typically rely on detecting a model's overfitting to a fixed target string (e.g., a specific answer in SFT or a preferred response in RLHF). However, RLVR does not optimize toward a predetermined response; instead, it optimizes the policy to maximize the probability of generating any response that passes a verifier. Consequently, standard likelihood-based MIAs fail because there is no fixed target string to score against.

The paper addresses the need to audit whether specific prompts (often proprietary benchmarks or private pipelines) were used in RLVR post-training, even in the absence of fixed target responses.

2. Methodology: Divergence-in-Behavior Auditing (DIBA)

The authors propose DIBA (Divergence-in-Behavior Auditing), a white-box, query-level auditing framework. DIBA operates by comparing the released RLVR-fine-tuned model ($\pi_{ft}$) against its pre-RLVR reference checkpoint ($\pi_{ref}$).

Core Insight

The authors observe that while RLVR does not fit a fixed string, it leaves observable behavioral traces along two dimensions:

1. Reward-side Evidence: RLVR changes the frequency with which the model achieves verifiable success on a prompt.
2. Policy-side Evidence: RLVR induces behavioral drift in the probability distribution of generated trajectories relative to the reference policy.

The DIBA Framework

DIBA aggregates signals from multiple stochastic rollouts ($N$ samples) for a given query $q$ to produce a stable auditing signal. It constructs features along two axes:

• Reward-side Signal (Advantage):

  • Computes the expected reward for the fine-tuned model ($FT_{score}$) and the base model ($Base_{score}$) using a rule-based verifier $R$.
  • $s_{Adv} = [ \frac{1}{N}\sum R(o^{ft}_i), \frac{1}{N}\sum R(o^{ref}_i) ]$.
  • This captures whether RLVR improved the model's ability to solve the task.

• Policy-side Signal (Divergence):

  • K3 Divergence: Measures the token-level divergence between $\pi_{ft}$ and $\pi_{ref}$ on the fine-tuned model's own rollouts using an unbiased KL-divergence estimator.
  • Confidence Shift: Computes the log-likelihood ratio ($\log \pi_{ft} - \log \pi_{ref}$) and a Zlib-normalized variant to capture shifts in model confidence and redundancy.
  • Crucially, the reference model acts as an active baseline for counterfactual comparison, not just a calibrator for a fixed target.

Prediction

These features are concatenated and fed into a stacking predictor (Random Forest + Logistic Regression) to output a prompt-level membership score.

3. Key Contributions

1. Systematic Study of RLVR Auditing: The paper establishes that exposure risk in RLVR is better associated with prompt-conditioned behavioral traces rather than fixed-target fitting.
2. Proposal of DIBA: A comparative auditing framework that estimates exposure signals by analyzing reward-side changes and policy-side behavioral divergence relative to a reference model.
3. Comprehensive Evaluation: The authors characterize the boundaries of RLVR auditing, identifying when it succeeds (active learning traces) and when it fails (already-mastered prompts or saturated rewards). They also extend the analysis to vision-language models (VLMs) and various transfer settings.

4. Experimental Results

Performance

• White-Box Setting: DIBA consistently outperforms strong transferred baselines (including calibrated and self-generated variants of LOSS, Zlib, Min-K%, etc.).
  • AUC: Achieves approximately 0.8 across various models (Qwen-3B/7B) and algorithms (GRPO, DAPO).
  • High-Precision Regime: Achieves a TPR@0.1%FPR that is an order of magnitude higher than baselines (e.g., 0.154 for DAPO-3B vs. ~0.01 for baselines).
• Feature Analysis:
  • FT Score (Reward) provides strong overall ranking (high AUC) but fails in low-FPR regimes.
  • Divergence is the most effective feature for high-precision detection (low FPR), highlighting the importance of policy drift.
  • The combination of reward and policy signals yields synergistic improvements.

Auditing Regimes and Boundaries

• Actively-Learned Prompts: DIBA is most effective here, where RLVR induces non-trivial behavioral changes.
• Already-Mastered Prompts: Detection is weakest. If the base model already solves the prompt perfectly, RLVR induces minimal additional updates, leaving no detectable trace.
• Never-Learned Prompts: Detection is also weak if the model fails to learn the task entirely.
• Training Dynamics: Auditing signal strength correlates with the widening gap between training and test rewards; it emerges progressively as training advances.

Transferability and Generalization

• Model Size: Transfer across model sizes (3B vs. 7B) within the same RLVR algorithm is robust.
• Algorithm Mismatch: Transfer between different RLVR algorithms (e.g., GRPO to DAPO) is less stable, particularly in high-precision regimes.
• Grey-Box Setting: With carefully chosen shadow data, DIBA remains useful (AUC > 0.7) even without exact distribution matching.
• Multimodal (VLMs): DIBA extends to Vision-Language Models (AUC ~0.7), though high-precision detection (TPR@0.1%FPR) is harder due to noise from visual encoders.
• Task Transfer: Signals persist in instruction-following and tool-using datasets.

Mitigations

• Regularization: Increasing KL regularization ($\beta$) offers limited protection; it often leads to underfitting rather than genuine privacy preservation.
• Post-hoc Obfuscation: Paraphrasing outputs can reduce text-level auditability (lowering AUC/TPR) but does not eliminate exposure if the auditor has access to the model weights or logits.

5. Significance and Claims

The paper positions DIBA as a practical auditor for white-box RLVR compliance settings. Its primary significance lies in shifting the paradigm of membership inference from "fitting to a fixed target" to "detecting behavioral divergence."

The authors claim that:

1. Exposure is Detectable: RLVR training leaves observable, prompt-specific traces that can be surfaced through targeted auditing, even without fixed target strings.
2. Context Matters: The audibility of data exposure is not uniform; it is strongest when training induces observable reward changes and behavioral shifts, and weakest when the base model already possesses the capability.
3. Limitations: The method is configuration-sensitive (dependent on the RL algorithm and model family) and cannot detect the use of prompts that were already perfectly solved by the base model (the "pre-existing knowledge" limitation).

The work highlights a critical privacy risk in the emerging RLVR paradigm, suggesting that proprietary prompt suites used for reasoning enhancement are vulnerable to exposure auditing, necessitating new privacy-preserving strategies for RLVR training.