Technical Summary: When the Reward Suite is Leaky

1. Problem Statement

Reinforcement Learning from Verifiable Rewards (RLVR) trains code models using test suites as ground truth. However, deployed test suites often contain natural false positives (FPs): per-task, persistent, asymmetric errors where a specific wrong program consistently passes the same weak tests. Unlike the symmetric, resampled noise assumed by existing robustness analyses (which average out over training), these natural FPs provide a standing reward for specific incorrect solutions.

The paper addresses a missing causal question: What happens when models train against these persistent, natural false positives? Specifically, does the model learn to exploit these leaks (learned exploitation), or does it merely select pre-existing error modes (selection)? Furthermore, does training on a "leaky" suite degrade held-out capability compared to a "hardened" suite?

2. Methodology

The authors conducted a preregistered two-arm causal contrast across three model families (Qwen2.5-Coder-1.5B, DeepSeek-Coder-1.3B, and Llama-3.2-1B).

Experimental Design

• Training Arms: Both arms used identical models, tasks (250 MBPP tasks), seeds (5), hyperparameters, and compute (400 GRPO steps). The only difference was the reward signal:
  • Leaky Arm: Rewarded based on the original, deployed MBPP test suite.
  • Hardened Arm: Rewarded based only on the "extra" tests generated by the MBPP+ project (which are disjoint from the original tests on 313/378 tasks).
• Static Audit: Before training, a cheap static audit sampled 16 solutions from the base model per task. A task's "leakiness" was defined as the fraction of samples that passed the original suite but failed the extra tests.
• Evaluation: Held-out performance was scored exclusively by the extra-test suite (MBPP+ "plus" tests) to ensure a fair, non-leaky assessment.
• Preregistration: The study design, including non-inferiority margins and analysis plans, was preregistered before data collection. Claims are tagged [C] (confirmatory/preregistered) or [E] (exploratory/post-hoc).

Key Metrics

• Rewarded FP Mass: The fraction of rollouts in the leaky arm that passed the original suite but failed the extra suite.
• Verified-Wrong Share: An exhaustive human audit of rewarded FPs to distinguish genuinely wrong code from artifacts (e.g., extra-test false negatives).
• Held-Out Gap: The difference in pass rates between the hardened and leaky arms on unseen tasks.

3. Key Results

Confirmatory Findings [C]

1. Bounded Average Effect: At 400 steps, the leaky arm did not suffer a statistically significant drop in held-out capability compared to the hardened arm. The gap was 0.20 points, with a one-sided 95% upper bound of 0.75 points, staying well within the preregistered non-inferiority margin of 1.5 points.
2. Static Audit Predicts Leakage: There is a strong correlation (Spearman $\rho \approx 0.80$) between the pre-training static leakiness score and the actual rewarded FP mass observed during training. This suggests the leak is a property of the task/suite, not something the model learns to create.
3. Replication: The core findings (large verified-wrong share, leakiness tracking, and train-side stratification) replicated across all three model families.

Exploratory Findings [E]

1. Reward Pays for Real Bugs: Approximately 47.57% of the rewarded false positives in the primary family were verified as genuinely wrong code (real bugs), not just suite artifacts. This share was large in replication families (45.37% and 62.78%).
2. Selection, Not Learning: The mechanism appears to be selection of pre-existing error modes rather than learned exploitation.
   • The FP rate was already at its long-run level at Step 0 and did not grow over the training horizon.
   • Untrained base models produced the same characteristic wrong outputs when filtered by the leaky suite.
   • There was no evidence of "hacking signatures" or parameter-level sharpening toward the false positives beyond behavioral resolution.
3. Meta-Audit of Frontier Judges: An exploratory audit of frontier models (DeepSeek-V4-Pro, MiMo-v2.5-Pro) acting as judges found they could separate their own false positives from genuine passes only weakly (AUC ~0.62–0.64), significantly below their ability to judge the 1.5B policy's errors. A deconfounded test of self-assessment bias returned UNRESOLVED.

4. Contributions and Significance

The paper's primary contribution is a measurement layer that isolates the causal impact of natural false positives in RLVR.

• Measurement vs. Capability: The study demonstrates that a "clean" aggregate held-out score does not certify a "clean" reward stream. The leaky arm accumulated 8.37 points more training reward than the hardened arm, yet this inflation did not translate into measurable capability gains or losses on held-out tasks within the studied scale (1–1.5B parameters, 400 steps).
• Audit Economics: A cheap static audit performed before training can predict where reward leakage will occur. This allows practitioners to identify high-risk tasks without running expensive training runs.
• Nature of the Leak: The paper argues that the reward stream is "paying for" pre-existing errors in the base model distribution that the suite fails to filter, rather than the model learning to hack the suite. The leak acts as a selection mechanism, not a teaching signal for new exploits.
• Practical Boundary: The findings suggest that for current scales and task lengths, the risk of natural false positives is primarily a measurement problem (inflated reward accounting) rather than a capability problem (learned misalignment). However, the authors explicitly state they claim nothing beyond their specific scale and horizon, noting that exploitation could grow with larger models or longer tasks.

5. Limitations and Scope

The authors are explicit about the boundaries of their claims:

• Scale: Results are specific to 1–1.5B models and 400 GRPO steps. The paper does not rule out exploitation growing at larger scales or longer horizons.
• Tasks: The study is limited to MBPP and HumanEval+ tasks.
• Hardening: The "hardened" arm uses only extra tests, which are not perfect (they contain their own false negatives), though the error floor is smaller than the original suite.
• Meta-Audit: The findings regarding frontier judges are descriptive and limited to two specific models; they do not generalize to all frontier models.

In summary, the paper provides a rigorous, preregistered demonstration that while deployed test suites leak reward to genuinely wrong code, this leakage currently manifests as a measurement inflation rather than a learned capability to exploit the suite, at least within the observed scale and horizon.