Technical Summary: Auditing Reward Hackability in Code RL Training Environments

1. Problem Statement

Reinforcement Learning from Execution Feedback (RLFE) in code generation relies entirely on the correctness of the verifier (test suite). If a benchmark's test suite accepts an incorrect solution as correct, the policy receives a reward for the wrong behavior, leading to misalignment, alignment faking, or sabotage. While previous audits (e.g., OpenAI's February 2026 note) have identified flawed tests in benchmarks like SWE-bench Verified, they lack an operational metric answering a critical question for curators: What fraction of tasks in a code RL training set actually reward models for incorrect solutions?

This paper addresses the "reward hackability" of code environments, defined as the rate at which a task's test suite accepts an incorrect patch as passing. It further proposes and validates a procedure to harden these broken tasks before they are used for training.

2. Methodology

The paper introduces a four-component pipeline (Figure 1) to audit and repair environments:

2.1 Exploit Generation

For each task, the system generates $K$ candidate incorrect source patches using a frontier LLM (Claude Sonnet 4).

• Format: Unified diffs (SWE-bench) or file replacement mappings (R2E-Gym).
• Process: Round 1 uses single-shot generation. Rounds 2 and 3 use an iterative-attacker pattern, biasing generation toward exploits not yet blocked by the suite using failure logs.
• Verification: Candidates are applied inside the task's Docker container. A task is "hackable" if at least one candidate passes the project's native test runner despite being incorrect.

2.2 Environment Quality Score (EQS)

Tasks are scored on a [0, 1] scale using four weighted dimensions:
$$EQS(t) = 0.35V(t) + 0.30(1 - H(t)) + 0.20F1(t) + 0.15D(t)$$

• $V(t)$: Verifier discrimination score (NVIDIA-style).
• $H(t)$: Empirical hackability (fraction of successful exploits).
• $F1(t)$: Agreement between a semi-formal LLM judge and execution (treating "partial" as incorrect).
• $D(t)$: Learnability signal derived from external Pass@1 data.
• Verdict: Tasks are classified as KEEP (EQS > 0.70), FIX (0.40 ≤ EQS ≤ 0.70), or DROP (EQS < 0.40).

2.3 Sanity-Gated Repair Loop

For tasks marked FIX, a closed-loop procedure attempts to generate blocking tests:

1. Augmenter: Proposes a test to block the exploit.
2. Gold-Sanity Gate (Docker): The proposed test is run against the gold solution in Docker. If the test fails on the gold solution (indicating the test itself is broken or the logic is inverted), it is discarded, and a diversity-biased retry is triggered.
3. LLM Judge: If the test passes the gate, a 3-sample self-consistency LLM judge (using semi-formal reasoning) determines if the test successfully blocks the exploit.
4. Termination: The loop ends when all target exploits are covered or the iteration budget is exhausted.

3. Key Contributions and Results

3.1 Cross-Benchmark Audit Rates

The paper measures the rate of reward hackability across two benchmarks:

• SWE-bench Verified ($n=49$): 28.5% of tasks are Docker-verified hackable (14/49). This includes 18.4% found in single-shot generation and additional tasks found via iterative refinement.
• R2E-Gym ($n=20$): 25.0% of tasks are hackable (4/16 decisive tasks). This is considered a lower bound as it used single-shot generation ($K=1$) compared to the iterative approach used for SWE-bench.

3.2 Meta-Analytic Validation of Impact

Using a random-effects meta-analysis on 134 frontier model submissions to SWE-bench Verified, the paper demonstrates that hackability directly inflates performance metrics:

• Result: Within the same human-rated difficulty stratum, model Pass@1 is +14.14 percentage points higher on flagged-hackable tasks compared to robust ones (95% CI [+11.80, +16.48]; one-sided $p < 10^{-6}$).
• Uniformity: The effect is uniform across submission eras, model families, and Pass@1 quartiles ($I^2 = 0\%$).
• Sign Test: 123 of 134 models showed higher Pass@1 on hackable tasks.

3.3 The 61.9% Augmenter Defect Rate

A critical finding is the failure rate of LLM-generated tests when not validated against the gold solution:

• Initial Failure: Without the Docker gate, an LLM judge endorsed 10 of 11 broken tasks as "upgraded."
• Gate Discovery: Upon adding the Docker gold-sanity gate, 65 of 105 decisive augmentations were flagged as failing on the gold patch itself (a 61.9% defect rate).
• Nature of Defects: The LLM judge correctly reasoned about the intended logic of the generated tests but failed to detect that the tests themselves were broken (e.g., wrong imports, inverted assertions, uncollectable test structures).
• Repair Success: With the gate and diversity-biased retry, the loop successfully converged 9 of 11 tasks to a "gated upgrade" (FIX $\to$ KEEP).

4. Significance and Claims

The paper claims three primary empirical contributions:

1. Quantification: It provides the first operational measurement of reward hackability rates (28.5% and 25.0%) in code RL environments, demonstrating that a substantial fraction of benchmark tasks reward incorrect behavior.
2. Validation: It establishes a causal link between test-suite weakness and inflated model performance metrics, showing that "Pass@1" on these benchmarks is significantly confounded by the ability to game weak verifiers rather than solve the problem.
3. Methodological Necessity: It demonstrates that relying solely on LLM judges to generate or validate hardening tests is insufficient. A Docker gold-sanity gate is required to catch a 61.9% defect rate where generated tests fail to run or contradict the gold solution.

The authors explicitly state modesty regarding causal training outcomes: while fixing broken tasks is necessary, the paper does not claim that fixing them improves training outcomes without a direct causal A/B test (which is proposed as future work). The primary contribution is the identification of the problem and the validation of a repair pipeline that prevents the propagation of broken tasks into training sets.