Technical Summary: Beyond Pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generation

Problem Statement

Large Language Models (LLMs) for code generation are typically evaluated in repeated-sampling settings using the Pass@k metric, which measures the probability that at least one of $k$ sampled programs passes unit tests. While recent Verifier-based Reinforcement Learning (RLVR) methods have successfully improved executable correctness by optimizing against test feedback, the behavior of these models regarding implementation-level redundancy remains poorly understood.

The core problem addressed is that standard RL objectives (optimizing for isolated correctness) and Pass@k-aware objectives (optimizing for group success) may induce qualitatively different generation behaviors. Specifically, it is unclear whether performance gains under a finite sampling budget stem from discovering diverse, distinct implementation patterns or from concentrating generations around a narrow set of near-duplicate solutions. Current metrics like Pass@k do not distinguish between these two scenarios, potentially masking a lack of diversity that could limit performance at larger sampling budgets ($k$).

Methodology

1. Redundancy Diagnostics: JPlag

To quantify implementation redundancy, the authors employ JPlag, a plagiarism detection system designed for code. Unlike lexical overlap metrics (e.g., n-grams) which are sensitive to superficial edits (variable renaming, formatting), JPlag uses language-aware tokenization and matching to detect structural similarities.

• Metric: The paper defines JPlag Diversity (JDiv) as $1 - \text{mean pairwise similarity}$. Higher JDiv indicates lower redundancy.
• Utility: JPlag serves as a diagnostic tool to analyze how different RL objectives utilize the sampling budget and as a basis for a new training signal.

2. Baseline RLVR Objectives

The study compares three existing approaches:

• Base-RLVR (Correctness-only): Assigns additive rewards for every correct sample within a group. Credit is assigned independently to successful generations.
• Pass@k-RLVR: Uses a group-level reward where success depends on whether at least one sample in the group is correct. Credit is assigned via a leave-one-out advantage, rewarding samples that change the group's success status.
• PKPO (Pass@k Policy Optimization): Extends Pass@k principles to groups larger than $k$ by averaging leave-one-out advantages over all $k$-subsets.

3. Proposed Method: JPlag-RLVR

The authors propose a Redundancy-Aware RLVR objective that augments standard executable rewards with a direct anti-redundancy signal.

• Reward Function: $R(x, Y^{(n)}) = R_{\text{corr}}(x, Y^{(n)}) + \lambda_{\text{div}} R_{\text{div}}(x, Y^{(n)})$.
  • $R_{\text{corr}}$ is the standard correctness reward.
  • $R_{\text{div}}$ is a scalar anti-redundancy reward derived from JPlag similarity (specifically, the inverse of mean pairwise similarity).
• Credit Assignment: To ensure the training signal depends on a sample's marginal contribution to the group's diversity, the authors use a leave-one-out advantage construction for the diversity component. A sample receives a positive advantage if its inclusion reduces the group's redundancy, and a negative advantage if it duplicates existing generations.
• Implementation: The method is applied across three models (Qwen3-4B, Qwen3-8B, Olmo3-7B) and three benchmarks (MBPP, Code-Contest, TACO-Cobalt).

Key Results

1. Analysis of Existing Objectives

• Correctness-Only RLVR: While improving Pass@1, this approach systematically increases redundancy. Training dynamics show that as Pass@1 improves, JPlag diversity decreases, indicating the model concentrates on a narrow set of near-duplicate implementations.
• Pass@k-Aware Objectives: These methods (Pass@k-RLVR, PKPO) implicitly maintain lower redundancy and higher JPlag diversity compared to correctness-only baselines. They tend to improve performance more significantly at larger budgets (Pass@10, Pass@100) by covering a broader set of implementation patterns.

2. Performance of JPlag-RLVR

• Finite-Budget Gains: Directly discouraging near-duplicate generations via JPlag-RLVR reliably improves finite-budget executable performance. Across models and benchmarks, it often matches or outperforms specialized Pass@k-aware objectives.
  • Example: On MBPP with Qwen3-8B, JPlag-RLVR improved Pass@1 from 79.1 to 90.0 and Pass@10 from 88.0 to 98.5, while simultaneously increasing JPlag diversity from 0.305 to 0.927.
• Trade-off Optimization: The method achieves a superior trade-off, moving models "up and right" in the performance-redundancy space (higher Pass@k, higher JDiv) compared to baselines.
• Robustness: Improvements persist across different random seeds and are not driven by a few outlier tasks. The gains are distributed across prompts.

3. Ablation Studies

• Not Just Entropy: Increasing sampling temperature improves Pass@100 but degrades Pass@1 and yields smaller gains than explicit anti-redundancy training.
• Not Just Verbosity: The gains are not caused by longer completions or superficial text variations; executable code lengths remain comparable across methods.
• Correctness is Essential: Training with diversity rewards only (without correctness) increases JDiv but causes executable performance to collapse. The anti-redundancy signal must be paired with the verifier reward.
• Signal Specificity: While simple lexical (1-gram) diversity rewards offer some improvement, JPlag-based rewards yield the strongest results, suggesting that code-structure-aware similarity is a more effective signal for reducing repeated implementations than surface-level variation.

Significance and Claims

The paper claims to identify implementation redundancy as both a critical diagnostic and a practical optimization target for repeated-sampling code generation.

1. Diagnostic Value: The study demonstrates that models optimizing for similar executable outcomes can exhibit vastly different redundancy structures. Correctness-only RLVR tends to concentrate generations, whereas Pass@k-aware objectives naturally encourage diversity.
2. Optimization Target: The authors show that explicitly penalizing near-duplicate generations via JPlag similarity is a simple yet effective intervention. It improves finite-budget performance without requiring complex credit assignment schemes or relying on superficial output variations.
3. Practical Implication: Finite-budget code generation should not be understood solely through marginal correctness. How a model allocates its sampling budget across distinct implementation patterns is a key factor in maximizing Pass@k performance.

The authors remain modest regarding limitations, noting that JPlag is an approximate structural similarity measure rather than a semantic equivalence oracle, and that their experiments are limited to specific open-weight models and short-horizon competitive programming benchmarks. They do not claim the method solves all diversity issues in code generation but highlight its efficacy in the specific context of verifier-based RL.