Technical Summary: Extrapolative Weight Averaging Reveals Correctness–Efficiency Frontiers in Code RL

Problem Statement

In Reinforcement Learning (RL) for code generation, particularly in competitive programming, models face a dual objective: functional correctness (passing all test cases) and computational efficiency (solving within time and memory limits). Standard RL training often struggles to balance these, as optimizing for strict efficiency can lead to correctness failures, while optimizing for correctness on small inputs may yield inefficient solutions that fail on larger stress tests.

Existing work on model merging (e.g., "Rewarded Soups") has shown that linearly interpolating weights between checkpoints trained under different reward objectives can trace a Pareto frontier between those objectives. However, it remains unclear whether extrapolative weight averaging (averaging weights with coefficients outside the $[0, 1]$ interval) can extend these frontiers to new, useful checkpoints without additional training. Specifically, can extrapolation navigate beyond the limits of individual RL training runs to discover policies that are neither purely correct nor purely efficient, but complementary?

Methodology

Experimental Setup

The authors investigate this question using competitive programming as a controlled setting where verifier strictness serves as a natural experimental axis.

• Models: Experiments are conducted on 7B and 32B parameter models (Qwen 2.5 7B and CWM-SFT 32B).
• Training Environment: Three distinct RL settings are used:
  1. Pure Reasoning: Single-turn generation.
  2. Tool Use: Multi-turn reasoning with Python tool calls.
  3. Agentic Coding: Multi-turn interaction with a sandboxed terminal.
• Reward Strictness Sweep: Starting from a shared Supervised Fine-Tuning (SFT) checkpoint, independent RL runs are trained on the CodeContestsPlus dataset. The reward function is a nested binary signal based on input-length thresholds ($k$):
  • $R_k(y) = 1$ if the solution passes all tests with input length $< 10^k$.
  • $R_\infty(y) = 1$ if the solution passes all tests (full suite).
  • This creates an ordered family of checkpoints from "low-coverage" (relaxed, small inputs) to "high-coverage" (strict, full stress tests).

Weight Averaging

The core operation is linear weight averaging between two endpoints, $\theta_{low}$ (low-coverage) and $\theta_{high}$ (high-coverage):
$$\theta_\alpha = \alpha \theta_{high} + (1 - \alpha) \theta_{low}$$

• Interpolation: $\alpha \in [0, 1]$.
• Extrapolation: $\alpha \notin [0, 1]$ (e.g., $\alpha < 0$ or $\alpha > 1$).

Evaluation

Checkpoints are evaluated on the LiveCodeBench (LCB) benchmark (held-out, full test coverage). Outcomes are categorized into:

1. Correct: Passes all tests.
2. Optimization Failure: Passes smaller tests but exceeds time/memory limits on larger ones.
3. Correctness Failure: Fails due to logic errors or runtime exceptions within limits.
4. Format Error: Unparsable output.

Key Contributions and Results

1. Emergence of a Correctness–Efficiency Frontier

The study reveals that varying verifier strictness does not produce a simple ranking of checkpoints. Instead, on hard problems, it traces a correctness–efficiency frontier:

• Low-coverage training reduces correctness failures but increases optimization failures (inefficient solutions).
• High-coverage training reduces optimization failures but increases correctness failures (models attempt efficient algorithms but fail logic).
• Solve Rate Stability: The overall solve rate remains nearly constant across the frontier; the trade-off is primarily between how the model fails (inefficiency vs. incorrectness), not if it solves the problem.

2. Interpolation Recovers the Frontier

Linear interpolation between low- and high-coverage endpoints successfully recovers the frontier traced by the independently trained checkpoints. A single interpolation direction in weight space is sufficient to traverse the trade-off between optimization and correctness failures.

3. Extrapolation Extends the Frontier

Crucially, extrapolative weight averaging extends this frontier beyond the endpoints reached by any single RL run:

• Beyond RL Limits: For $\alpha > 1$ and $\alpha < 0$, the extrapolated checkpoints continue the same trade-off curve, achieving levels of optimization/correctness failure ratios that no individual RL checkpoint reached.
• Stability: This continuation is stable over a broad range (e.g., $\alpha \in [-0.5, 1.5]$). Extreme extrapolation ($|\alpha| > 2$) leads to degradation primarily due to format errors (token exhaustion) rather than a collapse of the frontier logic.
• Generalization: This phenomenon holds across all three inference settings (reasoning, tool use, agentic coding) and both model scales (7B, 32B), provided the problem difficulty is matched to the model's capability.

4. Inference-Time Scaling via Ensembles

The paper demonstrates that extrapolated checkpoints are complementary policies:

• Per-Problem Turnover: Different points along the extrapolated axis solve different subsets of hard problems. While the overall solve rate is flat, the specific problems solved vary significantly.
• Ensemble Gains: By ensembling checkpoints along the extended frontier (interpolation + extrapolation), the authors improve the pass@250 metric on LCB/hard by 3.3% over the best single checkpoint at a matched sample budget.
• Diversity: The extrapolated pool provides unique solutions for problems that interpolation-only pools miss, particularly on the hardest problems.

5. What Moves the Frontier?

The authors contrast static reward shaping with failure-conditioned training:

• Static Reward Shaping: Assigning partial credit for passing smaller tests (discrete or continuous) keeps the model on the same frontier. It merely reweights existing outcomes without introducing new information.
• Failure-Conditioned Training: Using failure context (failed code + execution stats) to train a repair policy (online-fixing) shows potential to shift the frontier outward, improving both fix rates and original pass rates, suggesting that richer failure signals are required to move beyond the existing trade-off.

Significance and Claims

The paper claims that nested unit-test coverage in code RL induces a correctness–efficiency frontier that can be navigated, extended, and exploited via extrapolative weight averaging.

• Novelty: It establishes that extrapolation is not just a mathematical curiosity but a practical method to access a "continuation" of the Pareto frontier, generating diverse policies that no single training run produces.
• Practical Utility: Extrapolated checkpoints serve as a cheap, training-free mechanism for inference-time scaling, broadening the coverage of solved problems without additional RL compute.
• Limitations: The frontier is induced by the specific trade-off between correctness and efficiency in competitive programming. The authors note that moving the frontier itself (improving the Pareto front) likely requires training signals that reveal why programs fail (e.g., failure-conditioned training) rather than just stricter tests.

The work suggests a new paradigm for RL in code: rather than seeking a single "optimal" checkpoint, practitioners can leverage weight averaging to explore a spectrum of complementary behaviors, extending the capabilities of existing models through simple arithmetic operations on weights.