Technical Summary: Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair

Problem Statement

The paper addresses the challenge of training code agents using Group Relative Policy Optimization (GRPO) in weak-feedback environments. In agentic code repair (specifically "compile-fix" tasks), the online execution feedback (e.g., compilation success) is reliable but semantically incomplete. A patch may compile successfully yet fail to fix the intended semantic error (e.g., by deleting necessary code or adding unsafe stubs), or it may fail to compile due to environmental jitter rather than policy failure.

Standard GRPO relies on comparing multiple rollouts from the same prompt to generate advantages. However, in weak-feedback settings, three critical signal distortions prevent meaningful within-group comparisons:

1. Outcome-Semantics Failure: Binary rewards (compile success/fail) conflate syntactic shortcuts with genuine repairs, causing the agent to optimize for surface-level success rather than semantic correctness.
2. Diffuse Process Credit: Long-horizon trajectories receive a single scalar reward, making it impossible to distinguish effective repair steps from uninformative repetitions or repetitions.
3. Broken Within-Group Comparability: Infrastructure issues (e.g., container timeouts, compilation queue contention) and policy degeneration (e.g., catastrophic repetition) mix with genuine repair failures in the zero-reward bin, distorting the relative ranking of rollouts.

The authors argue that the solution is not to rewrite the RL objective, but to reshape the training signals fed into the existing GRPO framework to restore the validity of within-group comparisons.

Methodology

The proposed approach, Signal Reshaping, introduces three minimal interventions that modify the input signals to GRPO without altering its group-normalized advantage calculation mechanism.

1. Layered Outcome Rewards (Restoring Semantic Ranking)

Instead of a binary {0, 1} reward based solely on compilation, the authors propose a three-tiered reward structure:

• $R=0$: Compilation fails.
• $R=0.5$: Compilation succeeds, but the patch is semantically inconsistent with the ground-truth (GT) minimal sufficient repair.
• $R=1$: Compilation succeeds and is semantically consistent with the GT repair.
  The semantic consistency is determined by a strong LLM judge (Kimi-K2.5) comparing the agent's patch against a refined GT patch. This intermediate tier ($R=0.5$) preserves the ordering signal during training when fully correct semantic repairs are sparse, preventing the collapse of the ranking signal.

2. Step-Level Process Scores (Localizing Credit)

To address the diffusion of credit in long trajectories, the method introduces step-level process scores ($s_i \in [0, 1]$).

• These scores are assigned by a judge to individual tool-call steps within a trajectory that has successfully compiled.
• Crucially, these scores do not enter the GRPO scalar reward normalization. Instead, they act as token-level loss weights.
• High-scoring steps (e.g., successful edits, verification) receive higher gradient weights, while low-scoring steps (e.g., redundant file inspections) are down-weighted. This reallocates update strength within a trajectory without changing the relative advantage between different trajectories.

3. Failure-Cause-Aware Rollout Governance (Ensuring Comparability)

To ensure that rollouts in the same group are execution-comparable, the system implements a routing mechanism based on the failure cause:

• Full-Trajectory Masking: Samples failing due to non-learnable environmental exceptions (e.g., container initialization failure, context overflow, system timeouts) are masked entirely from the loss objective. They still contribute to the group mean/variance for normalization but generate no gradients.
• Last-Step Retention: Samples exhibiting policy degeneration (e.g., catastrophic repetition, excessive tool calls) are masked up to the final step, retaining only the last step as a penalty sample to align negative feedback with the specific point of failure.
• Normal Termination: Valid trajectories proceed to semantic judgment.
  Additionally, the system decouples concurrency for GPU inference, sandbox I/O, and compilation to prevent resource contention from biasing the sampling distribution.

Boundary Comparison: Token-Level Distillation

The paper also investigates whether token-level privileged distillation (using hints available during training but not at test time) can replace step-level process credit. Two methods, $\pi$-Distill (forward KL) and OPSD (reverse KL), are tested. The results show that token-level alignment fails to improve outcome correctness in long trajectories, as the signal is diluted by non-critical tokens and disrupts the trajectory-level optimization dynamics.

Key Results

Experiments were conducted on a compile-fix task using the Qwen3-Coder-30B model with a dataset of minimal sufficient repairs derived from internal CI logs.

• End-to-End Gain: The full signal-reshaped GRPO improved strict compile-and-semantic accuracy from the base model's zero-shot performance of 0.385 to 0.535 (a +15.0 percentage point gain).
• Role of Layered Rewards: Comparing a binary reward (compile-only) against the layered reward showed that binary rewards caused the compile rate to spike early and then degrade, failing to improve semantic correctness. The layered reward ($R=0.5$ tier) was essential for stable training and trajectory control.
• Role of Process Scores: Adding step-level process scores to the layered reward baseline further improved accuracy from 0.48 to 0.53 and reduced the average evaluation steps from 23.50 to 17.02. This demonstrates that process credit improves efficiency and correctness simultaneously, rather than trading exploration for success.
• Distillation Failure: Token-level distillation methods ($\pi$-Distill and OPSD) underperformed the GRPO baseline. $\pi$-Distill led to premature termination (learning to submit early), while OPSD led to low entropy and degraded tool skills. This confirms that local token alignment cannot replace outcome semantics or process credit in this setting.

Significance and Claims

The paper claims that for weak-feedback agentic tasks, signal reshaping is a more effective strategy than modifying the RL objective or adding mismatched supervision.

1. Reframing the Problem: The authors posit that GRPO's failure in weak-feedback settings is not due to the algorithm itself, but to the invalidity of the within-group comparison caused by incomplete signals.
2. Minimal Intervention: The proposed method leaves the core GRPO advantage calculation unchanged. It simply ensures that the inputs (rewards, process weights, and sample selection) satisfy the prerequisites for meaningful relative optimization.
3. Granularity Mismatch: The negative results on token-level distillation highlight a fundamental mismatch: in long-horizon agentic tasks, success is determined by a few critical decision points (tool calls, termination), while token-level supervision is diluted by non-critical tokens. Therefore, step-level process credit and outcome semantics are necessary and cannot be replaced by dense token-level alignment.

The work suggests a design principle for future agentic RL: when the objective is correct but feedback is incomplete, reshaping the signal to recover semantic ordering, localized credit, and sample comparability is more effective than adding auxiliary supervision that does not align with the decision granularity.