Technical Summary: Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward (VIGOR)

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful post-training paradigm for Large Language Models (LLMs), particularly in reasoning tasks like mathematics and coding. However, RLVR relies heavily on external, domain-specific verifiers (e.g., exact-match checkers for math answers or unit tests for code). This dependency limits scalability to new tasks or domains where such verifiers are unavailable or difficult to construct.

Existing verifier-free alternatives face significant challenges:

• Majority Voting Methods: These derive pseudo-labels from consensus among multiple samples. They often assume final answers can be reliably extracted and normalized, limiting their applicability to open-ended generation.
• Intrinsic Signal Methods: These use internal model signals like entropy or self-certainty. While promising, they can become brittle as training progresses, leading to reward degeneration, instability, and performance collapse.

The core challenge is to develop a verifier-free reward signal that is broadly applicable to free-form outputs, stable during reinforcement learning, and relies solely on the policy model itself without auxiliary components.

2. Methodology: VIGOR

The authors propose VIGOR (Verifier-free Intrinsic Gradient-Norm Reward), a method that computes rewards using only the policy model's internal training dynamics.

Core Intuition

The method is grounded in first-order optimization theory. In a loss landscape, a smaller gradient norm typically indicates that the current parameters are near a stationary region, implying milder updates and smoother training dynamics. VIGOR posits that completions inducing smaller $\ell_2$ norms of the teacher-forced negative log-likelihood (NLL) gradients are more aligned with the current policy and should receive higher rewards.

Algorithmic Steps

For a given prompt $x$, the policy $\pi_\theta$ samples a group of $G$ completions $\{y_i\}_{i=1}^G$. The reward calculation proceeds as follows:

1. Gradient Norm Computation: For each completion $y$, the average token-level NLL $\ell_{mean}(x, y)$ is computed. The gradient $g(x, y) = \nabla_\theta \ell_{mean}(x, y)$ is calculated with respect to the current parameters $\theta$ (without updating the model). The gradient is detached from the computation graph to serve as a scalar signal.
2. Length Correction ($\sqrt{T}$ Scaling): The raw gradient norm $\|g\|_2$ exhibits a systematic bias where it shrinks as sequence length $T$ increases, even if quality does not improve. To counteract this "length hacking," the signal is scaled by $\sqrt{T}$:
   $$S_{GN}(x, y) = -\sqrt{T} \|g(x, y)\|_2$$
   The negative sign converts the minimization of gradient norms into a reward maximization objective.
3. Rank-Based Normalization: To stabilize reward scales across different prompts and ensure comparability, the raw signals within a group are sorted and mapped to a rank-based reward $R_{GN} \in [-1, +1]$. The worst completion (highest gradient norm) receives $-1$, and the best (lowest gradient norm) receives $+1$.
4. Policy Optimization: These rank-normalized rewards are used to compute group-relative advantages $\hat{A}_i$ within the Group Relative Policy Optimization (GRPO) framework. The policy is updated to maximize the objective, encouraging completions with lower gradient norms.

3. Key Contributions

• VIGOR Framework: A novel verifier-free intrinsic reward mechanism that utilizes the $\ell_2$ norm of teacher-forced gradients as a proxy for policy alignment and training stability.
• Length Correction: The introduction of a $\sqrt{T}$ scaling factor to neutralize the systematic length bias inherent in averaged token-level gradient norms, preventing the model from exploiting length inflation to maximize rewards.
• Rank-Based Shaping: A normalization technique that stabilizes reward scales across prompts by relying on within-group relative ordering rather than absolute gradient magnitudes.
• Empirical Validation: Extensive experiments demonstrating that VIGOR outperforms state-of-the-art verifier-free baselines (specifically RLIF/INTUITOR) on mathematical reasoning and code generation benchmarks, while maintaining stable training dynamics.

4. Experimental Results

Experiments were conducted on Qwen2.5-3B-Base and Qwen2.5-7B-Base models, trained on the MATH dataset and CodeContests dataset respectively.

Performance on Mathematics (Trained on MATH)

• Qwen2.5-7B: VIGOR improved average math accuracy by +3.31% over the INTUITOR baseline. Specifically, it achieved 76.20% on MATH500 and 44.42% on AMC.
• Cross-Domain Transfer: When trained solely on math data, VIGOR improved average code accuracy by +1.91% over the INTUITOR baseline, demonstrating effective cross-domain transfer.
• Stability: Unlike INTUITOR, which showed late-stage regression in accuracy, VIGOR maintained stable improvement throughout training.

Performance on Code (Trained on CodeContests)

• Qwen2.5-3B: VIGOR improved average code performance from 16.98% to 23.64%.
• Cross-Domain Transfer: Training on code data improved the math average from 48.34% to 56.57%.
• Note: On the CodeAvg metric, VIGOR (23.64%) slightly underperformed INTUITOR (25.43%), particularly on the CRUX benchmark. The authors attribute this to the nature of competitive programming, where correctness often depends on discrete algorithmic choices not fully captured by gradient smoothness.

Training Dynamics Analysis

• Stability: VIGOR exhibited more stable training dynamics compared to INTUITOR, which suffered from reward degeneration (where high-reward samples became less aligned with correctness over time).
• Length Control: VIGOR effectively curbed length inflation. While INTUITOR showed a pronounced increase in generation length, VIGOR produced substantially shorter outputs, focusing optimization on reasoning quality rather than length hacking.
• Rank-Accuracy Monotonicity: Analysis confirmed a strong monotonic relationship between the gradient-norm ranking and task accuracy; completions ranked higher by VIGOR consistently achieved higher accuracy.

Ablation Studies

• Without $\sqrt{T}$ Correction: Removing this component caused catastrophic degradation, particularly in the 3B model, resulting in near-zero accuracy and severe length hacking.
• Without Rank Shaping: Removing rank normalization led to instability, particularly in the 7B model, causing a significant drop (8.90%) in MMLU-Pro performance due to the influence of extreme outliers in raw gradient magnitudes.

5. Significance and Limitations

Significance:
The paper claims that VIGOR provides a robust, verifier-free alternative for RL post-training that does not rely on external labels or auxiliary models. By leveraging the intrinsic geometry of the loss landscape (gradient norms), it offers a stable signal for policy optimization that generalizes across domains (math to code and vice versa) and maintains training stability where other intrinsic methods fail.

Limitations:

• Task Scope: The study primarily targets verifiable reasoning tasks. The authors note it is unclear how well VIGOR transfers to more open-ended generation settings (e.g., long-form writing or dialogue) without additional constraints.
• Computational Cost: VIGOR requires per-sample gradient-norm computation via automatic differentiation, which introduces non-trivial overhead compared to forward-only signals like likelihood or entropy.
• Proxy Nature: The gradient norm is a proxy objective. The authors acknowledge it may not consistently track downstream utility, leaving room for potential misalignment or reward exploitation.