💬 NLP

Constrained Group Relative Policy Optimization

This paper introduces Constrained GRPO, a Lagrangian-based extension of Group Relative Policy Optimization that improves constraint enforcement and training stability by scalarizing standardized advantages rather than raw rewards to eliminate the detrimental coupling effects caused by within-group normalization.

Original authors: Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azalée Robitaille, Christopher Pal, Liam Paull

Published 2026-09-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azalée Robitaille, Christopher Pal, Liam Paull

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, researchers are teaching large computer models to solve complex problems, from navigating self-driving cars to solving difficult math puzzles. These models learn by trial and error, a process known as reinforcement learning. Imagine a student trying to solve a maze; they receive a reward for reaching the exit and a penalty for hitting a wall. Over time, the student learns to maximize rewards and avoid penalties. However, a major challenge arises when we want the model to follow strict rules, such as "never hit a pedestrian" or "always use correct grammar," while still trying to be helpful. If the rules are too rigid, the model might become useless; if they are too loose, it might break them. To solve this, scientists use a mathematical framework that balances the desire to succeed with the need to obey constraints, adjusting the importance of each rule as the model learns.

A popular method for teaching these models, called Group Relative Policy Optimization, has become a favorite because it is efficient and does not require a separate "judge" model to evaluate every step. Instead, it compares a group of answers generated for the same question to decide which ones are better. While this method works well for general tasks, researchers found that applying it to strict safety rules was tricky. In a new study, a team from the Mila – Quebec AI Institute and École Polytechnique de Montréal discovered that the standard way of combining different goals into a single score was actually breaking the system's ability to follow rules. They introduced a new approach, Constrained Group Relative Policy Optimization, which fixes this flaw and allows models to learn complex behaviors while strictly adhering to safety limits.

The core problem the researchers identified was how the computer handles multiple goals at once. In the standard approach, the model takes all its rewards and penalties, mixes them together into one single number, and then normalizes that number to make it easier to learn from. The researchers showed that this mixing process creates a hidden interference. When the computer adjusts the weight of one rule, it unintentionally changes the relative importance of all the other rules as well. It is like trying to tune the volume of a single instrument in an orchestra by turning a knob that also shifts the balance of the entire band; you might try to make the violins louder, but in doing so, you accidentally make the drums too quiet and the flutes too loud. This makes it very difficult for the model to learn exactly which rule to follow, often causing it to ignore safety constraints or become unstable during training.

To fix this, the researchers changed the order of operations. Instead of mixing the rewards and penalties first, they let the model calculate the value of each rule separately and normalize them individually. Only after each rule has been treated fairly on its own do they combine them using the learned weights. This simple switch removes the hidden interference. By keeping the signals separate until the very end, the model can clearly see how much it is improving on each specific rule. The result is a learning process that is much more stable and predictable. The researchers tested this new method in three very different environments: a simple grid-based game where an agent had to avoid lava and manage a battery, a realistic simulation of autonomous driving with thousands of complex traffic scenarios, and a mathematical reasoning task involving grade-school word problems.

In the grid-based game, the new method allowed the agent to learn much more smoothly. The standard approach caused the agent to become overly cautious, avoiding the lava so aggressively that it barely moved, while the new method allowed the agent to use its available "budget" of risk effectively, reaching the goal while staying safe. In the autonomous driving simulation, the new approach produced drivers that were not only safer but also more effective at completing their routes. The models trained with the new method achieved higher scores in safety compliance and route progress compared to previous methods that mixed the signals first. They successfully avoided collisions and followed traffic laws without sacrificing their ability to drive forward, a balance that other methods struggled to maintain.

The final test involved teaching a language model to solve math problems while ensuring the answers were short, formatted correctly, and contained valid numbers. Here, the new method again proved superior. Models trained with the standard mixing approach often sacrificed correctness to make their answers shorter or to fit a specific format. In contrast, the new method ensured that the model prioritized getting the math right, while still maintaining high standards for formatting and length. Across different sizes of computer models, from smaller ones with 1.5 billion parameters to larger ones with 7 billion, the new approach consistently produced more accurate results without needing expensive additional training components.

The findings suggest that the way we combine different learning signals is just as important as the signals themselves. By simply changing the order in which the computer processes its rewards and rules, the researchers were able to create a system that respects constraints much more reliably. This work does not just offer a small improvement; it provides a clearer path for training artificial intelligence to operate safely in the real world, where following the rules is often just as important as achieving the goal. The study confirms that when we want AI to be both capable and safe, we must be careful not to let the way we measure success confuse the model about what it is actually supposed to do.

Technical Summary: Constrained Group Relative Policy Optimization

Problem Statement

Group Relative Policy Optimization (GRPO) has emerged as a dominant, critic-free approach for fine-tuning Large Language Models (LLMs) and Vision-Language Models (VLMs), particularly in domains like mathematical reasoning and embodied decision-making. GRPO eliminates the need for a learned value function (critic) by estimating advantages through within-group normalization of rewards obtained from multiple sampled outputs for a single query.

However, the compatibility of GRPO with constrained policy optimization—essential for safety-critical domains where specific behavioral thresholds must be enforced—has not been rigorously examined. Existing multi-objective GRPO methods often rely on manually weighting reward terms, which lacks the principled framework of Constrained Markov Decision Processes (CMDPs). CMDPs enforce constraints via explicit thresholds rather than fixed weights, typically using Lagrangian relaxation. The core challenge addressed in this work is integrating Lagrangian-based constraint enforcement into the GRPO framework without introducing instability or distorting the intended trade-offs between objectives.

Methodology: Constrained GRPO

The authors propose Constrained GRPO, a framework that combines GRPO for policy optimization with Lagrangian relaxation for constraint handling. The method utilizes indicator cost functions to define behavioral constraints (e.g., safety violations) and learns multipliers to enforce target behavior rates.

The Critical Failure Mode: Scalarizing Rewards

A straightforward approach to combining GRPO with Lagrangian methods involves scalarizing rewards before normalization. In this approach, the reward and cost terms are linearly combined using the current Lagrangian multipliers ( $\lambda$ ) to form a single scalar signal, which is then standardized (mean-subtracted and divided by standard deviation) within the group.

The paper identifies a critical failure mode in this approach: shared-denominator coupling.

Normalization Dependency: The standardization denominator (the group standard deviation, $\hat{\sigma}_{S_\lambda}$ ) depends on the scalarized signal, which is a function of the multipliers.
Coupled Sensitivity: Changing a single multiplier $\lambda_k$ alters not only the weight of its corresponding constraint but also the normalization factor for all components (rewards and other constraints).
Consequence: This introduces covariance-mediated coupling where the effective weight of a constraint becomes a complex, non-linear function of all multipliers and the current batch statistics. This leads to unstable multiplier dynamics, where increasing a penalty multiplier might inadvertently suppress the reward signal or other constraints, preventing effective enforcement of the specified behavior rates.

The Proposed Solution: Scalarizing Advantages

To resolve this, the authors propose scalarizing standardized advantages instead of rewards. The procedure is as follows:

Component-wise Standardization: Each reward and cost component is standardized independently within the group (subtracting the component mean and dividing by the component standard deviation).
$Z_{R,i} = \frac{R_i - \bar{R}}{\hat{\sigma}_R}, \quad Z_{C_k,i} = \frac{C_{k,i} - \bar{C}_k}{\hat{\sigma}_{C_k}}$
Linear Combination: The standardized components are then combined using the Lagrangian multipliers to form the final advantage:
$A_i = \lambda_R Z_{R,i} - \sum_{k=1}^K \lambda_k Z_{C_k,i}$

Theoretical Justification:
The authors provide a sensitivity analysis (Theorem 4.1 and Corollary 4.2) showing that scalarizing advantages removes the shared-denominator coupling. While softmax-normalized multipliers still introduce competition between components (as increasing one logit reduces the mass of others), the directions of the components are already normalized. Consequently, the primal update responds directly to the multiplier changes without the confounding effect of a changing normalization denominator. This yields a better-conditioned update and ensures that increases in Lagrangian multipliers translate reliably into stronger enforcement of the corresponding constraints.

Key Contributions

Framework Introduction: The introduction of Constrained GRPO, a policy optimization framework that enforces user-specified behavioral constraints via indicator costs and learned Lagrangian multipliers, eliminating the need for manual reward weight tuning.
Theoretical Analysis: A formal characterization of the local response of the primal update to changes in dual variables. The authors prove that scalarizing rewards before normalization introduces a specific coupling that distorts the effective weights of objectives, whereas scalarizing advantages avoids this issue.
Empirical Validation: Demonstration across three distinct domains that scalarizing advantages leads to more stable learning and faithful constraint enforcement compared to scalarizing rewards.

Experimental Results

The authors evaluated Constrained GRPO on three benchmarks:

1. Gridworld Environment

In a controlled 10x10 gridworld with lava and battery constraints:

Scalarized Advantages: Produced stable learning curves and allowed constraint violation rates to adapt naturally toward the specified thresholds (0.01). The Lagrangian multipliers adjusted effectively to utilize the available cost budget.
Scalarized Rewards: Caused constraint rates to collapse to near-zero early in training, regardless of multiplier values. The effective weights became noisy and decoupled from the learned multipliers, leading to inconsistent behavior.

2. NAVSIM-v2 (Autonomous Driving)

Evaluated on the NAVSIM-v2 "Navhard" split using the Extended PDM Score (EPDMS), which aggregates safety penalties (collisions, drivable area, etc.) and performance metrics.

Performance: Constrained GRPO with scalarized advantages achieved the highest EPDMS on Stage 1 (real scenes) and second-highest on Stage 2 (synthetic scenes) among GRPO baselines, outperforming both unconstrained GRPO variants and a hard-constraint GRPO baseline.
Constraint Enforcement: Unlike scalarized rewards, where multipliers failed to drive constraint satisfaction (e.g., drivable area compliance remained low despite high multipliers), scalarized advantages allowed the multipliers to exert their intended influence, rapidly improving constraint satisfaction (e.g., DAC improved within ~1000 steps).
Comparison: The method outperformed unconstrained GRPO and matched or exceeded the performance of PPO-Lagrangian, despite not requiring a learned critic.

3. GSM8K (Mathematical Reasoning)

Evaluated on grade-school math problems using Qwen2.5-Instruct models (1.5B, 3B, 7B). Constraints included correctness, format adherence, and integer validity; the primary reward was minimizing length.

Correctness: Constrained GRPO with scalarized advantages consistently improved correctness over the Multi-Objective GRPO baseline (GDPO) across all scales (e.g., 0.770 $\to$ 0.802 at 3B; 0.768 $\to$ 0.887 at 7B).
Trade-offs: The method prevented the model from over-optimizing length at the expense of correctness, a common failure mode in unconstrained or poorly weighted multi-objective setups.
Efficiency: At 1.5B and 3B scales, Constrained GRPO achieved competitive or better performance than PPO-Lagrangian without the computational cost of training a value function.

Significance and Claims

The paper claims that Constrained GRPO provides a robust, critic-free framework for safety-critical fine-tuning of large models. Its primary significance lies in identifying and resolving a subtle but critical interaction between Lagrangian multipliers and GRPO's within-group normalization.

The authors assert that the standard practice of scalarizing rewards before normalization is fundamentally flawed for constrained optimization because it introduces shared-denominator coupling, which decouples the learned multipliers from the actual policy update dynamics. By shifting to scalarized advantages, the method ensures that the Lagrangian multipliers function as intended: directly controlling the trade-off between objectives and constraints. This results in more stable training, reliable adherence to safety thresholds, and improved task performance without the need for expensive critic models or manual reward weighting.