🔢 mathematics

Position: Adopt Constraints Over Fixed Penalties in Deep Learning

This position paper argues that deep learning problems with non-negotiable requirements should be addressed by directly solving the formulated constrained optimization, rather than relying on a fixed weighted sum penalty method that offers no guarantee of constraint satisfaction, softens hard requirements into weak compromises, and requires costly trial-and-error tuning.

Original authors: Juan Ramirez, Meraj Hashemizadeh, Simon Lacoste-Julien

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juan Ramirez, Meraj Hashemizadeh, Simon Lacoste-Julien

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Argument: Do Not Trade Your Rules for a Better Score

Imagine you are training a robot to drive a car. You have two goals:

Drive fast (Task performance).
Never hit a pedestrian (An immutable requirement).

The authors of this paper argue that in the world of Deep Learning (AI), we often approach the second goal incorrectly. Instead of treating "never hit a pedestrian" as a fixed rule, we usually convert it into a "penalty point."

Here is the breakdown of their argument using simple analogies.

The Current Approach: The Error of the "Fixed Penalty"

The Analogy: Imagine a video game where you earn points for speed but lose points when you hit a wall.

The Setup: The game designer says: "If you hit a wall, I will deduct 10 points from your total score."
The Problem: The AI (the player) realizes that hitting the wall once costs only 10 points, whereas driving 100 meters faster yields 50 points. So the AI decides: "It is worth hitting the wall a few times to achieve the high score."
The Reality: In Deep Learning, this is called Fixed Weighted Sum Penalties. We take the rule "hit wall," convert it into a number (a penalty), add it to the "speed" score, and tell the AI to minimize the total result.

Why the authors hate this:

It is not the same problem: In complex, chaotic environments (non-convex settings), minimizing the score "speed minus wall penalty" does not guarantee that you find the solution that actually adheres to the rule. You might find a "fast" solution that crashes, or a "safe" solution that is too slow, but you will never find the perfect balance where you are both fast and safe.
The "Goldilocks" Tuning Nightmare: For the penalty to work, you must guess the right number.
- If the penalty is too low (1 point), the AI ignores the wall.
- If the penalty is too high (1,000 points), the AI drives so slowly that it never reaches the finish line.
- Searching for the "just right" number requires endless trial and error. It is like trying to guess the exact temperature for baking a cake by baking it 50 times and changing the oven temperature by 1 degree each time.
It weakens the rule: By converting a hard rule ("Do not hit") into a soft penalty ("Try not to hit, but it is okay if you pay the price"), you are essentially telling the AI that safety is just another tradeable commodity. If the AI can achieve a slightly better score by breaking the rule, it will do so.

The Proposed Solution: The "Hard Constraint" Approach

The Analogy: Imagine a strict driving instructor who says: "If you hit a wall, the lesson ends immediately. You must stay on the road."

The Setup: Instead of deducting points, the system is designed to enforce the rule. The AI is only allowed to explore paths that stay on the road.
The Method: The authors propose using Constrained Optimization (specifically Lagrange methods).
- Think of this as an "intelligent penalty" that changes itself.
- If the AI begins to drift toward the wall, the penalty automatically becomes huge and forces it back.
- If the AI is far from the wall, the penalty becomes small, allowing it to focus on driving fast.
- The system learns the correct "pressure" automatically during training, rather than requiring a human to guess the number in advance.

Why This Matters (The "So What?")

The authors are not saying you should never use penalties.

Use penalties when: You have a "soft preference." (e.g., "I prefer the car to be slightly smaller, but it is okay if it is slightly larger.")
Use constraints when: You have a "hard requirement." (e.g., "The car must not exceed 60 mph" or "The medical diagnosis must not miss a tumor".)

If you have a hard requirement, using a fixed penalty is like trying to hold a heavy box with a rubber band. Sometimes it works, but often the box slips through. Using a constraint is like packing the box inside a crate. It stays in place.

The Trade-off

The paper admits that the "Hard Constraint" method is somewhat more complicated to set up. It is like using a specialized tool instead of a hammer.

The Cost: It might require slightly more computer time (the paper says about 1% to 5% more) and demands slightly more programming expertise.
The Benefit: You actually solve the problem you said you wanted to solve. You do not have to spend weeks guessing numbers, and you do not have to worry that the AI has found a "loophole" where it breaks the rules just to achieve a better score.

Summary

The paper argues that when we have immutable rules for AI (such as safety or fairness), we should stop treating them as optional "penalties" that we can trade away. Instead, we should embed them directly into the training process as hard constraints. This ensures that the AI actually follows the rules, rather than merely calculating that breaking them "pays off."

Technical Conclusion: Position: Adoption of Constraints Instead of Fixed Penalties in Deep Learning

Problem Statement
The article addresses the growing need to integrate explicit, non-negotiable requirements (e.g., fairness limits, safety thresholds, sparsity budgets) into deep learning systems. While the standard practice for handling such requirements is the fixed weighted sum penalty—where constraints are added to the task loss function with fixed coefficients to generate a scalarized objective—the authors argue that this approach is fundamentally unsuitable for deep learning problems where requirements constitute true constraints rather than soft preferences.

The core problem is that the fixed penalty transforms a constrained optimization problem into an unconstrained surrogate problem. In non-convex settings characteristic of deep learning, the solution to the penalized problem is not guaranteed to solve the original constrained problem. Furthermore, the process of selecting penalty coefficients is not a standard hyperparameter tuning task; rather, it defines the specific trade-off to be optimized. A poor choice of coefficients can lead to solving the "wrong" problem altogether, rather than merely solving the intended problem poorly.

Methodology and Theoretical Analysis
The article does not propose a single new algorithm, but rather a shift in formulation strategy, advocating to begin with a constrained formulation and select a solution method based on the problem's structure.

Theoretical Limitations of Fixed Penalties:
- Non-Equivalence in Non-Convexity: The authors demonstrate that in non-convex settings, the set of solutions reachable via a fixed weighted sum of the objective and constraints does not necessarily contain the constrained minimizer. They provide a concave 1D counterexample where no choice of a fixed penalty coefficient can recover the true constrained minimizer; the penalized objective either ignores the constraint or drives the solution to a suboptimal boundary.
- Tuning Effort: Selecting penalty coefficients proves costly and unstable. Unlike learning rates (which influence convergence speed), penalty coefficients determine the specific trade-off between task performance and constraint violation. The article illustrates that achieving a specific target (e.g., 50% sparsity) often requires repeated retraining and bisection search, whereas constraint-based methods can reach the target in a single run.
- Softening of Requirements: Fixed penalties transform hard constraints into soft penalties that can be traded off against performance. This removes feasibility from the optimization procedure itself and makes compliance something that must be verified post-hoc, rather than enforced by construction.
Proposed Alternative: The Lagrange Approach:
The article advocates for primal-dual methods, particularly the Lagrange approach, as a principled alternative.
- Mechanism: Instead of fixed coefficients, the method introduces adaptive multipliers (dual variables) learned during optimization. The algorithm alternates between minimizing the Lagrangian with respect to the primal variables (model parameters) and maximizing with respect to the dual variables (multipliers).
- Adaptivity: When a constraint is violated, its corresponding multiplier increases, placing more weight on fulfillment. When the constraint is satisfied with slack, the multiplier decreases. This enables the method to automatically learn the appropriate trade-off to satisfy the constraint level $\epsilon$ without manual coefficient adjustment.
- Computational Cost: The article notes that for first-order methods, the computational overhead for Lagrange updates is moderate (e.g., ~5% for rate-constrained classification, ~1.6% for sparsity-constrained training) compared to the costs of repeated retraining required to adjust fixed penalties.

Main Results and Evidence
The article supports its position through theoretical examples and empirical illustrations:

Concave Counterexample: In a 1D concave optimization problem, the fixed penalty fails to recover the constrained minimizer for any coefficient value, often yielding infeasible or overly conservative solutions. In contrast, an Augmented Lagrange method consistently recovers the true constrained minimizer across various constraint levels.
Hyperparameter Sensitivity: Experiments on a rate-constrained binary classification task show that the fixed penalty yields satisfactory results only within a narrow range of coefficients. Conversely, the primal-dual approach achieves satisfactory solutions over a much broader range of dual step sizes, highlighting that the dual step size is an optimization parameter, whereas the fixed penalty coefficient defines the problem itself.
Sparsity Constraints: A visualization of a sparsity-constrained classification task shows that achieving a target sparsity level via fixed penalty requires multiple iterations of retraining and coefficient adjustment, whereas a constraint-based method reaches the target in a single run.

Significance and Claims
The primary contribution of the article is a formulation-level claim: If a deep learning problem specifies non-negotiable requirements, the fixed weighted sum penalty is often a discrepancy between the problem specification and the optimization problem to be solved.

Distinction from Previous Work: The authors clarify that they do not criticize penalties in general (which remain valid for soft preferences or regularizers) nor argue that every problem must be constrained. Their critique specifically targets the standard use of fixed penalties for true requirements.
Practical Guidance: The article offers a decision framework:
1. Determine whether the quantity is a hard requirement (use constraints) or a soft preference (use penalties).
2. If a constraint is required, utilize specific structures (e.g., projections, reparameterizations) if available.
3. For large-scale deep learning without specific structure, use primal-dual gradient-based methods (such as Lagrange approaches) as a baseline instead of fixed penalties.
Outlook: The authors acknowledge that constraint-based methods currently face challenges regarding stability, generalization to unseen data, and integration into standard pipelines. However, they argue that these are reasons to improve constrained optimization procedures, not to retreat to fixed penalties that solve a surrogate problem. The goal is to make constrained training as robust and routine as standard deep learning without constraints.

Conclusion
The article concludes that for non-negotiable requirements, the constrained formulation should be the starting point. Fixed penalties should be regarded as a surrogate that risks solving the wrong problem, whereas constrained optimization procedures (particularly primal-dual approaches) directly address the stated requirement and ensure that feasibility is integrated into the optimization process rather than verified post-hoc.