Technical Summary: Neural Slack Variables for Shape Constraints

Problem Statement

In scientific computing and industrial applications (e.g., control systems, quantitative finance), neural networks are often required to approximate functions that must satisfy structural laws, such as monotonicity, convexity, or the absence of arbitrage. These requirements manifest as functional inequality constraints (e.g., $C[f_\theta](x) \geq 0$) that must hold everywhere on the domain.

Existing approaches face significant limitations:

1. Architectural Constraints: Methods like Input-Convex Neural Networks (ICNN) or Constrained Monotonic Neural Networks (CMNN) guarantee feasibility by construction but restrict the hypothesis class, often leading to training stiffness, low-frequency bias, and poor fit to complex data.
2. Soft Constraints (Penalty & Primal-Dual): The standard deep learning approach involves adding a penalty term to the loss or using primal-dual (Lagrangian) methods. However, the paper identifies a shared failure mode termed "constraint drifting."
   • In penalty methods, once the network becomes feasible at sampled points, the violation term vanishes, providing no gradient to maintain feasibility. Subsequent training steps can reintroduce violations, which are only corrected after they appear.
   • Primal-dual methods suffer similarly: under complementary slackness, Lagrangian multipliers become zero in feasible regions, eliminating the constraint gradient.
   • The result is fragile satisfaction where models exhibit spurious violations, which is unacceptable in high-stakes domains like finance (where arbitrage-free surfaces are required) or control (where barrier certificates must be exact).

Methodology: Neural Slack Variables

The authors propose Neural Slack Variables (NSV), a primal-side soft constraint method that converts constraint enforcement into a regression problem.

Core Mechanism

Instead of penalizing violations of $C[f_\theta](x) \geq 0$, the method introduces an auxiliary neural network $s_\phi$ (the "neural slack variable") that is jointly trained with the primary network $f_\theta$.

1. Feasible Target: The auxiliary network $s_\phi$ is architecturally constrained to output non-negative values (e.g., via an activation like $\epsilon + (\cdot)^2$), ensuring $s_\phi(x) \in K = [\epsilon, \infty)^m$.
2. Matching Loss: The primary network's constraint profile $c_\theta(x) = C[f_\theta](x)$ is trained to match the slack variable $s_\phi(x)$ via a quadratic matching loss:
   $$\mathcal{L}_{slack}(\theta, \phi) = \frac{1}{2} \mathbb{E}_{x \sim U(\Omega)} \left[ \| C[f_\theta](x) - s_\phi(x) \|_2^2 \right]$$
3. Joint Optimization: The total loss is $\mathcal{L}(\theta, \phi) = \mathcal{L}_{data}(\theta) + \rho \mathcal{L}_{slack}(\theta, \phi)$.

Key Dynamics

• Gradient Persistence: Unlike penalty methods where gradients vanish upon feasibility, the finite capacity of the learned slack network $s_\phi$ ensures that the matching residual $c_\theta - s_\phi$ generally does not vanish completely. This leaves a persistent "anchoring" gradient even when $c_\theta \geq \epsilon$, preventing the constraint profile from drifting back into violation.
• Variable Splitting Interpretation: The method can be viewed as a learned relaxation of variable splitting. Instead of projecting $c_\theta$ exactly onto the feasible set (which requires an intractable architecture for general constraints), the method learns a feasible surrogate $s_\phi$ and steers $c_\theta$ toward it.
• Inductive Bias: The architecture of $s_\phi$ imposes an explicit inductive bias on the shape of the constraint profile. By choosing spectrally expressive architectures for $s_\phi$ (e.g., SIRENs), the method can stabilize constraint satisfaction between collocation points, even when the primary network $f_\theta$ is highly expressive.

Implementation Details

• Scale-Balanced Matching: To handle the wide dynamic range of constraints, the authors employ a relative scaling mechanism where the matching loss is normalized by the magnitude of $s_\phi$, preventing large satisfied margins from dominating the gradient.
• Stabilization: An $\text{asinh}$ transform is applied to the arguments before differencing to improve training stability.

Key Contributions

1. Novel Method: Introduction of Neural Slack Variables, a soft constraint technique that couples the primary network with a jointly learned, non-negative auxiliary network.
2. Diagnosis of Drifting: Identification and visualization of the "constraint drifting" failure mode in penalty and primal-dual methods, demonstrating that NSV overcomes this in data-driven settings.
3. Regularity Transfer: Demonstration that the slack loss transfers the regularity of $s_\phi$ to the constraint profile $c_\theta$, enabling the use of spectrally expressive primary networks (SIRENs, Fourier features) without losing constraint satisfaction.
4. Empirical Superiority:
   • Synthetic Benchmarks: On monotonicity and convexity tasks, NSV achieves zero measured violations on dense grids, whereas penalty and Lagrangian baselines leave residual violations.
   • Data-Free Certification: In the FOSSIL Barr3 benchmark (neural barrier certificates), NSV successfully induces feasibility where penalty methods fail due to the lack of a data-fitting term to anchor the solution.
   • Industrial Application: Applied to implied volatility surface modeling in quantitative finance. NSV is the only method tested that produces robustly arbitrage-free surfaces across training, interpolation, and prior sampling regimes, while simultaneously achieving the lowest data fit error.

Results Summary

• Monotonicity (1D): NSV achieved 0 violations across 100 seeds, compared to 2/100 for penalty methods and 73/100 for Augmented Lagrangian. When using SIRENs for the primary network, NSV outperformed architectural baselines (CMNN) in data fit while maintaining 100% feasibility.
• Convexity (High-D): As dimensionality increased ($d=2$ to $5$), penalty methods degraded significantly due to the curse of dimensionality on sparse grids. NSV maintained feasibility longer than penalty methods, though both eventually struggled at high dimensions ($d=5$), while ICNNs remained feasible but underfitted the data.
• Volatility Surfaces: In the generative modeling of NDX option surfaces, NSV eliminated arbitrage violations ($\eta_{rate}=0$) in all three evaluation regimes (TRAIN, INTERPOLATION, PRIOR), whereas penalty and Augmented Lagrangian methods exhibited residual violations. NSV also achieved the lowest Mean Absolute Error (MAE) on the data fit.

Significance and Claims

The paper claims that Neural Slack Variables offer a robust alternative to both architectural constraints and traditional soft constraints. By decoupling the feasibility requirement from the primary network's architecture and instead learning a valid target profile, NSV avoids the "constraint drifting" that plagues penalty and primal-dual methods.

The authors position this as a solution to a high-impact industrial challenge: the generation of arbitrage-free volatility surfaces in quantitative finance. They argue that while architectural constraints are theoretically sound, they are often too rigid for complex data, and soft constraints are too fragile. NSV bridges this gap, enabling the use of flexible, high-capacity neural architectures while guaranteeing (in practice) strict adherence to functional inequalities.

The paper acknowledges limitations: the method scales poorly with input dimension due to the need for dense constraint grids, and it does not provide formal off-grid certification (unlike architectural constraints). However, for low-to-moderate dimensional problems where data fidelity and constraint satisfaction are both critical, NSV presents a superior empirical performance.