Technical Summary: CAffNet: Hard Constraint-Affine Neural Networks

Problem Statement

Neural networks (NNs) possess remarkable expressive power for approximating complex nonlinear functions, making them valuable for safety-critical domains such as autonomous vehicles, robotics, and aerospace. However, a significant challenge remains in ensuring that learned controllers provide formal hard constraint satisfaction.

Existing approaches suffer from specific limitations:

• Soft Constraints: Penalty-based methods encourage safe outputs but offer no guarantee of satisfaction, particularly with limited training data or out-of-distribution edge cases.
• Post-Processing: Enforcing constraints after training can lead to suboptimality.
• Input-Independent Hard Constraints: Methods like C-DGMs or RAYEN handle general constraints but assume the feasible set is independent of the input.
• Input-Dependent Hard Constraints: Recent works (e.g., POLICE, KKT-hPINN, HardNet-Aff) address input-dependent constraints but face critical restrictions. Specifically, HardNet-Aff assumes the constraint matrix $A(x)$ has full row rank. This limits the number of constraints to the output dimension and fails when constraints are linearly dependent, potentially projecting outputs outside the feasible region. Furthermore, these methods often struggle with non-unique projections when the constraint matrix is not full column rank.

Methodology: CAffNet Framework

The authors propose CAffNet (Hard Constraint-Affine Neural Networks), a closed-form framework that embeds a trainable Constraint-Affine (CAffine) layer into standard feedforward or transformer architectures. The framework enforces input-dependent affine inequality constraints of the form:
$$A(x)y \leq b(x)$$
where $x$ is the input, $y$ is the output, and there are no restrictions on the number of constraints ($m$) or the rank of $A(x)$.

Core Components

1. Constraint Decomposition:
   To handle an arbitrary number of constraints, the method decomposes the constraint set into smaller subsets (sub-constraints). Inspired by the geometry of polyhedra, the authors observe that a feasible region is defined by minimal faces, which require at most $n_{out}$ linearly independent constraints. The method considers combinations of $k$ constraints (where $1 \leq k \leq \min(m, n_{out})$) to form sub-matrices $A_\gamma$ and sub-vectors $b_\gamma$.

2. Trainable Projection with Null-Space:
   For each sub-constraint $\gamma$, the framework computes a projection $P_\gamma(x)$ using a closed-form expression:
   $$P_\gamma(x) = f_\theta(x) - A_\gamma^\dagger(x)(A_\gamma(x)f_\theta(x) - b_\gamma(x)) + (I - A_\gamma^\dagger(x)A_\gamma(x))w_\phi(x)$$

   • $f_\theta(x)$: Output of an unconstrained neural network.
   • $A_\gamma^\dagger(x)$: Moore-Penrose pseudoinverse of the sub-constraint matrix.
   • $w_\phi(x)$: A trainable null-space component parameterized by a neural network.

   The term $(I - A_\gamma^\dagger A_\gamma)w_\phi(x)$ is crucial. When constraints are linearly dependent (resulting in a non-unique solution space), this term allows the network to learn an optimal point within the null space (e.g., selecting a specific point on a feasible edge or face) rather than relying solely on orthogonal projection.

3. Feasible Candidate Selection:
   The final output $P^*(x)$ is selected from the set of all valid projections $S_P(x)$ (those satisfying $A(x)y \leq b(x)$) that minimize the distance to the unconstrained output $f_\theta(x)$. If the unconstrained output is already feasible, it is used directly.

Key Contributions

1. Closed-Form Hard Constraint Enforcement: CAffNet guarantees hard satisfaction of an arbitrary number of input-dependent affine inequality constraints without requiring the constraint matrix $A(x)$ to have full row rank.
2. Robust Projection via Decomposition: By introducing constraint decomposition and a trainable null-space module, the method handles linearly dependent constraints and non-unique projections, overcoming the limitations of previous projection-based methods like HardNet-Aff.
3. Universal Approximation Property: The authors prove that CAffNet preserves the universal approximation property of the underlying neural networks (both feedforward and transformers). The framework can approximate any continuous function within the constrained feasible set.
4. Scalability (CAffNet-Lite): A reduced variant, CAffNet-Lite, is proposed to mitigate computational complexity by restricting the search space to minimal faces (vertices) rather than all combinations, reducing complexity from exponential to polynomial in the number of constraints.

Experimental Results

The paper validates CAffNet across three scenarios, comparing it against standard NNs with soft constraints and HardNet-Aff:

1. Learning Nonlinear Functions with Piecewise Constraints:

   • CAffNet variants strictly satisfied all constraints, whereas NN and HardNet exhibited violations.
   • CAffNet-TF (Transformer-based) achieved the lowest Mean Squared Error (MSE), reducing it by 73.33% compared to the soft-constrained NN.
   • CAffNet demonstrated faster convergence rates and lower initial loss.

2. Learning Optimization Solvers:

   • In a problem involving inequality and equality constraints, CAffNet variants strictly satisfied all constraints (0% violation), while NN and HardNet failed significantly on equality constraints.
   • CAffNet-FF achieved objective values comparable to the benchmark solver IPOPT but with significantly faster inference times, making it suitable for real-time applications.
   • CAffNet-Lite achieved the same performance as the full version while reducing training and testing time by approximately 45%.

3. Learning Control Policies for Safety-Critical Systems:

   • In a unicycle robot control task with obstacle avoidance, CAffNet-FF successfully navigated to the goal without collisions or constraint violations.
   • Both NN and HardNet collided with obstacles due to constraint violations.
   • Crucially, a posteriori enforcement (applying projection only after training) resulted in the robot getting stuck, demonstrating the necessity of joint optimization between the network parameters and the projection layer.

Significance and Claims

The paper claims that CAffNet addresses a fundamental open challenge in deploying neural networks in safety-critical systems: simultaneously guaranteeing approximation accuracy and provable constraint satisfaction.

• Formal Guarantees: Unlike penalty-based methods, CAffNet provides mathematical guarantees that outputs will satisfy constraints for all inputs, provided the feasible set is non-empty.
• Architectural Versatility: The framework is compatible with standard architectures (feedforward NNs and Transformers) and does not restrict the dimensionality of the constraint space relative to the output.
• Optimality: By jointly training the network and the projection layer, CAffNet finds optimal feasible solutions, whereas post-processing methods may lead to suboptimal performance or infeasibility.

The authors conclude that while the framework provides strong guarantees under stated assumptions, it does not inherently ensure the completeness of the constraints or the accuracy of the system model. Therefore, independent validation and domain-specific safety analysis remain necessary for deployment in real-world safety-critical applications. Future work is directed toward extending the framework to non-affine constraints and validating performance on physical robotic platforms.