💻 computer science

Fixed Point Theory Beyond Contractions for Deep Equilibrium Models

This paper advances Deep Equilibrium Models by establishing a fixed point theory that relaxes the restrictive strict-contraction requirement, proving existence and uniqueness under Wardowski $F$ -contractions, demonstrating convergence of Krasnoselskii–Mann iteration for nonexpansive operators, and deriving Lipschitz stability bounds for robust training.

Original authors: Milad tahavor, Tayyebe Haqiri, Reza Memarbashi

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Milad tahavor, Tayyebe Haqiri, Reza Memarbashi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Deep learning, the technology behind modern artificial intelligence, often relies on stacking layer upon layer of mathematical operations to transform raw data into a final answer. Imagine a factory assembly line where a product passes through hundreds of stations, each one slightly altering the item before passing it to the next. For decades, researchers have tried to make these lines deeper and deeper to solve harder problems. However, building a line with thousands of stations is computationally expensive and difficult to manage. A clever alternative, known as a Deep Equilibrium Model, replaces this long assembly line with a single station that repeats its own work over and over until the product settles into a stable state. Instead of counting the number of stations, the system simply waits until the output stops changing, effectively simulating an infinitely deep network with just one set of instructions.

The success of this approach depends entirely on the system being able to find that stable state, or equilibrium, reliably. If the repeating instructions are too chaotic, the product might spin out of control or never settle down. For years, the mathematical rules guaranteeing that a stable state exists were very strict. They required the repeating instructions to be a "contraction," meaning that every time the system ran, the distance between any two possible outcomes had to shrink significantly. This ensured that no matter where you started, the system would always converge to a single, unique answer. While this worked, it forced engineers to design their networks in very specific, restrictive ways that often limited how powerful the models could become.

A team of researchers from Semnan University and Damghan University in Iran has now developed a new mathematical framework that removes these strict limitations. Their work, published in a recent research article, demonstrates that these equilibrium models can be stable and solvable even when the repeating instructions do not shrink the distance between outcomes. They proved that the system can still find a unique, stable answer using a broader class of mathematical rules called F-contractions, which are less demanding than the old strict rules. Furthermore, they showed that even when the instructions do not shrink distances at all—meaning the system is merely "nonexpansive," like a perfect mirror that reflects without distortion—the system can still be coaxed into finding a stable solution. This is achieved not by running the instructions blindly, but by using a specific, damped averaging technique that gently guides the system toward equilibrium, much like a heavy door that swings slowly to a stop rather than slamming shut.

The researchers did not just prove these ideas on paper; they tested them with concrete computer experiments to see how the theory held up in practice. In one test, they created a scenario where the old, strict rules would have predicted failure. They set up a system that rotated data in a circle without shrinking it. When they tried to solve this using the standard method, the system spun in place forever, never finding a solution. However, when they applied the new averaging technique, the system successfully converged to the correct answer, with the error shrinking steadily over time. This confirmed that their new method works even in cases where the traditional approach is mathematically guaranteed to fail.

Interestingly, the team also discovered a nuance in how these models behave in real-world applications. In a second experiment involving a more complex, non-linear system that mimics a typical neural network layer, they found that the standard method actually worked fine, even without the strict shrinking rules. This happened because the specific mathematical function they used naturally dampened the system's behavior, making it behave like a contraction on its own. This finding is valuable because it clarifies exactly when the new, more complex method is necessary: it is essential for systems that lack this natural damping, such as those using certain types of linear or near-linear operations, but it may not always be needed for every single type of network.

Beyond just finding the solution, the researchers also proved that these new models are stable when their internal settings are tweaked. In machine learning, the system must adjust its internal knobs during training to learn from data. The team showed that if the system is built according to their new rules, small changes to these knobs will result in only small, predictable changes to the final answer. This provides a safety guarantee that the model will not behave erratically during the learning process, a crucial requirement for training reliable artificial intelligence.

By expanding the mathematical toolkit available for these models, this work opens the door to designing deeper, more flexible, and more powerful artificial intelligence systems. It allows engineers to use architectural choices that were previously considered too risky because they did not fit the old, strict mathematical molds. The researchers provided a complete set of instructions for how to train these models using their new methods, including a modified way to calculate the necessary adjustments during learning. Their work suggests that the future of deep learning may not require forcing every system into a narrow box of strict constraints, but rather understanding and utilizing the broader, more flexible mathematical properties that allow these complex systems to find their balance.

Technical Summary: Fixed Point Theory Beyond Contractions for Deep Equilibrium Models

1. Problem Statement

Deep Equilibrium Models (DEQs) define the output of a neural network as the fixed point $z^*(x)$ of a single weight-tied layer $f_\theta$ , satisfying $z^* = f_\theta(z^*, x)$ . This approach replaces the explicit stacking of $L$ layers with an implicit equilibrium formulation, enabling infinite-depth networks with constant memory cost.

However, the theoretical guarantees for DEQs—specifically the existence, uniqueness, and computability of the equilibrium $z^*$ —rely almost exclusively on the Banach Contraction Principle. This requires the operator $f_\theta(\cdot, x)$ to be a strict Lipschitz contraction (i.e., Lipschitz constant $k < 1$ ). The authors identify three critical limitations of this assumption:

Architectural Restriction: Many modern stable architectures utilize orthogonal or isometric weight matrices, which render $f_\theta$ merely nonexpansive ( $k=1$ ) rather than strictly contractive. Strict contractions are often avoided to prevent vanishing/exploding behaviors and to maintain expressivity.
Training Constraints: Enforcing a strict contraction constant $k < 1$ via spectral normalization with a hard margin is difficult to maintain under stochastic gradient updates and may bias the model away from useful configurations.
Lack of Stability Analysis: Existing convergence proofs do not provide quantitative bounds on how the equilibrium $z^*$ shifts as parameters $\theta$ are updated when $k=1$ is allowed.

The paper addresses the gap in applying fixed point theory beyond strict contractions to the DEQ setting.

2. Methodology

The authors develop a framework that relaxes the strict contraction requirement by leveraging two distinct classes of operators from fixed point theory:

A. Wardowski F-Contractions

Instead of requiring a global Lipschitz constant $k < 1$ , the authors utilize F-contractions. An operator $T$ is an F-contraction if there exists a function $F: (0, \infty) \to \mathbb{R}$ satisfying specific monotonicity and limit conditions, and a constant $\tau > 0$ , such that:
$\|Tx - Ty\| > 0 \implies \tau + F(\|Tx - Ty\|) \leq F(\|x - y\|)$
This class strictly contains Lipschitz contractions (e.g., by choosing $F(t) = \ln t$ ) but allows for operators that are not globally Lipschitz-contractive.

B. Krasnoselskii–Mann (KM) Iteration for Nonexpansive Operators

For operators that are merely nonexpansive (where the Banach principle fails and standard Picard iteration $z_{n+1} = f(z_n)$ may not converge), the authors propose replacing the forward solver with the Krasnoselskii–Mann iteration:
$z_{n+1} = (1 - \lambda)z_n + \lambda f(z_n)$
where $\lambda \in (0, 1)$ is a step size. This "damped" iteration is designed to converge to a fixed point for the broader class of nonexpansive operators.

C. Metric Subregularity for Stability

To analyze the sensitivity of the equilibrium to parameter changes ( $\theta \to z^*(\theta)$ ) without a contraction constant, the authors introduce Metric Subregularity. This assumption posits that the distance to the fixed point is bounded by the residual:
$\|z - z^*(\theta)\| \leq \mu \|f_\theta(z, x) - z\|$
This condition is weaker than requiring a contraction and allows for stability analysis in the nonexpansive regime.

3. Key Contributions and Theoretical Results

I. Existence and Uniqueness under F-Contractions (Theorem 5.1)

The authors prove that if $f_\theta$ is an F-contraction on a closed set $D \subseteq \mathbb{R}^n$ :

A unique fixed point $z^*$ exists.
Standard Picard iteration converges to $z^*$ .
Convergence Rate: The convergence is guaranteed with a polynomial rate of $O(n^{1-1/k})$ (where $k \in (0,1)$ is from the F-contraction definition), which is generally slower than the geometric rate of strict contractions but sufficient for existence.

II. Convergence for Nonexpansive Operators (Theorem 5.5)

For operators that are merely nonexpansive (e.g., orthogonal layers):

Existence: If $D$ is compact and convex, a fixed point exists (via Brouwer's theorem).
Convergence: The KM iteration converges to a fixed point.
Rate: The authors establish a non-asymptotic convergence rate for the residual $\|f(z_n) - z_n\|$ of $O(1/\sqrt{n})$ .
Algorithmic Implication: This provides a computable forward solver for architectures where strict contraction cannot be guaranteed.

III. Stability and Data Dependence (Theorem 5.10)

Under the assumption that $f_\theta$ is nonexpansive and satisfies metric subregularity:

The equilibrium map $\theta \mapsto z^*(\theta)$ is Lipschitz stable.
Specifically, $\|z^*(\theta_1) - z^*(\theta_2)\| \leq \mu L \|\theta_1 - \theta_2\|$ , where $L$ is the Lipschitz constant of the operator with respect to parameters.
This provides a first-principles robustness bound for DEQ training in the non-contractive regime.

IV. KM-Compatible Training Algorithm (Section 6)

The authors propose a modified training algorithm:

Forward Pass: Uses KM iteration instead of Broyden's method or standard Picard iteration.
Backward Pass: Solves the adjoint equation $v = \lambda (\frac{\partial f}{\partial z})^\top v + (1-\lambda)v + \frac{\partial L}{\partial z^*}$ using the same damped iteration scheme. This avoids the need to invert $(I - \frac{\partial f}{\partial z})$ , which may be singular when the spectral radius is 1.

4. Experimental Results

The authors validate the theory using synthetic numerical experiments (NumPy, double precision):

Affine Counterexample (E1):
- Setup: A 20D affine map $f(z) = Rz + c$ where $R$ is a block-diagonal rotation matrix (orthogonal, non-contractive).
- Result: Standard Picard iteration failed to converge (residual remained constant). KM iteration converged with a residual decreasing to $10^{-3}$ , strictly adhering to the theoretical $O(1/\sqrt{n})$ bound.
- Significance: Confirms that KM iteration is necessary for isometric operators.
Nonlinear DEQ Layer (E1'):
- Setup: A layer $f(z) = \tanh(Wz + Ux)$ with $W$ orthogonal (non-contractive) vs. $W' = 0.9W$ (strictly contractive).
- Result: Surprisingly, Picard iteration converged in both cases (mean 52.6 iterations for orthogonal $W$ ), outperforming KM iteration.
- Insight: The saturating nonlinearity ( $\tanh$ ) provided local contraction near the equilibrium, making the strict contraction assumption of the linear part unnecessary in this specific nonlinear regime. The authors note this as an "honest negative finding," clarifying that the KM relaxation is "load-bearing" primarily for near-linear regimes or non-saturating activations (e.g., ReLU).
Stability Check (E3):
- Setup: Perturbed parameters $U$ in the orthogonal- $W$ $\tanh$ layer and measured the ratio of output change to parameter change.
- Result: The ratio remained bounded (mean 0.716) across perturbation magnitudes, consistent with the Lipschitz stability predicted by Theorem 5.10.

5. Significance and Claims

The paper claims to provide the first fixed-point theoretic framework for DEQs that operates beyond strict contractions. Its significance lies in:

Theoretical Expansion: It extends DEQ theory to include orthogonal and isometric architectures, which are increasingly popular for stability but were previously excluded by the Banach contraction requirement.
Practical Solver: It offers a concrete, provably convergent alternative (KM iteration) for forward and backward passes in non-contractive regimes.
Robustness: It establishes a stability bound for training dynamics in the absence of a contraction constant, relying instead on metric subregularity.
Modesty: The authors explicitly acknowledge that while the theory guarantees convergence for nonexpansive operators, practical nonlinear layers (like $\tanh$ ) may still converge via standard methods due to local contraction properties. The framework is presented as a necessary safety net for architectures where such local contraction cannot be guaranteed (e.g., near-linear regimes or non-saturating activations).

The authors note that future work includes extending these guarantees to spiking neural networks (time-dimension fixed points) and applying the KM-based solver to large-scale image classification benchmarks (CIFAR-10/ImageNet) to compare accuracy and efficiency against contractive baselines.