💻 computer science

Local Second-Order Adjoint Dynamics for Implicit Neural Networks

This paper introduces Causal Adjoint Transport (CAT), a local second-order adjoint dynamics method that significantly reduces the computational cost of training implicit and recurrent neural networks near stability boundaries by requiring substantially fewer Jacobian actions compared to first-order relaxation and other solvers.

Original authors: Dino Vlahek, Dijana Oreški, Matija Novak, Darko Andročec

Published 2026-09-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dino Vlahek, Dijana Oreški, Matija Novak, Darko Andročec

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, computers learn by adjusting their internal settings to minimize mistakes. For standard networks that process information in a straight line, this learning process is like a well-rehearsed relay race: a signal travels forward to make a prediction, and then a correction signal travels backward, passing from one runner to the next in a precise, ordered chain. This method, known as backpropagation, is efficient and reliable. However, a different class of neural networks, called implicit networks, does not follow a straight line. Instead, these systems settle into a state of balance, or equilibrium, where the output is determined by a complex web of interactions that loop back on themselves. To learn from these systems, the computer must solve a difficult mathematical puzzle to figure out how to adjust the settings. As these networks become more complex and their internal loops approach a point of instability, the standard method of sending that correction signal backward becomes painfully slow, requiring thousands of tiny steps to reach a solution.

Researchers at the University of Zagreb have developed a new way to speed up this backward learning process for these looping networks. They introduced a method called Causal Adjoint Transport, which adds a small amount of "momentum" to the correction signal. Imagine a runner who, instead of just reacting to the person ahead, also remembers their own previous step to maintain a smoother, more direct path. By keeping track of this extra bit of history, the new method allows the correction signal to travel much faster through the network's loops. In their experiments, the researchers found that this approach could reduce the number of steps needed to solve the learning puzzle by up to ten times compared to the standard method, especially when the network was operating near the edge of stability.

The study focused on a specific challenge: how to efficiently calculate the "credit" for a mistake in a system where the parts are constantly influencing each other. In a standard feed-forward network, the path of influence is fixed and finite, so the correction signal simply follows the path backward. But in an implicit network, the signal must be found by solving a system of equations that describes the network's equilibrium state. The researchers tested their new two-state method against the traditional first-order approach, which only looks at the immediate neighbor. They discovered that while the new method offered little advantage for simple, straight-line networks, it provided a massive boost for the looping, implicit ones. When the network's internal connections were strong and the system was close to becoming unstable, the traditional method slowed down significantly, whereas the new method maintained its speed.

To verify their findings, the team ran extensive tests on various datasets, including image recognition tasks and synthetic data. They measured how many times the computer had to perform a specific calculation, known as a Jacobian action, to reach the correct answer. In the most difficult scenarios, where the network was nearly critical, the new method required up to 8.83 times fewer calculations than the best-tuned version of the old method. Even when the network's internal weights were allowed to change during training, the new method consistently used fewer steps, reducing the workload by a median factor of 2.13 times. Crucially, the researchers confirmed that this speedup did not come at the cost of accuracy; the final learning outcomes were identical to those achieved by the slower method, proving that the new approach simply found the same answer more efficiently.

The researchers also explored what happens when the network's behavior becomes more complex, involving patterns that do not fit simple real-number ranges. They found that the standard way of setting the method's parameters could fail in these cases, causing the system to become unstable. However, by adjusting the calibration to account for these complex patterns—using a technique they described as an elliptic spectral enclosure—they were able to restore stability and convergence. This demonstrated that the core idea of using a two-state memory was robust, provided the settings were tuned correctly for the specific shape of the network's behavior.

This work highlights a fundamental difference between how we teach straight-line networks and how we teach looping ones. For the latter, the difficulty of learning is directly tied to how close the system is to a tipping point. The study shows that by adding a second state to the learning process, we can navigate these difficult regions much more effectively. The results suggest that for implicit neural networks, which are increasingly used for modeling complex physical systems and long-term dependencies, this second-order approach offers a practical and significant reduction in the computational cost of learning. The findings are not just theoretical; they were measured across dozens of training runs and multiple datasets, showing a consistent and predictable relationship between the network's stability and the speed of the learning process.

The researchers also compared their method to other advanced mathematical solvers used in engineering and physics. While some of these global solvers could solve the problem with even fewer steps, they required the computer to store large amounts of history and perform complex calculations that involve the entire system at once. The new method, by contrast, works locally, using only the information available to immediate neighbors. This makes it particularly suitable for distributed systems where information cannot be easily gathered from the whole network. The study concludes that while the method offers little benefit for simple, linear networks, it becomes an essential tool for implicit networks as they approach the limits of their stability, turning a potentially slow and expensive process into a manageable one.

Technical Summary: Local Second-Order Adjoint Dynamics for Implicit Neural Networks

Problem Statement

Implicit and recurrent neural networks (RNNs) define hidden states via fixed-point equations rather than explicit layer sequences. Computing exact gradients for these models requires solving a linear adjoint system involving the transpose of the recurrent Jacobian ( $J^\top$ ). Standard first-order local relaxation methods, such as Activation Relaxation (AR), become inefficient as the recurrent dynamics approach the stability boundary (criticality). In this regime, slowly decaying modes necessitate a large number of iterative steps to converge to the exact gradient, increasing the computational cost of "local credit assignment" (backward information propagation via neighboring interactions).

While general-purpose solvers (e.g., GMRES, Broyden) can reduce iteration counts, they often rely on global algebraic operations, growing solver states, or stored history, which violate the constraints of a purely local dynamical system where state variables communicate only through neighboring couplings. The paper seeks a method that accelerates convergence within a strictly local computation model—using only neighboring Jacobian actions and a fixed, small amount of persistent local state—without requiring global synchronization or external scheduling signals.

Methodology

The authors propose Causal Adjoint Transport (CAT), a two-state local adjoint dynamics designed to solve the implicit backward problem.

The CAT Dynamics

Unlike AR, which maintains a single adjoint state and performs a first-order update, CAT maintains an additional inertial state. For a frozen implicit system, the update rule is a second-order stationary iteration:
$\lambda_{t+1} = \lambda_t + \beta(\lambda_t - \lambda_{t-1}) + \alpha [b - (I - J^\top)\lambda_t]$
where $\lambda$ is the adjoint state, $b$ is the loss derivative, and $\alpha, \beta$ are fixed coefficients determined by spectral calibration.

Key Distinctions

Local Autonomy: Unlike Chebyshev semi-iteration, which achieves similar spectral acceleration but requires iteration-dependent coefficients (necessitating a global synchronization signal or shared iteration index), CAT uses a single, time-homogeneous pair of coefficients ( $\alpha, \beta$ ) after initial calibration. This allows each unit in a distributed system to update autonomously based solely on its current and previous local state.
Spectral Calibration: The method is calibrated using the spectral properties of the operator $A = I - J^\top$ $A = I - J^{⊤}$ .
- For real spectra, coefficients are derived from the spectral interval $[\mu, L]$ .
- For complex spectra (directed operators), the paper introduces elliptic calibration, fitting an axis-aligned ellipse to the spectrum to restore stability where real-interval tuning fails near criticality.
Scope: The study focuses on exact implicit gradients, distinguishing itself from approximate methods like Jacobian-Free Backpropagation or phantom gradients.

Key Contributions

Formulation of Local Second-Order Dynamics: The authors formulate a common local adjoint dynamics for both feed-forward and implicit networks, identifying the frozen implicit recurrence with the classical linear second-degree stationary family (Manteuffel).
Architecture-Dependent Utility: The study demonstrates that the second state provides negligible benefit for feed-forward networks (where the adjoint is a finite substitution) but offers significant acceleration for implicit networks as the recurrent Jacobian approaches criticality.
Quantitative Spectral Prediction: The paper establishes that the measured reduction in backward actions during learning closely follows asymptotic spectral predictions. Across eight fixed-recurrent datasets, the measured-to-theoretical action ratio is consistent (median factor $\approx 0.82$ ), and the Jacobian radius is perfectly rank-correlated ( $\rho_s = 1.000$ ) with the action reduction.
Robustness to Complex Spectra: The authors show that while real-interval tuning becomes unstable for directed operators with complex spectra, elliptic calibration restores convergence without altering the underlying two-state recurrence structure.

Results

Spectral Benchmarks

Criticality: As the recurrent spectral radius ( $\rho$ ) approaches 1.0, CAT significantly outperforms first-order AR. At $\rho = 0.99$ , CAT requires up to 8.83× fewer neighboring Jacobian actions than oracle-tuned AR (using a conservative bound) and 10.13× fewer when both use oracle intervals.
Comparison with Solvers: While GMRES and tuned Anderson acceleration can achieve lower action counts, they utilize global operations or larger solver states. CAT achieves a similar spectral scale to Chebyshev semi-iteration but with a simpler, time-homogeneous update rule.

Learning Experiments

Fixed Recurrent Weights: Across 24 dataset-seed pairs on fixed-recurrent datasets, CAT consistently used fewer local actions than AR (median reduction 2.67×). Test accuracy remained identical, confirming that both solvers reached the same exact implicit gradient target.
Trainable Recurrent Weights: In experiments where recurrent edges were trained (60 pairs), CAT maintained a median 2.13× reduction in local actions. The performance gain persisted even when the recurrent operator changed materially during training.
Complex Spectra: For directed operators, real-interval coefficients failed to converge near criticality ( $\rho=0.99$ ). Elliptic calibration restored convergence in all trials, with median action counts comparable to direct discrete spectrum optimization.

Feed-Forward Networks

In feed-forward networks, the second state provided no practical advantage over matched first-order relaxation. The results align with the theoretical expectation that first-order local relaxation is sufficient to propagate the exact finite substitution through the graph, leaving little room for second-order acceleration.

Significance and Claims

The paper claims that recurrent criticality serves as a reliable empirical indicator of the local backward cost for exact implicit credit assignment. The primary significance of the work lies in demonstrating that a second-order local adjoint dynamics can substantially reduce the computational cost of solving implicit neural networks without violating the locality constraints of the system.

The authors emphasize that CAT is most relevant when:

The backward computation is an iterative implicit solve (not a finite substitution).
The system operates under a local computation model where global aggregation or external scheduling signals are undesirable.
The recurrent spectrum approaches the stability boundary.

The study concludes that while general solvers may offer further reductions via global algebra, CAT provides a robust, autonomous, and theoretically grounded solution for local credit assignment in implicit and recurrent architectures, with performance predictable by classical spectral theory even during training.