Forward-mode automatic differentiation for the tensor renormalization group and its relation to the impurity method

This paper proposes a forward-mode automatic differentiation framework for tensor renormalization group methods that offers superior accuracy in calculating thermodynamic quantities compared to conventional impurity methods, while establishing a theoretical link between the two approaches and providing practical procedures for extracting critical exponents.

The Gist

Imagine you are trying to understand how a massive, complex machine works by taking it apart, piece by piece, to see how it behaves when you turn a specific dial (like temperature). This is what physicists do when they study materials like magnets or superconductors. They use a mathematical tool called the Tensor Renormalization Group (TRG) to simplify this giant machine into smaller, manageable chunks.

However, there's a catch: they don't just want to know how the machine runs; they want to know how sensitive it is. If they tweak the dial slightly, how much does the machine's energy change? Or how much does its "heat capacity" (how hard it is to heat up) change?

This paper introduces a new, smarter way to calculate these sensitivities. Here is the breakdown using simple analogies:

1. The Problem: The "Backwards" vs. "Forwards" Mess

Traditionally, to figure out how a system reacts to changes, scientists used two main methods:

• The "Finite Difference" Method: This is like trying to guess how steep a hill is by walking a tiny step forward, measuring the height, walking back, and measuring again. It's messy, prone to errors, and you have to guess how small your step should be.
• The "Impurity" Method: This is like inserting a tiny, special "spy" sensor into the machine to measure the reaction. It works well, but it's a bit rigid. It assumes the rest of the machine stays exactly the same, which isn't always true.
• The "Reverse-Mode" (Backpropagation): This is the method used by AI (like the neural networks that power your phone). It works by running the machine forward, then running it backwards to trace where the changes came from.
  • The Problem: In physics simulations, the machine is so deep and complex that running it backwards requires remembering every single step you took. It's like trying to walk up a 100-story building while carrying a backpack full of bricks for every floor you've already climbed. Eventually, you run out of memory (RAM) and crash.

2. The Solution: The "Forward-Mode" Superpower

The author, Yuto Sugimoto, proposes a new approach called Forward-Mode Automatic Differentiation (AD).

Imagine you are driving a car.

• Reverse-Mode (Old Way): You drive to the destination, then you drive all the way back to the start, trying to remember every turn you made to figure out how fast you were going at the beginning. You need a huge notebook to write down every turn.
• Forward-Mode (New Way): As you drive forward, you carry a second, smaller notebook. Every time you turn the steering wheel, you immediately write down how that turn affects your speed right then and there. You don't need to go backwards. You just keep updating your speed as you go.

The Magic:
This new method calculates the "sensitivity" (derivatives) of the system while it is running forward.

• Memory: It only needs a tiny bit more memory (just enough to hold the original data plus the sensitivity data). It doesn't care how deep the simulation goes.
• Speed: It takes a little more time to do the math (about 3 times longer for simple changes, 6 times for complex ones), but it is incredibly fast compared to the memory cost of the old "backwards" method.

3. The "Impurity" Connection

The paper also reveals a beautiful secret: The old "Impurity Method" (the spy sensor) is actually just a simplified, "lazy" version of this new Forward-Mode method.

Think of the Impurity Method as a sketch artist who draws a picture but ignores the shadows. The new Forward-Mode method is the same artist, but they also paint the shadows and the lighting.

• When the author turned off the "shadow" calculations in their new method, it perfectly matched the old Impurity Method.
• The Result: Because the new method includes the "shadows" (the subtle changes in how the machine parts adjust), it is much more accurate. In tests, it was millions of times more accurate than the old method for calculating things like specific heat, without costing much more time.

4. Why This Matters

• Better Physics: Scientists can now calculate how materials behave near critical points (like when ice melts or a magnet loses its magnetism) with extreme precision.
• 3D Possibilities: The old "backwards" method was too heavy to use for 3D simulations (like a cube of atoms). This new "forward" method is light enough to handle 3D, opening the door to studying more complex real-world materials.
• No Black Boxes: Unlike using pre-made AI software (which can be slow or limited), this method builds the "sensitivity calculator" directly into the physics code. It's like building a custom engine instead of buying a generic one.

Summary

The paper presents a new way to do physics math that is like carrying a running tally of changes as you go, rather than trying to remember everything after you finish. It's lighter on memory, faster for complex 3D problems, and significantly more accurate than previous methods, effectively upgrading the "spy sensor" technique to a high-definition camera.

Technical Summary

1. Problem Statement

The Tensor Renormalization Group (TRG) is a powerful numerical method for studying classical and quantum many-body systems, approximating the partition function via coarse-graining and Singular Value Decomposition (SVD). While TRG effectively computes the free energy, calculating physical observables (e.g., internal energy, specific heat) and critical exponents requires derivatives of the partition function with respect to parameters like temperature ($\beta$).

Existing methods for these derivatives face significant limitations:

• Numerical Differentiation: Suffers from instability and accuracy dependence on step size selection.
• Impurity Method: Involves inserting "impurity tensors" into the network. While it yields smoother data than numerical differentiation, it typically relies on projectors optimized for the bulk tensor, neglecting the parameter dependence of the SVD projectors (isometries). This introduces systematic errors, particularly for higher-order derivatives.
• Reverse-mode Automatic Differentiation (AD): While precise, reverse-mode AD (backpropagation) requires storing all intermediate tensors and projectors generated during the forward pass. In TRG, the computational graph depth grows logarithmically with system size ($O(\log V)$), leading to prohibitive memory costs for large volumes or higher-dimensional networks.

2. Methodology

The author proposes a Forward-mode Automatic Differentiation (AD) framework specifically tailored for TRG algorithms, including Higher-order TRG (HOTRG) and Bond-Weighted TRG (BWTRG).

Core Algorithm

• Forward Propagation: Instead of backpropagating gradients, the method propagates derivative information alongside the renormalization flow. At each coarse-graining step $i$, the algorithm computes both the renormalized tensor $T^{(i)}$ and its derivatives $\dot{T}^{(i)}, \ddot{T}^{(i)}, \dots, T^{(i)[k]}$ with respect to the parameter $\beta$.
• Chain Rule Application: The update rule for the derivative tensors is derived by applying the chain rule to the RG map $R$. For a first-order derivative, the update involves terms like $R(\dot{T}, T, E, F) + R(T, \dot{T}, E, F) + \dots$, where $E$ and $F$ are projectors derived from SVD.
• SVD Differentiation: A crucial component is the explicit differentiation of the SVD process. The paper utilizes Lorentzian broadening ($1/x \to x/(x^2+\eta)$) to handle degenerate singular values and avoid divergences in the derivatives of isometries.
• Contraction Tree Optimization: To minimize computational overhead, the authors employ a contraction-tree-based differentiation strategy. Rather than performing independent contractions for every term in the chain rule, they differentiate intermediate tensors along the contraction tree.
  • Cost Scaling: Computing derivatives up to order $k$ increases the matrix multiplication cost by a factor of $(k+1)(k+2)/2$ compared to the free energy calculation alone.
  • Memory Scaling: The memory footprint increases only by a factor of $k+1$, independent of the system size or graph depth. This is a significant advantage over reverse-mode AD.

Theoretical Connection to Impurity Method

The paper establishes a rigorous theoretical link between the proposed forward-mode AD and the conventional impurity method:

• The impurity method is shown to be a limiting case of the forward-mode AD where the derivatives of the SVD projectors ($\dot{E}, \dot{F}$) are set to zero (or $\eta \to \infty$).
• In this limit, the forward-mode update rules reduce exactly to the systematic summation rules used in impurity methods.
• This implies that the impurity method effectively ignores the $\beta$-dependence of the isometric tensors, which is the source of its systematic error.

3. Key Contributions

1. Novel AD Framework: Development of a forward-mode AD scheme for TRG that avoids the memory bottlenecks of reverse-mode AD while maintaining machine-precision accuracy.
2. Theoretical Unification: Proving that the impurity method is a specific approximation of the forward-mode AD (neglecting SVD derivatives), thereby explaining the systematic errors in the former.
3. Efficient Implementation: Derivation of update rules for HOTRG and BWTRG that scale efficiently ($O(D^7)$ bottleneck for HOTRG, $O(D^6)$ for BWTRG) with a manageable constant factor overhead for derivatives.
4. Critical Exponent Extraction: A practical procedure to extract critical exponents (specifically $1/\nu$) by analyzing the volume dependence of the derivative of the Gu-Wen ratio ($\partial X / \partial T$) directly from the renormalized tensors.

4. Numerical Results

The method was tested on the 2D and 3D Ising models using HOTRG and BWTRG.

• Accuracy:

  • Internal Energy & Specific Heat: The forward-mode AD results are significantly more accurate than the impurity method. For the specific heat (second derivative), the impurity method showed relative errors of $O(10^{-1})$ over a wide temperature range, whereas the forward-mode AD (with $\eta = 10^{-20}$) achieved errors $< 10^{-5}$.
  • Convergence: As the regularization parameter $\eta \to \infty$, the AD results converge to the impurity method results, validating the theoretical connection.
  • BWTRG: In the 2D BWTRG case, the forward-mode AD with a small bond dimension ($D=30$) outperformed the multi-impurity method with a much larger bond dimension ($D=128$).

• Computational Cost:

  • The bottleneck contraction cost for derivatives matches the theoretical scaling factor $(k+1)(k+2)/2$.
  • Total runtime is slightly higher than the impurity method due to the overhead of differentiating the SVD (calculating derivatives of squeezers/isometries), but the scaling remains comparable.

• Critical Exponents:

  • By fitting the volume dependence of $\partial X / \partial T$ at the critical temperature, the method extracted $1/\nu$ for the 2D Ising model as $1.000053(8)$ (exact value is 1), demonstrating high precision.
  • For the 3D Ising model, the method yielded $\nu \approx 0.571$, though deviations from the exact value ($\approx 0.63$) were attributed to limited bond dimensions and SVD truncation instabilities in 3D.

5. Significance

• Overcoming Memory Limits: This approach enables high-precision derivative calculations for large system sizes and higher-dimensional tensor networks where reverse-mode AD is infeasible due to memory constraints.
• Systematic Error Reduction: By explicitly including the parameter dependence of SVD projectors, the method eliminates a major source of systematic error inherent in the impurity method, leading to more reliable thermodynamic quantities.
• Versatility: The framework is generalizable to arbitrary tensor networks and optimization-based TRG algorithms (e.g., TNR), offering a robust tool for studying critical phenomena and phase transitions.
• Future Directions: The paper highlights the need for symmetry-blocking techniques to handle degenerate singular values in higher dimensions and suggests extending the method to correlation functions and magnetization.

In summary, Sugimoto presents a mathematically rigorous and computationally efficient forward-mode AD framework that not only surpasses the accuracy of existing impurity methods but also provides a deeper theoretical understanding of their relationship, paving the way for more precise studies of critical phenomena in lattice models.