💻 computer science

Tensor-Train Layers for Physics-Informed Neural Networks

This paper demonstrates that integrating Tensor-Train layers into Physics-Informed Neural Networks significantly reduces parameter counts by up to 8.8× while maintaining 97% of the original accuracy, with compression efficiency heavily dependent on the regularity of the underlying PDE solution.

Original authors: Elias Javanmard, Yosef Javanmard, Rezvan Salehi

Published 2026-09-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Elias Javanmard, Yosef Javanmard, Rezvan Salehi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern science, a powerful tool has emerged that allows computers to learn the laws of nature without needing a pre-drawn map. These tools, known as physics-informed neural networks, act as digital surrogates for complex physical systems. Instead of relying on a rigid grid of points to calculate how heat spreads, how fluids flow, or how prices change, these networks are trained to satisfy the fundamental equations of physics directly. They weave the rules of the universe into their very structure, allowing them to find solutions in a mesh-free way that is incredibly flexible. However, this flexibility comes with a heavy price. To capture the sharp, sudden changes that often occur in real-world phenomena—like a shockwave in a fluid or a sudden shift in market value—these networks must be built with massive, wide layers containing millions of adjustable settings. This size makes them difficult to run on smaller computers or to use in situations where speed and memory are limited, creating a bottleneck for their practical use.

Researchers Elias Javanmard, Yosef Javanmard, and Rezvan Salehi have tackled this problem by introducing a method to compress these giant networks without losing their ability to solve difficult problems. They applied a technique called tensor-train decomposition to the internal layers of these networks. Imagine a network as a series of rooms where information passes from one to the next; in a standard network, the door between rooms is a massive, solid wall of data. The researchers replaced these solid walls with a clever, lightweight structure that holds the same information but uses far fewer resources. By breaking down the massive data connections into a chain of smaller, linked components, they were able to shrink the number of settings required by up to nearly nine times. Remarkably, even with this drastic reduction, the compressed networks retained almost all of their original accuracy, solving the same physical puzzles with nearly the same precision as the giant, uncompressed versions.

The team tested this approach on two very different types of physical problems to see how well the compression held up under pressure. The first was a fluid dynamics problem involving a viscous fluid, which is known for developing sharp, sudden spikes in its behavior, similar to a breaking wave. The second was a financial model used to price options, which describes a much smoother, more gradual change. They found that the smooth financial problem was significantly easier to compress than the fluid problem. When the solution to a problem is smooth and predictable, the network can be shrunk down much further while still keeping its accuracy. However, when the solution involves sharp, chaotic features like shockwaves, the network needs to keep more of its original size to capture those details correctly. This distinction revealed a key insight: the quality of the solution itself dictates how much the network can be compressed.

A major hurdle in making this compression work was the way these networks are trained. Because the networks must satisfy physical laws, the training process requires the computer to calculate not just the answer, but also how that answer changes in response to tiny shifts in the input, and even how those changes shift again. This requires a complex, multi-step calculation that usually slows things down significantly when the network is compressed. The researchers discovered a specific bottleneck where the computer was unnecessarily repeating the same heavy calculations over and over. To fix this, they developed a simple caching system that stores the results of these calculations once and reuses them, preventing the computer from wasting time recalculating the same steps. This small adjustment removed a major slowdown, making the compressed networks run almost as fast as the original ones, despite the added complexity of the new structure.

The results of this study offer a clear path forward for using these advanced networks in the real world. By carefully choosing how much to compress the network based on the smoothness of the problem, scientists can create models that are small enough to fit on portable devices or fast enough to run thousands of times in a single second. For smooth problems, the savings are enormous, allowing for highly efficient models that are nearly indistinguishable from the massive originals. For problems with sharp, sudden changes, the savings are still significant, though the models must remain slightly larger to preserve accuracy. This work demonstrates that it is possible to have both high performance and low resource usage, provided the compression strategy is matched to the nature of the physical problem being solved. The researchers have shown that with the right mathematical tools, the heavy burden of these large networks can be lifted, opening the door to wider and more practical applications of scientific machine learning.

Technical Summary: Tensor-Train Layers for Physics-Informed Neural Networks

Problem Statement

Physics-Informed Neural Networks (PINNs) embed partial differential equation (PDE) constraints directly into the loss function, enabling mesh-free solutions for forward and inverse problems. However, their deployment on resource-constrained platforms is hindered by the large parameter counts required in wide hidden layers to resolve sharp solution features (e.g., shocks, boundary layers). While tensor-train (TT) decomposition has been successfully applied to compress weights in standard feed-forward, recurrent, and convolutional networks, its application to PINNs presents unique challenges:

Higher-Order Differentiation: PINN training requires first and second spatial derivatives of the network output via automatic differentiation (autograd). Without optimization, this necessitates repeated contractions of the TT core chain for every derivative calculation, tripling the computational cost per training step compared to standard inference.
Solution Regularity: PINN targets often contain localized, high-curvature features (shocks) that are intrinsically difficult to represent with low-rank approximations, unlike the smoother functions typically found in classification tasks.

Methodology

The authors propose TT-PINN, a framework that replaces the dominant hidden-to-hidden weight matrices in a PINN with tensor-train layers. The methodology addresses the specific computational and optimization challenges of PINNs through three key components:

1. Architecture Design

The network retains dense input and output projections but factorizes the internal hidden-to-hidden layers ( $H \times H$ ) into an $L$ -site tensor-train chain.

Factorization: A weight matrix $W \in \mathbb{R}^{H \times H}$ is decomposed into a chain of cores $A^{[k]}$ with bond dimension $\chi$ .
Configuration: In the experiments, $H=128$ and $L=2$ , with factor dimensions $d_{out}=d_{in}=[8, 16]$ . This isolates compression to the parameter-dominant part of the network while keeping low-dimensional input/output maps dense.

2. Per-Step Weight Caching (Autograd Optimization)

To resolve the computational bottleneck of higher-order autograd, the authors introduce a per-step weight cache.

Mechanism: Before the forward pass, the TT cores are contracted once into a dense matrix $\hat{W}^{(\ell)}$ . This cached matrix is then passed to the three required autograd passes (forward, first derivative, second derivative) for the PDE residual calculation.
Benefit: This reduces the number of core contractions per step from $3L n_h$ to $L n_h$ (a 3x reduction for second-order PDEs) with negligible memory overhead ( $n_h H^2$ floats).
Gradient Computation: Standard PyTorch autograd correctly computes gradients with respect to individual TT cores via the environment-gradient-environment factorization, requiring no custom backward pass implementation.

3. Initialization Strategy

Rather than training TT layers from random initialization, the authors employ TT-SVD initialization:

Train a standard dense PINN to convergence.
Decompose the converged dense weights into TT cores using the TT-SVD algorithm, truncating to the target bond dimension $\chi$ .
Fine-tune the compressed TT-PINN.
This approach is found to be essential for recovering high accuracy at low bond dimensions.

Benchmark Experiments

The method was evaluated on two PDEs with contrasting regularity:

1D Viscous Burgers Equation: A hyperbolic PDE featuring a sharp shock wave ( $\nu = 0.01/\pi$ ).
Black-Scholes Equation: A parabolic PDE for option pricing, characterized by global smoothness but high curvature near the "at-the-money" region.

Both benchmarks utilized identical architectures ( $n_h=4, H=128$ ), optimizers (Adam with cosine decay), and training budgets (5,000 steps) to isolate the effect of solution regularity on compressibility.

Key Results

Accuracy vs. Compression Trade-off

Burgers Equation: At a bond dimension of $\chi=16$ , the TT-PINN achieved 3.1× parameter compression (from 83,201 to 26,753). The model retained 97% of the dense model's performance, with a residual loss ( $L_f$ ) of $7.9 \times 10^{-5}$ compared to $2.7 \times 10^{-5}$ for the dense baseline. While this represents a significant compression, the shock front exhibits slightly more deviation than the dense solution.
Black-Scholes Equation: The smooth solution proved significantly more compressible. At $\chi=16$ , the model achieved a final loss of $5.2 \times 10^{-6}$ , which is within a factor of three of the dense baseline ( $1.8 \times 10^{-6}$ ). This corresponds to a 3.1× parameter compression. While the loss is higher than the dense baseline, the smooth nature of the solution allows for a lower-rank representation with less penalty in PDE residual compared to the shock-dominated Burgers case. Even at $\chi=8$ , the loss remained within an order of magnitude of the dense baseline.

Regularity as a Determinant

The comparison revealed that solution regularity is a key driver of compression quality. The Black-Scholes frontier (smooth solution) lay uniformly below the Burgers frontier (shock-dominated) at fixed $\chi$ , indicating that smoother PDE solutions admit lower-rank representations with less penalty in PDE residual.

Computational Overhead

Wall-Clock Time: The per-step training time showed only a modest improvement (approx. 1.04× speedup) with caching on a single-core CPU. The authors attribute this to the fact that the dominant cost is the batched matrix multiplication over collocation points ( $N_f$ ), not the TT-core contraction itself.
Algorithmic Efficiency: The primary benefit of the caching mechanism is the elimination of redundant computation-graph construction and repeated contractions, which becomes more significant at larger network depths or on hardware with higher kernel launch overheads.

Significance and Claims

The paper positions TT layers as a practical low-rank parameterization for PINNs, specifically addressing the "hidden-to-hidden" parameter bottleneck. The authors claim three primary contributions:

Resolution of the Autograd Bottleneck: The identification and solution of the specific computational cost incurred by higher-order derivatives in TT-PINNs via per-step weight caching.
Initialization Protocol: Demonstrating that TT-SVD initialization from a pre-trained dense network is superior to random initialization for achieving high accuracy at low bond dimensions.
Regularity-Compressibility Link: Establishing that solution regularity (smoothness vs. shocks) is a controllable axis for compression quality, with smooth solutions being significantly more compressible than shock-dominated ones.

The authors conclude that while parameter reduction does not automatically translate to proportional wall-time speedups due to tensor algebra overheads, TT-PINN offers a viable path for deploying scientific machine learning surrogates in many-query settings (e.g., uncertainty quantification, model calibration) where memory footprint and parameter count are critical constraints.