🤖 machine learning

Advanced Linear Algebra with Applications - Part I (Numerical linear algebra for PDEs, machine learning, and data assimilation)

These master's-level lecture notes introduce advanced numerical linear algebra by connecting classical algorithms to modern applications in PDEs, machine learning, and data assimilation, emphasizing efficient solutions for large, structured systems through matrix-vector products.

Original authors: Victorita Dolean, Jemima Tabeart

Published 2026-08-24

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Victorita Dolean, Jemima Tabeart

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world, science and engineering rely heavily on solving massive puzzles made of numbers. Whether predicting the weather, designing a bridge, or training an artificial intelligence to recognize a face, these tasks often boil down to finding the solution to a system of equations with millions or even billions of unknowns. For decades, the standard way to solve these puzzles was to break them down into smaller, manageable pieces using direct methods, much like solving a complex algebra problem step-by-step on paper. However, as the problems have grown to encompass the entire atmosphere or the sum of human knowledge on the internet, these traditional step-by-step approaches have become too slow and too memory-hungry to be useful. The numbers involved are simply too vast to write down or manipulate all at once.

This is where a different philosophy takes over: instead of trying to find the exact answer immediately, researchers use iterative methods. These are techniques that start with a rough guess and then repeatedly refine it, getting a little closer to the truth with every pass. The challenge has always been that these guesses can get stuck or move too slowly, especially when the underlying data is messy or the connections between numbers are weak. A new set of lecture notes, prepared for advanced students, brings together the latest thinking on how to make these iterative guesses not just faster, but reliable enough to handle the most difficult problems in physics, network analysis, and machine learning. The work unifies three seemingly different worlds—solving equations for physical laws, analyzing the structure of networks, and training computer models—by showing that they all share the same mathematical DNA.

The authors, Victorita Dolean and Jemima Tabeart, begin by explaining that the difficulty in solving these giant systems often comes from the shape of the data itself. In many real-world scenarios, such as a weather model or a social network, each piece of information is only connected to a few neighbors. This creates a "sparse" structure, where most of the numbers in the giant grid are zero. While this sparsity saves memory, it also creates a specific kind of mathematical landscape where the solution is hidden in a way that makes it hard to find. The notes detail how traditional methods, which work well for smaller, dense problems, fail to scale because they try to fill in all the zeros, destroying the efficiency that sparsity provided.

To overcome this, the text introduces a family of advanced techniques known as Krylov subspace methods. Rather than treating the problem as a static block of numbers to be cracked open, these methods view the solution as a path that can be explored. They build a small, manageable space of possibilities based on the initial guess and the direction of the error, then search for the best answer within that space. The most famous of these is the Conjugate Gradient method, which is shown to be far superior to older techniques for problems involving physical laws like heat flow or fluid dynamics. The authors demonstrate that this method can solve problems in a number of steps that grows much more slowly than the size of the problem, making it possible to handle systems with millions of variables that would have been impossible just a few years ago.

The notes then reveal a surprising connection: the same mathematical tools used to solve equations for physical phenomena are also the engines behind modern machine learning. When a computer learns to recognize patterns, it is essentially solving a massive least-squares problem to fit a model to data. The authors show that the process of training a neural network is mathematically identical to the iterative methods used for solving differential equations. They explain that the speed at which a machine learning model learns is governed by the same properties that determine how fast a weather forecast converges. This insight leads to a powerful realization: techniques developed for physics can be directly applied to improve how artificial intelligence learns, and vice versa. For instance, stopping a learning algorithm early, a common trick in machine learning, is shown to be a form of mathematical filtering that removes noise, a concept that has been understood in physics for decades.

A significant portion of the work is dedicated to the problem of "conditioning," which describes how sensitive a solution is to small errors in the data. In many real-world applications, from the stability of an oil platform to the accuracy of a weather prediction, a small rounding error can lead to a catastrophic failure. The authors explain that some problems are inherently difficult because their structure amplifies these tiny errors. To fix this, they introduce the concept of "preconditioning." This is a technique where the original difficult problem is transformed into a slightly different, easier version that has the same solution but is much more stable to solve. They describe how this can be done by breaking the problem into smaller overlapping pieces, solving each piece independently, and then stitching the results back together. This approach, known as domain decomposition, allows the work to be spread across many computers simultaneously, making it possible to solve problems that are too large for any single machine.

The text also explores how these methods apply to the structure of networks, such as the internet or social media. By treating a network as a giant mathematical object, the authors show how iterative methods can quickly identify communities or clusters within the data. They explain that the same algorithms used to smooth out errors in a physical simulation can be used to find the most important nodes in a network, a technique that was central to the original PageRank algorithm used by search engines. The notes emphasize that while the applications look different on the surface, the underlying mathematics is identical: a sparse matrix representing connections, a spectrum of values that dictate the speed of convergence, and a need for clever shortcuts to avoid getting bogged down.

Throughout the notes, the authors stress that the key to success is not just having a powerful computer, but understanding the geometry of the problem. They show that by looking at the distribution of values within the data, one can predict how fast a solution will be found and choose the right tool for the job. Whether it is a weather model with a billion unknowns, a graph of billions of web pages, or a dataset of millions of images, the principles remain the same. The work serves as a bridge between classical numerical analysis and modern data science, proving that the tools developed to solve the equations of the physical world are exactly what is needed to navigate the complex data landscapes of the twenty-first century.

The authors conclude by providing a unified framework that treats these diverse fields as variations of the same fundamental challenge. They demonstrate that the old distinction between solving equations for physics and optimizing models for machine learning is artificial. In both cases, the goal is to find a solution in a high-dimensional space where the data is sparse and the path to the answer is not obvious. By using iterative methods, preconditioning, and a deep understanding of the spectrum of the data, researchers can now tackle problems that were previously out of reach. The notes do not claim to have solved every problem, but they provide a clear, rigorous, and practical guide to the methods that are currently driving progress in science and technology. The message is clear: the future of computation lies not in brute force, but in smart, adaptive strategies that respect the structure of the data.

Technical Summary: Advanced Linear Algebra with Applications (Part I)

Problem Statement

The paper addresses the computational challenges inherent in solving large-scale linear systems ( $Ax=b$ ) arising from three distinct but structurally similar domains: the discretization of partial differential equations (PDEs), the analysis of graph Laplacians in network science, and the optimization of machine learning models (specifically least squares and regularized regression).

The core problem is that modern applications generate systems that are too large for direct factorization ( $O(n^3)$ complexity) and often too ill-conditioned for standard iterative methods to converge efficiently. These systems share common characteristics: they are sparse, structured, and accessible primarily through matrix-vector products. The paper identifies a gap in traditional pedagogy where these domains are often treated in isolation, despite sharing the same underlying numerical linear algebra principles. The goal is to provide a unified foundation that connects classical algorithms (direct solvers, stationary iterations) with modern applications (spectral clustering, data assimilation, deep learning) to demonstrate that a small set of core ideas—sparsity, spectral properties, Krylov subspaces, and preconditioning—governs all these fields.

Methodology

The text is structured as a foundational course for master's students, progressing from basic numerical analysis to advanced iterative solvers and preconditioning techniques. The methodology is built on four pillars:

Foundations of Numerical Stability and Conditioning:
The work begins by establishing the necessary theoretical framework, including vector and matrix norms, floating-point arithmetic (IEEE 754), and the concept of condition numbers ( $\kappa(A)$ ). It emphasizes the distinction between forward and backward error and the stability of algorithms like Gaussian Elimination with Partial Pivoting (GEPP) and Cholesky factorization for Symmetric Positive Definite (SPD) matrices.
Discretization and Sparsity Patterns:
The paper derives how continuous problems (1D and 2D Poisson equations) and discrete structures (graph Laplacians, machine learning design matrices) translate into sparse linear systems. It details finite difference approximations, the resulting stencil structures (tridiagonal and block-tridiagonal), and the critical role of storage formats (CSR/CSC) and reordering strategies in managing fill-in during factorization.
Iterative Solvers and Krylov Subspaces:
The core methodology shifts from direct methods to iterative approaches.
- Stationary Methods: It analyzes Jacobi, Gauss-Seidel, and Successive Over-Relaxation (SOR), deriving their convergence rates based on the spectral radius of the iteration matrix. It highlights their role as "smoothers" rather than standalone solvers for large systems.
- Krylov Subspace Methods: The text introduces the Conjugate Gradient (CG) method for SPD systems and GMRES/Arnoldi for non-symmetric systems. It establishes that these methods minimize the error or residual over a growing Krylov subspace $\mathcal{K}_k(A, r^{(0)})$ . A key theoretical contribution is the equivalence between the CG recurrence and the Lanczos algorithm, showing how CG implicitly builds a tridiagonal projection of the matrix.
- Non-Symmetric Systems: For non-normal matrices (e.g., convection-diffusion, PageRank), the paper explains why eigenvalues alone are insufficient predictors of convergence, introducing the "field of values" (numerical range) as the governing geometric object.
Preconditioning and Multigrid:
To overcome the slow convergence of Krylov methods on ill-conditioned PDE matrices (where $\kappa(A) \sim O(h^{-2})$ ), the paper details preconditioning strategies. It covers:
- Algebraic Preconditioners: Jacobi and Incomplete Factorizations.
- Domain Decomposition: The Schwarz methods (alternating and additive), analyzing convergence in terms of overlap width.
- Multigrid: The two-level and V-cycle methods, which combine local smoothing (damping high-frequency errors) with coarse-grid correction (eliminating low-frequency errors) to achieve mesh-independent convergence rates.

Key Contributions and Applications

The paper's primary contribution is the synthesis of these methods across three application domains, demonstrating that the same algorithmic machinery applies to all:

PDE Discretization: It shows how the 1D/2D Poisson matrices serve as the canonical testbed for analyzing the spectral properties of iterative methods. It derives explicit convergence factors for Jacobi, GS, and SOR, demonstrating that while SOR with optimal relaxation improves the rate to $O(1/n)$ per iteration, it remains inferior to modern solvers like Krylov methods (e.g., CG) or Multigrid.
Graph Laplacians and Spectral Analysis: The text interprets iterative methods on graphs as diffusion processes. It connects the Jacobi iteration to random walks and demonstrates how the Lanczos algorithm can efficiently compute the Fiedler vector (the eigenvector corresponding to the second smallest eigenvalue) for spectral clustering, effectively identifying community structures in networks.
Machine Learning and Data Assimilation:
- Gradient Descent: It establishes that Richardson iteration is mathematically equivalent to gradient descent on quadratic losses, linking the step size to the spectrum of the Hessian ( $X^TX$ ).
- Spectral Bias and Regularization: The paper interprets early stopping in iterative solvers (like CG) as a form of regularization. It draws a parallel between the "smoothing" property of PDE solvers (damping high-frequency errors) and "spectral bias" in machine learning (learning low-frequency patterns first).
- LSQR and Truncated SVD: For rectangular systems (least squares), it introduces the Golub-Kahan bidiagonalization and the LSQR algorithm, showing that stopping LSQR early acts as a spectral filter similar to truncated SVD, suppressing noise amplification without explicitly computing the SVD.
- PageRank: It formulates PageRank as an eigenproblem for the Google matrix, analyzing the convergence of the power iteration and the role of the damping factor.

Results

The paper provides rigorous theoretical bounds and convergence analyses:

Convergence Rates: It quantifies the iteration counts required for various methods. For a 1D Poisson problem with $n$ unknowns, Jacobi requires $O(n^2)$ iterations, while SOR with optimal relaxation requires $O(n)$ iterations. The text notes that even $O(n)$ is poor compared to modern solvers like Multigrid or Krylov methods, which offer superior scaling (often $O(1)$ or $O(\sqrt{\kappa})$ ) particularly in higher dimensions or with effective preconditioning.
Spectral Filtering: It proves that CG acts as an optimal spectral filter, minimizing the error in the $A$ -norm at each step. It demonstrates that the filter factors of CG approximate the ideal step function of truncated SVD, but are adapted to the specific data vector.
Non-Normality: It provides counterexamples (companion matrices) where eigenvalues are tightly clustered but GMRES stagnates, proving that the field of values is the correct metric for convergence in non-normal cases.
Preconditioning Efficacy: It demonstrates that while simple preconditioners (like Jacobi) fail to improve the condition number of Laplacian systems, domain decomposition and multigrid preconditioners can cluster eigenvalues near 1, rendering the solver mesh-independent.

Significance and Claims

The authors claim that this work serves as a "unified foundation" for advanced numerical linear algebra. The significance lies in the observation that despite the diversity of applications—from weather forecasting (PDEs) to ranking web pages (graphs) to fitting models (ML)—the underlying mathematical structures are identical.

The paper asserts that:

Universality of Core Concepts: Sparsity, the spectrum, Krylov subspaces, and preconditioning are the few essential ideas needed to solve problems across these disparate fields.
Interchangeability of Methods: A method designed for a discretized Laplacian (e.g., Multigrid or CG) is often the correct approach for a graph Laplacian or a design matrix in machine learning.
Educational Necessity: The text argues that understanding the spectral mechanisms (smoothing vs. coarse correction, spectral bias vs. regularization) is more critical than memorizing specific algorithms, as these mechanisms dictate performance in the era of massive datasets and high-resolution simulations.

The paper explicitly positions itself as the first part of a larger course, focusing on foundational algorithms and their theoretical underpinnings, leaving the second part (not included) to take these foundations for granted in more advanced applications. It emphasizes that while direct factorization is the "black box" for small systems, iterative methods and preconditioning are the only viable path for the large-scale, structured systems defining modern scientific computing.