🤖 machine learning

Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models

This paper proposes Generalized Fisher-Weighted SVD (GFWSVD), a scalable post-training compression method for large language models that utilizes a Kronecker-factored approximation of the full Fisher information matrix to capture parameter correlations and significantly outperform existing diagonal-based compression techniques.

Original authors: Viktoriia Chekalina, Daniil Moskovskiy, Tatiana Matveeva, Andrey Kuznetsov, Evgeny Frolov

Published 2026-07-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Viktoriia Chekalina, Daniil Moskovskiy, Tatiana Matveeva, Andrey Kuznetsov, Evgeny Frolov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to shrink a massive, intricate library down to fit inside a backpack without losing the stories inside. This is the daily challenge for scientists working in the field of artificial intelligence, specifically with "neural networks"—computer programs designed to learn like brains. These programs are built from millions of tiny switches called "parameters." To make these programs run faster and on smaller devices, researchers try to cut out the switches that aren't doing much work. But here's the tricky part: the switches don't work alone. They are like a complex dance troupe; if you pull one dancer out, the whole routine might collapse because that dancer was holding hands with someone else.

For years, the standard way to decide which dancers to cut was to look at each one individually, ignoring the hand-holding. It was like checking if a dancer was tired without noticing they were supporting a partner. This method was fast, but it often ruined the performance. The paper you are about to read tackles this problem by introducing a new way to see the "dance" of the parameters. It uses a mathematical tool called the "Fisher Information Matrix," which acts like a map showing how every switch is connected to every other switch. The goal is to use this map to shrink the library (the AI model) while keeping the stories (the intelligence) perfectly intact.

The Big Idea: Seeing the Whole Dance, Not Just the Dancers

The authors of this paper, Viktoriia Chekalina and her team, realized that the old maps were too blurry. They wanted a map that showed not just which dancers were important, but how they were linked together. To do this, they invented a new algorithm called Matrix-free Fisher Factorization (MFF).

Think of the Fisher Information Matrix as a giant, dense fog covering the entire dance floor. In the past, trying to see through this fog to find the connections was impossible because the fog was too thick and the dance floor too huge. The old methods just guessed the connections were simple (like a straight line), which missed the complex curves of the real dance.

The team's new trick, MFF, is like having a special pair of glasses that lets you see the fog's structure without ever having to clear the whole fog away. Instead of trying to write down every single connection (which would take too much memory), their algorithm calculates the connections on the fly, focusing only on the specific "layers" of the dance. It's a "matrix-free" approach, meaning it never actually builds the giant, heavy map; it just uses the map's shape to guide the cuts.

The Solution: A New Way to Shrink the Model

Using this new way of seeing connections, the team developed a method called GFWSVD (Generalized Fisher-Weighted SVD). If you imagine the AI model as a block of clay, standard methods might just slice off the edges. GFWSVD, however, understands the internal grain of the clay. It knows that some parts of the clay are tightly woven together and must be cut in a specific way to keep the shape.

The paper proves that under certain mathematical conditions (specifically, if the connections follow a pattern called a "Matrix-Variate Normal distribution"), their method is the unique, optimal way to shrink the model. It's not just a guess; it's the mathematically perfect way to minimize the damage to the model's performance when you remove parameters.

What They Found: Shaving Off Half the Model

The team tested their new method on some of the most famous AI models out there, including Llama 2 and Llama 3.1, which are massive language models used for everything from writing code to chatting. They also tested it on BERT, a model used for understanding text.

Here is what they discovered:

The Compression Power: They were able to shrink these giant models by up to 50%. That means cutting the number of parameters in half.
The Performance: Even with half the size, the models performed just as well as, or sometimes even better than, the original versions. In many tests, GFWSVD beat the current best methods (like diagonal approximations and activation-based methods) across the board.
Avoiding the Collapse: When they tried to compress the models by 40%, standard methods started to fail, causing the AI to lose its ability to reason or answer questions correctly. GFWSVD, however, stayed robust and reliable.
The Speed: Because the models are smaller, they run faster. On a powerful computer chip (an NVIDIA A100), the compressed models processed text 1.34 times faster than the original, uncompressed models.

Why This Matters

The authors showed that by paying attention to the hidden connections between parameters (the off-diagonal elements), you can shrink AI models much more aggressively without breaking them. They proved that ignoring these connections, as most other methods do, leaves a lot of potential performance on the table.

They also showed that this method works as a great "starter" for other training processes. If you use GFWSVD to shrink a model first, and then let the model learn a little bit more (fine-tune), it holds onto its accuracy much better than if you used standard shrinking methods.

In short, this paper provides a new, mathematically sound "scissors" for cutting down giant AI models. It allows us to keep the intelligence while throwing away the bulk, making powerful AI accessible on smaller devices and cheaper to run, all without losing the magic of the original model.

Technical Summary: Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity

1. Problem Statement

The Fisher Information Matrix (FIM) offers a principled geometric framework for understanding parameter sensitivity in neural networks, particularly for tasks like post-training compression. However, the full FIM is computationally infeasible to compute or store for high-dimensional deep learning models due to its quadratic scaling ( $O(d^2)$ ). Consequently, existing methods typically rely on diagonal approximations (e.g., Frankle & Carbin, 2019; Wu et al., 2024), which assume parameter independence. This assumption discards critical off-diagonal correlation structures between model parameters, leading to suboptimal compression and potential accuracy degradation, especially in dense architectures like Large Language Models (LLMs).

2. Methodology

Matrix-free Fisher Factorization (MFF)

The authors propose Matrix-free Fisher Factorization (MFF), a GPU-tractable algorithm designed to approximate the FIM without materializing the full matrix.

Kronecker Structure: MFF approximates the empirical FIM ( $I_F$ ) as a Kronecker product of two smaller matrices, $A$ and $B$ , such that $I_F \approx A \otimes B$ . This structure captures both row-wise and column-wise dependencies within a linear layer's weight matrix.
Matrix-Free Computation: To avoid the $O(m^2n^2)$ complexity of constructing the permuted Fisher matrix $\tilde{I}_F$ , MFF utilizes the Lanczos method for truncated Singular Value Decomposition (SVD). It relies solely on matrix-vector multiplication (matvec) operations.
Efficient Matvec: By leveraging the property $(K \otimes L)\text{vec}(C) = \text{vec}(K^\top CL)$ , the algorithm computes the action of the permuted Fisher matrix on a vector using a sequence of compact matrix multiplications involving gradient matrices ( $G_i$ ) from mini-batches. This reduces the time complexity to $O(mn^2 + m^2n)$ , making it feasible for large transformer layers.

Generalized Fisher-Weighted SVD (GFWSVD)

Building on MFF, the authors derive GFWSVD, a closed-form low-rank layer decomposition method for post-training compression.

Theoretical Foundation: Under the assumption that layer weights follow a Matrix-Variate Normal (MVN) distribution and the FIM admits a Kronecker factorization, the authors prove that minimizing the expected second-order loss increase (parameter sensitivity) yields a unique optimal decomposition.
Algorithm: The optimal rank- $r$ approximation of a weight matrix $W^*$ is given by $W_r = L_B^{-\top} \hat{W}_r L_A^{-1}$ , where $\hat{W}_r$ is the rank- $r$ truncated SVD of an auxiliary matrix $\hat{W} = L_B^\top W^* L_A$ . Here, $L_A$ and $L_B$ are Cholesky factors of the Kronecker components $A$ and $B$ .
Generalization: This approach generalizes previous methods:
- Standard SVD corresponds to identity transformations.
- Fisher-Weighted SVD (FWSVD) corresponds to diagonal transformations on one side.
- GFWSVD employs full, non-diagonal transformations on both sides, capturing richer correlation structures.

3. Key Contributions

Algorithmic Primitive: The derivation of MFF, a scalable, GPU-tractable algorithm for Kronecker-structured FIM approximation that avoids explicit matrix storage.
Theoretical Optimality: A proof that under MVN and Kronecker-structured FIM assumptions, GFWSVD provides the unique optimal low-rank compression minimizing expected second-order loss increase.
Empirical Performance: The proposal of GFWSVD as a compression method that consistently outperforms diagonal Fisher approximations (FWSVD) and activation-based baselines (ASVD, SVD-LLM) across various compression regimes.

4. Experimental Results

The authors evaluated their method on synthetic datasets, encoder-only models (BERT), and decoder-only LLMs (Llama 2 7B, Llama 3.1 8B).

Synthetic Benchmarks: In controlled settings with explicit correlation structures (isotropic, anisotropic, and Toeplitz), GFWSVD demonstrated substantial advantages over diagonal methods as correlations increased, confirming the importance of capturing off-diagonal dependencies.
BERT Compression: On the GLUE benchmark, GFWSVD achieved state-of-the-art results, particularly at lower ranks (higher compression). For instance, at a 40% compression ratio, GFWSVD maintained a macro-averaged GLUE score of 0.51, significantly outperforming SVD (0.37) and FWSVD (0.38).
LLM Compression:
- Llama 2 7B & Llama 3.1 8B: GFWSVD substantially outperformed independent layer decomposition methods (FWSVD, ASVD) and cross-layer methods (Basis Sharing) at compression ratios up to 50%.
- Robustness: At 40% parameter reduction, GFWSVD reliably avoided the performance collapse observed in dense architectures where standard methods degrade.
- Fine-Tuning Initialization: When used to initialize optimization pipelines like Dobi-SVD, GFWSVD preserved accuracy better than standard SVD initialization, showing only moderate degradation at 40% compression compared to substantial drops in baselines.
Efficiency: The runtime for computing Kronecker factors for a single layer in Llama 3.1 8B was approximately 249 seconds on a GPU, demonstrating tractability even for large models. Inference throughput speedups of up to 1.34 $\times$ were observed at 50% compression.

5. Significance and Claims

The paper positions MFF and GFWSVD as foundational algorithmic primitives for scalable, second-order-aware neural network approximation.

Theoretical Advancement: The work establishes a direct link between Fisher geometry and optimal low-rank approximation, generalizing prior diagonal Fisher-based methods to a theoretically grounded, full-matrix sensitivity framework.
Practical Impact: By avoiding the explicit construction of the full FIM, the method enables the leveraging of second-order information in large neural networks, offering a practical pathway for efficient model compression without the prohibitive memory costs of full curvature estimation.
Scope: The authors emphasize that while their evaluation focuses on post-training compression, the underlying formulation targets parameter sensitivity via Fisher-weighted quadratic loss, suggesting applicability to other sensitivity-driven tasks. They note that the method assumes a local optimum and MVN distribution, which are analytical conveniences that hold well empirically but may not perfectly reflect the complex dynamics of stochastic training.

The authors declare no financial conflicts of interest and note that all evaluated models (BERT, Llama 2, Llama 3.1) are publicly available.