🤖 machine learning

LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Models

The paper proposes LASER, a low-rank compression framework for vision-language models that utilizes loss-aware singular-value decomposition guided by curvature information and a cross-layer rank allocation strategy to achieve significant decoding speedups while maintaining accuracy under low-precision inference.

Original authors: Haiyu Wang, Yutong Wang, Leshu Li, Yihui Ren, Sai Qian Zhang

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Haiyu Wang, Yutong Wang, Leshu Li, Yihui Ren, Sai Qian Zhang

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Giant Brain" That Won't Fit in Your Pocket

Imagine a Vision-Language Model (VLM) as a super-smart, giant brain that can look at a picture and answer questions about it. It's incredibly powerful, but it's also huge. It's like trying to carry a library of encyclopedias in your backpack. Because it's so big, it takes a lot of energy to run, needs a massive hard drive to store, and moves very slowly on regular phones or laptops.

Scientists have tried to shrink these models before using two main tricks:

Pruning: Cutting out parts of the brain that seem useless.
Quantization: Translating the brain's thoughts into a simpler, shorter language (like using abbreviations).

But there's a third, promising trick called Low-Rank Decomposition. Imagine taking a complex painting and realizing it's actually just a few simple brushstrokes layered on top of each other. If you can find those few strokes, you can recreate the painting without storing every single pixel. This is what "Low-Rank" does: it simplifies the math inside the model.

The Problem with Old Methods: Previous attempts at this were like trying to shrink a painting by just guessing which brushstrokes to keep. They often focused on making the picture look the same (reconstruction) rather than making sure the brain still thinks correctly. They also treated every part of the brain the same, even though some parts are more important than others.

The Solution: LASER (Loss-Aware Singular-value dEcomposition and Rank allocation)

The authors of this paper created LASER, a new method to shrink these giant brains without losing their smarts. Think of LASER as a smart, loss-aware sculptor.

Here is how LASER works in three simple steps:

1. The "Curvature" Compass (Loss-Aware SVD)

Old methods asked: "If I remove this part, does the picture look the same?"
LASER asks: "If I remove this part, does the model get the wrong answer?"

Imagine you are editing a movie. A standard editor might cut a scene just because it looks similar to the next one. LASER is like a director who knows exactly which scenes are crucial for the plot. It uses a mathematical "compass" (called K-FAC) that measures how much the model's confidence or accuracy would drop if a specific part were changed. It doesn't just look at the weight of the numbers; it looks at how those numbers affect the final result. This ensures that when they shrink the model, the "brain" stays smart.

2. The "Fair Share" Budget (Cross-Layer Rank Allocation)

Imagine you have a budget of 100 dollars to fix a house with 10 rooms.

Old Method: Give every room exactly $10.
LASER Method: Walk through the house first. The kitchen is falling apart (very sensitive), so it gets $40. The closet is fine (not sensitive), so it gets $5. The rest is split among the other rooms based on need.

LASER realizes that not all parts of the AI are equally important. Some layers are "fragile" and need to keep more of their original complexity, while others can be shrunk down significantly. It uses a special "calibration" process to figure out exactly how much "space" (rank) to give to each layer so the whole model stays balanced and accurate.

3. The "Hybrid" FFN (The Heavy Lifter)

Inside these AI brains, there are two main types of workers:

The Attention Team: They look at the image and text to figure out what to focus on.
The FFN Team (Feed-Forward Network): They do the heavy lifting of processing and reasoning. This team makes up a huge chunk of the model's size.

Old methods mostly shrank the "Attention Team" and ignored the "FFN Team." LASER goes into the FFN team and says, "We can't shrink everyone equally, or the team will fail."
It uses a hybrid strategy:

It identifies the specific workers (channels) that are good at being simplified and shrinks them using the "brushstroke" technique (SVD).
It leaves the stubborn, difficult workers alone (keeping them dense).
Then, it translates the whole team into a simpler language (quantization) to save even more space.

The Result: Fast, Small, and Smart

The paper claims that by using this smart sculpting approach:

Speed: The model runs 2.3 times faster than previous top methods and 4.7 times faster than standard high-precision methods.
Accuracy: Even though the model is shrunk and simplified, it still answers questions correctly almost as well as the giant, unshrunk version.
Efficiency: It saves a massive amount of memory, making it possible to run these powerful AI brains on devices that usually couldn't handle them.

In short: LASER is a smart way to shrink a giant AI brain. Instead of blindly cutting pieces off, it carefully measures which pieces are essential, gives more space to the important parts, and simplifies the rest, resulting in a model that is fast, small, and still very smart.

Technical Summary: LASER (Loss-Aware Singular-value Decomposition and Rank Allocation)

Problem Statement

Vision-Language Models (VLMs) have demonstrated strong multimodal reasoning capabilities but face significant deployment challenges due to their large computational costs, high parameter counts, and substantial memory bandwidth requirements during autoregressive decoding. While low-rank decomposition has emerged as a promising compression technique, existing methods suffer from three primary limitations:

Local Optimization: Most methods optimize for local matrix reconstruction error or layer-wise output error, ignoring the end-to-end impact on model loss.
Inefficient Rank Allocation: Current approaches often rely on uniform or heuristic rank allocation, failing to account for the varying sensitivity of different layers and attention heads to compression.
Limited Scope: Existing efforts focus primarily on attention projections (Q, K, V), leaving Feed-Forward Networks (FFNs)—which constitute a large fraction of model parameters and storage—largely underexplored for low-rank compression.

Methodology

The paper proposes LASER (Loss-Aware Singular-value dEcomposition and Rank allocation), a framework designed to align low-rank compression with downstream VLM performance rather than mere reconstruction fidelity. The methodology consists of four core components:

1. Loss-Aware Singular Value Decomposition (SVD)

Instead of minimizing the Frobenius norm of weight reconstruction error, LASER derives a curvature-weighted SVD objective from a second-order approximation of the model loss.

K-FAC Surrogate: Utilizing Kronecker-factored Approximate Curvature (K-FAC), the method approximates the loss increase caused by weight perturbation $\Delta W$ as $\Delta L \approx \frac{1}{2}\text{vec}(\Delta W)^\top (G \otimes X)\text{vec}(\Delta W)$ , where $X$ and $G$ are the activation and gradient covariance matrices estimated on a calibration set.
Transformation: The weight matrix $W$ is transformed into a curvature-weighted space $\tilde{W} = S_x W S_g$ (where $S_x$ and $S_g$ are Cholesky factors of $X$ and $G$ ). SVD is performed on $\tilde{W}$ , and the resulting low-rank factors are mapped back to the original space. This ensures that the truncated components correspond to the smallest estimated increase in model loss.
Quantization-Aware Whitening (QAW): To accommodate low-precision inference, the method estimates K-FAC factors using quantized calibration activations, aligning the whitening objective with the deployed low-precision layer.

2. Loss-Aware Cross-Layer Rank Allocation

The paper identifies that singular values derived from K-FAC are not directly comparable across different layers or attention heads due to varying scaling factors in the curvature estimation.

Empirical-Fisher Importance Score: To enable global rank allocation, LASER introduces an importance score based on calibration gradients. For each singular component, the score measures its contribution to the empirical-Fisher quadratic loss approximation, combining the singular value magnitude with its alignment to the calibration gradients in the original parameter space.
Global Optimization: Ranks are allocated by greedily selecting singular components with the highest importance scores per parameter cost until a target parameter budget is reached, allowing for non-uniform rank distribution across layers and heads.

3. Hybrid SVD-Quantization for FFNs

Recognizing that uniform low-rank decomposition of FFNs can cause severe accuracy degradation, LASER employs a hybrid scheme:

Channel Selection: A temporary SVD is applied to gate and up projections to identify "SVD-friendly" hidden channels (those with low reconstruction error).
Hybrid Architecture: Selected channels are compressed via SVD, while the remaining channels are kept dense.
Quantization: An orthogonal rotation is inserted between SVD factors to smooth outliers, followed by Round-to-Nearest (RTN) quantization for both the low-rank and dense branches.

4. System Implementation

To realize latency benefits, the authors implement fused Triton kernels for the hybrid low-rank FFN and low-precision Self-Attention (SA) layers. These kernels fuse input projections, latent activation generation, SwiGLU operations, and quantization/dequantization steps, significantly reducing memory movement and kernel launch overhead.

Key Contributions

Loss-Aware Objective: Derivation of a curvature-weighted SVD objective that aligns decomposition with downstream VLM performance using K-FAC, moving beyond local reconstruction error.
Robust Rank Allocation: Identification of the incomparability of K-FAC singular values across layers and the introduction of a calibration-gradient-based importance score for reliable global rank allocation.
FFN Extension: A hybrid SVD-quantization scheme that extends low-rank compression to FFN layers by selectively applying SVD to tolerant channels while preserving dense computation for sensitive ones.
System Optimization: Development of fused Triton kernels that integrate low-rank reconstruction, quantization, and non-linear operations, achieving significant decoding speedups.

Experimental Results

The framework was evaluated on five VLMs (LLaVA-v1.5 7B, LLaVA-Next 7B/13B, Qwen2-VL 7B, SmolVLM 2B) across ScienceQA-IMG, SEED-Bench-IMG, and MMBench.

Accuracy Preservation: Under FP16 inference, LASER-noQ (SVD only) consistently outperformed baselines (SVD-LLM, QSVD, WSVD) across various parameter ratios ( $\rho_1$ ). On LLaVA-Next 13B with $\rho_1=50\%$ , LASER maintained accuracy comparable to the full FP16 model, with less than a 1% drop.
Low-Precision Performance: In W8A8 quantized settings, LASER achieved an average accuracy drop of only ~0.5% relative to FP16 baselines while reducing the KV-cache size to 25% of the FP16 model. It outperformed other quantization-aware SVD methods (QSVD, WSVD) and pure quantization methods (DuQuant, QVLM).
Latency and Speedup: System-level evaluations on RTX 4090 and 5090 GPUs demonstrated that LASER achieves a 4.7× decoding speedup over Flash Decoding and a 2.3× speedup over WSVD, attributed to the fused kernels and reduced memory traffic.

Significance and Claims

The paper claims that LASER provides a principled approach to low-rank compression for VLMs by shifting the optimization focus from local reconstruction to global loss sensitivity. By addressing the limitations of uniform rank allocation and the neglect of FFN layers, LASER enables efficient, low-precision inference on resource-constrained devices without sacrificing multimodal reasoning capabilities. The work demonstrates that combining loss-aware decomposition, gradient-based rank allocation, and hybrid compression strategies can yield substantial gains in both accuracy retention and inference speed.