💻 computer science

Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer

This paper proposes Fisher-Guided Quantization (FGQ), a post-training quantization method that leverages the diagonal Fisher information matrix to identify and preserve task-specific sensitivities across transformer blocks and channels, thereby significantly improving the accuracy of billion-scale Visual Geometry Grounded Transformer (VGGT) models for 3D reconstruction tasks compared to existing baselines.

Original authors: Yipu Zhang, Jintao Cheng, Weilun Feng, Jiehao Luo, Chuanguang Yang, Zhulin An, Yongjun Xu, Wei Zhang

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yipu Zhang, Jintao Cheng, Weilun Feng, Jiehao Luo, Chuanguang Yang, Zhulin An, Yongjun Xu, Wei Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, super-smart robot chef (the Visual Geometry Grounded Transformer, or VGGT) that can look at a few photos and instantly figure out three things at once:

Where the camera was when the photo was taken (Pose).
How deep everything is (Depth).
Exactly what the 3D shape of the objects looks like (Point Map).

This robot is incredibly talented, but it's also a "giant." It has billions of ingredients (parameters) in its pantry, which makes it slow, heavy, and impossible to fit into a small kitchen (like a smartphone or a robot on a factory floor).

To make this robot fit in a small kitchen, scientists usually try to shrink its ingredients. They take the massive, precise measurements (like using a scale that weighs to the milligram) and round them down to simple numbers (like using a scale that only weighs in whole grams). This is called Quantization.

The Problem: "One Size Does Not Fit All"

The researchers in this paper noticed a funny problem with how people were shrinking this robot chef.

Imagine the robot is cooking a complex meal where:

Task A (The Camera Pose) is like stirring a pot. It's a big, smooth motion. If you round your measurements a little bit, the pot still stirs fine. It's tough.
Task B (The Point Map) is like arranging delicate, tiny sugar crystals into a perfect 3D sculpture. If you round your measurements even a tiny bit, the whole sculpture collapses into a messy pile. It's extremely sensitive.

Previous methods treated the robot like it was doing just one job. They applied the same "rounding rules" to the stirring and the sugar crystals equally.

The Result: The stirring (Pose) stayed perfect, but the sugar sculpture (Point Map) got ruined. The robot was great at guessing where the camera was, but terrible at reconstructing the 3D shape.

The Solution: The "Fisher-Guided" Chef

The authors, Yipu Zhang and his team, invented a new method called FGQ (Fisher-Guided Quantization).

Think of FGQ as a smart sous-chef who knows exactly which parts of the recipe are fragile and which are sturdy.

The "Fisher" Score: Before shrinking the ingredients, the sous-chef runs a quick test. It asks: "If I mess up this specific channel of information, how much does it hurt the sugar sculpture? How much does it hurt the stirring?"
- It turns out that different parts of the robot's brain (different "blocks" and "channels") are responsible for different tasks. Some channels are the "sugar crystal guardians," while others are just "stirring helpers."
The Customized Shrinking: Instead of using the same rounding rules for everyone, FGQ uses this score to be gentle with the fragile parts and ruthless with the sturdy parts.
- For the "sugar crystal" channels, it keeps the numbers very precise.
- For the "stirring" channels, it shrinks them down aggressively to save space.

The Results: Saving the Sculpture

The paper tested this new method on the robot chef with a very aggressive shrinking (4-bit quantization, which is like turning a high-definition photo into a tiny pixel art).

Old Methods: The 3D sugar sculpture (Point Map) looked blurry and broken. The robot lost its ability to see fine details.
FGQ Method: The 3D sculpture remained sharp and detailed. The robot could still see the fine edges of objects, even though it was using much less memory.

In fact, the paper claims that for the 3D shape reconstruction task, their method improved the results by up to 39% compared to the previous best methods. It was like taking a blurry, pixelated image and suddenly making it crisp again, just by being smarter about where to save the details.

Summary

The paper argues that you can't treat all parts of a multi-task AI model the same way when you try to shrink it. Some parts are like a brick (hard to break), and some are like glass (easy to shatter). FGQ is a tool that identifies the glass and protects it, allowing the whole robot to fit in a small pocket without losing its ability to see the world in 3D.

Technical Summary: Fisher-Guided Quantization for Visual Geometry Transformer

Problem Statement

Feed-forward 3D reconstruction models, exemplified by the Visual Geometry Grounded Transformer (VGGT), have achieved state-of-the-art performance by jointly predicting multiple geometric tasks—depth estimation, camera pose prediction, and point map reconstruction—in a single forward pass. However, these models rely on billion-scale parameters, creating prohibitive memory and computation overheads that hinder on-device deployment.

While Post-Training Quantization (PTQ) offers a solution to reduce this overhead, existing methods for feed-forward 3D models face a critical limitation: they treat the model as a single-task entity. Current PTQ approaches primarily focus on mitigating heavy-tailed activation distributions (caused by special tokens like camera or register tokens) and constructing diverse calibration datasets. They overlook the inherent multi-task architecture of VGGT.

The authors observe that different transformer blocks and hidden channels contribute distinctly to each downstream task. Consequently, these components exhibit substantially different sensitivities to quantization errors. A uniform quantization strategy, which weights all tasks equally during calibration, inadvertently over-emphasizes insensitive tasks (e.g., camera pose estimation) while causing significant accuracy degradation in highly sensitive tasks (e.g., point map reconstruction). As demonstrated in the paper, applying uniform quantization can lead to a Normalized Performance Degradation (NPD) of 200%–500% for point map reconstruction, whereas pose estimation remains relatively robust.

Methodology: Fisher-Guided Quantization (FGQ)

To address the multi-task sensitivity problem, the authors propose Fisher-Guided Quantization (FGQ), a task-aware PTQ framework. The method operates on three granularities: tasks, transformer blocks, and hidden channels.

1. Sensitivity Measurement via Fisher Information

FGQ quantifies the sensitivity of each component to quantization errors using the diagonal Fisher information matrix.

Theoretical Basis: The method derives sensitivity scores from a second-order Taylor expansion of the task loss. Assuming the model is near a local optimum, the expected loss change is governed by the Hessian term. To avoid the prohibitive computational cost of calculating the full Hessian, FGQ utilizes the Fisher information matrix as a tractable surrogate, based on the identity that the expected Hessian of the negative log-likelihood equals the Fisher information matrix under standard regularity conditions.
Diagonal Approximation: To ensure efficiency and scalability, the authors approximate the full Fisher matrix with its diagonal form. This reduces storage complexity from $O(C^2)$ to $O(C)$ (where $C$ is the channel dimension) and treats channel perturbations as locally independent.
Task-Aware Scoring: The Fisher score $F_k[l, c]$ measures the sensitivity of task $k$ to squared errors on channel $c$ of block $l$ . These scores are computed using first-order gradients from a small calibration set, requiring only one backward pass per task.

2. Integration into Learnable Affine Transformations

FGQ incorporates these sensitivity scores into the calibration of Learnable Affine Transformations, a technique used to suppress outliers before quantization.

Weighted Calibration Objective: Standard calibration minimizes the reconstruction error between full-precision and quantized block outputs using a uniform loss. FGQ replaces this with a Fisher-weighted objective.
Normalization and Aggregation: To prevent tasks with larger raw gradient magnitudes from dominating the calibration, the Fisher scores are normalized per task and then aggregated across tasks. A per-block normalization ensures the loss scale remains comparable across different transformer layers.
Flooring: A small floor value is applied to the weights to ensure that channels with low Fisher scores are not completely ignored, preserving the integrity of the full model.

The resulting objective function allocates more calibration capacity to channels and blocks that are critical for specific tasks, thereby preserving the features most sensitive to quantization noise.

Key Contributions

Identification of Multi-Task Sensitivity: The paper identifies that feed-forward 3D reconstruction models exhibit distinct quantization sensitivities across tasks, blocks, and channels. Uniform treatment of these components leads to disproportionate performance degradation in sensitive tasks like point map reconstruction.
Fisher-Guided Quantization (FGQ): The authors propose a novel PTQ framework that uses the diagonal Fisher information matrix to quantify per-task, per-block, and per-channel sensitivity. These sensitivities are integrated into the Learnable Affine Transformation objective to guide calibration toward the most critical components.
Empirical Validation: Extensive experiments demonstrate that FGQ consistently outperforms state-of-the-art PTQ baselines (including QuantVGGT, FlatQuant, and QuaRot) under 4-bit quantization, with the most significant improvements observed in dense geometric reconstruction tasks.

Experimental Results

The authors evaluated FGQ on the VGGT-1.2B model across three primary tasks: camera pose estimation, point map reconstruction, and depth estimation.

Camera Pose Estimation: On Co3Dv2 and Re10K benchmarks, FGQ achieved competitive or superior results compared to baselines under both W4A8 and W4A4 settings. The improvements were modest, suggesting that global pose estimation is less sensitive to the specific channel selection than dense outputs.
Point Map Reconstruction: This task showed the most dramatic improvement. Under W4A4 quantization, FGQ significantly reduced error metrics compared to QuantVGGT. For instance, on the 7-Scenes dataset, FGQ reduced the mean completeness error from 0.085 (QuantVGGT) to 0.059 and improved mean accuracy. Qualitative results showed that FGQ preserved finer structural details, whereas baseline methods produced blurry reconstructions.
Depth Estimation: On KITTI (monocular and video), FGQ consistently outperformed baselines in W4A8 and W4A4 settings, achieving the best results in AbsRel, SqRel, and RMSE metrics.
Efficiency: Runtime analysis on an NVIDIA A100 GPU showed that while the learnable affine transformation introduces a slight overhead compared to simple round-to-nearest quantization, FGQ still achieves a 1.81× speedup over full-precision inference in the W4A4 setting, retaining most of the efficiency benefits of low-bit quantization.

Significance and Claims

The paper claims that the primary significance of FGQ lies in its ability to resolve the "one-size-fits-all" limitation of current PTQ methods for multi-task 3D vision models. By leveraging Fisher information to identify and protect task-critical channels, FGQ enables the deployment of billion-parameter 3D reconstruction models on resource-constrained devices without sacrificing the fidelity of sensitive geometric outputs.

The authors emphasize that their approach does not require architectural changes or retraining from scratch, making it a practical solution for existing feed-forward 3D models. The results suggest that future quantization strategies for multi-modal or multi-task models must move beyond uniform calibration to account for the heterogeneous sensitivity of different output heads.