🤖 AI

Evaluating the Impact of Post-Training Quantization on Reliable VQA with Multimodal LLMs

This paper presents the first systematic study demonstrating that while Post-Training Quantization degrades the accuracy and reliability of Multimodal Large Language Models in Visual Question Answering, combining data-aware quantization with an adapted Selector confidence estimator effectively mitigates these issues, achieving near-uncompressed performance with approximately 75% reduced memory demand.

Original authors: Paul Jonas Kurz, Tobias Jan Wieczorek, Mohamed A. Abdelsalam, Rahaf Aljundi, Marcus Rohrbach

Published 2026-02-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Paul Jonas Kurz, Tobias Jan Wieczorek, Mohamed A. Abdelsalam, Rahaf Aljundi, Marcus Rohrbach

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart robot assistant (a Multimodal Large Language Model) that can look at a picture and answer questions about it. It's incredibly powerful, but it has two big problems:

It's too heavy: It's like a giant, gold-plated statue. You can't carry it in your pocket or put it on a small phone; it needs a massive server room to run.
It's too confident: Even when it's wrong, it speaks with 100% certainty. It's like a student who guesses "The capital of France is London" and insists, "I am absolutely sure!"

This paper asks: Can we shrink this robot to fit in your pocket without making it even more overconfident and unreliable?

Here is the breakdown of their experiment, using some everyday analogies.

1. The Problem: Shrinking the Robot (Quantization)

To make the robot fit on a phone, the researchers used a technique called Post-Training Quantization (PTQ).

The Analogy: Imagine the robot's brain is written in high-definition, 4K color. To save space, they translate it into a low-resolution, black-and-white sketch.
The Result: The robot becomes much smaller and faster (using 75% less memory!), but the sketch is a bit blurry. Because the brain is blurry, the robot starts making more mistakes and, worse, it gets more confused about when it is right or wrong. It might guess wildly and still sound very sure.

2. The Solution: The "Second Opinion" (The Selector)

The researchers realized that just shrinking the robot wasn't enough. They needed a way to tell the robot, "Hey, you sound unsure, maybe you should just say 'I don't know' instead of guessing."

They built a small, lightweight add-on called The Selector.

The Analogy: Think of the main robot as a loud, confident tour guide. The Selector is a quiet, observant manager standing next to him. The manager doesn't know the answers, but they are really good at reading the tour guide's body language. If the guide is stuttering or looking confused (even if the guide says they are sure), the manager steps in and says, "Stop! Don't answer that one. It's too risky."
The Magic: This manager works even when the tour guide's brain has been shrunk into a low-res sketch.

3. The Experiment: Testing Different "Sketches"

The researchers tried two ways to shrink the robot's brain:

Method A (Data-Free/HQQ): Like trying to shrink a photo using a generic, automatic setting on your phone. It's fast, but it might miss important details.
Method B (Data-Aware/MBQ): Like hiring a professional editor who looks at the specific photo and carefully adjusts the colors to keep the important parts sharp. This takes a bit more effort but keeps the quality much higher.

They tested these methods on different levels of "shrinkage" (8-bit, 4-bit, and 3-bit).

4. What They Found

Shrinking hurts: As they made the robot smaller (going from 8-bit down to 3-bit), it got worse at answering questions and worse at knowing when to stay silent. The "3-bit" version was so blurry it became almost useless.
The "Professional Editor" wins: The Data-Aware (MBQ) method kept the robot much smarter than the generic method, especially when shrinking it down to 4-bit.
The Manager saves the day: When they added The Selector to the 4-bit robot, it fixed the confidence problem. The robot started saying "I don't know" at the right times, just like the giant, uncompressed version.

The Big Takeaway

You can shrink a massive AI model down to fit on a small device (saving 75% of the memory!) without losing its reliability, IF you do two things:

Use the smart, careful shrinking method (Data-Aware/MBQ).
Add a small "manager" (The Selector) to double-check the answers and stop the robot from guessing when it's unsure.

In short: You don't have to choose between a fast, small robot and a reliable one. With the right tools, you can have a tiny robot that is just as honest and trustworthy as the giant one.

1. Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly deployed in critical domains (healthcare, autonomous systems) where reliability (the ability to abstain from answering when uncertain) and efficiency (deployment on resource-constrained edge devices) are paramount. However, two major challenges exist:

Overconfidence: Current MLLMs often produce highly certain but incorrect answers, failing to adhere to the principle of selective prediction.
Deployment Constraints: The massive size of these models necessitates compression techniques like Post-Training Quantization (PTQ) to reduce memory and computation.

The Gap: While PTQ is known to affect task accuracy, its specific impact on model reliability and confidence calibration in multimodal settings remains unexplored. Prior work focuses on unimodal accuracy or assumes uncompressed networks for reliability estimation. This paper investigates the intersection: How does quantization degrade the reliability of MLLMs, and can reliability be restored without retraining?

2. Methodology

Models and Datasets

Models: Two state-of-the-art MLLMs were evaluated: Qwen2-VL-7B and Idefics3-8B.
Datasets:
- In-Distribution (ID): VQAv2.
- Out-of-Distribution (OOD): AdVQA (linguistic shift) and VizWiz (multimodal shift, lower quality images).
Quantization Schemes: The models were quantized post-training using two methods across bit widths (int8, int4, int3):
- HQQ (Half-Quadratic Quantization): A data-free method using heuristics to optimize scaling.
- MBQ (Modality-Balanced Quantization): A data-aware method using a calibration dataset (ShareGPT4V) to capture activation dynamics and apply modality-aware conditioning.

Reliability Estimation

The study compares two confidence estimation strategies:

Intrinsic (MaxProb): The joint softmax probability of generated tokens. Known to be overconfident and poorly calibrated.
Extrinsic (Selector): A lightweight, two-layer MLP trained to predict the likelihood of correctness based on multimodal signals (image/question embeddings, first output token, and joint probability). This acts as a "Selector" to decide whether to accept or abstain from an answer.

Evaluation Metrics

Accuracy: Standard VQA accuracy.
Calibration: Expected Calibration Error (ECE).
Selective Prediction: Coverage-at-Risk (C@R) at various risk levels (0.5%, 1%, 5%), Area under the Risk–Coverage Curve (AUC), and Effective Reliability ( $\Phi_c$ ).

3. Key Contributions

First Systematic Study: This is the first work to systematically link quantization levels with reliability metrics in multimodal settings.
Quantization vs. Reliability Analysis: It establishes that PTQ degrades both accuracy and reliability (calibration), with lower bit widths causing severe confidence instability.
Method Comparison: It demonstrates that data-aware quantization (MBQ) significantly outperforms data-free methods (HQQ) in preserving both accuracy and calibration, particularly at 4-bit precision.
Reliability Restoration: It introduces and validates the Selector estimator for quantized MLLMs, showing it can effectively mitigate reliability losses without retraining the base model.

4. Key Results

RQ1: Effect of Quantization on Performance

Degradation: As bit width decreases, accuracy drops and ECE increases.
- int8: Minimal impact on accuracy; calibration remains relatively stable.
- int4: Accuracy drops slightly (~2% from bf16), but calibration error increases noticeably.
- int3: Severe degradation in both accuracy and reliability, with high confidence instability.
Data-Aware Advantage: MBQ consistently maintains higher accuracy and lower ECE than HQQ, especially at int4 and int3. MBQ preserves multimodal sensitivity better than the heuristic-based HQQ.

RQ2: Selector Compensation

Restoration: The Selector significantly improves reliability for both quantized and uncompressed models.
Performance: Compared to the intrinsic MaxProb baseline, the Selector lowers ECE and improves C@1% across all bit widths.
Int4MBQ + Selector: This combination achieves the best trade-off. It recovers reliability to levels comparable to the uncompressed (bf16) baseline while reducing memory demand by ~75%. It preserves ~98% of the original bf16 accuracy.
Limitation: At int3, intrinsic noise becomes too severe for the Selector to fully compensate.

RQ3: Robustness to OOD

Distribution Shifts: Moving from VQAv2 to AdVQA and VizWiz causes performance drops for all models.
Quantization Impact: Quantization amplifies the degradation but does not fundamentally alter the trend of robustness.
Selector Role: The Selector enhances coverage and effective reliability across OOD scenarios, delaying the "collapse" of performance under severe shifts. Its behavior remains correlated with intrinsic confidences, suggesting it stabilizes distorted activation patterns rather than learning a completely new uncertainty signal.

5. Significance and Conclusion

This paper provides a critical roadmap for deploying efficient, trustworthy MLLMs on edge devices.

Practical Implication: It proves that high-efficiency deployment (int4) is feasible without sacrificing reliability, provided that data-aware quantization (MBQ) is used alongside a lightweight Selector.
Efficiency-Reliability Trade-off: The combination of int4MBQ and the Selector offers the optimal balance, achieving near-uncompressed performance with a 75% reduction in memory footprint.
Future Directions: The authors suggest exploring quantization-aware reliability training and modeling intrinsic uncertainty under compression for broader multimodal reasoning tasks.

In summary, the study confirms that while quantization introduces reliability risks, these can be effectively managed through advanced quantization techniques and external confidence estimators, enabling the safe deployment of large multimodal models on resource-constrained hardware.