Technical Summary: Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation

Problem Statement

Vision-language models (VLMs) are increasingly deployed on consumer hardware (e.g., edge devices, consumer GPUs) where memory budgets are fixed and input images are often degraded (compressed, blurred, underexposed). In these settings, a reliable uncertainty signal is critical for "selective prediction"—the ability of a system to defer to a human when it is likely to be wrong.

Practitioners face a trade-off under a fixed memory budget (e.g., 16 GB):

1. Small, Full Precision: High precision, low parameter count.
2. Small, Quantized: Low precision, low parameter count (fastest/cheapest).
3. Large, Quantized: Low precision, high parameter count (fits the same budget).

It is unknown how the choice between model scale and numerical precision affects the two primary routes to uncertainty in generative VLMs:

• Verbalized Confidence: The model explicitly stating a confidence score (e.g., "Confidence: 85").
• Internal Confidence: The mean token probability of the generated answer.

This paper investigates whether increasing model size improves these signals, whether 4-bit quantization degrades them, and which configuration offers the best deployment trade-off.

Methodology

The study employs a controlled three-arm experiment using the Qwen2-VL family of models on a fixed dataset of 100 Food101 images posed as four-option multiple-choice questions.

Experimental Arms

• Arm A: Qwen2-VL-2B-Instruct (fp16) — Small, Full Precision.
• Arm B: Qwen2-VL-2B-Instruct (NF4 4-bit) — Small, Quantized.
• Arm C: Qwen2-VL-7B-Instruct (NF4 4-bit) — Large, Quantized.

This design isolates two orthogonal effects:

• Scale Effect (B vs. C): Precision held constant (4-bit), parameters varied.
• Quantization Effect (A vs. B): Parameters held constant (2B), precision varied.

Data and Degradations

The models were tested on 5,700 predictions across six realistic photographic degradations at three severity levels each:

1. JPEG compression
2. Motion blur
3. Low light
4. Glare
5. Rotation
6. Resampling (downscale/upscale)

Metrics

• Primary Metric: Error-detection AUROC (Area Under the Receiver Operating Characteristic curve). This measures the probability that a correct answer is assigned a higher confidence score than an incorrect one. It is chosen over calibration error because it directly measures the signal's utility for deferral.
• Secondary Metrics: Accuracy, Expected Calibration Error (ECE), Brier score, and Risk-Coverage curves (AURC).
• Parsing Rate: The percentage of times the model successfully followed the requested two-line format (Answer + Confidence).

Key Contributions

1. Controlled Scale Measurement: A measurement of how model scale (2B vs. 7B) affects both verbalized and internal confidence under realistic image degradation, revealing that scale improves internal self-knowledge significantly more than verbalized reporting.
2. Isolated Quantization Effect: A measurement of 4-bit quantization's impact on uncertainty signals while holding model size constant, demonstrating that quantization severely degrades confidence signals while preserving accuracy.
3. Deployment Recommendation: A concrete recommendation for fixed-budget deployments, expressed via selective-prediction operating points (risk-coverage curves and threshold-transfer behavior).
4. Methodological Argument: Evidence that error-detection AUROC is a superior diagnostic metric to calibration error for confidence signals with near-zero variance, as a signal can appear well-calibrated (low ECE) while being useless for ranking correct vs. incorrect answers.

Results

1. Accuracy vs. Confidence Signals

Accuracy remained high and comparable across all arms (0.88–0.94). However, the quality of the confidence signals varied drastically:

• Scale Effect: Increasing scale from 2B to 7B (both 4-bit) improved internal error-detection AUROC from 0.80 to 0.98. In contrast, verbalized confidence AUROC only improved from 0.61 to 0.69, remaining near chance level in many conditions (including low light).
• The Gap Widens: The discrepancy between what the model "knows" (internal signal) and what it "says" (verbalized signal) increased with size. At 7B, the internal signal was near-perfect (0.98), while the verbalized signal remained weak (0.69).

2. Quantization Impact

Moving from fp16 to 4-bit on the 2B model:

• Accuracy: Dropped only slightly (0.904 → 0.888).
• Internal Signal: Dropped significantly (AUROC 0.95 → 0.80).
• Verbalized Signal: The parse rate collapsed from 99% to 64%. The quantized model frequently failed to follow the formatting instructions, returning bare labels instead of confidence scores.

3. Deployment Recommendation

Under a fixed 16 GB memory budget, the 7B-4bit configuration is superior to both the 2B-fp16 and 2B-4bit configurations.

• It achieves the highest accuracy (0.935).
• It achieves the highest internal uncertainty signal (AUROC 0.98), outperforming the full-precision 2B model (AUROC 0.95).
• Threshold Transfer: A deferral threshold calibrated on clean images transfers safely to degraded conditions for the 7B model (maintaining ~100% selective accuracy), whereas the 2B-4bit model silently admits errors under degradation.

4. Calibration Error vs. AUROC

Expected Calibration Error (ECE) was low across all arms (0.01–0.07), suggesting models were "well-calibrated." However, this metric failed to distinguish the utility of the signals. For example, the 2B-fp16 verbalized confidence had an ECE of 0.011 but an AUROC of 0.56 (chance). The paper argues that ECE is misleading for these signals because a constant high-confidence score on a high-accuracy task is well-calibrated but useless for ranking errors.

Significance and Claims

The paper claims that for VLMs deployed on edge hardware:

1. Scale is the primary lever for uncertainty: Increasing model size dramatically sharpens the internal probability distribution (the model's ability to distinguish correct from incorrect answers) but does not make the model more truthful or discriminative in its verbalized confidence.
2. Quantization is "cheap" for accuracy but "expensive" for signals: While 4-bit quantization preserves raw accuracy, it significantly degrades the model's ability to signal uncertainty, both internally and verbally.
3. Practical Recommendation: Practitioners should prioritize parameters over precision. A larger quantized model (e.g., 7B-4bit) provides both better accuracy and a more reliable uncertainty signal than a smaller full-precision model.
4. Signal Selection: Internal token probability should be used for deferral decisions, not verbalized confidence. Verbalized confidence in these models appears to be a learned surface behavior that becomes more fluent with scale but does not necessarily become more truthful.
5. Limitation: No configuration is reliable under severe low-light conditions (severity 3), where accuracy collapses and internal signals weaken. In such cases, an upstream image-quality check is required rather than relying on downstream confidence thresholds.

The study concludes that the gap between a model's internal self-knowledge and its verbalized self-report widens with size, and that error-detection AUROC is the necessary metric to expose this divergence.