Technical Summary: Rethinking Small VLM Quantization

Problem Statement

The emergence of small vision-language models (sVLMs) with fewer than 3 billion parameters has accelerated the potential for on-device multimodal intelligence in mobile, offline, and low-power environments. However, deploying these models on heterogeneous edge System-on-Chip (SoC) platforms, such as the NVIDIA Jetson Orin series, faces fundamental constraints regarding memory bandwidth and VRAM capacity. While model quantization is a standard compression technique, existing research predominantly focuses on end-to-end evaluation in high-performance server-grade GPU environments. There is a lack of systematic, component-wise analysis to guide engineers on which specific model components (vision encoder, projector, or LLM backbone) should be quantized and how bit-widths should be allocated to balance accuracy, latency, and energy efficiency on edge hardware.

Methodology

The authors propose a systematic evaluation framework to empirically validate five hypotheses regarding component-wise quantization. The study utilizes five architecturally diverse sVLMs (Qwen3-VL-2B, DeepSeek-VL2-Tiny, PaliGemma2-3B, LLaVA-OV-0.5B, and Kosmos-2.5) and evaluates them on two heterogeneous edge platforms: Jetson Orin NX (16 GB VRAM, 102.4 GB/s bandwidth) and Jetson Orin AGX (64 GB VRAM, 204.8 GB/s bandwidth).

Experimental Design:

• Configurations: Six quantization configurations were defined to isolate component effects:
  • Single-component ablations: Quantizing only the LLM (cfg1), Projector (cfg2), or Vision Encoder (cfg4) while keeping others at FP16.
  • Composite configurations: Simultaneously quantizing the Projector and LLM (cfg3), or Vision Encoder and LLM (cfg5).
• Quantization Strategy: INT4 quantization was applied exclusively to the LLM backbone to reduce VRAM, while vision-related components were kept at INT8 or higher to preserve modality alignment. The BitsAndBytes framework was used for quantization.
• Metrics: Evaluation included MME benchmark scores (Perception and Cognition), Peak VRAM usage, component-wise latency (Vision Encoder, Projector, and Token Per Output Time - TPOT), and Intelligence-per-Joule (IPJ).

Key Contributions

1. Systematic Component-Wise Analysis: This is the first study to independently separate and quantify the sensitivity of the vision encoder, projector, and LLM backbone in sub-3B sVLMs, moving beyond holistic model evaluation.
2. Cross-Hardware Empirical Validation: By testing on both Jetson Orin NX and AGX, the study distinguishes between platform-invariant accuracy rankings and hardware-specific latency/energy profiles.
3. Identification of Hardware-Software Stack Friction: The paper identifies and analyzes abnormal latency phenomena specific to SigLIP-based architectures when using INT8 quantization on Ampere-based edge SoCs.
4. Practical Deployment Guidelines: The study provides evidence-based guidelines for precision allocation strategies in edge environments, challenging the assumption that smaller models are inherently more sensitive to quantization.

Key Results and Findings

1. Quantization Sensitivity is Architecture-Dependent, Not Just Scale-Dependent (H1)

Contrary to the hypothesis that smaller models degrade more severely under INT4 quantization, the results show that the structural paradigm (MoE vs. Dense) is the governing factor.

• MoE Backbones: Models like Qwen3-VL (MoE-based) and DeepSeek-VL2 (MoE-based) actually demonstrated improvements in MME accuracy (e.g., +56.04 points for Qwen3-VL) when the LLM was quantized to INT4.
• Dense Backbones: Dense models like PaliGemma2 and LLaVA-OV suffered significant accuracy degradation (e.g., -220.02 points for LLaVA-OV).
• Conclusion: MoE architectures mitigate INT4 noise, whereas dense backbones degrade, regardless of parameter scale.

2. SigLIP-Specific Latency Anomaly (H2)

Applying INT8 quantization to SigLIP-based vision encoders resulted in a disproportionate increase in latency (2.43x to 4.66x) on Jetson Orin platforms, despite minimal impact on accuracy.

• Cause: This is attributed to a hardware-software stack incompatibility where the BitsAndBytes INT8 kernel fails to efficiently accelerate the specific computational patterns of SigLIP vision transformers on the Ampere architecture.
• Contrast: Non-SigLIP encoders (e.g., Kosmos-2.5) showed much smaller latency increases (1.15x–1.20x).

3. VRAM Savings vs. Inference Efficiency Trade-off (H3)

While INT4 quantization of the LLM backbone reduced VRAM consumption by 21.8%–47.5%, it consistently increased Token Per Output Time (TPOT) by 10.4%–56.3%.

• Mechanism: The dequantization overhead introduced by BitsAndBytes outweighs the native computation speedup of INT4 on the Ampere architecture.
• Energy Impact: For larger models (e.g., Qwen3-VL), the increased inference time led to a significant rise in energy consumption (+45% to +55%), whereas smaller models (LLaVA-OV) saw negligible energy increases. Thus, INT4 via BitsAndBytes should be viewed as a VRAM-saving technique rather than a latency-reduction strategy.

4. Non-Additive Interaction in Composite Quantization (H4)

• Projector + LLM (cfg3): The accuracy degradation was largely additive, closely matching the linear sum of individual component errors.
• Vision + LLM (cfg5): The interaction was non-additive and architecture-dependent. For instance, PaliGemma2 showed significant additional degradation (residual of -16.33), while DeepSeek-VL2 showed a super-additive pattern. This suggests that the modality-alignment path is a critical bottleneck sensitive to specific architectural combinations.

5. Platform Invariance of Accuracy vs. Specificity of Efficiency (H5)

• Accuracy: The ranking of model performance (Qwen3-VL > DeepSeek-VL2 > PaliGemma2 > LLaVA-OV > Kosmos-2.5) remained strictly consistent across both Jetson NX and AGX platforms, regardless of quantization configuration.
• Efficiency: Energy efficiency (IPJ) and latency were highly platform-specific. For example, Qwen3-VL achieved up to 2.5x higher IPJ on AGX compared to NX due to the wider memory bandwidth alleviating bottlenecks, whereas other models did not show such gains.

Significance and Claims

The paper claims to provide the first systematic, controlled evidence that algorithmic architecture (MoE vs. Dense) is a more critical determinant of quantization stability than model scale for sVLMs. It highlights that standard quantization tools (like BitsAndBytes) may introduce severe, architecture-specific latency penalties on edge hardware that are not apparent in server-grade evaluations.

The authors conclude that achieving peak efficiency for edge-AI requires an integrated hardware-software co-design strategy rather than relying solely on standalone model compression. They emphasize that deployment decisions must account for specific hardware-software stack interactions (e.g., SigLIP on Ampere) and that accuracy rankings are transferable across edge platforms, while efficiency profiles are not. The study serves as a cautionary guide for engineers, suggesting that blindly applying quantization without component-wise analysis can lead to unexpected performance degradation or latency spikes.