Technical Summary: Quantizing Whisper: How Design Choices Affect ASR Performance

Problem Statement

Large-scale Automatic Speech Recognition (ASR) models, such as OpenAI's Whisper family, achieve state-of-the-art accuracy but face significant barriers to deployment in resource-constrained environments (e.g., edge devices, embedded systems). These models, often containing hundreds of millions of parameters, demand high memory and computational resources that exceed the budgets of many low-resource and on-device settings. While Post-Training Quantization (PTQ) offers a pathway to compress these models without retraining, the impact of specific design choices—such as quantization schemes, libraries, and bit-widths—on ASR performance remains underexplored, particularly for large-scale Transformer architectures like Whisper. Existing literature often focuses on computer vision or Large Language Models (LLMs), leaving a gap in understanding how aggressive quantization affects speech recognition robustness across different hardware backends.

Methodology

The authors present a unified, cross-library evaluation of PTQ applied to Whisper-small (244M parameters), with validation experiments on Whisper-base and Whisper-tiny. The study compares four widely used PTQ libraries:

1. PyTorch: Native dynamic int8 quantization (CPU-focused).
2. Optimum-Quanto (Quanto): Supports integer and low-precision floating formats across CPU, GPU, and MPS.
3. HQQ: Based on half-quadratic optimization with configurable group sizes.
4. bitsandbytes (BNB): GPU-only implementations of normalized formats (e.g., nf4) with optional double quantization.

Experimental Design:

• Datasets: English subsets of the LibriSpeech corpus: test-clean (easy) and test-other (acoustically challenging).
• Variables: The study systematically varies quantization scheme (dynamic vs. static), method (symmetric vs. asymmetric), granularity (per-tensor vs. per-channel), and bit-width (int8, int4, int3, nf4, fp8).
• Metrics: Performance is evaluated on three dimensions:
  • Accuracy: Word Error Rate (WER) and Character Error Rate (CER).
  • Efficiency: Real-Time Factor (RTF) on both CPU and GPU.
  • Compression: Model size reduction relative to the full-precision (fp32) baseline.
• Protocol: All configurations were evaluated under a unified measurement protocol to ensure direct comparability. Static quantization used a 10% calibration subset, while dynamic quantization computed scales at runtime.

Key Contributions

1. Unified Evaluation Framework: A systematic, cross-library comparison of Whisper PTQ under a controlled protocol, enabling direct comparison of trade-offs specific to different libraries.
2. Comparative Analysis of Design Choices: An empirical assessment of how quantization scheme, method, granularity, and bit-width interact to affect ASR performance.
3. Device-Specific Insights: Parallel evaluation on CPU and GPU revealing that optimal configurations differ based on the hardware backend and acoustic conditions (clean vs. challenging).
4. Cross-Model Validation: Verification that the primary findings regarding dynamic vs. static quantization and bit-width trade-offs hold across Whisper-small, Whisper-base, and Whisper-tiny.

Results

The experiments yielded several distinct findings regarding the trade-offs between compression, speed, and accuracy:

• Dynamic vs. Static Quantization: Contrary to theoretical expectations that static quantization reduces runtime overhead, dynamic quantization consistently outperformed static quantization in both accuracy and speed for Whisper-small. Static quantization (e.g., Quanto int8/fp8) resulted in substantial performance degradation, likely due to the lack of efficient low-bit implementations for operations like LayerNorm and Softmax, forcing repeated dequantization.
• CPU Performance:
  • PyTorch dynamic int8 offered the best speed-accuracy trade-off, achieving the fastest inference (36.4% faster than baseline) with a modest accuracy drop.
  • HQQ dynamic int4 provided the highest compression (69%) while preserving accuracy on clean speech, though it was slower than PyTorch.
• GPU Performance:
  • Quanto dynamic int8 emerged as the most robust configuration. It reduced model size by 57% and, notably, outperformed the fp32 baseline on the challenging test-other split (WER: 10.65 vs. 11.88). This suggests a potential regularization effect where quantization stabilizes predictions in noisy conditions.
  • Aggressive Formats (nf4, int3): Libraries like BNB (nf4) and HQQ (int3) achieved extreme compression (up to 71% size reduction) but suffered significant accuracy degradation, particularly on test-other.
• Granularity and Scheme:
  • Per-channel quantization (used by Quanto) favored accuracy by assigning independent scales to channels, aligning well with Whisper's weight distributions.
  • Per-tensor quantization (used by PyTorch) favored efficiency by using a single scale per tensor, simplifying computation but offering less robustness on diverse acoustic data.
• Model Size Scaling: The trends observed in Whisper-small were consistent across Whisper-base and Whisper-tiny. Dynamic int8 remained the most robust operating point, while aggressive low-bit formats (nf4, int3) consistently traded robustness for compression.

Significance and Claims

The paper concludes that carefully chosen PTQ methods can substantially reduce model size and inference cost without retraining, enabling the deployment of Whisper on constrained hardware. The authors emphasize that:

• Dynamic quantization is the most reliable choice for Whisper architectures, outperforming static approaches in both speed and accuracy.
• 8-bit precision (int8) is a safe minimum for real-world deployment, especially in acoustically challenging environments. Lower-bit formats (int4, int3, nf4) introduce significant robustness gaps that may not be acceptable for general ASR tasks.
• Library and hardware context matter: Optimal configurations are device-specific. PyTorch is preferable for low-latency CPU inference, while Quanto is superior for GPU deployment where accuracy and robustness are prioritized.
• Quantization can act as regularization: In specific cases (Quanto dynamic int8 on GPU), quantization improved performance on noisy data compared to the full-precision baseline, though this benefit did not persist at bit-widths below 8-bit.

The study provides practical guidelines for developers, suggesting that quantization strategies must be tailored not only to the model architecture but also to the specific deployment constraints (device type, memory limits) and expected acoustic conditions.