Technical Summary: An Empirical Study of OpenPangu Quantization on Ascend NPUs

Problem Statement

OpenPangu models represent a significant family of domestic large language models (LLMs) intended for private and domain-specific deployment. However, their robustness under aggressive post-training quantization (PTQ) specifically on Huawei Ascend NPUs has not been systematically characterized. While mature PTQ methods exist for CUDA-centric environments, the unique system context of Ascend NPUs—encompassing specific operator support, device memory constraints, and framework compatibility—necessitates a re-examination of whether standard quantization recipes remain reliable. The core challenge is to identify which bit-widths and methods maintain credibility under a consistent NPU-oriented calibration and evaluation protocol, particularly for the 1B and 7B parameter variants of OpenPangu.

Methodology

The authors conducted a controlled empirical study evaluating OpenPangu-1B and OpenPangu-7B on Huawei Ascend 910B1 NPUs (64GB HBM). The study utilized a unified protocol across 18 evaluation tasks, including language modeling (WikiText2, C4), English commonsense reasoning (PIQA, ARC, HellaSwag, Winogrande, BoolQ), Chinese knowledge benchmarks (CMMLU, C-Eval), mathematical reasoning (GSM8K, MATH-500), code generation (MBPP, HumanEval), and instruction following.

Quantization Methods Evaluated:
The study covered a broad spectrum of PTQ techniques spanning 1 to 8 bits:

• Weight-Only: Round-To-Nearest (RTN), GPTQ, AWQ, GPTAQ.
• Weight-Activation: SmoothQuant (evaluated at W8A8 and W4A4).
• Ultra-Low Precision: BiLLM (binary) and SliM-LLM (2-bit).

Experimental Protocol:

• Calibration: Methods requiring calibration used 128 Alpaca-style instruction samples (sequence length 512). RTN did not use calibration data.
• Configuration: Weight-only methods employed symmetric per-group quantization (group size 128) with 16-bit activations. GPTQ utilized act-order, 0.01 damping, and block size 128. SmoothQuant used per-channel weight and per-token activation quantization with a smoothing coefficient of 0.8.
• Granularity Ablation: A specific ablation study compared per-channel versus per-group scaling to determine the optimal granularity for weight reconstruction error and task performance.
• Software Stack: PyTorch 2.5.1, torch-npu 2.5.1, Transformers 4.53.2, and lm-evaluation-harness.

Key Results

1. 8-bit Weight-Only Quantization is Effectively Lossless

Across both model sizes, 8-bit weight-only quantization (W8A16) demonstrated performance nearly indistinguishable from the FP16 baseline. Methods like RTN, AWQ, and GPTQ preserved perplexity and benchmark accuracy with negligible degradation, establishing W8 as a safe, low-risk default for deployment.

2. 4-bit Quantization: Model-Dependent Viability

• OpenPangu-7B: 4-bit weight-only quantization proved practical. AWQ-W4 maintained C4 perplexity close to FP16 (80.60 vs. 80.43) and showed minimal drops in commonsense reasoning. RTN-W4 also performed competitively without requiring calibration data.
• OpenPangu-1B: 4-bit quantization resulted in visible performance degradation, particularly in reasoning, math, and code tasks. While commonsense classification remained relatively stable, capabilities in GSM8K, MATH-500, MBPP, and HumanEval collapsed significantly (e.g., HumanEval dropping from ~20% to ~9%). This indicates that the 1B model lacks the redundancy to support aggressive W4 compression for general-purpose tasks.

3. Ultra-Low Precision Failure

Settings at 2-bit, binary (1-bit), and W4A4 (4-bit weight, 4-bit activation) largely failed:

• Collapse to Randomness: Most 2-bit and binary configurations resulted in near-random behavior, with perplexity values becoming non-finite or extremely large and task accuracies approaching chance levels.
• W4A4 Instability: SmoothQuant W4A4 produced non-finite perplexity on both models. While W8A8 was usable for the 7B model, the transition to W4A4 caused immediate failure, highlighting that activation quantization error is a more significant bottleneck than weight compression alone in this context.

4. Granularity and Method Sensitivity

• Per-Group Scaling: Ablation studies confirmed that per-group scaling (group size 128) consistently outperformed per-channel scaling in terms of weight reconstruction error (SQNR gain of ~2.6–3.2 dB in the W3–W7 range) and downstream task accuracy.
• Method Ranking: At W4, AWQ emerged as the strongest weight-only method for the 7B model. At W3, only AWQ kept the 7B model in a partially usable regime; other methods collapsed. For the 1B model, even AWQ-W3 was insufficient for general-purpose use.

Significance and Claims

The paper provides an NPU-oriented accuracy map for selecting OpenPangu quantization settings, offering concrete guidance for practitioners deploying these models on Ascend hardware.

• Practical Recommendations: The authors conclude that W8 weight-only quantization is the safest default for both models. W4 weight-only is viable for the 7B model (preferably with AWQ) but should be treated as task-specific for the 1B model, suitable only for lightweight classification rather than complex reasoning or code generation.
• Bottleneck Identification: The study highlights that activation quantization remains the primary unresolved bottleneck. The failure of W4A4 settings suggests that current flows cannot reliably preserve accuracy when compressing activations to 4 bits, necessitating future work on activation-aware kernels and outlier handling.
• Ultra-Low Precision Reality: The results indicate that ultra-low-bit settings (W2, binary) are not currently reliable for OpenPangu without additional recovery mechanisms, as they consistently lead to model collapse.

The authors explicitly state that these findings are based on an accuracy-focused evaluation using simulated quantization and do not claim latency or throughput improvements. They emphasize that while the results provide a robust accuracy baseline, full deployment decisions require further validation with packed kernels and end-to-end throughput profiling.