Technical Summary: E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring

1. Problem Formulation: Post-Merge Quantization (PMQ)

The paper addresses a specific low-resource deployment scenario where multiple task-specialized "expert" models are first merged into a single model, and this resulting merged model is subsequently quantized for efficient inference. The authors define this setting as Post-Merge Quantization (PMQ).

The core challenge identified is that standard Post-Training Quantization (PTQ) methods, such as GPTQ, are unreliable when applied directly to merged models. In standard PTQ, the full-precision model is treated as a reliable reconstruction target. However, in PMQ, the full-precision merged model ($W_m$) is not an independently optimized multi-task model; it is a composition of parameters that may already deviate from the behaviors of the source experts.

The authors argue that naive PMQ (directly quantizing $W_m$) couples two distinct deviations:

1. Quantization Deviation: The error introduced by reconstructing weights in a low-bit space.
2. Expert-Relative Merging Deviation: The error inherent in the merged model itself, representing how far $W_m$ has drifted from the specific behaviors of the source experts ($W_i$).

When a naive PTQ algorithm minimizes the reconstruction error of $W_m$ alone, it treats this potentially deviated model as the ground truth. Consequently, the quantization process fails to correct the pre-existing merging deviations and instead compounds them with quantization errors, leading to significant performance degradation, especially under aggressive low-bit settings (e.g., 4-bit) or when merging many tasks.

2. Methodology: E-PMQ Framework

To mitigate these coupled deviations, the authors propose E-PMQ, an expert-guided framework that integrates source expert information during the calibration phase while maintaining the deployment efficiency of a single merged model.

Core Components

E-PMQ operates via layer-wise quantization in a forward order. For each layer $\ell$, it utilizes:

• The merged weights ($W_m^\ell$).
• The source expert weights ($\{W_i^\ell\}_{i=1}^K$).
• Calibration activations collected from the current partially quantized model.

The method introduces two key mechanisms:

A. Expert-Guided Output Targets

Instead of reconstructing the merged model's output ($W_m^\ell X^\ell$) as the sole target, E-PMQ constructs output targets based on the source experts. For each expert $i$, the target is $Y_i^\ell = W_i^\ell X_i^\ell$, where $X_i^\ell$ are activations collected from the calibration subset associated with that expert.
The objective minimizes the discrepancy between the quantized layer output and these expert-induced targets:
$$\min_{Q^\ell} \sum_{i=1}^K \| Q^\ell X_i^\ell - W_i^\ell X_i^\ell \|_F^2$$
This forces the quantization process to align with the behaviors of the original experts, thereby mitigating the expert-relative merging deviation.

B. Merged-Weight Anchoring

Relying solely on expert targets risks over-correcting the merged layer, potentially causing the quantized model to drift toward a specific expert and lose the integrated capabilities achieved by the merging process. To prevent this, E-PMQ adds a regularization term that anchors the quantized weights ($Q^\ell$) to the full-precision merged weights ($W_m^\ell$):
$$\min_{Q^\ell} \sum_{i=1}^K \| Q^\ell X_i^\ell - W_i^\ell X_i^\ell \|_F^2 + \lambda^\ell \| Q^\ell - W_m^\ell \|_F^2$$
The anchor strength $\lambda^\ell$ is adaptive, scaled by the total activation energy of the layer to ensure stability across different layers.

Solver Implementation

The authors adapt the GPTQ-style sequential rounding solver to handle this new objective. The solver utilizes modified quadratic statistics:

• Effective Curvature: $H_{E-PMQ}^\ell = \sum H_i^\ell + \lambda^\ell I$
• Right-Hand Side: $R_{E-PMQ}^\ell = \sum W_i^\ell H_i^\ell + \lambda^\ell W_m^\ell$
  This allows the method to be implemented efficiently without end-to-end retraining, using only a calibration set.

3. Key Contributions

The paper outlines three primary contributions:

1. Problem Formulation: The authors formally define Post-Merge Quantization (PMQ) and identify the failure mode of naive approaches: the coupling of quantization deviation with expert-relative merging deviation.
2. Algorithm Proposal: They introduce E-PMQ, which utilizes source expert weights to provide guided output targets during calibration and employs merged-weight anchoring to stabilize the solution and preserve the merged model's integrated behavior.
3. Empirical Validation: Extensive experiments demonstrate that E-PMQ consistently outperforms naive PMQ baselines (RTN, GPTQ, AWQ) across various merging methods (Task Arithmetic, TIES-Merging, Simple Averaging), model architectures (CLIP-ViT, FLAN-T5, Llama-3.1), and task scales (8 to 20 tasks).

4. Experimental Results

The paper evaluates E-PMQ on vision (CLIP) and language (FLAN-T5, Llama-3.1) models using 4-bit weight-only quantization.

• CLIP-ViT-B/32 (8 Tasks): Under Task Arithmetic, E-PMQ improves 4-bit GPTQ accuracy from 65.0% to 73.6%. Under TIES-Merging, it improves from 69.1% to 74.8%.
• CLIP-ViT-L/14 (20 Tasks): In this harder setting with more interference, E-PMQ shows dramatic gains, improving GPTQ from 34.8% to 76.7% under Task Arithmetic.
• FLAN-T5-base (GLUE): E-PMQ improves the average score from 78.26% to 83.34% under Task Arithmetic and from 79.98% to 83.48% under TIES-Merging.
• Llama-3.1 (3B and 8B): E-PMQ achieves the best average performance among quantized variants, improving scores over GPTQ and AWQ baselines.
• Ablation Studies: Removing the merged-weight anchor ($\alpha=0$) causes severe performance collapse (e.g., dropping to ~5% accuracy), confirming that expert guidance alone is insufficient without anchoring to the merged model. E-PMQ also demonstrates robustness across bit-widths (3-bit to 8-bit) and with reduced calibration budgets.

5. Significance and Claims

The paper claims that E-PMQ enables effective low-bit deployment of merged models, a setting previously considered unreliable due to the accumulation of deviations.

• Efficiency: The method requires access to source experts only during the pre-deployment quantization stage. The final deployed model is a single low-bit merged model with no additional inference-time modules or expert weights, maintaining the same parameter count and inference speed as naive baselines.
• Generalizability: The approach is shown to be effective across different modalities (vision and text), model scales, and merging strategies.
• Limitations: The authors modestly note that E-PMQ requires access to source experts during quantization, which increases pre-deployment compute and memory costs compared to naive PTQ. If only the merged model is available, the method cannot construct expert-guided targets. Additionally, the method's performance depends on the representativeness of the calibration set.

In summary, the paper argues that to successfully quantize merged models, one must look beyond the merged model itself and utilize the source experts to guide the calibration, while simultaneously anchoring the solution to the merged weights to preserve the benefits of the fusion.