Technical Summary: TileQ - Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling

1. Problem Statement

Mixture-of-Experts (MoE) models have become a dominant paradigm for scaling Large Language Models (LLMs) by sparsely activating specialized "expert" sub-networks. While this architecture reduces computational costs during inference, it introduces a critical memory bottleneck: the entire set of expert parameters must remain resident in memory, even though only a small subset is active per token. This large memory footprint severely limits deployment on resource-constrained hardware and increases cloud inference costs.

Existing Post-Training Quantization (PTQ) methods, particularly low-rank approaches, offer a path to compression. However, current state-of-the-art methods face three significant limitations when applied to MoE models:

1. Accuracy Degradation: Simple quantization schemes suffer severe performance drops at extremely low bit-widths (e.g., 2-bit).
2. Memory Overhead: Traditional low-rank PTQs often incur non-negligible additional memory overhead due to storing low-rank factors for each expert independently or using 1D-sharing strategies that do not fully exploit expert redundancy.
3. Inference Latency: High-accuracy quantization methods often introduce significant latency. In MoE models, where experts are small and sparse, the overhead of managing disjoint low-rank matrices for each expert can degrade inference speed by over 50%, negating the benefits of sparsity.

Furthermore, fine-tuning-based methods (like LoRA) degrade model generalization and are unsuitable for rapid task adaptation without retraining. The goal is to achieve high accuracy and compression ratios with minimal inference latency without fine-tuning.

2. Methodology: TILEQ

The authors propose TILEQ, a fine-tuning-free PTQ method that employs 2D-tiling structured low-rank quantization. The core innovation lies in sharing low-rank factors across both input and output dimensions of MoE experts, rather than treating experts independently or sharing only along a single dimension.

2.1. 2D-Tiling Structured Low-Rank Quantization

The method approximates the weight matrix $W_k$ of each expert $k$ by arranging all $K$ experts into a 2D tiled layout. This allows for the sharing of low-rank factors $U$ and $V$ across the grid.

The process involves five key steps:

1. Scaling: A diagonal scaling matrix is applied to each expert's weight matrix based on calibration data to align singular vectors with input/output feature spaces.
2. Extracting: Approximate rank-$r_0$ SVD factors ($U_k, V_k$) are extracted for each scaled expert. These are vectorized and normalized to create feature embeddings.
3. Clustering: A biclustering algorithm (K-Means) is applied to the normalized embeddings to assign each expert to an ideal grid coordinate $(m, n)$ in an $M \times N$ tile grid. This groups experts with similar singular subspaces.
4. 2D-Tiling Placement: To resolve conflicts where multiple experts map to the same tile, a greedy spatial allocation strategy is used. This minimizes the distance between an expert's ideal cluster position and its actual physical tile position. The experts are then placed into a large block matrix $W_{big}$.
5. Quantization: The large tiled matrix $W_{big}$ undergoes a global low-rank decomposition (approximated via Gaussian Sketching to avoid costly SVD). The resulting shared low-rank factors ($U, V, \Sigma$) are stored, and the residual error for each expert is quantized using Hessian-aware methods (GPTQ/GPTVQ) and rotation techniques (LOPRO).

Key Technical Insight: By structuring experts into a 2D grid, TILEQ enables experts in the same row to share a left basis (output space) and experts in the same column to share a right basis (input space). This reduces storage overhead significantly compared to 1D-sharing or element-wise methods.

2.2. Efficient Fused Sparse Low-Rank Inference

To address the latency bottleneck, TILEQ introduces LoTileMoE, an inference algorithm that fuses multiple low-rank expert computations into a single-pass operation.

• Mechanism: Instead of performing irregular, per-token sparse dispatch for each expert, the algorithm projects the entire input batch onto the shared left factor $(U\Sigma)$ once. It then reshapes this projection into a 3D tensor, applies routing weights to select active tiles, and performs a single batched matrix multiplication with the shared right factor $V$.
• Optimization: The method avoids explicitly constructing large intermediate 3D tensors by using an in-place accumulation buffer. This replaces irregular memory accesses with regular, dense tensor operations (GEMMs) that are highly optimized on modern hardware (GPUs/TPUs).
• Complexity: The approach reduces the complexity from handling $K$ separate small GEMMs to two large, well-shaped GEMMs, significantly improving hardware utilization and reducing kernel launch overhead.

3. Key Contributions

1. 2D-Tiling Low-Rank PTQ: A novel, fine-tuning-free quantization method that leverages expert similarity via 2D tiling to enable efficient shared representations across both input and output dimensions, achieving higher compression ratios than 1D-sharing or independent methods.
2. Fused Inference Algorithm: An optimized inference technique (LoTileMoE) that fuses low-rank expert computations into a single-pass operation, eliminating the latency overhead associated with sparse expert dispatch in MoE models.
3. Performance Efficiency: The method achieves state-of-the-art quantization accuracy while reducing low-rank extra memory usage by up to 10$\times$ and inference latency to approximately 5% of the baseline overhead, without requiring fine-tuning.

4. Experimental Results

The authors evaluated TILEQ on several representative MoE models, including Qwen1.5-MoE-A2.7B, Qwen3-30B-A3B, Qwen3-Next-80B-A3B, Mixtral-8$\times$7B, and Deepseek-MoE-16B.

• Accuracy: TILEQ demonstrates robustness under extreme low-bit settings (2-bit and 3-bit).
  • On Mixtral-8$\times$7B at 2-bit, TILEQ achieves a perplexity of 4.78, significantly outperforming GPTQ (15.3) and GPTVQ (5.28).
  • At 3-bit, TILEQ approaches FP16 performance (e.g., perplexity 4.10 vs. 3.87 for FP16 on Mixtral) and achieves MMLU scores comparable to unquantized models.
  • It consistently outperforms baselines like GPTQ, GPTVQ, MOEQUANT, and LOPRO across various downstream tasks (ARC, PIQA, Winogrande, MMLU, HellaSwag).
• Memory Efficiency: TILEQ reduces the additional memory overhead required for low-rank factors by up to 90% (10$\times$ reduction) compared to per-expert methods. For example, on Qwen3-30B-A3B, it achieves an 8$\times$ reduction in memory overhead compared to per-expert low-rank quantization.
• Inference Latency:
  • Prefill: TILEQ reduces latency to less than 5% by replacing irregular sparse dispatch with structured GEMMs.
  • Decode: Despite being memory-bound, TILEQ maintains low latency (around 5-10% overhead) by utilizing contiguous weight layouts and vectorized access, whereas element-wise and 1D-shared methods suffer from 15-75% latency increases.
• Ablation Studies: Experiments confirm that the 2D-tiling mechanism is the primary driver of performance, as removing it increases bit-width requirements by 8$\times$ while maintaining similar accuracy. The method is scalable across models with varying sparsity and expert sizes.

5. Significance and Claims

The paper claims that TILEQ provides a scalable and hardware-friendly solution for deploying MoE models under strict memory and computational constraints. By jointly structuring experts into a 2D-tiling layout and fusing inference operations, TILEQ overcomes the trade-offs between accuracy, compression, and latency that plague existing low-rank PTQ methods.

The authors emphasize that TILEQ serves as a lightweight, general-purpose extension to existing PTQ frameworks (like GPTQ), enabling the deployment of next-generation sparse MoE models without the need for computationally expensive fine-tuning or retraining. The method is particularly significant for scenarios requiring ultra-low-bit quantization (2-3 bits) where traditional methods fail to maintain fidelity.