Technical Summary: GQLA – Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding

1. Problem Statement

Modern autoregressive Large Language Model (LLM) decoding is fundamentally bottlenecked by Key-Value (KV) cache traffic, as every generated token requires reading the entire history of cached keys and values from off-chip memory. While recent architectures like Multi-head Latent Attention (MLA), used in DeepSeek-V2/V3, achieve state-of-the-art KV cache reduction by jointly compressing keys and values into a low-rank latent, they suffer from three coupled hardware drawbacks when deployed on diverse inference hardware:

1. Hardware Coupling: MLA's trained weights expose only a single decoding path (an "absorbed" Multi-Query Attention or MQA form). This path is perfectly tuned for the compute-to-bandwidth ratio of high-end GPUs like the NVIDIA H100 (ridge $\approx$ 295 FLOPs/byte). However, on export-restricted or commodity GPUs like the H20 (which retains bandwidth but cuts compute by $\sim$7x, lowering the ridge to $\approx$ 37 FLOPs/byte), this path becomes severely compute-bound, resulting in poor throughput.
2. Loss of Tensor Parallelism (TP): The absorbed MQA form funnels all query heads through a single shared latent KV. This structure prevents tensor parallelism along the head axis, as every device must replicate the entire latent, limiting scalability.
3. Zero Multi-Token Prediction (MTP) Gain: MTP, which generates multiple tokens per step to increase arithmetic intensity, pushes the already compute-bound H20 decoding further past its efficiency ridge, yielding zero throughput gain. Conversely, on H100, MTP overshoots the ridge, reducing the efficiency of the gain.

2. Methodology: Group-Query Latent Attention (GQLA)

The authors propose Group-Query Latent Attention (GQLA), a minimal architectural modification of MLA that preserves the joint low-rank latent compression while exposing two algebraically equivalent decoding paths over the same trained parameters.

Architecture

GQLA indexes the up-projection matrices ($W_{UK}, W_{UV}$) by $g$ groups rather than replicating them across all $h_q$ query heads. This design enables two distinct execution paths:

1. MQA-Absorb Path (Memory-Friendly):

   • Mechanism: Identical to MLA. The up-projections are absorbed into the query and output projections. All $h_q$ heads attend directly to the compact latent cache ($cKV$).
   • Cache: Stores only the latent ($r_{kv}$) and a shared RoPE key ($d^R_h$).
   • Target Hardware: Optimized for H100-class hardware where the arithmetic intensity ($\approx$ 242 FLOPs/byte for $s_q=1$) sits just below the memory-compute ridge.

2. GQA Path (Compute-Friendly):

   • Mechanism: The latent is expanded into $g$ key/value groups. Decoding runs vanilla Grouped-Query Attention (GQA) without per-step latent expansion.
   • Cache: Stores per-group expanded keys and values ($2gd_h$) plus the shared RoPE key.
   • Target Hardware: Optimized for H20-class hardware. By adjusting the group count ($g$) and the number of query tokens per step ($s_q$), the arithmetic intensity can be pinned to the H20 ridge ($\approx$ 37 FLOPs/byte).
   • Scalability: Supports up to 8-way zero-redundancy tensor parallelism along the group axis.

Path Selection

The runtime selects the path based on the target hardware without retraining or custom kernels:

• H100: Uses the MQA-absorb path ($s_q=1$).
• H20: Uses the GQA path (e.g., $g=8, s_q=2$ or $g=4, s_q=1$).
• MTP: On H20, the GQA path with $s_q=2$ recovers near-ideal throughput gains, whereas MLA gains nothing.

TransGQLA: Conversion from Pretrained Models

To avoid pretraining from scratch, the authors introduce TransGQLA, an extension of the TransMLA pipeline. It converts a pretrained GQA checkpoint into a GQLA model via a single targeted modification:

• Head Merging: Instead of merging grouped heads into a single latent and replicating up-projections across all query heads (as in TransMLA), TransGQLA keeps the up-projections indexed by group.
• Result: The model retains the GQA execution path structurally while enabling the MQA-absorb path via the standard absorb operation. This preserves tensor parallelism along the group axis.

Sparse GQLA

The design extends to fine-grained sparse attention. Because the canonical GQLA configuration ($h_q/g = 16$) matches the Tensor-Core MMA tile dimension ($m=16$), sparse GQLA can preserve the GQA path on H20-class hardware. In contrast, sparse MLA is structurally locked to the absorbed MQA path due to tile constraints, inheriting the same compute overhead issues.

3. Key Contributions

1. Identification of MLA Drawbacks: The paper identifies that MLA's single-path design couples it to H100-class ratios, forfeits head-axis tensor parallelism, and yields zero MTP gains on commodity GPUs.
2. GQLA Architecture: A minimal variant of MLA that exposes two algebraically equivalent paths (MQA-absorb and GQA) over the same weights. A single configuration ($h_q=128, g=8$) with one MTP head pins the rooflines of both H100 and H20 simultaneously.
3. TransGQLA Pipeline: A conversion method that transforms pretrained GQA checkpoints into GQLA models with minimal architectural changes, retaining tensor parallelism and enabling a smooth transition to the latent form.
4. Roofline Analysis: A theoretical verification showing that GQLA weights can achieve near-peak throughput on both H100 and H20 by switching paths, whereas MLA is inefficient on H20.

4. Results

The authors evaluated TransGQLA on the LLaMA-3-8B model:

• Capability Preservation: Converting the GQA weights to the GQLA latent form (without further training) resulted in a loss of only $\sim$9.7 average points on commonsense reasoning benchmarks (MMLU, ARC, etc.) compared to the original 15T-token pretrained model.
• Cache Compression: On the MQA-absorb path, the per-token KV cache was compressed to 28.125% of the original GQA baseline (a 71.875% reduction).
• Throughput Projections:
  • H100: The MQA-absorb path achieves 354K tokens/sec.
  • H20: The GQA path (with $g=8, s_q=2$) achieves 221K tokens/sec, a 3.4x improvement over MLA on the same device (which is stuck at 65K tokens/sec due to being compute-bound).
• Continued Pretraining: The authors project that continued pretraining (following the TransMLA trajectory) will recover the lost capability to within 0.5 points of the original model after 30B tokens, while maintaining the KV cache compression.

5. Significance and Claims

The paper claims that GQLA offers a practical, hardware-agnostic design principle for LLM inference. By exposing multiple algebraically equivalent decoding paths over a single set of trained weights, GQLA allows a model to adapt to different hardware constraints (compute-rich vs. memory-rich) at deployment time without retraining or custom kernels.

The authors position GQLA as a solution to the "hardware coupling" problem, enabling efficient deployment on both flagship accelerators (H100) and commodity/export-restricted hardware (H20). They emphasize that this approach supports Multi-Token Prediction and Tensor Parallelism on hardware where MLA fails to do so efficiently.

Limitations Acknowledged:

• Hardware claims are based on Roofline modeling; full kernel-level benchmarks on actual H20/H100 hardware are pending.
• Continued pretraining results for TransGQLA are currently in progress and not yet finalized in the paper.
• Evaluation is currently limited to the LLaMA-3-8B backbone and commonsense reasoning benchmarks.
• Sparse GQLA is described structurally but not yet benchmarked against existing sparse attention methods.