A 35B Hybrid-Attention Mixture-of-Experts Model on a 6GB 2011 GPU: Hand-Written 4-bit CUDA Inference for Fermi

Technical Summary: Hand-Written 4-bit CUDA Inference for Fermi (sm_20)

Problem Statement
The paper addresses the challenge of running a contemporary, frontier-class Large Language Model (LLM) on hardware that predates modern inference accelerators. Specifically, the authors attempt to execute Qwen3.6-35B-A3B, a 35-billion-parameter Mixture-of-Experts (MoE) model with a hybrid gated-delta-net (GDN) and full-attention backbone, on an NVIDIA Tesla C2075. Released in 2011, this Fermi-generation GPU (sm_20) possesses only 6 GB of GDDR5 memory, lacks tensor cores, has no native FP16 arithmetic, and lacks the DP4A integer dot-product instruction. Crucially, the 4-bit quantized model requires approximately 10.5 GB of memory, exceeding the device's capacity by nearly a factor of two. Furthermore, the model's hybrid architecture includes a recurrent state (GDN) that complicates standard prefix-caching strategies, and the host CPU (Intel Xeon X5690) lacks AVX instructions, limiting SIMD capabilities to SSSE3.

Methodology
To overcome these constraints, the authors built an inference engine from first principles, avoiding modern toolchains and off-the-shelf kernels. The core strategy is a hybrid execution model that splits the workload between the GPU and the host CPU based on the specific bottlenecks of each phase:

1. Hybrid GPU-Prefill / CPU-Decode:

   • Prefill (GPU): The GPU handles the batched processing of the input prompt. Since the 10.5 GB model cannot fit in 6 GB VRAM, expert weights are streamed layer-by-layer from host RAM to the GPU. The GPU performs the GDN recurrence, softmax attention, and a grouped 4-bit GEMM for the MoE routing.
   • Decode (CPU): The CPU handles single-token generation. Streaming weights across PCIe 2.0 for a single token is latency-prohibitive; therefore, the CPU reads 4-bit weights directly from RAM. The authors implemented a hand-written W4A8 (4-bit weights, 8-bit activations) integer kernel using the SSSE3 pmaddubsw instruction to perform integer multiply-adds, bypassing the lack of AVX.

2. Memory and Quantization:

   • Quantization: The model uses group-wise symmetric 4-bit quantization (Q4_0 family) for weights, paired with 8-bit activations for the decode path.
   • Expert Pruning: To reduce the resident footprint, REAP (Router-weighted Expert Activation Pruning) is applied, removing 50% of the experts (reducing from 256 to 128 per layer) without significant loss in accuracy.
   • Memory Management: Experts are stored in page-locked, NUMA-interleaved host memory to maximize bandwidth and minimize PCIe transfer costs.

3. Prefix Caching for Recurrent Architectures:

   • Standard KV caches cannot be reused for GDN layers because the recurrent state is only valid at the end of a sequence. The authors implemented a position-indexed snapshot cache, checkpointing the full recurrent state at block boundaries. This allows new requests to resume from the nearest valid snapshot if they share a prefix, rather than recomputing the entire sequence.

4. Optimizations:

   • Expert Pinning: Moving experts to page-locked memory reduced transfer time significantly.
   • Single-Pass Prefill: Processing the entire prompt in one pass (rather than chunking) eliminated redundant expert re-streaming and maximized GEMM batch size.
   • NUMA Interleaving: Spreading memory allocation across both CPU sockets improved decode throughput slightly.

Key Contributions

1. Hybrid Execution Engine: A working system for sm_20 that offloads prefill to the GPU (streaming experts) and decode to the CPU (integer SIMD), dictated by memory ceilings and the absence of fast integer dot products on the GPU.
2. Integer-SIMD Decode Path: A custom W4A8 kernel utilizing pmaddubsw that achieves decode speeds on a non-AVX CPU comparable to modern optimized libraries, tripling throughput over a scalar baseline.
3. Recurrent Prefix Cache: A novel caching mechanism for gated-delta-net architectures that enables prefix reuse via state snapshots, reducing repeated prefill latency from 78s to 0.5s.
4. Empirical Hardware Floor: A detailed measurement study identifying the structural limits of Fermi hardware, including negative results that define the practical ceiling of the platform.
5. Multimodal Support: A from-scratch vision tower ported to sm_20 that successfully discriminates visual inputs, demonstrating the engine's capability beyond text.

Results

• Latency Reduction: Through expert pinning, single-pass prefill, and NUMA interleaving, prefill latency for a 947-token prompt was reduced by 34% (from 57.2s to 37.5s).
• Throughput Improvement: The integer-SIMD decode kernel increased decode throughput from 2.8 to 8.6 tokens/second (approx. 3× improvement).
• Prefix Reuse: Repeated prompts utilizing the snapshot cache saw prefill time drop from 78.4s to 0.5s.
• Negative Results (Hardware Limits):
  • Offloading the language model head to the GPU was slower (116ms → 139ms/token) due to PCIe 2.0 latency.
  • Hyper-threading (using 24 logical cores vs. 12 physical) was catastrophic, increasing latency to 1,722ms/token due to resource contention.
  • Further kernel rewrites (splitting GDN recurrence, moving state to shared memory) provided no benefit or were slightly slower, indicating the bottlenecks are structural (thin-M GEMM and sequential recurrence) rather than tunable.

Significance and Claims
The authors explicitly state that this work is not a speed record nor a claim of a new algorithm. Instead, its significance lies in:

• Engineering Rigor: Demonstrating what is required to run a 2026-class MoE model on 14-year-old silicon by writing the entire engine by hand for an unsupported compute capability.
• Constraint-Driven Design: Showing how specific hardware limitations (lack of tensor cores, small VRAM, no AVX) force specific architectural decisions (hybrid CPU/GPU split, integer SIMD, snapshot caching).
• Reality Check: Providing an "honest hardware floor" by documenting what fails (offloading, hyper-threading, kernel rewrites), thereby defining the practical limits of the Fermi architecture.
• Legacy Viability: Concluding that a surprising amount of contemporary capability is reachable on hardware abandoned by the software ecosystem, provided one is willing to trade convenience for manual, low-level optimization.