BiSCo-LLM: Lookup-Free Binary Spherical Coding for Extreme Low-Bit Large Language Model Compression

Technical Summary: BiSCo-LLM

Problem Statement
Large Language Models (LLMs) face significant deployment constraints due to terabyte-scale checkpoint storage, memory bandwidth limitations, and the latency costs associated with weight loading during autoregressive decoding. While post-training quantization (PTQ) has advanced to 4-bit and 3-bit regimes, extending compression to 2 bits or lower remains challenging. Existing scalar or group-wise quantization methods suffer from limited representation capacity at such low bit budgets, failing to preserve the dominant structure of weight vectors. Conversely, vector quantization (VQ) methods offer richer representations but typically rely on explicit codebooks, index lookups, and additional storage overhead, which complicates the storage accounting and deployment efficiency. There is a need for a compression framework that achieves extreme low-bit compression (targeting ~2 bits) without explicit codebooks, while explicitly accounting for all auxiliary storage components (decoders, metadata, compensation modules) to provide a realistic measure of the compressed model size.

Methodology
The paper proposes BiSCo-LLM (Lookup-Free Binary Spherical Coding for Large Language Model Compression), a storage-aware pipeline designed for extreme low-bit weight compression. The framework consists of three primary components:

1. Codebook-Free Binary Spherical Coding (BSQ):
   Instead of mapping weight chunks to explicit centroids in a codebook, BiSCo-LLM maps local weight chunks onto a unit hypersphere and binarizes them into compact spherical codes. The encoder normalizes the latent vector $z$ and applies a sign function to produce a binary code $q \in \{-1/\sqrt{b}, +1/\sqrt{b}\}^b$. This eliminates the need to store explicit codebook entries. The reconstruction is performed by a compact neural decoder $D_\theta$. The training objective combines reconstruction loss, latent commitment, and entropy regularization to ensure efficient use of the binary code space.

2. Second-Stage Residual BSQ:
   To address the inefficiency of simply enlarging a single-stage binary code space (where most theoretical sign patterns remain unoccupied by finite weight chunks), BiSCo-LLM employs a two-stage approach. The first stage reconstructs the dominant component of the weight chunk. The second stage is trained specifically on the residual error left by the base codec. This allocates additional bit capacity to the reconstruction error rather than expanding the primary code space, providing a practical rate–distortion path without increasing codebook storage.

3. Category-Wise Recovery Distillation and Sensitivity Protection:

   • Sensitivity Protection: Recognizing that a small number of channels are highly sensitive to quantization, the method identifies these channels using activation statistics and weight magnitudes. A small subset (e.g., 1%) of these sensitive channels is stored using an auxiliary 8-bit quantizer. This path is counted separately from the main BSQ payload.
   • Category-Wise Distillation: To align the compressed weights with the final model behavior, the pipeline replaces Transformer modules category-by-category (e.g., all q_proj layers). After replacing a category, a recovery distillation step is performed where the partially compressed model (student) is trained to match the original model (teacher). This ensures the recovery objective reflects the interaction between the compressed category and the remaining full model, rather than just minimizing local layer-wise reconstruction error.
   • LoRA Compensation: Optional Low-Rank Adaptation (LoRA) adapters can be applied during the recovery phase to further mitigate errors, with their storage cost explicitly reported.

Key Contributions

• BiSCo-LLM Pipeline: A novel 2-bit-oriented compression pipeline that integrates codebook-free binary spherical coding, residual code allocation, and category-wise recovery distillation. It optimizes and evaluates models under an explicit, real-size storage budget.
• Residual BSQ Design: The introduction of a second-stage residual codec that assigns extra bits to the reconstruction error of the base stage. This approach is shown to be more effective than simply increasing the granularity of a single-stage code, as it better utilizes the available bit budget for recoverable information.
• Category-Wise Optimization: A training and recovery strategy that operates at the Transformer module category level. This approach reduces the mismatch between local weight reconstruction and global model behavior, outperforming purely layer-wise recovery objectives.
• Comprehensive Storage Accounting: The framework explicitly accounts for all storage components, including bit-packed BSQ streams, neural decoder parameters, optional 8-bit protected payloads, LoRA adapters, and metadata, providing a fair comparison baseline against explicit VQ and projection-based methods.

Experimental Results
The method was evaluated on the Qwen3-8B model under a real storage budget:

• Perplexity: On the WikiText-2 dataset, BiSCo-LLM achieved a perplexity of 10.18, compared to 9.73 for the full-precision (FP16/BF16) model. This outperforms scalar baselines (e.g., GPTQ, SpinQuant) and vector-based baselines (e.g., AQLM, QuIP#, UniSVQ) reported in the study.
• Downstream Accuracy: On a seven-task evaluation set (including BoolQ, RTE, WinoGrande, ARC, OpenBookQA, and PIQA), the compressed model achieved an average accuracy of 68.05, compared to 69.92 for the full-precision baseline. This represents a gap of approximately 1.87 percentage points.
• Ablation Studies:
  • Residual Coding: Adding a second-stage residual codec consistently improved perplexity across module categories compared to single-stage configurations.
  • Protected Channels: Using a 1% protected-channel path (stored in 8-bit) significantly reduced perplexity, particularly for sensitive categories like down_proj.
  • Decoder Architecture: Compact symmetric nonlinear decoders were found to be superior to linear decoders for this task.
  • Granularity: Diagnostics showed that simply enlarging the binary code space without residual stages does not guarantee monotonic performance gains.

Significance and Claims
The paper claims that codebook-free spherical coding is a viable representation for extreme low-bit (2-bit) LLM compression when residual coding, sensitivity-aware protection, and recovery distillation are jointly considered. The significance lies in shifting the storage burden from explicit vector codebooks to shared neural decoders, thereby removing the overhead of index lookups and large codebook storage. The authors emphasize that the method preserves model behavior under an extreme low-bit budget only when the entire pipeline—including the cost of decoders, residual stages, and compensation modules—is explicitly accounted for in the storage budget. The results suggest that this approach can maintain model performance within a narrow margin of full-precision models while drastically reducing the storage footprint, making it suitable for deployment scenarios with strict memory and bandwidth constraints.