Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model

This paper presents Bielik-Q2-Sharp, a systematic evaluation of six 2-bit quantization methods on a Polish 11B language model that identifies QuIP# as a high-performing variant comparable to the IQ2_XXS baseline while revealing a critical dissociation between log-likelihood preservation and autoregressive generation in rotation-based methods.

The Gist

Imagine you have a brilliant, 11-billion-paragraph encyclopedia written entirely in Polish. It's incredibly smart, capable of understanding complex grammar, jokes, and history. But there's a problem: this encyclopedia is so massive (about 22 gigabytes) that it won't fit on a standard laptop or a home computer. It's like trying to carry a library in a backpack; you need a truck (a massive, expensive server) to move it.

The goal of this research paper is to figure out how to shrink this encyclopedia down to the size of a paperback book (about 3 gigabytes) without losing its intelligence. The researchers call this process "Extreme 2-bit Quantization."

Here is the story of how they did it, explained simply.

The Challenge: Compressing a Masterpiece

Think of the model's "brain" (its weights) as a giant mosaic made of millions of tiny tiles. In its original form, every tile is a high-definition, full-color photograph (this is the "FP16" format). To make it smaller, the researchers wanted to replace those photos with simple, 2-bit sketches (like a stick figure or a basic shape).

The problem? If you just randomly turn photos into stick figures, the picture becomes unrecognizable. The model might forget how to speak Polish or start speaking in gibberish. The researchers wanted to find the smartest way to turn those photos into sketches so the meaning stays intact.

The Experiment: Six Different Compression Techniques

The researcher, working alone with a modest budget (about $285, roughly the cost of a nice dinner for four), tested six different "compression algorithms." Think of these as six different artists trying to shrink the encyclopedia:

1. The Lattice Artist (QuIP#): This method uses a very specific, mathematically perfect grid (like a honeycomb) to organize the data. It's like packing suitcases so efficiently that you can fit a month's worth of clothes in a carry-on.
2. The Rotation Artist (SpinQuant & Butterfly): These artists try to "twist" the data before shrinking it, hoping that a rotated image is easier to compress.
3. The Trellis Artist (QTIP): This method uses a complex, winding path (like a maze) to encode the information, requiring no extra storage space.
4. The Residual Artist (VPTQ): This artist takes a rough draft, then adds a second layer of tiny details to fix the mistakes.
5. The Additive Artist (AQLM): This method breaks the data into small chunks and reassembles them like a puzzle.

The Big Surprise: The "Two-Step" Trap

One of the most important discoveries in the paper is a phenomenon the researchers call "The MC-Generation Dissociation."

Imagine you have a robot that is taking a multiple-choice test.

• The Test (MC): The robot reads a question and picks the right answer. It gets a perfect score!
• The Conversation (Generation): You ask the robot to write a story. It starts fine, but after a few sentences, it starts repeating itself or speaking nonsense.

The researchers found that two of the "Rotation Artists" (SpinQuant and Butterfly) were great at taking the test but terrible at having a conversation. They preserved the facts but broke the flow. It's like having a dictionary that knows every word but can't form a sentence. This happened because the software needed to "un-twist" the data while speaking, and the standard tools didn't know how to do that.

The Winners: Who Shrunk the Encyclopedia Best?

After testing all six methods on Polish benchmarks (like grammar tests, emotional intelligence quizzes, and reading comprehension), here is what they found:

• The Best All-Rounder (QuIP#): This method shrank the model to 3.26 GB. It performed almost exactly as well as the current community standard (which was already quite good). It was particularly good at understanding emotions and complex reasoning.
• The Efficiency King (QTIP): This method was the smallest and most efficient. It achieved the same high quality as the others but used the least amount of "bits" per word. It's the most fuel-efficient car in the race.
• The Heavyweight (VPTQ): This method was the most accurate at answering multiple-choice questions, but it was slightly larger (5 GB) because it used a "two-step" compression that added extra data to fix errors.

Why This Matters for You

1. Polish Language is Hard: Polish has a very complex grammar system (words change form depending on their role in a sentence). The researchers proved that you can't just use English compression tricks; you have to calibrate the compressor specifically for Polish. They did this by feeding the model Polish text to "learn" how to shrink it correctly.
2. It's Affordable: The entire project was done by one person using rented cloud computers for less than $300. This proves that you don't need a billion-dollar lab to do cutting-edge AI research.
3. Running on Your Laptop: By shrinking the model from 22 GB to 3.26 GB, this research means that a powerful Polish AI could soon run on a standard laptop or even a high-end gaming PC, rather than requiring a massive data center.

The Bottom Line

The researchers successfully proved that you can shrink a massive Polish AI model down to a tiny size without losing its "soul." They found that while some compression methods break the model's ability to speak fluently, others (like QuIP# and QTIP) keep it smart and coherent.

They also discovered a "quality ceiling": no matter how clever the compression method is, there seems to be a limit to how small you can make the model before it starts forgetting things. But for now, they've pushed that limit further than anyone else has for the Polish language.

Technical Summary

Here is a detailed technical summary of the paper "Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model."

1. Problem Statement

Large Language Models (LLMs) face significant deployment barriers due to memory and compute requirements. An 11-billion parameter model in FP16 precision requires ~22 GB of VRAM, exceeding the capacity of most consumer GPUs. While Post-Training Quantization (PTQ) has advanced to 2-bit precision for English models (e.g., LLaMA, Mistral), its application to Polish (and Slavic) language models remains unexplored.

Polish presents unique challenges for extreme compression due to its rich morphological system (7 grammatical cases, 3 genders, complex conjugation). The model must preserve fine-grained distinctions between similar word forms (e.g., dom, domu, domowi) to maintain grammatical coherence. Existing community quantizations (e.g., IQ2_XXS) rely on general-purpose frameworks (llama.cpp) rather than state-of-the-art (SOTA) academic methods. This paper addresses the gap in systematic evaluation of extreme 2-bit quantization for morphologically rich, non-English LLMs.

2. Methodology

The study evaluates six distinct SOTA 2-bit quantization methods applied to Bielik-11B-v2.3-Instruct (an 11B parameter, Mistral-based architecture with 50 layers).

• Base Model: Bielik-11B-v2.3-Instruct (50 layers, 4096 hidden dimension, Grouped-Query Attention).

• Calibration Data: A shared Polish-language corpus (CulturaX-PL, 512 samples × 4096 tokens) was used to generate Hessian matrices (second-order derivative information) on an H200 GPU. This ensures language-specific calibration, critical for capturing Polish activation patterns.

• Evaluated Variants:

  1. Variant A (QuIP# E8P12): Uses Random Hadamard Transform + E8 lattice codebook (216 codewords).
  2. Variant B (SpinQuant + GPTQ): Uses learned rotation matrices (Cayley SGD) fused into weights, followed by GPTQ 2-bit quantization.
  3. Variant C (ButterflyQuant + E8P): Uses learned per-layer Butterfly transforms (Givens rotations) + E8P12 codebook.
  4. Variant D (QTIP): Uses Trellis Coded Quantization (TCQ) in 256 dimensions with computational codebooks (no storage overhead).
  5. Variant E (VPTQ): Uses Vector Post-Training Quantization with residual quantization (primary + secondary codebooks).
  6. Variant F (AQLM): Uses Multi-codebook Additive Quantization with adaptive per-module bitwidth allocation.

• Infrastructure: The entire project was executed by a single independent researcher on cloud GPUs (vast.ai) with a total budget of ~$285.

3. Key Contributions

1. First Academic 2-bit Study for Polish LLMs: The first systematic comparison of six SOTA quantization paradigms on a Slavic language model.
2. Language-Specific Calibration: Demonstrated that generating Hessian matrices from a Polish corpus is practical and affordable (~$2, ~40 mins on H200), significantly impacting quantization boundaries for morphologically complex languages.
3. Discovery of "MC-Generation Dissociation": Identified a critical failure mode where rotation-based methods (SpinQuant, ButterflyQuant) preserve log-likelihood (Multiple Choice) scores but fail catastrophically at autoregressive text generation due to missing runtime transforms (R3/R4).
4. Low-Budget Reproducibility: Proved that rigorous academic quantization research is accessible to independent researchers via cloud infrastructure.
5. Identification of a Quality Ceiling: Found that four distinct quantization paradigms converge to a narrow accuracy band (78.1%–79.4%) on Multiple Choice tasks, suggesting an information-theoretic limit for this model at extreme compression.

4. Key Results

A. Performance vs. Baseline (IQ2_XXS)

The best academic variant, QuIP# E8P12 (Variant A), achieved near-parity with the community baseline IQ2_XXS:

• Raw Average (22 tasks): QuIP# 71.92% vs. IQ2_XXS 72.07% (difference within statistical noise).
• Model Size: QuIP# (3.26 GB) vs. IQ2_XXS (~2.6 GB).
• Emotional Intelligence (eq_bench): QuIP# significantly outperformed the baseline (47.14% vs. 43.53%), suggesting better preservation of higher-order reasoning.
• Error Profiles: QuIP# excelled in reasoning and comprehension tasks, while IQ2_XXS dominated in classification and surface pattern matching.

B. Efficiency Leaders

• QTIP (Variant D): Achieved the best per-bit efficiency. It matched VPTQ's Multiple Choice accuracy (79.4%) at 35% smaller size (3.27 GB vs. 5.0 GB) and true 2-bit precision (2.4 bpw).
• VPTQ (Variant E): Achieved the highest raw MC scores (79.64% acc_norm) but at a higher effective bitrate (~3.58 bpw, 5.0 GB), making it less efficient for deployment.
• AQLM (Variant F): Demonstrated adaptive bitwidth allocation, assigning more bits to attention layers (2.5–3.0 bits) and fewer to FFN layers (2.3–2.4 bits). It achieved competitive MC scores (79.3%) with high stability.

C. Failure Modes

• SpinQuant (Variant B) & ButterflyQuant (Variant C): Both achieved reasonable MC scores but produced incoherent, looping text during generation. This was traced to missing runtime transforms (Hadamard on KV cache/FFN) required by the rotation methods, which standard inference engines do not support.

D. The Quality Ceiling

Four successful variants (QuIP#, QTIP, VPTQ, AQLM) converged to a 78.1% – 79.4% accuracy range on 10 shared Multiple Choice tasks. This suggests that for the Bielik-11B model, further algorithmic sophistication yields diminishing returns without increasing the effective bitrate or applying task-specific fine-tuning.

5. Significance and Implications

• Deployment Viability: The study demonstrates that a Polish 11B model can be compressed from 22 GB to ~3.26 GB (6.7x compression) while retaining >93% of its FP16 quality, enabling deployment on consumer GPUs with as little as 4 GB VRAM.
• Evaluation Protocol Warning: The "MC-Generation Dissociation" highlights a critical flaw in relying solely on log-likelihood (Multiple Choice) metrics for 2-bit quantization. Autoregressive generation testing is mandatory, especially for rotation-based methods.
• Non-English NLP: The success of language-specific Hessian calibration implies that "one-size-fits-all" English calibration data is insufficient for morphologically rich languages; local language data is essential for extreme compression.
• Future Research: The identified quality ceiling suggests that future gains may require modifying the model architecture or evaluation protocols rather than just refining quantization algorithms.

In summary, Bielik-Q2-Sharp establishes a new benchmark for Polish LLM compression, proving that academic SOTA methods can match community baselines while offering deeper insights into the trade-offs between compression efficiency, reasoning capability, and generation stability.