💻 computer science

Clustered Codebook Quantization for 2D Gaussian-based Image Compression

This paper introduces Cluster-Guided Vector Quantization (CGVQ), a method that partitions Gaussian parameters into homogeneous groups prior to quantization to achieve a 20% reduction in bits-per-pixel while maintaining visual quality comparable to the baseline for 2D Gaussian-based image compression.

Original authors: Runze Cheng, Yicheng Zhan, Josef Spjut, Kaan Akşit

Published 2026-07-08

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Runze Cheng, Yicheng Zhan, Josef Spjut, Kaan Akşit

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, high-resolution digital painting made up of millions of tiny, glowing, oval-shaped stickers (called "2D Gaussian primitives"). Each sticker has specific instructions on where to sit, how big to be, how to spin, and what color to glow. This method is great for making images look sharp and realistic, but there's a problem: storing the exact instructions for every single sticker takes up a huge amount of digital space, like trying to carry a library in your backpack.

The paper introduces a new trick called CGVQ (Clustered Codebook Quantization) to shrink the backpack without losing the picture's quality. Here is how it works, using simple analogies:

The Problem: One Size Doesn't Fit All

Previously, to save space, researchers tried to force all the stickers into a single, giant "instruction dictionary" (a global codebook). Imagine trying to describe every single object in a city—from a tiny ant to a skyscraper—using just one list of 1,000 words. You'd have to use very vague words like "big thing" or "small thing," which makes the description inaccurate. This leads to blurry images or "artifacts" (weird glitches) because the dictionary wasn't specific enough for the details.

The Solution: Sorting by Style

The authors' new idea is to sort the stickers into groups before trying to shrink them.

The Sorting (K-Means Clustering): Imagine you have a box of mixed-up LEGO bricks. Instead of trying to describe them all at once, you sort them into piles: one pile for red bricks, one for blue, one for tiny 1x1s, and one for giant 4x4s. In the paper, they sort the image stickers based on their "personality"—how they are rotated, how big they are, and what color they are.
The Specialized Dictionaries (Cluster-Specific Codebooks): Once the stickers are sorted into these neat piles, the system creates a small, specialized dictionary for each pile.
- The "Red Brick" pile gets a dictionary full of specific red shades.
- The "Tiny Brick" pile gets a dictionary full of precise small sizes.
- Because each dictionary only has to describe one type of sticker, it can be much more precise and use fewer words (bits) to do the job.

The Result: A Smaller Backpack, Same Picture

By using these specialized dictionaries, the system can describe the image much more efficiently.

The Claim: The paper states that this method shrinks the file size by about 20% compared to the previous best method (GI), while keeping the image looking just as sharp.
The Trade-off: There is a small cost. Sorting the stickers into groups and managing multiple dictionaries takes a bit more time to process. The paper notes that as they use more groups (to get better quality), the speed of saving and loading the image slows down. It's a trade-off between how small you want the file and how fast you want to open it.

In a Nutshell

Think of it like packing for a trip.

Old Way: You throw everything into one giant suitcase and try to label it "Stuff." It's heavy and messy.
New Way (CGVQ): You sort your clothes into "Shirts," "Pants," and "Shoes" and pack them into separate, perfectly sized bags. You can fit the same amount of stuff into a smaller total space, and you can find what you need faster (though sorting takes a minute at the start).

The paper proves that by organizing the image data into similar groups first, you can compress the image significantly more without making it look blurry.

Technical Summary: Clustered Codebook Quantization for 2D Gaussian-based Image Compression

Problem Statement

Gaussian-based image representations, such as those proposed by Zhang et al. [2025a], model image content using compact parametric primitives (2D Gaussians) to achieve high visual fidelity and resolution-independent rendering. However, storing the large number of unconstrained floating-point parameters required for each primitive (position, rotation-scale, and color) significantly degrades rate-distortion efficiency, particularly at high-fidelity targets.

While previous work (GI) attempted to address this using global Vector Quantization (VQ) and Residual Vector Quantization (RQ) to compress these attributes into discrete codebooks, a single global codebook struggles to map the immense variance of natural image parameters. This capacity constraint inevitably leads to quantization errors and visual artifacts. The core challenge is to preserve accurate Gaussian reconstruction while constraining parameter entropy to improve codebook efficiency.

Methodology: Cluster-Guided Vector Quantization (CGVQ)

The authors propose CGVQ, a method that partitions Gaussian parameters into homogeneous groups prior to quantization. The underlying principle is that for a fixed codebook capacity, narrower parameter distributions yield better quantization results.

1. Clustering Strategy

Before codebook training, the method groups 2D Gaussian primitives based on feature similarity using K-Means clustering:

Feature Vector: For each primitive $G_i$ , a feature vector $f_i$ is constructed by concatenating rotation ( $r_i$ ), scale ( $s_i$ ), and RGB color ( $c_i$ ) attributes.
Exclusion of Position: The 2D position ( $\mu_i$ ) is deliberately excluded from the clustering features because compression quality is highly sensitive to spatial accuracy; grouping based on position could degrade geometric fidelity.
Process: The normalized features are partitioned into $K$ disjoint clusters to minimize intra-cluster variance. The number of clusters $K$ is manually selected (typically between 4 and 16) to balance reconstruction fidelity and compression efficiency.
Consistency: Clustering is performed once after initial training converges, and the resulting cluster assignments are shared across all parameter-specific quantizers.

2. Per-Cluster Codebook Training

Instead of a single global codebook, CGVQ trains three independent codebooks per cluster to exploit the distinct statistical properties of each attribute type:

Position ( $\mu$ ): Modeled using 16-bit floating-point Quantization-Aware Training (QAT) with a "Straight-Through" FakeVQ estimator.
Rotation-Scale (RS): Modeled using a Uniform Vector Quantization (UQ) codebook.
Color ( $c$ ): Modeled using a Residual Vector Quantization (RQ) codebook for fine-grained residual refinement.

These codebooks are trained end-to-end by minimizing a joint loss function: $L = L_{recon} + L_{vq}$ . The reconstruction loss ( $L_{recon}$ ) combines pixel-wise $L_1$ loss and perceptual $L_1$ loss in feature space, while $L_{vq}$ acts as a commitment loss to regularize encoder outputs toward their assigned codebook entries.

3. Decoding Pipeline

During decoding, each compressed cluster is processed independently:

Quantization indices are used to look up values in the three dedicated codebooks (FP16 for position, UQ for RS, RQ for color).
The decoded primitives from all $K$ clusters are composed into a single Gaussian set for rendering.
Bitrate Reduction: The quantization index tables undergo Partial Bits Back Encoding to further reduce the final bitrate.

Key Contributions

Partitioned Quantization: The proposal of CGVQ, which partitions 2D Gaussian primitives into low-variance groups before quantization, rather than applying a global quantization strategy.
Localized Codebooks: Demonstration that localized, cluster-specific codebooks provide a superior compression–fidelity trade-off compared to global codebook baselines, improving PSNR and SSIM at comparable bitrates.
Clustering Optimization: Highlighting the potential of K-Means clustering as a direction for optimizing future codebook quantization in Gaussian representations.

Experimental Results

Experiments were conducted on the Kodak dataset with Gaussian primitives ranging from 10K to 25K.

Rate-Distortion Performance: CGVQ consistently outperforms the GI baseline. At similar reconstruction fidelity (approx. 31.0 dB), CGVQ achieves a 20% reduction in bits-per-pixel (bpp) (1.92 bpp vs. 2.40 bpp for GI).
High-Fidelity Regime: The advantage is most pronounced in high-quality regimes (above 30 dB), where CGVQ maintains higher PSNR and stronger structural similarity (SSIM).
Parameter Efficiency: With a fixed count of 15K primitives, CGVQ improves PSNR by 1.68 dB over GI. Furthermore, CGVQ with 15K primitives achieves visual quality comparable to GI with 19K primitives, representing a 21% reduction in the required primitive count.
Computational Trade-off: Increasing the number of clusters ( $K$ ) improves compression quality but introduces runtime overhead. As $K$ increases from 1 to 16, encoding FPS drops from $2.91 \times 10^{-2}$ to $1.9 \times 10^{-3}$ , and decoding FPS drops from 133.3 to 33.3.

Significance and Claims

The paper claims that CGVQ effectively addresses the storage cost of 2D Gaussian primitives by leveraging the principle that narrower parameter distributions are easier to quantize accurately. By partitioning primitives into homogeneous clusters based on appearance and anisotropy, the method constrains parameter variance within groups, allowing localized multi-codebook designs to capture high-frequency details more precisely than global approaches.

The authors position this work as an extension of the GI framework, offering a more efficient quantization strategy that improves rate-distortion performance. However, they explicitly acknowledge a trade-off: while compression quality improves with finer clustering, this comes at the cost of reduced encoding and decoding speeds. The method is presented as a step toward optimizing codebook quantization for Gaussian-based image compression, balancing visual fidelity with computational efficiency.