FreeAct: Freeing Activations for LLM Quantization

FreeAct is a novel quantization framework that improves Large Language Model performance by relaxing rigid one-to-one transformation constraints to dynamically allocate token-specific activation transformations, thereby addressing the distinct distribution patterns in diffusion and multimodal models.

The Gist

Imagine you have a massive, incredibly detailed library (a Large Language Model, or LLM) that can write stories, solve math problems, and understand pictures. This library is so huge that it takes up an entire warehouse to store and requires a giant power plant to run.

To make this library portable enough to fit in your backpack (your phone or laptop), you need to compress it. This process is called Quantization. It's like taking high-resolution photos and shrinking them to low-resolution JPEGs to save space.

However, there's a catch: if you shrink the photos too much, they become blurry and unrecognizable. The text turns into gibberish, and the math answers become wrong.

The Old Way: The "One-Size-Fits-All" Suit

Previous methods tried to fix this blurriness by using a special "smoothing" tool. Imagine you have a bumpy, jagged mountain range (the data inside the model). To make it easier to compress, you want to flatten the mountains into gentle hills.

Old methods used a rigid, one-to-one rule:

• They had one specific tool (a transformation matrix) to flatten the mountains.
• To make sure the math still worked, they had to use the exact opposite tool on the other side (the weights).
• The Problem: This is like trying to fit everyone in a family into a single, rigid suit. It might fit the dad okay, but it's terrible for the toddler or the grandmother.

In modern AI models, the "data" isn't uniform.

• In Multimodal models (like those that see and read), the "vision" data looks very different from the "text" data.
• In Diffusion models (which generate text step-by-step), the "masked" (hidden) tokens look very different from the "unmasked" (visible) tokens.

Using one rigid tool for all these different types of data causes the "suit" to tear, and the model breaks.

The New Way: FreeAct (The "Custom Tailor")

The paper introduces FreeAct, a new method that says: "Why force one tool to do everything? Let's use different tools for different jobs!"

Here is how FreeAct works, using a creative analogy:

1. The "Rank-Deficient" Secret

The authors discovered a hidden property of these AI models: the data isn't actually as complex as it seems. It's like a 3D sculpture that, when viewed from a certain angle, looks like a flat 2D drawing. The data is "rank-deficient," meaning it has a lot of empty space or redundancy.

Because of this, they realized they don't need a perfect, one-to-one inverse tool. They can use a flexible, custom-fit approach.

2. The "Custom Tailor" Approach

Instead of one rigid suit, FreeAct acts like a master tailor:

• The Weights (The Mannequin): The core structure of the model (the weights) stays static. We give it one standard, high-quality suit that fits the general shape.
• The Activations (The People): The incoming data (the people) are different.
  • If a Text Token walks in, the tailor uses a "Text-Specific" smoothing tool.
  • If a Vision Token (an image) walks in, the tailor uses a "Vision-Specific" smoothing tool.
  • If a Masked Token (a hidden part of a sentence) walks in, the tailor uses a "Mask-Specific" tool.

3. The Magic Trick: Zero-Padding

How do they make this work without breaking the math?
Imagine the tailor has a giant block of clay.

• For the Text data, they carve out a specific shape in the left side of the clay and leave the right side empty (zeros).
• For the Vision data, they carve out a shape in the right side and leave the left side empty.
• The Weights (the mannequin) are carved to match the entire block, combining both sides.

Because the "empty" parts (zeros) don't interfere with each other, the math still balances perfectly. The model gets the benefit of a custom fit for every type of data, while the core structure remains stable.

The Result: A Perfect Fit

By "freeing" the activation side from the strict one-to-one constraint, FreeAct allows the model to handle the messy, dynamic reality of real-world data.

• Before: Trying to force a square peg (vision data) into a round hole (text tool) resulted in a broken model.
• After: FreeAct gives the square peg a square hole and the round peg a round hole.

The Outcome:
The paper shows that this method allows the model to be compressed to extremely small sizes (4-bit) without losing its intelligence. In tests, it improved performance by up to 5.3% compared to the best existing methods. It's the difference between a blurry, unreadable map and a crisp, high-definition GPS guide.

In short: FreeAct stops trying to force a single solution on a complex problem. Instead, it recognizes that different types of data need different treatments, and it builds a flexible system that adapts to each one, keeping the AI smart even when it's tiny.

Technical Summary

Here is a detailed technical summary of the paper "FreeAct: Freeing Activations for LLM Quantization".

1. Problem Statement

Large Language Models (LLMs), particularly advanced paradigms like Diffusion LLMs (dLLMs) and Multimodal LLMs (MLLMs), face significant memory and computational overhead. Quantization (reducing precision, e.g., to 4-bit) is a primary solution, but it introduces errors, especially when activations have non-uniform distributions or significant outliers.

The Core Limitation:
Existing state-of-the-art transformation-based quantization methods (e.g., QuaRot, FlatQuant) rely on a rigid one-to-one transformation constraint. They use a single orthogonal matrix $P$ to transform activations and its unique inverse $P^{-1}$ to transform weights, ensuring mathematical equivalence ($P \times P^{-1} = I$).

• The Flaw: This static approach assumes activations behave consistently. However, in dLLMs (where masked vs. unmasked tokens have vastly different distributions) and MLLMs (where vision vs. text tokens differ), activations exhibit dynamic patterns.
• The Consequence: A single transformation matrix cannot simultaneously smooth the distributions of diverse token types, leading to suboptimal quantization quality and performance degradation in low-bit settings (W4A4).

2. Methodology: FreeAct

The authors propose FreeAct, a post-training quantization framework that relaxes the static one-to-one constraint to accommodate dynamic activation disparities.

A. Theoretical Foundation: Beyond Inverse Matrices

The paper leverages the rank-deficient nature of activations. In high-dimensional spaces, activations often occupy a lower-dimensional subspace.

• Proposition 1: The authors prove that the condition for preserving computation ($XP \tilde{P} W^T = XW^T$) does not strictly require $P = \tilde{P}^{-1}$.
• Solution Space: Because activations are rank-deficient, there exists a solution space strictly larger than the set of exact inverses. This allows for distinct transformation matrices for different activation types while maintaining a unified weight transformation.

B. Framework Design

FreeAct decouples activation transformations from weight transformations:

1. Token Indexing: Activations are categorized by type (e.g., Masked vs. Unmasked in dLLMs; Vision vs. Text in MLLMs).
2. Dynamic Allocation (Activation Side):
   • Instead of one matrix, FreeAct allocates distinct transformation matrices ($P$ and $P'$) for different token types.
   • These matrices are constructed using shared components ($U$) and unique components ($U_X, U_{X'}$).
   • Zero-Padding Strategy: To prevent information entanglement between unique subspaces, the unique components are padded with zeros in the other matrix's specific dimensions.
   • Example: $P = [U, U_X, 0]$ and $P' = [U, 0, U_{X'}]$.
3. Static Unification (Weight Side):
   • The weight transformation matrix ($\tilde{P}$) unifies all components: $\tilde{P} = [U, U_X, U_{X'}]^T$.
   • This ensures the weight side remains static and unified, while the activation side dynamically adapts.
4. Optimization: The parameters are optimized by minimizing the quantization error ($L_q$) between the original output and the quantized output for each token type separately.

C. Implementation Details

• Compatibility: FreeAct is compatible with existing enhancements like learnable clip thresholds and per-channel scaling.
• Efficiency: Since $P$ and $P'$ are derived by slicing the unified $\tilde{P}$, there is no additional memory cost to store multiple large matrices.
• Algorithm: The process involves identifying token indices, applying the specific transformation, quantizing, and minimizing the reconstruction error via an optimizer (AdamW).

3. Key Contributions

1. Paradigm Shift: The first work to relax the static one-to-one transformation constraint in LLM quantization, allowing for flexible, dynamic handling of varying activation patterns.
2. Unified Framework: Successfully unifies the quantization of two distinct advanced paradigms (dLLMs and MLLMs) under a single principle based on token-specific dynamics.
3. Theoretical Guarantee: Provides a theoretical proof (Proposition 1 and Theorem 2) that distinct activation transformations can coexist with a unified weight transformation due to the rank-deficiency of activations, ensuring mathematical equivalence.
4. Subspace Construction: Introduces a novel subspace-based construction with zero-padding to handle diverse token types without information entanglement.

4. Experimental Results

Experiments were conducted on dLLMs (LLaDA, Dream) and MLLMs (Qwen2.5-VL, InternVL2.5) using W4A4 (4-bit weights, 4-bit activations) quantization.

• Performance Improvement: FreeAct significantly outperforms baselines (RTN, SmoothQuant, QuaRot, FlatQuant).
  • Achieved up to 5.3% relative performance improvement over the best SOTA baseline (FlatQuant).
  • In several tasks, FreeAct recovered performance to levels comparable to W8A8 RTN and even 16-bit baselines.
• Robustness:
  • dLLMs: Successfully handled the disparity between masked and unmasked tokens, preventing the model from collapsing (which happened with RTN and SmoothQuant).
  • MLLMs: Effectively managed the distinct distributions of vision and text tokens.
• Ablation Studies:
  • Confirmed that rank-deficiency is crucial; removing too many dimensions degrades performance, but removing $d/32$ or $d/64$ approaches optimal performance.
  • Showed that the learnable transformation matrix is the primary driver of success, though learnable clip thresholds provide additional synergy.

5. Significance

• Enabling Ultra-Low Bit Quantization: FreeAct makes 4-bit quantization viable for complex, dynamic models (dLLMs and MLLMs) where previous methods failed or performed poorly.
• Hardware Efficiency: By maintaining a static weight transformation and only dynamically slicing it for activations, FreeAct offers a practical path to efficient deployment without excessive memory overhead.
• Future Directions: The framework opens new avenues for quantizing hybrid models (e.g., Multimodal Diffusion LLMs) and extending to more modalities (audio, etc.) by simply adding new unique subspaces to the transformation matrix.

In summary, FreeAct solves the "one-size-fits-all" limitation of current quantization methods by mathematically proving that activation transformations can be decoupled from weights, enabling superior compression for the next generation of dynamic LLMs.