🤖 machine learning

Low-Rank Ternary Adaptation for Fine-Tuning Transformers

This paper proposes "ternary multiplicative adaptation," a novel low-rank method that enables efficient fine-tuning of ternary transformers by representing discrete weight updates through small ternary matrices, thereby preserving the ternary domain and allowing direct merging without dequantization while significantly recovering performance across language and vision models.

Original authors: Alexandru-Dragos Manolache, Yunqiang Li, Jan van Gemert

Published 2026-08-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alexandru-Dragos Manolache, Yunqiang Li, Jan van Gemert

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence relies on massive computer programs called transformers, which have become the engine behind everything from writing assistants to image generators. These programs are incredibly powerful, but they are also heavy, requiring vast amounts of computer memory and energy to run. To make them usable on smaller devices like phones or laptops, researchers have spent years trying to shrink them down. One popular strategy is to compress the numbers inside the program, turning them from complex, high-precision values into simpler, low-bit versions. Imagine taking a high-resolution photograph and reducing it to a few colors; the image becomes smaller and faster to process, though some detail is inevitably lost. The most aggressive form of this compression involves limiting the program's internal numbers to just three possible values: negative one, zero, and positive one. This approach, known as ternary quantization, slashes the memory needed by a factor of ten, potentially turning a model that once required a large server into something that could run on a standard laptop.

However, a significant problem has stalled the progress of these ultra-compact models. While the models are small and efficient, they are difficult to teach new tasks. Standard methods for teaching these programs involve making tiny, continuous adjustments to their numbers. But when a model is restricted to only three values, it cannot make small adjustments; it can only flip a number from negative to positive, or turn it off completely. Existing techniques for teaching these compressed models often require temporarily expanding the numbers back to their full, heavy size to make the changes, and then squeezing them back down. This process of expanding and re-compressing introduces errors that degrade the model's performance, often making the final result worse than if no teaching had happened at all.

A team of researchers at Delft University of Technology and Amazon has developed a new way to teach these ultra-compact models that avoids this cycle of expansion and error. Instead of trying to add small, continuous numbers to the model, their method works directly within the strict three-value system. They designed a system that acts like a set of switches, allowing the model to flip the sign of its internal numbers or turn them off entirely, without ever leaving the compressed state. To make this efficient, they did not try to adjust every single number individually. Instead, they used a mathematical structure that allows a few small, simple patterns to control vast sections of the model at once. This approach, which they call a low-rank ternary adaptation, lets the model learn new skills while staying perfectly compressed.

The researchers tested this method on several different types of artificial intelligence, including language models capable of reasoning and vision models that recognize images. They started with models that had already been compressed to the three-value limit and then applied their new teaching technique. The results showed that the models recovered much of the accuracy they had lost during the initial compression. In tests involving language tasks, the adapted models performed significantly better than previous attempts that tried to force the model to learn through expansion and re-compression. For instance, on a language model with three billion parameters, the new method improved the average accuracy across nine different tasks from roughly 32 percent to 38 percent, while also making the model's predictions much more confident.

Perhaps most importantly, the final result of this process is a single, unified model that remains in its ultra-compact form. Unlike other methods that leave behind a separate, heavy set of instructions that must be loaded alongside the small model, this new approach merges the learning directly into the tiny weights. The researchers found that for vision tasks, such as identifying objects in images, their method produced a model that was far more accurate than those created by re-compressing after learning. They also discovered that the learning process worked by redistributing the signs of the existing numbers rather than trying to bring zeroed-out numbers back to life. This means the model learns by reorganizing what it already has, rather than trying to create new connections where none existed.

This work demonstrates that it is possible to teach highly compressed artificial intelligence models without sacrificing their efficiency or accuracy. By respecting the strict limits of the three-value system and finding a way to learn within those boundaries, the researchers have shown that these tiny models can be just as capable as their larger counterparts for many tasks. The method requires no extra memory or computing power during use, as the learning is fully integrated into the model's structure. This opens the door for running sophisticated artificial intelligence on devices with very limited resources, bringing powerful capabilities to places where they were previously impossible to deploy.

Technical Summary: Low-Rank Ternary Adaptation for Fine-Tuning Transformers

Problem Statement

Transformer models, particularly Large Language Models (LLMs) and Vision Transformers (ViTs), face significant deployment challenges due to high memory and compute requirements. Quantization offers a solution by compressing weights, with ternary quantization (restricting weights to $\{-1, 0, 1\}$ ) representing an extreme compression level (~1.58 bits per weight) suitable for low-end hardware. However, fine-tuning these ternary models presents a unique challenge: existing Parameter-Efficient Fine-Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA) and its quantized variants (QLoRA, QA-LoRA), rely on additive updates in continuous floating-point spaces.

Applying additive updates to ternary weights forces the model out of the ternary domain, necessitating dequantization to full precision for the update and subsequent re-quantization for inference. This process introduces quantization errors that degrade performance, particularly at sub-2-bit precisions. Furthermore, current methods either fail to preserve the ternary nature of the merged model or require dequantization steps that negate the memory and compute benefits of ternary quantization. There is a lack of a method that can perform PEFT natively within the discrete ternary domain while producing a merged model that remains strictly ternary.

Methodology

The authors propose Ternary Multiplicative Adaptation, a method designed to fine-tune ternary transformers without leaving the ternary domain. The core innovation lies in replacing continuous additive updates with discrete multiplicative updates.

1. Ternary Multiplicative Update

Instead of adding a low-rank matrix to the weights, the method applies an element-wise Hadamard product ( $\odot$ ) between the pretrained ternary weights ( $W_{tern}$ ) and a ternary adaptation mask ( $\Delta_{tern}$ ):
$W'_{tern} = W_{tern} \odot \Delta_{tern}$
The mask $\Delta_{tern}$ consists of values in $\{-1, 0, 1\}$ , performing three discrete operations on the base weights:

$+1$ : Retain the weight (identity).
$0$: Zero out the weight (pruning).
$-1$: Flip the sign of the weight.

This formulation ensures closure within the ternary domain; the merged weights $W'_{tern}$ remain strictly in $\{-1, 0, 1\}$ , eliminating the need for dequantization or re-quantization.

2. Low-Rank Kronecker Decomposition

To represent the high-dimensional adaptation mask $\Delta_{tern}$ efficiently, the authors decompose it using a Kronecker product of two smaller trainable ternary matrices, $A$ and $B$ :
$\Delta_{tern} = A \otimes B$
where $A \in \{-1, 0, 1\}^{p \times q}$ and $B \in \{-1, 0, 1\}^{r \times s}$ . The dimensions are chosen such that $p \cdot r = d_{out}$ and $q \cdot s = d_{in}$ , matching the shape of the original weight matrix. This structure allows the model to achieve high-rank expressivity (rank up to $\min(d_{out}, d_{in})$ ) while maintaining a parameter count of $O(d)$ for square layers, significantly lower than the $O(d^2)$ required for full fine-tuning or high-rank LoRA.

3. Training via Real-Valued Proxies and STE

Since the matrices $A$ and $B$ must be discrete, direct gradient-based optimization is impossible. The authors optimize real-valued proxies $\bar{A}$ and $\bar{B}$ and project them to the ternary domain during the forward pass using a thresholding function (based on the absolute mean of the matrix). Gradients are passed through this non-differentiable projection using a Straight-Through Estimator (STE).

4. Inference and Merging

Post-training, the adaptation is merged directly into the base weights via element-wise multiplication. The resulting model is fully ternary, requiring no additional parameters or latency overhead during inference.

Key Contributions

Ternary Multiplicative Adapter: A novel PEFT method that models discrete ternary updates (sign flips and zeroing) directly on ternary weights, preserving the ternary domain throughout training and inference.
Kronecker Decomposition: A compact, low-rank representation of the ternary update mask that balances parameter efficiency with high expressivity, enabling the adaptation to cover all weights in a layer.
Zero-Overhead Deployment: The method allows for the full merging of the adapter into the base model, resulting in a strictly ternary model that requires no dequantization or re-quantization steps.
Empirical Validation: Comprehensive evaluation across six models (LLaMA-3.2 1B/3B, Falcon-E 1B/3B, BitNet b1.58, and ViT-B/16) demonstrating recovery of performance lost to quantization.

Experimental Results

The authors evaluated their method on language and vision tasks, comparing against full-precision baselines, post-training quantization (PTQ) baselines, and QLoRA variants.

LLM Performance (Llama-3.2 1B/3B): On the Llama-3.2 1B model, the method improved the average benchmark score from 33.2 (ternary SpinQuant baseline) to 35.3, outperforming 2-bit baselines on seven of nine tasks. For Llama-3.2 3B, the method achieved an average accuracy of 38.3, surpassing the 2-bit SpinQuant baseline (32.6) and the 1.58-bit baseline (31.9) on all nine benchmarks.
Perplexity (PPL): The method significantly reduced perplexity compared to the ternary SpinQuant baseline (e.g., from 86.5 to 44.6 for Llama-1B) and outperformed QLoRA that was merged and re-quantized to 1.58 bits.
Downstream Adaptation: When fine-tuning pre-trained ternary models (Falcon-E, BitNet) on GSM8K, the method improved exact-match accuracy by +1.0 to +3.1 points over the frozen ternary backbones.
Vision (ViT-B/16): On ImageNet-100, the merged ternary ViT achieved 83.0% Top-1 accuracy, outperforming the re-quantized QLoRA baseline (78.9%) and narrowing the gap to full ternary fine-tuning (85.7%).
Weight Analysis: Transition analysis revealed that accuracy gains are derived primarily from redistributing signs among existing non-zero weights. The method cannot reactivate weights that were quantized to zero, as the multiplicative update cannot change a zero to a non-zero value.

Significance and Claims

The paper claims that this work addresses a critical gap in efficient model adaptation: enabling fine-tuning of extreme low-bit (ternary) models without sacrificing the memory and compute benefits of the ternary format. By avoiding dequantization and re-quantization, the method prevents the accuracy degradation associated with these steps at sub-2-bit precisions.

The authors emphasize that their approach is the first to produce a merged model that remains strictly ternary while using a parameter-efficient adaptation strategy. This enables the deployment of highly compressed, fine-tuned models on resource-constrained hardware without the inference overhead of separate adapter modules or the precision loss of re-quantization. The work demonstrates that ternary models can be effectively adapted to downstream tasks, recovering much of the performance lost during the initial quantization process.