💻 computer science

FTerViT: Fully Ternary Vision Transformer

This paper introduces FTerViT, a fully ternarized Vision Transformer that quantizes all weights and normalization parameters to achieve significant memory compression and enable efficient on-device deployment on microcontrollers while maintaining competitive ImageNet-1K accuracy.

Original authors: Szymon Ruciński, Pietro Bonazzi, Engin Türetken, Simon Narduzzi, Michele Magno, Nadim Maamari

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Szymon Ruciński, Pietro Bonazzi, Engin Türetken, Simon Narduzzi, Michele Magno, Nadim Maamari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, highly educated librarian (the Vision Transformer or ViT) who can look at a picture and tell you exactly what it is. This librarian is incredibly smart, but they carry a massive library of books in their head. If you try to put this librarian into a tiny backpack (a microcontroller like the one in a smartwatch or a cheap camera), the backpack rips open because the books are too heavy and take up too much space.

Usually, to fit a smart AI into a tiny device, engineers try to shrink the books. They might turn the text into a shorter code, but they often leave the most important, delicate pages (like the cover, the index, and the final summary) in their original, bulky format. The paper argues that this is like trying to fit a heavy encyclopedia into a pocket by only shrinking the middle chapters; the cover and index still weigh too much.

Here is what the authors of FTerViT did to solve this:

1. The "Three-Color" Dictionary

Instead of using complex numbers (like 3.14159...) to store information, the authors forced the AI to use only three simple symbols: -1, 0, and +1.

Think of this like a dictionary where every word is replaced by a red, green, or blue dot.
By using only these three options, they can pack the information incredibly tightly. It's like folding a giant map into a tiny square.
They call this Ternary (meaning "three").

2. The "Fragile" Parts

Previous attempts at shrinking these AI models stopped short. They shrank the main brain (the "encoder") but left the Patch Embedding (how the AI sees the first glimpse of the image), the LayerNorm (how the AI balances its thoughts), and the Classifier (the final decision-maker) in their heavy, full-size format.

The authors realized that in a tiny device, these "leftover" heavy parts actually take up the most space, even though they are small in number.
FTerViT is the first to shrink everything, including these fragile parts, into the simple -1, 0, +1 format.

3. The "Tutor and Student" Trick

Shrinking a giant brain into a tiny one usually makes it forget how to think, causing it to make silly mistakes. To fix this, the authors used a technique called Knowledge Distillation.

Imagine a Master Chef (the original, heavy AI) teaching a Junior Chef (the tiny, shrunk AI).
Instead of just telling the Junior Chef "This is a cat," the Master Chef shows them how they see the cat. "Look at the ears, the whiskers, the shape."
The Junior Chef learns by copying the Master's thought process, not just the final answer. This allowed the tiny AI to stay smart even after being shrunk down to its absolute smallest size.

4. The Result: A Smart AI in a $10 Camera

The authors tested this on a very cheap, common microcontroller chip called the ESP32-S3 (found in devices costing around $10).

Before: The original AI was 88.3 MB. It was too big to even fit on the chip.
After: Their new FTerViT model is only 5.81 MB. It fits perfectly.
Performance: Even though it is tiny, it is still very smart. It correctly identifies images about 79.6% of the time. This is much better than previous attempts at shrinking AI, which often dropped to 74% or lower.

5. Why It Matters for Your Pocket

The paper shows that you don't need a supercomputer to run smart vision. You can run it on a device with 8 MB of memory (about the size of a small text file).

Speed: Because the data is so small, the device doesn't have to work as hard to fetch information. It runs faster and uses less battery.
Efficiency: The authors built a custom "engine" (software) that runs this tiny AI entirely on the chip, without needing to connect to the internet or a cloud server.

In summary: The paper introduces a way to shrink the most advanced image-seeing AI models down to the size of a tiny pebble, allowing them to run on cheap, battery-powered devices that were previously too weak to handle them. They did this by simplifying the math to just three numbers and using a "tutor" to teach the tiny model how to think correctly.

Technical Summary: FTerViT: Fully Ternary Vision Transformer

Problem Statement

Vision Transformers (ViTs) are powerful image classifiers but are generally unsuitable for microcontroller-class devices due to their substantial memory footprint. A standard compact ViT like DeiT-Small requires approximately 88.3 MB in FP32 precision, whereas typical microcontroller units (MCUs) offer only a few megabytes of external RAM. This mismatch prevents the deployment of always-on, low-power vision applications on edge devices without cloud offloading.

While ternary quantization (constraining weights to $\{-1, 0, +1\}$ ) offers a theoretical 16 $\times$ compression over FP32, existing state-of-the-art methods only ternarize the transformer encoder layers. They retain the patch embedding, LayerNorm parameters, and classifier heads in full precision (FP32) or lower-bit integers (INT8). In compact models, these remaining full-precision components become the dominant bottleneck, consuming up to 38% of the model size despite representing a small fraction of parameters. Furthermore, prior attempts to fully ternarize these sensitive components (e.g., TerViT) resulted in significant accuracy drops (up to 22.4 percentage points), largely due to the known sensitivity of the first and last layers to quantization.

Methodology

The authors propose FTerViT, the first fully ternarized Vision Transformer where all weight matrices and normalization parameters are constrained to $\{-1, 0, +1\}$ . The approach consists of two main pillars: novel ternary operators and a specialized training protocol.

1. Ternary Primitives

To eliminate FP32 bottlenecks, the authors replace standard components with dedicated ternary modules:

TernaryBitLinear: Used for fully-connected layers. It quantizes weights to $\{-1, 0, +1\}$ using per-tensor absolute mean ($absmean$) scaling and clamped rounding. Activations are quantized to 8-bit integers using per-token absolute max scaling.
TernaryBitConv2d: Specifically designed for the patch embedding layer. It applies per-channel scaling to handle heterogeneous filter magnitudes, addressing the sensitivity of the patch embedding layer which prior work identified as the most critical component.
TernaryLayerNorm: Ternarizes the learnable affine parameters ( $\gamma$ and $\beta$ ) of LayerNorm using the same $absmean$ scheme. Per-token statistics (mean and variance) and biases remain in FP32 as their parameter count is negligible, but their ternarization would destroy positional information.

2. Training Protocol

The model is trained using a two-phase Knowledge Distillation (KD) strategy:

Phase 1 (Training): The ternary student is initialized from a pretrained FP32 teacher and trained using Quantization-Aware Distillation (QAD) with a learning rate of $10^{-4}$ (cosine decay) for 250 epochs. The loss function utilizes only the forward-KL divergence term between student and teacher logits, avoiding cross-entropy which can conflict with KL in low-bit networks.
Phase 2 (Recovery): A lightweight fine-tuning phase where the learning rate is restarted at $10^{-5}$ (cosine decay) for 10 epochs. This phase allows for the recalibration of scales and recovery of accuracy lost during the initial saturation of Phase 1.

Key Contributions

First Fully Ternary ViT: The authors demonstrate that the most fragile components of ViTs (patch embedding, LayerNorms, and classifier head) can be successfully ternarized to $\{-1, 0, +1\}$ when trained with knowledge distillation, pushing the boundaries of what was previously considered quantizable.
Novel Operators: Introduction of TernaryBitConv2d with per-channel scaling and TernaryLayerNorm to handle the specific sensitivity of non-encoder layers.
Training Strategy: A simple yet effective two-phase training strategy (high-LR training followed by low-LR recovery) that enables faster convergence and significantly higher final accuracy compared to prior ternary ViT approaches.
On-Device Implementation: The first standalone C implementation of a ternary Vision Transformer on a resource-constrained dual-core ESP32-S3 microcontroller (8 MB PSRAM).

Experimental Results

The method was evaluated on ImageNet-1K and other benchmarks:

Accuracy and Compression:
- FTerViT-DeiT-III-S (384 $\times$ 384): Achieves 82.43% top-1 accuracy at 6.09 MB (14.6 $\times$ compression), losing only 2.42 percentage points compared to its FP32 teacher. This outperforms prior ternary ViT methods (e.g., TerViT) by up to 8 percentage points.
- FTerViT-DeiT-III-S (224 $\times$ 224): Achieves 79.64% top-1 at 5.81 MB.
- FTerViT-DeiT-S (224 $\times$ 224): Achieves 77.47% top-1 at 5.81 MB, surpassing TerViT-DeiT-S by 3.27 percentage points.
- CIFAR Benchmarks: DeiT-Tiny variants achieve 97.43% (CIFAR-10) and 86.01% (CIFAR-100) with only 1.53 MB storage.
Component Sensitivity Analysis:
- Taylor first-order and Hessian-trace analyses confirm that LayerNorms and patch embeddings dominate model importance (34–39% and 4.6% respectively) despite occupying <0.2% of parameters. This explains why prior methods left them in higher precision and why their successful ternarization is critical.
On-Device Deployment (ESP32-S3):
- The 5.81 MB model fits entirely in the 8 MB PSRAM of the ESP32-S3, leaving ~4.5 MB free for camera and LCD buffers.
- Performance: A forward pass takes 21.06 seconds.
- Efficiency: Compared to FP32, the packed 2-bit ternary inference is 1.71 $\times$ faster (804 ms vs. 1376 ms for a specific layer) and reduces above-idle energy consumption by 55% (59 mJ vs. 130 mJ).

Significance

The paper claims that FTerViT substantially narrows the gap between standard ViT accuracy and the strict memory constraints of microcontroller-scale deployment. By successfully ternarizing the entire architecture, including the previously "untouchable" patch embeddings and normalization layers, the authors enable the deployment of high-accuracy vision models on commodity, low-cost ($10) MCUs. The work validates that with the correct training strategy (same-architecture KD and two-phase recovery), the sensitivity of these components can be managed, making fully ternary ViTs a practical reality for edge AI. The authors note that the accuracy gap to FP32 scales inversely with model capacity, with larger models (DeiT-III-S384) suffering less relative accuracy loss than smaller ones.