UtVAA: Ultra-tiny Vision Transformer with Affix Attention for Mobile Image Classification

Technical Summary: UtVAA: Ultra-tiny Vision Transformer with Affix Attention for Mobile Image Classification

Problem Statement
While Vision Transformers (ViTs) have demonstrated superior representation capabilities in image classification compared to Convolutional Neural Networks (CNNs), their deployment on resource-constrained mobile and edge devices is hindered by quadratic self-attention complexity and large parameter counts. Existing lightweight architectures (e.g., MobileNet, ShuffleNet) often struggle with effective multi-scale feature extraction and consistent high accuracy across diverse scenarios. Furthermore, non-architectural compression methods like pruning and quantization often degrade feature representation. There is a critical gap in the systematic design of "ultra-tiny" transformer models (sub-million parameters) that maintain strong discriminative performance while adhering to strict computational budgets.

Methodology
The paper proposes UtVAA (Ultra-tiny Vision Transformer with Affix Attention), a hierarchical architecture designed to balance global contextual modeling with local feature extraction under severe resource constraints. The framework integrates three core components:

1. Dilated Bottleneck Blocks: To efficiently expand the receptive field without increasing parameter counts, the authors introduce two variants of Dilated Bottleneck blocks (with and without residual connections). These utilize dilated depthwise and pointwise convolutions. The residual variant specifically addresses optimization stability and vanishing gradient issues. This design reduces computational complexity by approximately 8–9 times compared to standard convolutions while preserving multi-scale context.
2. Affix Attention Block: This is the novel core of the architecture, designed to mitigate spatial information loss common in linear self-attention. It employs a parallel architecture rather than a sequential one, consisting of:
   • Local Representation Block: Uses depthwise separable convolutions to capture fine-grained features.
   • Global Representation Block: Implements a linear transformer formulation to model long-range dependencies with linear scaling relative to the number of patches, avoiding the quadratic cost of standard self-attention.
   • Coordinate Attention: Encodes spatial dependencies by performing global pooling along height and width axes separately, capturing directional information.
   • Ternary Fusion: A lightweight strategy that concatenates the outputs of the global block, coordinate attention, and the residual connection, followed by a 1×1 convolution to integrate local and global representations.
3. Scalable Variants: The architecture is instantiated in three sizes: Tiny (204.67K parameters), Medium (609.36K), and Large (858.13K), allowing for flexibility in deployment based on specific hardware constraints.

Key Contributions

1. Affix Attention Block: A novel attention mechanism that jointly encodes global and local positional information using linear self-attention and coordinate attention, integrated via a ternary fusion strategy.
2. Dilated Bottleneck Blocks: Two efficient block designs that expand the receptive field using dilated convolutions while maintaining low FLOPs and stable optimization through residual connections.
3. Ultra-tiny Architecture: The systematic integration of these components into UtVAA, achieving competitive accuracy with sub-million parameter counts.
4. Empirical Validation: Comprehensive evaluation on standard benchmarks (CIFAR-10, CIFAR-100) and real-world agricultural datasets (PlantVillage-Tomato, SLIF-Tomato).

Experimental Results
The authors evaluated UtVAA against state-of-the-art lightweight CNNs (MobileNet-V2, ShuffleNet-V2, SqueezeNet) and ViTs (MobileViT, DeiT, PiT).

• CIFAR Benchmarks: UtVAA-Large achieved the highest Top-1 accuracy on both CIFAR-10 (93.34%) and CIFAR-100 (70.97%), outperforming all compared methods while maintaining a significantly lower parameter count than large ViTs. UtVAA-Tiny achieved 90.46% on CIFAR-10 with only 204.67K parameters and 53.95M FLOPs, demonstrating a superior accuracy-complexity trade-off.
• Plant Disease Datasets: On the PlantVillage-Tomato dataset, UtVAA-Tiny achieved 99.61% Top-1 accuracy, surpassing MobileNet-V2 (99.45%) and MobileViT (99.28%) with fewer parameters. On the more challenging in-field SLIF-Tomato dataset, it achieved 99.66% accuracy, outperforming all baselines.
• Efficiency: UtVAA-Tiny demonstrated the lowest inference latency (8.41ms on PlantVillage, 7.45ms on SLIF) among all tested models, including transformer-based baselines, confirming its suitability for real-time edge deployment.
• Ablation Studies: Removing the coordinate attention module resulted in a notable accuracy drop (from 70.97% to 67.91% on CIFAR-100), validating its contribution to spatial feature representation.

Significance and Claims
The paper claims that UtVAA demonstrates that transformer-based vision models can be systematically redesigned into ultra-tiny architectures without significant loss in discriminative performance. By replacing computationally prohibitive self-attention with the proposed Affix Attention and utilizing dilated bottlenecks, the model achieves a balance between global context modeling and computational efficiency. The authors assert that this approach offers a principled architectural alternative to post-hoc compression, making high-performance vision transformers viable for mobile and edge devices where memory and FLOPs are strictly limited. The work suggests that such ultra-compact designs are particularly effective for applications like precision agriculture, where real-time inference on resource-constrained hardware is critical. Future work is identified as focusing on hardware-aware optimization for microcontrollers and extending the architecture to dense prediction tasks like object detection and semantic segmentation.