Beyond MACs: Hardware Efficient Architecture Design for Vision Backbones

1. Problem Statement

The paper addresses a critical limitation in modern computer vision research: the reliance on Multiply-Accumulate (MAC) operations as the primary metric for model efficiency.

• The Flaw: While MACs count the total number of arithmetic operations, they fail to account for hardware-specific factors such as memory access costs (latency) and the degree of parallelism available on specific devices.
• The Consequence: A model with fewer MACs does not necessarily execute faster, especially on edge devices (e.g., mobile phones, Jetson TX2) where memory bandwidth and serialization bottlenecks dominate performance.
• The Goal: The authors aim to design vision backbone architectures that optimize for actual execution time (latency) and throughput across diverse hardware platforms, rather than simply minimizing theoretical operation counts.

2. Methodology

The authors employed a two-phase methodology: extensive empirical analysis of architectural components followed by the design of a new architecture family based on these insights.

Phase 1: Empirical Hardware Efficiency Analysis

The authors conducted exhaustive experiments on a diverse set of hardware (Nvidia A40, Titan RTX, Jetson TX2, iPhone 13, Raspberry Pi 5) to contrast MAC counts with actual latency. Key findings included:

• Depthwise vs. Ungrouped Convolutions: Contrary to common belief, depthwise convolutions (often used in mobile nets) are often less hardware efficient than standard ungrouped convolutions on edge devices. Depthwise convolutions suffer from high memory access overhead and poor parallelization, leading to significantly higher latency despite having fewer MACs.
• Fused MBConv Blocks: The Fused Mobile Inverted Bottleneck (MBConv), which merges depthwise and pointwise convolutions into a single ungrouped convolution, is generally faster than the standard unfused MBConv, particularly in early network stages with lower channel dimensions (<256).
• Resolution vs. Channel Dimension: For a fixed MAC count, architectures with lower operating resolution and higher channel dimensions are more efficient than those with high resolution and low channels. High-resolution operations incur disproportionate memory costs.
• Multi-Head Self-Attention (MHSA): Standard MHSA scales quadratically with input resolution. The authors found that reducing the resolution and channel dimension of the attention mechanism (via downsampling and channel compression) drastically reduces latency with minimal accuracy loss.

Phase 2: LowFormer Architecture Design

Based on the above insights, the authors designed LowFormer, a new family of vision backbones.

• Macro Design:
  • Adopts a 5-stage structure similar to EfficientViT/MobileViT.
  • Early Stages: Uses purely convolutional layers with fused MBConv blocks and fewer layers to maximize efficiency at high resolutions.
  • Late Stages: Concentrates computation in lower-resolution stages with higher channel dimensions.
• Micro Design (Lowtention):
  • Introduces Lowtention, a lightweight alternative to MHSA.
  • Mechanism: It encapsulates Scaled Dot-Product Attention (SDA) with depthwise convolutions. Crucially, it applies channel compression (halving channels) and resolution downsampling (2x) before the attention operation, then upscales/restores them afterward.
  • Benefit: This reduces the quadratic complexity of attention significantly while maintaining representational power.
• Edge GPU Variants (LowFormer-E1/E2/E3):
  • Specifically optimized for edge GPUs (like Jetson TX2) which have high parallelism but limited memory bandwidth.
  • Modifications: Removes the MLP (which is memory-bound), removes the attention mechanism entirely in some variants, and reduces network depth to favor wide, parallelizable convolutional blocks over deep sequential layers.

3. Key Contributions

1. Hardware Efficiency Analysis: A comprehensive study demonstrating that MAC counts are poor predictors of execution time on edge devices, highlighting the negative impact of depthwise convolutions and high-resolution attention.
2. LowFormer Architecture: A new backbone family (B0–B3) that achieves state-of-the-art speed-accuracy trade-offs by prioritizing latency over MAC reduction.
3. Lowtention Mechanism: A novel, efficient attention block that combines channel compression and resolution downsampling, proving superior to standard MHSA and linear attention variants.
4. Edge-Optimized Variants: Three specific variants (E1, E2, E3) tailored for edge GPUs, achieving massive speedups by removing memory-bound components (MLP/Attention).
5. Broad Applicability: Validation across multiple downstream tasks, proving the architecture's versatility beyond simple classification.

4. Results

The authors evaluated LowFormer on ImageNet-1K and various downstream tasks, comparing it against state-of-the-art models (e.g., MobileOne, EfficientMod, FastViT, RepViT).

• ImageNet-1K Classification:
  • LowFormer-B0 achieves 78.4% top-1 accuracy with 8.5ms latency on Jetson TX2, outperforming EfficientMod-xs (76.0% acc, 15.0ms latency) and MobileOne-S2.
  • LowFormer-B3 achieves 83.6% accuracy, running 36% faster on TX2 and 46% faster on ARM CPU than comparable models like FastViT-SA36, while matching its accuracy.
  • LowFormer models consistently show 2x to 3x higher GPU throughput compared to competitors with similar accuracy.
• Edge GPU Variants:
  • LowFormer-E1 achieves 78.8% accuracy with 6.2ms latency on TX2, significantly faster than MobileOne-S3 (11.8ms) with higher accuracy.
  • LowFormer-E2 matches the accuracy of FastViT-SA24 (81.6%) but with 47% lower TX2 latency.
• Downstream Tasks:
  • Object Detection (COCO): LowFormer backbones integrated into RetinaNet outperform PVTv2 and FAT in mAP while offering 2x–4x speedups.
  • Semantic Segmentation (ADE20K): Achieves superior mIoU with significantly lower latency than EfficientFormerV2 and FastViT.
  • Image Retrieval & Tracking: Demonstrated superior throughput on GPR1200 and improved tracking metrics (AUC/Precision) on GOT10K and LaSOT when used in the SMAT framework.

5. Significance

• Paradigm Shift: The paper challenges the community to move beyond MAC-centric design, advocating for latency-driven architecture search that considers memory access patterns and hardware parallelism.
• Edge AI Deployment: By providing models specifically tuned for edge GPUs (Jetson) and mobile devices, LowFormer enables real-time, high-accuracy computer vision applications in robotics, autonomous driving, and surveillance where previous "efficient" models were too slow.
• Efficiency vs. Accuracy: LowFormer demonstrates that it is possible to achieve both higher accuracy and lower latency simultaneously, breaking the traditional trade-off curve where speed gains usually come at the cost of accuracy.
• Open Source: The authors provide code and models, facilitating immediate adoption and further research into hardware-aware neural architecture design.