💻 computer science

ReGLA: Efficient Receptive-Field Modeling with Gated Linear Attention Network

ReGLA is a series of lightweight hybrid networks that combines efficient convolutions with ReLU-based gated linear attention and multi-teacher distillation to achieve state-of-the-art accuracy and low latency on high-resolution images, outperforming existing models in both ImageNet classification and downstream detection and segmentation tasks.

Original authors: Junzhou Li, Manqi Zhao, Yilin Gao, Zhiheng Yu, Yin Li, Dongsheng Jiang, Li Xiao

Published 2026-02-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junzhou Li, Manqi Zhao, Yilin Gao, Zhiheng Yu, Yin Li, Dongsheng Jiang, Li Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize objects in a photo. The photo is huge (high-resolution), like a 4K image, but the robot is small and running on a battery-powered device, like a smartphone or a smart camera.

The problem is that the current "smart" robots (called Transformers) are like brilliant but clumsy giants. They can see the whole picture and understand how distant parts relate to each other, but they take forever to think and drain the battery quickly. On the other hand, the "fast" robots (called CNNs) are like sprinters; they are great at spotting local details (like the texture of a cat's fur) but are bad at understanding the big picture (like realizing the cat is sitting on a sofa across the room).

ReGLA is a new robot design that tries to be the best of both worlds: a sprinter with the brain of a giant, but without the slowness. Here is how it works, broken down into simple parts:

1. The Big Idea: "Less but Better"

The authors wanted to build a model that is efficient (fast and low battery) but still smart (accurate). They call this new family of models ReGLA. Think of it as a hybrid car that uses electric power for city driving (local details) and a turbo engine for highway cruising (global context), but it's designed so the engine never overheats.

2. The Two Secret Ingredients

ReGLA uses two special tools to solve the speed vs. smarts problem:

A. The "Super-Sniffer" (ELRF Module)

The Problem: In the early stages of looking at a photo, the robot needs to see fine details (like edges and textures) without getting confused by the whole image at once. Traditional methods use huge "lenses" to see a wide area, but these lenses are heavy and slow.
The ReGLA Solution: They built a tool called ELRF (Efficient Large Receptive Field).
The Analogy: Imagine trying to read a book. Instead of using a giant magnifying glass that covers the whole page (which is heavy and hard to move), you use a stack of smaller, lightweight magnifying glasses that you slide over the text one by one. You still see the whole page eventually, but you do it much faster and with less effort. ELRF does this for images, capturing a wide view while staying light and fast.

B. The "Smart Gatekeeper" (RGMA Module)

The Problem: To understand the whole image, the robot needs to connect distant dots (e.g., the sky connects to the horizon). Standard methods do this by comparing every single pixel to every other pixel. If you have a million pixels, that's a trillion comparisons! It's like trying to introduce every person in a stadium to every other person individually.
The ReGLA Solution: They built a tool called RGMA (ReLU Gated Modulated Attention).
The Analogy: Instead of introducing everyone to everyone, imagine a bouncer at a club.
- The "Bouncer" (the Gating mechanism) quickly checks who is important and who can be ignored.
- The "Linear Attention" part is a fast, straight-line conversation that only happens between the important people.
- By using a simple "Yes/No" switch (ReLU) instead of a complex calculation (Softmax), the robot can process the whole room in a straight line rather than a tangled web. It keeps the speed of a linear process but adds a "gate" to make sure it doesn't miss important local details.

3. The "Study Group" (Multi-Teacher Distillation)

Even with a great design, the robot needs to learn. The authors didn't just teach ReGLA from scratch; they used a Multi-Teacher Distillation strategy.

The Analogy: Imagine ReGLA is a student. Instead of having just one teacher, they put ReGLA in a classroom with seven different experts (teachers).
- One teacher is great at recognizing cats.
- Another is great at finding cars.
- Another is great at understanding shapes.
- ReGLA listens to all of them at once, combining their wisdom.
The Result: This helps ReGLA become a "super-generalist" that is better at everything than if it had learned from just one teacher.

4. The Results: Fast and Accurate

The paper tested ReGLA on standard benchmarks (like ImageNet for photos, COCO for finding objects, and ADE20K for understanding scenes).

Speed: On a high-end iPhone, ReGLA processed images in 4.98 milliseconds. That is incredibly fast—faster than a human blink.
Accuracy: It achieved 80.85% accuracy on standard photo tests, beating other models of the same size.
Downstream Tasks: When they used ReGLA to find objects (like in self-driving cars) or segment images (like in medical imaging or mapping), it beat similar-sized competitors by a significant margin (up to 3.6% better).

Summary

ReGLA is a new way to build AI vision models that are:

Lightweight: They fit on phones and small devices.
Fast: They use "linear" math instead of "quadratic" math, meaning they don't slow down as the image gets bigger.
Smart: They use a "gate" to focus on what matters and a "stacked lens" to see details clearly.
Well-Trained: They learn from a team of expert teachers to get the best possible performance.

The authors conclude that you don't have to sacrifice speed for intelligence. With ReGLA, you can have sophisticated visual intelligence that is also efficient and accessible.

Technical Summary: ReGLA – Efficient Receptive-Field Modeling with Gated Linear Attention Network

1. Problem Statement

The deployment of Vision Transformers (ViTs) on resource-constrained platforms (e.g., mobile devices, embedded systems) is hindered by the quadratic computational complexity of standard softmax-based self-attention mechanisms. While lightweight Convolutional Neural Networks (CNNs) offer efficiency, they often struggle to capture long-range dependencies effectively due to limited receptive fields. Conversely, hybrid CNN-Transformer architectures frequently inherit the inefficiencies of conventional attention or sacrifice modeling fidelity when attempting to linearize attention. The core challenge addressed by this work is balancing high accuracy on high-resolution images with low latency and linear computational complexity, without compromising the stability of training or the expressiveness of the model.

2. Methodology

The authors propose ReGLA, a lightweight hybrid CNN-Transformer architecture designed to optimize the trade-off between local feature extraction and global modeling. The architecture consists of a stem block followed by four hierarchical stages, scaling in channel depth and spatial resolution. The design is built upon three core principles: linear computational complexity, large receptive fields with enhanced local focus, and numerical stability.

Key Architectural Components

1. ReLU Gated Modulated Attention (RGMA)
To replace the computationally expensive softmax attention, ReGLA introduces RGMA, a memory-efficient attention mechanism that achieves linear complexity $O(Nd^2)$ .

Mechanism: It replaces the softmax kernel with a ReLU-based linear kernel and incorporates a lightweight gating unit.
Operation: Given a feature map $x$ $x$ , the output $X_o$ $X_{o}$ is computed as $X_o = G(x) \odot C(x)$ $X_{o} = G (x) ⊙ C (x)$ , where:
- $G(x)$ is a convolutional gated mechanism (using a convolution and sigmoid activation) that generates local feature weights.
- $C(x)$ is the ReLU linear attention component that extracts global context.
- $\odot$ denotes element-wise multiplication.
Benefit: This design decouples token interactions to reduce complexity while the gating mechanism selectively focuses on local patterns, ensuring stable gradients during training and inference.

2. Efficient Large Receptive Field (ELRF)
Designed for the early stages of the network where high-resolution inputs require expansive spatial coverage, ELRF aims to capture local information while maintaining a large receptive field without the cost of large kernels.

Design: Instead of using a single large kernel (e.g., $7 \times 7$ ), ELRF decomposes the operation into cascaded depthwise convolutions ( $3 \times 3$ and $5 \times 5$ ) interleaved with Feed-Forward Network (FFN) layers.
Benefit: This staged approach achieves an equivalent $7 \times 7$ receptive field but optimizes computational efficiency and minimizes memory access bottlenecks by leveraging depthwise convolutions.

3. Multi-Teacher Knowledge Distillation
To enhance generalization across downstream tasks, the authors employ a multi-teacher distillation strategy leveraging seven diverse pre-trained models (including SAM2, DeiT-III, DINOv2, iBOT, ViTamin, and AIMv2).

Adaptations: Unlike standard distillation frameworks (e.g., UNIC) focused on classification, this strategy is optimized for heterogeneous downstream tasks. It introduces an adaptive average pooling to construct a global CLS token from patch tokens for better spatial coherence in dense prediction tasks.
Training: The strategy involves a 30-epoch distillation phase on ImageNet-21K followed by fine-tuning on ImageNet-1K, utilizing synchronized batch normalization and cosine similarity loss to ensure stable knowledge transfer.

3. Key Contributions

ReGLA Architecture: The introduction of a highly efficient CNN-Transformer architecture tailored for mobile and edge deployment, featuring the ELRF module for local representation and RGMA (a softmax-free, gated linear attention) for global modeling.
Distillation Strategy: A novel multi-teacher distillation framework that addresses challenges in model selection, conflict resolution, and architectural heterogeneity between plain ViT teachers and pyramidal student networks.
Empirical Validation: Comprehensive evaluation across ImageNet-1K classification, COCO object detection, and ADE20K semantic segmentation, demonstrating state-of-the-art performance with reduced latency and parameter counts.

4. Experimental Results

The authors evaluated ReGLA variants (T, S, M, L, X) against recent lightweight models (e.g., MobileNetV4, iFormer, EfficientFormer) on multiple benchmarks.

ImageNet-1K Classification:
- ReGLA-M achieves 80.85% Top-1 accuracy at 224px resolution with only 4.98 ms latency on an iPhone 16 Pro (512px input).
- In the 10M parameter category, ReGLA-M outperforms iFormer-M by 0.45% (without distillation) and 0.8% (with distillation).
- ReGLA-L (19.9M params) achieves 82.9% accuracy, surpassing EfficientViT-B2 (24.3M params) by 0.7% while using 4.4M fewer parameters.
- With multi-teacher distillation, ReGLA-X reaches 85.2% Top-1 accuracy, outperforming FastViT-MA36 by 0.5% with 3M fewer parameters.
Downstream Tasks (COCO & ADE20K):
- Object Detection (COCO): ReGLA-M (distilled) achieves 43.9% APbox, outperforming iFormer-M-distilled by 3.1% AP. Notably, ReGLA-M-distilled exceeds the larger iFormer-L-distilled by 1.7% APbox.
- Semantic Segmentation (ADE20K): ReGLA-M-distilled achieves a 3.6% mIoU gain over iFormer-M-distilled. ReGLA-L-distilled shows a 3.9% mIoU improvement over iFormer-L-distilled.
- Latency: ReGLA models consistently demonstrate significantly lower latency on mobile CPUs compared to similarly scaled iFormer models (e.g., ReGLA-M at 4.98ms vs. iFormer-M at 19.24ms on iPhone 16 Pro).

5. Significance and Claims

The paper positions ReGLA as a practical and scalable solution for "sustainable computer vision," addressing the environmental and economic costs of large-scale AI. The authors claim that ReGLA embodies a shift toward "less but better," proving that sophisticated visual intelligence does not require a trade-off with efficiency or accessibility.

Specifically, the work asserts that:

Global attention can be efficient: RGMA surpasses softmax-based methods in both speed and accuracy under resource constraints.
Co-design is critical: The integration of local operators (ELRF) and attention mechanisms (RGMA) is essential for high-resolution efficiency.
Distillation efficacy: The proposed multi-teacher strategy successfully enhances generalization, allowing smaller models to outperform larger, non-distilled counterparts in dense prediction tasks.

The authors conclude that ReGLA establishes a new state-of-the-art for high-resolution visual applications, particularly where latency and parameter efficiency are paramount.