💻 computer science

Neural Architecture Search on Large Kernel Convolution Search Space for Computer Vision Tasks

This paper introduces LarKNAS, a novel Neural Architecture Search method featuring a large kernel convolution search space and an efficient gradient estimation strategy that discovers large-scale convolutional models with performance comparable to hand-designed architectures while significantly reducing search costs.

Original authors: Lbin Hou, Linyuan Wang, Tianyuan Liu, Nuolin Sun, Bin Yan

Published 2026-09-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lbin Hou, Linyuan Wang, Tianyuan Liu, Nuolin Sun, Bin Yan

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer vision, machines learn to see by processing images through layers of mathematical filters, much like a human eye passing light through a lens to focus on a scene. For years, the most successful designs for these digital eyes relied on small, repetitive filters that examined tiny patches of an image at a time. While these small filters worked well for basic tasks, researchers began to suspect they were hitting a ceiling when trying to build models large enough to understand complex scenes, such as a crowded street or a vast landscape. The prevailing wisdom suggested that to make these systems smarter, one simply needed to make them bigger or deeper, stacking more layers of these small filters. However, this approach often led to diminishing returns, where adding more computing power yielded only tiny improvements in accuracy. The question arose: perhaps the issue was not the size of the machine, but the size of the tools it used to look at the world.

A team of researchers at the PLA Information Engineering University set out to challenge this assumption by designing a new method to automatically discover better ways for computers to see. They focused on a specific type of filter known as a convolutional kernel, which is the mathematical tool that scans an image to detect features like edges or textures. While modern designs had largely settled on using very small kernels, the researchers hypothesized that using much larger ones could allow the machine to see a broader context in a single glance, similar to how a human might step back to see the whole picture rather than focusing on a single brick. To test this, they developed a system called LarKNAS, which stands for Neural Architecture Search on Large Kernel Convolution Search Space. This system does not rely on human intuition to pick the best design; instead, it acts as an automated architect, testing thousands of different combinations of these large filters to find the most efficient and powerful structure.

The researchers faced a significant hurdle: searching for the best design among such a vast array of possibilities is incredibly expensive in terms of time and computing power. Traditional methods often require testing each design one by one, a process that can take thousands of days of computing time. To solve this, the team created a new search strategy that estimates the potential of a design without fully training it first. They looked at how the mathematical signals within the network changed as it learned, using the average strength and consistency of these signals to predict which designs would perform best. This allowed them to navigate the search space much faster than previous methods, effectively filtering out poor designs early and focusing resources on the most promising candidates.

The search space they built was not just about making filters bigger; it was about arranging them intelligently. They designed a structure where different parts of the network used different sizes of filters. The early layers, which handle simple details like edges, used small filters, while the middle layers, responsible for understanding complex shapes and relationships, used very large filters, some as wide as fifty-one pixels. They also introduced a way to make these large filters more efficient by making them "sparse," meaning they only processed a selection of the data points rather than every single one, saving energy while maintaining the ability to see long distances across the image. This approach resulted in a family of models that could be scaled up or down depending on the task, from lightweight versions for mobile devices to massive versions for high-end servers.

When the researchers tested their best-found model, known as LarKNASNet-L, the results were striking. On the ImageNet dataset, a standard benchmark for image recognition, the model achieved an accuracy of 86.6 percent. This performance was comparable to, and in some cases better than, the best hand-designed models created by human experts, including those based on the latest transformer architectures that have dominated the field recently. The model also excelled in other vision tasks, such as detecting objects in images and segmenting them into distinct parts, achieving top scores on the COCO and ADE20K benchmarks. Perhaps most importantly, the team achieved these results with a search cost of only 96 days of computing time on specialized graphics processors. This is a fraction of the cost required by other large-scale search methods, which often consume hundreds of days to find a model of similar size.

The study explicitly ruled out the idea that simply adding more layers of small filters was the best path forward for large-scale models. By comparing their results with other methods that relied on re-parameterization techniques or dense large kernels, the researchers found that their approach of using sparse, large kernels within a phased architecture was superior. They demonstrated that trying to use re-parameterization methods, which combine multiple branches of filters, often led to memory issues that made the search process impossible on standard hardware. Their findings suggest that the key to unlocking the next generation of computer vision models lies not just in scaling up the number of parameters, but in rethinking the fundamental geometry of how these models process visual information.

The implications of this work extend beyond just higher accuracy numbers. The researchers showed that their models developed a better understanding of the shapes of objects, rather than just relying on texture, which is a trait that aligns more closely with human vision. This suggests that by allowing machines to look at larger portions of an image at once, they can form a more robust and human-like understanding of the visual world. The success of LarKNAS indicates that the field of computer vision may be entering a new era where the design of the search space itself—the rules and tools used to find the best model—is just as important as the model it produces. By proving that large kernels can be searched for efficiently and effectively, the team has opened a new door for creating powerful, efficient, and intelligent visual systems without the prohibitive costs that have previously limited such exploration.

Technical Summary: Neural Architecture Search on Large Kernel Convolution Search Space (LarKNAS)

Problem Statement

Neural Architecture Search (NAS) has evolved to target large-scale models for computer vision tasks. However, existing NAS methods face performance bottlenecks when scaling up model parameters. The primary limitation identified is that current search spaces remain anchored in design paradigms from the 2010s, specifically relying on small convolution kernels (e.g., 3×3 or 5×5) and outdated cell-based or block-based structures derived from ResNet and MobileNet. While recent hand-designed ConvNets (e.g., ConvNeXt, RepLKNet) have demonstrated the efficacy of large kernels for establishing effective receptive fields (ERF), these designs have not been effectively integrated into automated NAS frameworks. Furthermore, existing large-scale NAS methods often incur prohibitive search costs (thousands of GPU days) and struggle with the computational instability and memory overflow associated with bilevel optimization when large kernels are introduced.

Methodology

The paper proposes LarKNAS (Neural Architecture Search on Large Kernel Convolution Search Space), a framework designed to discover large-scale convolutional backbones by rethinking the search space and the optimization algorithm.

1. The LarKSpace Search Space

LarKNAS introduces a modern search space, LarKSpace, specifically tailored for large-scale ConvNets. Key design elements include:

Phased Stage Architecture: The model is divided into four stages (N1–N4) to accommodate varying feature processing requirements.
- N1 (Early Stage): Uses small kernels (3×3) for low-level feature extraction.
- N2 & N3 (Middle Stages): Utilizes large kernels (21×21, 31×31, 51×51) to capture long-range dependencies.
- N4 (Final Stage): Employs medium kernels (7×7, 13×13).
ULarK Block: A unified module integrating prevalent modern designs (ConvNeXt, ExtraDW, IBDW, and FFN) into a single searchable structure. It combines Pointwise (PW) and Depth-wise (DW) convolutions to balance search complexity and representation capacity.
Sparse Large Kernels: To mitigate the quadratic growth of parameters and memory usage associated with dense large kernels, LarKNAS incorporates sparse convolution. The search space includes a sparsity degree parameter ( $s$ ) optimized via a dynamic pruning strategy (Algorithm 1) that adjusts sparsity based on gradient magnitude (SNIP) within computational constraints.
ConvFFN: A feature processing module similar to Transformers but implemented with convolutions, Batch Normalization (BN), and GELU activation, used to increase network depth and inter-channel computation.

2. Gradient Estimation Search Method

To avoid the memory overflow and instability of bilevel optimization (e.g., DARTS) when dealing with large kernels, LarKNAS employs a gradient estimation search method:

Mechanism: Instead of alternating gradient backpropagation between architecture and weight parameters, the method estimates the performance of candidate subnetworks within a single batch.
Optimization Objective: The algorithm selects subnetworks by maximizing the absolute mean of gradients ( $\mu_{\nabla}$ ) and minimizing the standard deviation of gradients ( $\sigma_{\nabla}$ ) across batches. High mean gradients indicate strong learning signals, while low standard deviation suggests stability.
Constraints: The search is constrained by FLOPs and GPU memory budgets to ensure feasibility.
Pipeline: The process involves constructing an initial architecture, performing the gradient estimation search to identify optimal architecture parameters (kernel size $K$ , block type $T$ , sparsity $s$ ), and finally retraining the discovered architecture from scratch.

Key Contributions

LarKNAS Framework: The proposal of a NAS method specifically designed to search for large-scale backbones using large convolution kernels, addressing the performance limitations of existing NAS methods that rely on small kernels.
LarKSpace Design: The introduction of a modern search space featuring phased stage architecture, a "small-medium-large" kernel range, optimized module combinations (ULarK Block), and searchable sparse large kernels.
Efficient Search Algorithm: The development of a gradient estimation search method that eliminates the need for expensive bilevel optimization, enabling the search of large kernel configurations within a reasonable computational budget.

Experimental Results

The authors evaluated LarKNAS on ImageNet-1K classification, COCO object detection, and ADE20K semantic segmentation.

Search Efficiency: LarKNAS achieved a 335M parameter model (LarKNASNet-L) with a search cost of 96 GPU days, which is only 36% of the cost required by ScaleNet for a similar scale.
ImageNet-1K Classification:
- LarKNASNet-L achieved 86.6% top-1 accuracy (448×448 input), outperforming ConvNeXt-B and Swin-B by approximately 1% and competing with MAE-ViT-H (87.8%) while using roughly half the parameters.
- LarKNASNet-B (384×384 input) achieved 85.4% accuracy, surpassing Swin-B and DeiT-B.
Downstream Tasks:
- COCO Object Detection: LarKNASNet-L achieved 55.2% box AP and 48.3% mask AP, outperforming Swin-L (pretrained on ImageNet-22K) by 1.3% and 1.6% respectively, despite not using ImageNet-22K pretraining.
- ADE20K Segmentation: LarKNASNet-L achieved 53.4% mIoU, matching ConvNeXt-L (ImageNet-22K pretrained) and outperforming Swin-L by 1.3%.
Ablation Studies:
- The "small-medium-large" kernel design significantly improved inference throughput compared to uniform large kernel designs.
- Sparse large kernels were shown to be essential for managing GPU memory and preventing search failure, whereas re-parameterization techniques caused Out-Of-Memory (OOM) errors during search.
- The gradient estimation search outperformed random and grid search baselines.

Significance and Claims

The paper claims that LarKNAS successfully pushes the boundaries of large-scale NAS by demonstrating that automated search can discover architectures comparable to, or surpassing, hand-designed large-kernel ConvNets. The authors emphasize that their work validates the large kernel search space as a viable and efficient direction for future NAS research.

Key claims regarding the significance of the work include:

Overcoming Bottlenecks: LarKNAS addresses the performance plateau of existing NAS methods by moving beyond 2010s-era small-kernel designs.
Efficiency: The gradient estimation method allows for the exploration of complex, large-kernel search spaces without the prohibitive memory costs of bilevel optimization.
Visual Properties: The discovered models exhibit larger Effective Receptive Fields (ERF) and a shape bias closer to human visual perception compared to previous NAS methods and ResNet-based models, suggesting that large kernels inherently improve the model's ability to capture global context and shape features.
Scalability: The method scales effectively to hundreds of millions of parameters (335M) with significantly reduced search costs compared to prior art.