Technical Summary: Structuring Open-Ended NAS with Semi-Automated Design Knowledge

Problem Statement

Current Neural Architecture Search (NAS) methods face a fundamental trade-off between search space flexibility and exploration efficiency. Traditional NAS approaches rely on closed, predefined search spaces (e.g., combinations of fixed operators like 3x3 convolutions), which restrict the discovery of novel architectures. Conversely, recent Large Language Model (LLM)-assisted NAS methods enable "open-ended" search by directly generating model source code, allowing for the creation of new blocks and topologies. However, these open-ended approaches often suffer from inefficient exploration. They frequently rely on unstructured knowledge pools derived from LLM priors or raw paper mining, leading to:

1. Biased and Low-Quality Ideas: The search space may be contaminated by overly domain-specific, task-specific, or irrelevant design concepts (e.g., loss function design or 3D point cloud processing) that do not apply to visual backbones.
2. Inefficient Sampling: Without structured organization, the sampling of design ideas lacks diversity, often overlooking compact yet high-performing models or failing to explore promising regions of the Pareto frontier.
3. Hardware Constraint Violations: Naive mutation strategies often generate models that exceed hardware budgets (FLOPs, parameter count), leading to their immediate discard and a reduction in candidate diversity.

Methodology

The authors propose a two-pronged approach to address these inefficiencies: Semi-Automated Model Design Attribute Structuring and the FairNAD framework.

1. Semi-Automated Model Design Attribute Structuring

To curate high-quality, diverse design knowledge, the authors introduce a hierarchical Model Design Attribute Tree ($D_{attr}$).

• Structure: The tree is organized into three levels: Granularity (Operation, Block/Connectivity, Network), Main Category (e.g., Feature Extraction, Normalization), and Sub-Category (e.g., Grouped Convolution, Layer Normalization).
• Construction: The top two levels are predefined by experts to ensure broad coverage. The sub-attributes are generated semi-automatically by prompting an LLM with reference state-of-the-art (SOTA) model codes (e.g., ResNet, ViT, EfficientNet) and a predefined attribute template.
• Knowledge Extraction: This structured tree is used to filter and extract design ideas from academic papers. Papers are first screened to ensure they align with the attribute tree. Valid papers are then processed to extract fine-grained ideas, which are tagged with their corresponding attributes in $D_{attr}$. This process filters out irrelevant or overly specific knowledge, resulting in a curated Model Design Knowledge Database ($D_{kn}$).

2. The FairNAD Framework

FairNAD is an evolutionary search framework that utilizes the structured knowledge database to perform efficient open-ended Neural Architecture Design (NAD). It employs a multi-type mutation strategy and a feedback loop:

• Mutation with Fair Idea Sampling: To ensure unbiased exploration, ideas are sampled uniformly from $D_{kn}$ based on their attribute tags (granularity, main category). This prevents the search from being dominated by popular but potentially suboptimal trends (e.g., excessive attention mechanisms). The granularity tag also dictates the mutation target (e.g., modifying an operation vs. the network structure).
• Pareto-Aware Mutation: To address the issue of overlooking compact models, this stage samples parent models from the Pareto frontier (balancing accuracy and hardware constraints). It employs a dual strategy:
  • Upscaling: Models below the hardware budget threshold are scaled up (increasing width/depth).
  • Hyperparameter Adjustment: Models near or exceeding the threshold undergo hyperparameter tuning (e.g., kernel size, stride).
• LLM-Driven Iterative Mutation: To accelerate convergence, the framework performs iterative refinement on high-performing models. The LLM analyzes the history of successful mutations and generates new ideas within similar attribute categories to further optimize promising candidates.
• Feedback Loop: A critical component for maintaining diversity and validity. Generated code undergoes three verification stages:
  1. Execution Verification: Ensures code is compilable and executable; if errors occur, the LLM is prompted to debug.
  2. Budget Verification: Checks FLOPs and parameter size. If constraints are violated, the model is automatically downscaled (e.g., reducing depth or width) rather than discarded.
  3. Structural Verification: Ensures the model is multi-layered and structurally distinct from the parent.

Key Contributions

1. Semi-Automated Knowledge Structuring: The paper introduces a method to organize unstructured design knowledge from papers into a hierarchical attribute tree. This allows for the filtering of noisy, domain-specific ideas and the creation of a diverse, high-quality knowledge base for NAS.
2. FairNAD Framework: A novel search framework that integrates multi-type mutations (Fair Idea Sampling, Pareto-Aware, and LLM-Driven Iterative) with a feedback loop. This approach balances exploration (diversity) and exploitation (performance) while respecting hardware constraints.
3. State-of-the-Art Performance: The method demonstrates significant improvements over existing hand-crafted and LLM-based NAS methods on standard benchmarks.

Experimental Results

The authors evaluated FairNAD on the NAS-Bench-201 benchmark using CIFAR-10, CIFAR-100, and ImageNet16-120 datasets. The search was constrained to specific parameter counts (1.5M) and FLOPs limits.

• Performance Gains: Compared to the current state-of-the-art open-ended NAS method (NADER), FairNAD achieved test accuracy improvements of:
  • 0.84% on CIFAR-10
  • 2.17% on CIFAR-100
  • 2.35% on ImageNet16-120
• Ablation Studies:
  • Structured Knowledge: Using structured inspiration (vs. raw paper inspiration or LLM priors) improved test accuracy by 0.37 points on CIFAR-100.
  • Fair Sampling: Implementing fair idea sampling improved accuracy by 1.81 points, highlighting the importance of diverse attribute exploration.
  • Multi-Type Mutation: The combination of Pareto-aware and LLM-driven iterative mutations yielded the best results. Notably, LLM-driven mutation alone was detrimental, confirming the need for the diversity provided by Pareto sampling.
  • Feedback Loop: Removing the budget verifier significantly degraded performance and pass rates, demonstrating its role in preserving diverse candidates that would otherwise be discarded.

Significance and Claims

The paper claims that FairNAD establishes a crucial foundation for future research in open-ended NAS by demonstrating that structuring design knowledge is essential for efficient exploration. The authors argue that simply relying on the "open-ended" capability of LLMs is insufficient; the search process must be guided by curated, hierarchical knowledge and balanced by multi-stage mutation strategies.

The discovered architectures exhibit unique characteristics not typically found in standard NAS, such as:

• Customized attention mechanisms (e.g., DPCA using both average and max pooling).
• Dynamic feature sharing mechanisms (e.g., FeaturePool).
• Non-standard channel sizes and layer depths (e.g., using 51, 102, 204 channels instead of powers of two).
• Hybrid approaches combining CNNs with positional encodings or specialized activation functions (e.g., BandLimitedReLU).

The authors position this work as a step toward reducing the reliance on manual expert design and enabling the discovery of more powerful, efficient, and novel neural architectures by leveraging external knowledge in a structured, systematic manner. They acknowledge the high computational cost of the search but frame it as a necessary investment to achieve performance levels unattainable by conventional methods.