Technical Summary: From Regression to Inference: Meta-Learning Predictors for Neural Architecture Search

Problem Statement

Neural Architecture Search (NAS) aims to automate the design of deep learning architectures, a task necessitated by the high dependency of model performance on architectural choices. However, the vastness of modern search spaces renders exhaustive evaluation computationally prohibitive. Prediction-based NAS has emerged as a solution, utilizing learned predictors to estimate the performance of candidate architectures without full training.

Despite their utility, existing predictors typically rely on supervised regression to learn a fixed mapping from architecture representations to performance metrics. This approach faces critical limitations in the NAS context:

1. Data Scarcity: NAS operates in a regime where only a tiny fraction of the search space can be observed.
2. Overfitting and Generalization: Fixed mappings trained on limited samples often fail to generalize to unseen architectures, leading to poor out-of-distribution performance.
3. Misalignment with Deployment: Conventional regression does not explicitly reflect the deployment scenario, where predictions must be conditioned on partial observations of the search space rather than a global fixed function.

Methodology

The authors propose a fundamental reformulation of NAS performance prediction, shifting from deterministic regression to conditional function inference within a meta-learning framework.

Core Framework: Convolutional Neural Process (ConvNP)

Instead of learning a static function $y = f(x)$, the predictor models the conditional distribution of target outputs given observed architecture–performance pairs (the context set). The authors employ a Convolutional Neural Process (ConvNP), which consists of:

• Encoder: Aggregates information from the context set into latent representations.
• Decoder: Predicts target outputs conditioned on these latent representations.
• Mechanism: The model learns a distribution over functions, enabling it to infer unseen outputs from partial observations while accounting for uncertainty. This leverages the translation equivariance of convolutional structures to extrapolate beyond the support of observed inputs.

Synthetic Task Construction

To train the ConvNP under a meta-learning paradigm, the authors generate a distribution of tasks from a single available dataset ($D$) via:

1. Resampling: Creating synthetic tasks by resampling the observed data with replacement.
2. Shuffling: Randomly ordering data points to encourage permutation invariance.
3. Context-Target Splitting: Each synthetic task is split exclusively into a context set ($C$) and a target set ($T$). This simulates the NAS inference scenario where the model must predict performance based on a subset of observed architectures.

Meta-Feature Extraction

The authors design a set of simple yet effective meta-features for cell-based architectures, focusing on three aspects:

• Statistical Information: Counts of specific operations (e.g., $3\times3$ Conv, $1\times1$ Conv) and nodes.
• Architectural Complexity: Path-based weights assigned to operations based on complexity (e.g., $3\times3$ Conv = 3, Max-pooling = 1), aggregated into measures like maximum/minimum path weights and total parameters.
• Structural Information: Metrics derived from the adjacency matrix, including edge counts, edge-to-node ratios, and max in/out degrees.
  These features are normalized to the $[0, 1]$ range.

Learning Objective

Recognizing that NAS prioritizes identifying high-performing architectures over perfect distributional prediction, the authors combine two objectives:

1. Regression Loss ($L_{MSE}$): Encourages accurate prediction of validation accuracy.
2. Ranking Loss ($L_{rank}$): A pairwise margin-based loss that enforces the correct relative ordering of candidate architectures in the target set.
   The total objective is $L_{total} = L_{MSE} + \lambda L_{rank}$. The authors note that likelihood-based objectives (standard in NPs) were found empirically to be less effective and efficient than direct mean prediction with ranking supervision for architecture selection.

Key Contributions

1. New Formulation: The paper redefines NAS prediction as a conditional function inference problem using ConvNP, moving away from fixed mappings to models conditioned on partial observations.
2. Meta-Learning Strategy: It introduces a strategy for generating synthetic tasks via resampling and shuffling, enabling the predictor to generalize under extreme data scarcity.
3. Evaluation Perspective: The authors argue that strong global ranking metrics (e.g., Kendall's tau) do not necessarily translate to better architecture selection. They advocate for selection-oriented metrics like Top-K performance.
4. Empirical Validation: Comprehensive experiments on NAS-Bench-101 and NAS-Bench-201 demonstrate the method's efficacy.

Experimental Results

The method was evaluated on NAS-Bench-101 (CIFAR-10) and NAS-Bench-201 (CIFAR-10, CIFAR-100, ImageNet16-120) against various baselines, including regression models (LR, RF, MLP, XGB), Graph Neural Network-based predictors (NP, CAP, CARL), and search algorithms (RS, RE).

• Predictive Accuracy: ConvNP achieved competitive or superior validation and test accuracies. On NAS-Bench-101, it reached 94.76% validation and 94.09% test accuracy, outperforming most regression baselines and search algorithms.
• Top-K Performance: While ConvNP did not always achieve the highest global rank correlation (Kendall's tau) compared to specialized ranking methods like CARL, it consistently demonstrated superior Recall@k. It identified high-performing architectures more effectively as $k$ increased, particularly in low-data regimes.
• Data Efficiency: Experiments varying training set sizes (from 50 to 860 samples) showed that ConvNP benefits significantly from increased data in low-data regimes, with performance gains saturating as data volume grows.
• Ablation Studies: Removing context-target splitting or synthetic task generation led to significant performance drops, confirming the necessity of the meta-learning formulation. Combining regression and ranking losses proved superior to using either alone.

Significance and Claims

The paper claims that its primary significance lies in aligning the training procedure of NAS predictors with the actual deployment scenario. By modeling prediction as inference from partial observations, the method addresses the inherent data scarcity of NAS more effectively than traditional supervised regression.

A key insight highlighted is that global ranking metrics are not always aligned with the practical goal of NAS. The authors demonstrate that optimizing for selection-oriented metrics (Top-K) yields more effective architecture selection, even if global rank correlation is slightly lower than that of specialized ranking models. The work suggests that future NAS predictors should prioritize probabilistic and meta-learning approaches that better reflect the conditional nature of the search process.

The authors acknowledge limitations, including slightly lower global ranking performance compared to specialized ranking predictors, sensitivity to synthetic task design, and additional computational overhead compared to simple regressors. They also note that their evaluation is currently limited to specific benchmark search spaces.