SEAL: Searching Expandable Architectures for Incremental Learning

Technical Summary: SEAL (Searching Expandable Architectures for Incremental Learning)

Problem Formulation
Incremental Learning (IL) requires models to learn from a sequential stream of tasks while balancing plasticity (the ability to learn new information) and stability (the preservation of past knowledge). A primary challenge in this domain is catastrophic forgetting, where new learning overwrites previous representations. While Neural Architecture Search (NAS) has shown success in static settings, existing NAS-based IL approaches often suffer from high computational costs by repeating the search process for every task and from impractical resource consumption due to unbounded model expansion. Furthermore, many current methods rely on replay buffers or fixed expansion thresholds that do not adapt well to data-incremental scenarios, where disjoint data samples arrive sequentially without future access.

Methodology
The authors propose SEAL, a NAS framework designed specifically for data-incremental learning. Unlike prior methods that treat architecture design and expansion policy as separate or sequential processes, SEAL jointly optimizes both the neural network architecture and its expansion policy within a single search phase.

Key components of the SEAL framework include:

1. Unified Search Space: SEAL utilizes an OFA (Once-For-All) supernet based on MobileNetV3 to define the search space. This allows for the fast evaluation of candidate architectures via fine-tuning pre-trained weights rather than training from scratch. The search space is constrained to ensure that any expansion results in a valid OFA subnetwork.
2. Dynamic Expansion Policy: Instead of expanding the model at every task, SEAL employs a capacity estimation metric to determine when expansion is necessary. The model is expanded only if the current cross-entropy loss on new data exceeds a relative threshold compared to previous iterations (Eq. 10). This prevents unnecessary growth and retains model complexity.
3. Multi-Objective Optimization: The framework formulates the problem as a bi-objective optimization task solved via the NSGA-II genetic algorithm. The two objectives are:
   • Incremental Accuracy: A function balancing average accuracy on the final task set against the number of parameters (Eq. 12).
   • Robustness (Flatness): A rescaling-invariant flatness metric that measures the stability of the loss landscape under multiplicative noise perturbations (Eq. 14). This aims to guide the search toward flatter minima, which are more robust to distribution shifts.
4. Training Strategy: To mitigate catastrophic forgetting during expansion, SEAL employs cross-distillation training. After an expansion step, the model is trained using a combined loss function consisting of the standard cross-entropy loss and a Kullback-Leibler (KL) divergence loss against the soft targets of the pre-expansion model (Eq. 11).
5. Surrogate Modeling: To accelerate the search, SEAL uses adaptive-switching mechanisms (similar to MSuNAS) to select the most accurate surrogate model (among Gaussian Processes or Radial Basis Functions) for predicting performance metrics, avoiding the need for full retraining of every candidate.

Key Contributions

• Joint Optimization: SEAL is presented as the first NAS framework to jointly search for the optimal architecture and its expansion policy in a single phase, significantly reducing computational overhead compared to task-by-task search methods.
• Resource Efficiency: By integrating OFA subnetworks and conditional expansion, SEAL allocates additional capacity only when required, making it suitable for resource-constrained environments like IoT and TinyML.
• Flatness Integration: The work integrates a flatness-based robustness metric directly into the NAS objective for IL, demonstrating that optimizing for flat minima improves stability and generalization in sequential learning scenarios.
• Data-Incremental Focus: The framework specifically targets the data-incremental setting (disjoint data, same label space) without relying on explicit data storage or replay buffers.

Experimental Results
SEAL was evaluated on three benchmark datasets: CIFAR-10, CIFAR-100, and ImageNet16-120. The experiments compared SEAL against standard IL baselines (Naive, EWC, SI, LwF) and a Joint Training upper bound.

• Performance: SEAL achieved competitive or superior Average Accuracy (AA) compared to baselines. For instance, on CIFAR-10, SEAL achieved 95.35% accuracy, outperforming EWC (93.87%) and SI (94.31%). On CIFAR-100, it reached 83.94%, surpassing LwF (83.21%).
• Forgetting and Transfer: The framework demonstrated reduced Average Forgetting compared to several baselines (e.g., 4.81% on CIFAR-100 vs. 5.50% for LwF). However, on CIFAR-10, the Forward Transfer (FWT) was lower than some baselines, which the authors attribute to potential overfitting on the latest tasks when tasks are highly similar.
• Multi-Objective vs. Single-Objective: Results indicated that the multi-objective approach (optimizing for both accuracy and flatness) yielded better trade-offs than single-objective optimization. Specifically, models optimized for flatness tended to have fewer parameters while maintaining high accuracy.
• Ablation Studies: Experiments confirmed that initializing expanded weights from the OFA supernet (rather than random weights) significantly improved accuracy and reduced forgetting. Similarly, the use of cross-distillation loss was shown to be critical for maintaining performance and reducing forgetting after expansion.
• Statistical Significance: Friedman tests confirmed that the performance differences between SEAL and the baselines were statistically significant across all datasets and metrics.

Significance and Claims
The paper claims that SEAL offers a more practical solution for real-world deployment by aligning with key desiderata for NAS in IL: efficiency, scalability, and adaptability. By avoiding redundant retraining and unchecked model growth, SEAL addresses the pitfalls of current NAS-based IL approaches. The authors highlight that the integration of flatness as a design objective helps create models that are robust to the perturbations introduced by new tasks. While the study focuses on data-incremental learning, the authors suggest the framework could be extended to other IL settings, though they note that assumptions about architecture generalization across data splits may need adjustment for domain- or class-incremental scenarios. The work positions the combination of NAS and selective expansion as a promising direction for efficient, adaptive learning.