🤖 machine learning

Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs

This contribution introduces Delta-based Neural Architecture Search, a token-efficient method in which fine-tuned LLMs generate compact code diffs to refine base models, thereby achieving significantly higher validity and accuracy rates across multiple datasets compared to traditional full model synthesis, while drastically reducing output lengths.

Original authors: Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov

Published 2026-05-07

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Santosh Premi Adhikari, Radu Timofte, Dmitry Ignatov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to improve a complex machine, such as a car engine.

The Old Way (Full Generation):
In the past, when researchers used AI to design better neural networks (the "brains" of AI), they asked the AI to write the entire operating manual for a brand-new engine from scratch every single time.

The Problem: This is like asking an author to retype an entire book just to change a single comma. It requires an enormous amount of time, consumes a massive amount of computing power, and often leads to messy, redundant code. If the AI makes a small mistake in the middle, the entire engine fails.

The New Way (Delta-based Search):
This work introduces a smarter approach called Delta Code Generation. Instead of writing an entirely new manual, the AI is asked to create a tiny "patch" or "diff" (a list of specific changes) to repair an existing, functioning engine.

The Analogy: Imagine using the "Track Changes" feature in a word processor. Instead of rewriting the entire document, you simply highlight the specific sentence you want to correct and say, "Change this word to that."
The Result: The AI writes only about 30 to 50 lines of code (the patch) instead of 200+ lines (the entire manual). This saves approximately 75–85% of computing power.

How They Tested It

The researchers treated this like a cooking competition with three different AI chefs (all "7B" class models, which are like very intelligent but not oversized assistants):

DeepSeek-Coder (A chef specialized in programming).
Qwen2.5-Coder (Another programming specialist).
Mistral-7B (A general-purpose chef not specifically trained only for code).

They asked these chefs to improve recipes (neural network architectures) for six different types of dishes (datasets such as CIFAR-10, MNIST, CelebA, etc.), ranging from simple digits to complex faces.

The Results

Efficiency: The "patch" method was incredibly efficient. The AI chefs produced much shorter, cleaner instructions.
Performance: Surprisingly, the chefs who only wrote patches performed just as well (and often better) as the chefs who wrote full manuals.
- In a standard test (CIFAR-10), the patch chefs achieved accuracy rates of about 85%, while the old "full manual" method reached only about 64%.
- Even the general-purpose chef (Mistral), who was not a programming specialist, performed just as well as the programming specialists. This proves that the method of writing patches is the secret, not just the specific AI model used.
Reliability: The "patch" method was far more reliable. About 75% of the patches worked perfectly, whereas the old method worked only about 50% of the time.

Why This Matters

The work argues that this "Delta" approach is a turning point because:

It is cheaper: Far less computing power (tokens) is needed to achieve better results.
It is flexible: It works across many different types of problems (datasets) without requiring the AI to be retrained for each individual one.
It is smarter: By building upon existing, functioning code, the AI does not need to "reinvent the wheel" every time; it simply focuses on making the necessary improvements.

In short, the work shows that fine-tuning an existing solution is often better, faster, and more reliable than trying to build a new one from scratch.

Technical Summary: Delta-based Neural Architecture Search via Code-Diffs

Problem Statement

Neural Architecture Search (NAS) aims to automate the design of deep neural networks. Although recent approaches leveraging Large Language Models (LLMs) to generate executable PyTorch code are promising, they suffer from a critical limitation: the Full-Generation Paradigm. Existing methods require the LLM to synthesize complete model implementations from scratch at every iteration. This approach is computationally expensive, produces verbose outputs (typically 200+ lines of code), and fails to utilize the structural knowledge embedded in existing, functioning architectures. Furthermore, previous LLM-based NAS work has been largely restricted to evaluations on single datasets (e.g., CIFAR-10) or limited multi-dataset scopes, constraining the assessment of generalization capabilities.

Methodology: Delta-Code Generation

The authors propose Delta-Code Generation, a paradigm where fine-tuned LLMs generate compact unified Diffs (Deltas) to refine existing base architectures, rather than synthesizing entire models. This approach draws inspiration from software engineering practices where code is modified via patches instead of being rewritten from scratch.

Core Pipeline

The framework operates via an iterative 22-cycle protocol:

Base Architecture Selection: The system samples base architectures from the LEMUR dataset, a curated collection of neural networks with performance annotations across multiple datasets.
Delta Generation: The LLM receives the source code of the base architecture and a dataset descriptor. Instead of outputting a complete file, it generates a structured XML output containing:
- <delta>: A unified Diff in standard patch format (context lines, additions, deletions).
- <hp> and <tr>: Structured tags for hyperparameters and data transformations (following NNGPT conventions), although the evaluation utilizes the original hyperparameters of the base architecture to isolate architectural changes.
Application and Validation: The generated Diff is applied to the base code using standardized patch utilities. Successful patches undergo syntax validation, instantiation checks, and training over one epoch.
Filtering: To maintain structural diversity and quality, admitted architectures must pass two filters:
- Accuracy Filter: The model must achieve at least 40% Top-1 validation accuracy after one epoch.
- Novelty Filter: Using MinHash-Jaccard Similarity, the generated code must exhibit a structural distinctness of at least 90% ( $\tau_{nov} = 0.90$ ) from the existing corpus to prevent mode collapse.
Fine-tuning: The LLM is iteratively fine-tuned using LoRA (Low-Rank Adaptation) on the static, curated LEMUR corpus. Crucially, the training data for LoRA remains fixed; the expanding pool of generated architectures serves exclusively as a diverse set of base architectures for future Delta generation, not as training data for the LLM itself.

Experimental Setup

The study evaluates three 7B-class LLMs across six diverse image classification datasets (CIFAR-10, CIFAR-100, MNIST, SVHN, ImageNette, CelebA):

DeepSeek-Coder-7B-Instruct-v1.5 (Code-specialized)
Qwen2.5-Coder-7B-Instruct (Code-specialized)
Mistral-7B-Instruct-v0.3 (General instruction, not code-specialized)

All models undergo 22 fine-tuning cycles with 50 candidates per cycle (totaling 1,100 candidates per LLM). Performance is measured via first-epoch validation accuracy as a proxy, consistent with prior LLM-based NAS literature.

Key Results

1. Efficiency and Output Length

The Delta paradigm achieves a massive reduction in token consumption. Across all three LLMs, output lengths are uniformly compact:

DeepSeek: 30.4 lines (85% reduction vs. Full Generation)
Qwen: 31.4 lines (84% reduction)
Mistral: 49.5 lines (75% reduction)
This represents a 75–85% reduction compared to the typical ~200+ lines of Full-Generation baselines, corresponding to an aggregated saving of approximately 5.4x output tokens.

2. Generation Reliability and Accuracy

Under a unified cluster protocol with a balanced pool of base datasets across six datasets, all three LLMs significantly outperform the Full-Generation baseline (Khalid et al. [24]) and the iterative approach with a frozen LLM (Gu et al. [13]):

Valid Generation Rate: DeepSeek (75.3%), Qwen (72.1%), Mistral (66.6%) vs. Full Generation (50.6%).
Mean First-Epoch Accuracy: DeepSeek (65.8%), Qwen (64.6%), Mistral (66.1%) vs. Full Generation (42.3%).
Best First-Epoch Accuracy (CIFAR-10): Mistral (85.5%), DeepSeek (85.2%), Qwen (80.6%) vs. Full Generation (63.98%) and Gu et al. (71.5%).

3. Generalization Across Multiple Datasets

In contrast to prior work restricted to CIFAR-10 or three datasets, the Delta approach successfully generalizes across all six datasets, including challenging fine-grained classification (CIFAR-100) and facial attributes (CelebA). All three LLMs achieved best first-epoch accuracies of 99.5% on MNIST and ~95% on SVHN.

4. Model Agnosticism

Performance gains are driven by the Paradigm (Delta Generation) rather than the specific choice of LLM. Despite differences in pre-training (code-specialized vs. general instruction), the three models cluster tightly across all primary metrics with overlapping confidence intervals. Even the general Mistral-7B, which generates slightly longer diffs, achieves the highest CIFAR-10 accuracy (85.5%).

Significance and Claims

The work positions Delta-based Neural Architecture Search as a token-efficient, multi-domain, and LLM-agnostic alternative to full-model synthesis. The authors present three primary claims regarding the significance of their work:

Paradigm over Model: The shift from full-file generation to diff-based generation is the primary driver of performance gains. This is evidenced by the fact that swapping the output format for the same base LLM (DeepSeek-Coder-7B) and the same iterative protocol shifts the valid generation rate outside the confidence interval of the Full-Generation baseline.
Cross-Dataset Coverage as a Design Dimension: By operating on Diffs, the pipeline preserves the functioning, dataset-specific scaffolding structure of the baselines, enabling generalization across six diverse datasets without retraining the LLM per domain. This contrasts with prior work that evaluated almost exclusively on CIFAR-10.
Asymptotic Efficiency: The 75–85% reduction in output length accumulates over iterations, making the approach significantly more practical for local deployment on consumer GPUs, where LLM inference constitutes the dominant cost.

The authors note that all accuracy claims rely on the proxy of first-epoch validation accuracy, which they validate via a 50-epoch full-training study on the top-20 architectures. This study confirms that the 1-epoch proxy reliably preserves performance rankings (Spearman $\rho = 0.926$ for Mistral), placing the method at the upper end of NAS proxy correlations reported in the literature.

The work concludes that while the Diff format introduces a unique failure mode of "context incongruence," the overall benefits regarding efficiency, structural integrity, and multi-domain generalization establish Delta-based generation as a robust advancement in LLM-driven NAS.