🤖 machine learning

LLM-Driven Neural Network Generation with Same-Family Architecture Guidance: Disentangling Transfer and Adaptation

This paper proposes a source-guided protocol where Large Language Models (LLMs) generate valid neural network modifications by leveraging a stronger model from the same family, demonstrating significant accuracy improvements over non-source controls and confirming that the LLMs effectively adapt rather than simply copy source architectures.

Original authors: Kabir Dev Paul Baghel, Radu Timofte, Dmitry Ignatov

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kabir Dev Paul Baghel, Radu Timofte, Dmitry Ignatov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Hiring a Mentor to Fix a Weak Student

Imagine you have a student (a Weak Target Model) who is struggling to pass a test. You want to help them improve. You have two options:

Ask a smart AI (a Large Language Model or LLM) to guess new study tips or change the student's study schedule from scratch.
Ask that same AI to look at a Star Student (a Strong Source Model) who already aced the test, and ask the AI to figure out how to adapt the Star Student's success to help the struggling student.

This paper asks a specific question: Does looking at the Star Student actually help the AI give better advice than if the AI just guessed on its own?

The researchers found that, in many cases, yes, it does. But it's not magic; it depends heavily on how the AI uses that information.

The Experiment: The "Four-Arm" Protocol

To make sure the results were fair, the researchers set up a controlled experiment with four different "arms" (groups). Think of this like a cooking competition where every chef gets the same amount of time and ingredients, but they are given different instructions:

The "Guesswork" Group (Non-Source Controls): The AI tries to improve the weak student's recipe by changing the learning rate or batch size, but it doesn't know about the Star Student. It's just guessing based on general knowledge.
The "Copy-Paste" Group (hp Copy): The AI takes the Star Student's exact recipe (learning rate, momentum, etc.) and forces it onto the weak student. No thinking, just copying.
The "Adaptation" Group (Source-Guided): The AI looks at the Star Student's recipe and the weak student's current state, then rewrites the recipe to fit the weak student. It adapts the advice.
The "Architecture" Group: The AI tries to change the actual structure of the student's brain (the neural network layers), not just the study habits.

The Results: When Does It Work?

The paper found that the "Adaptation" group usually wins, but the way it wins changes depending on the situation.

1. The "Recipe Transfer" Scenario (Like CIFAR-10)

In some cases, the Star Student's recipe is already very good and fits the weak student perfectly.

Analogy: Imagine the Star Student uses a perfect study schedule. When you copy that schedule directly to the weak student, they immediately get much better grades. The AI's job is just to make sure the schedule fits the student's desk.
Result: The "Copy-Paste" method worked well here, but the "Adaptation" method did even better because the AI tweaked the details slightly to squeeze out extra points.

2. The "Recipe Adaptation" Scenario (Like SVHN)

In other cases, the Star Student's recipe is actually bad for the weak student if you just copy it.

Analogy: The Star Student is a marathon runner training for a 26-mile race. The weak student is a toddler. If you give the toddler the marathon runner's exact training plan (run 20 miles a day), the toddler will fail.
What happened: When the researchers just "Copy-Pasted" the Star Student's recipe to the weak student, the results were terrible (the toddler collapsed).
The Fix: The AI looked at the Star Student's plan, realized, "Oh, this is too intense for the toddler," and rewrote the plan to be manageable. The AI didn't copy; it translated the success into something the weak student could actually handle. This resulted in a massive jump in performance.

Key Findings in Plain English

It's not just about copying: The AI isn't just a photocopier. Sometimes it has to be a translator. If the source is too advanced, the AI must "dumb it down" or adjust it to work for the weaker model.
Family matters: The method works best when the Star Student and the Weak Student are "cousins" (same architecture family, like both being AlexNet). If they are too different, the advice doesn't transfer well.
Validity is key: A huge part of the challenge is just making sure the AI's code actually runs. Sometimes the AI writes code that crashes. The researchers found that using the Star Student as a guide actually helped the AI write more valid (working) code, not just better code.
One epoch vs. Five epochs: The researchers tested the models quickly (1 epoch of training) to see who was the fastest learner. The winners in the quick test usually stayed winners when they trained longer (5 epochs), proving the improvement was real.

The Bottom Line

The paper proves that using a "Stronger Sibling" to guide an AI in fixing a "Weaker Sibling" is a powerful strategy.

However, the AI needs to be smart enough to know when to copy and when to adapt.

If the source is compatible, the AI transfers the knowledge.
If the source is too advanced, the AI adapts the knowledge to fit the weaker model.

The paper concludes that this "Source-Guided" approach is a reliable way to boost weak models, provided you compare it fairly against models that don't have access to the Star Student's secrets.

Technical Summary: LLM-Driven Neural Network Generation with Same-Family Architecture Guidance

1. Problem Statement

The paper addresses the limitations of using Large Language Models (LLMs) for unrestricted neural network generation, which often yields invalid code or low-quality models. The specific scientific question posed is: Given a weak target model, does access to a stronger, same-family source model enable an LLM to generate better valid candidates than a non-source baseline, under an equal evaluation budget?

The study aims to disentangle two potential effects:

Non-source baseline effect: Improvements arising simply from the LLM modifying standard training parameters (e.g., learning rate, batch size) without external guidance.
Source-guided effect: Improvements specifically attributable to the LLM leveraging the retrieved source model's architecture, hyperparameters, and training recipes.

2. Methodology

The authors propose a source-guided candidate-generation protocol that treats the process as a controlled algorithm. The setup involves a weak target model $T$ and a stronger same-family source model $S$ (where $a(S) > a(T)$ ) retrieved from the LEMUR neural network database.

Experimental Arms:
The protocol compares four distinct arms, grouped into non-source controls and source-conditioned arms:

Non-Source Controls:
- hp default: Non-source hyperparameter/transform control using only the target.
- baseline edit: Non-source structured edit using only the target.
Source-Conditioned Arms:
- hp transfer: The target architecture remains fixed. The LLM is exposed to the source's training recipe (hyperparameters and transforms) and generates a new recipe for the target.
- analogical edit: The LLM performs source-conditioned structured architecture edits.

Validation and Repair:
To ensure fair comparison, the study employs minimal deterministic validation. Both arms receive identical cleanup: schema parsing, removal of unsupported hyperparameter keys, and dataset-compatible transform guards. The protocol explicitly excludes broader semantic repairs, fallback candidates, or accuracy-based rewriting to isolate the generation capability.

Evaluation Metrics:
The paper reports metrics separately to avoid conflating code-generation reliability with model quality:

Validity Rate: The percentage of candidates that parse, train, and evaluate successfully.
Best Valid Accuracy: The highest accuracy achieved by any valid candidate in a group.
Mean Valid Accuracy: The average accuracy of all valid candidates.
Source-Guided Advantage: The difference between the best source-guided accuracy and the best non-source accuracy.

3. Key Contributions

Protocol Definition: A controlled candidate-generation protocol featuring non-source controls, source-conditioned candidates, and a no-LLM hp copy ablation. The ablation directly copies source hyperparameters to the target without LLM intervention to distinguish between recipe transfer and LLM adaptation.
Mechanistic Decomposition: Evidence for two distinct regimes:
- Recipe-Transfer: Where the source recipe is inherently compatible with the target, and direct copying yields significant gains.
- Recipe-Adaptation: Where the source recipe requires LLM-mediated modification (e.g., adjusting learning rates for a weaker model) to be effective.
Architecture-Family Analysis: Reporting win rates at the architecture-family level (e.g., AlexNet, alt nn1) rather than just dataset level, identifying where source guidance is reliable.

4. Experimental Results

Experiments were conducted on datasets including CIFAR-10, SVHN, CIFAR-100, Imagenette, and CelebA-Gender using various LLMs (DeepSeek-Coder, Qwen2.5-Coder, OlympicCoder).

CIFAR-10 (Weak Target $a=0.1254$ ):
- The strongest source-guided candidate (DeepSeek-Coder-6.7B with archbest source) reached 0.5049 accuracy, compared to 0.2398 for the best non-source candidate.
- This represents a +0.2651 advantage.
- A five-epoch re-evaluation preserved the gain, with source-guided reaching 0.7686 versus 0.4839 for non-source.
- Source-guided candidates won best valid accuracy in 4/6 settings and mean valid accuracy in 6/6 settings.
SVHN AlexNet (Weak Target $a=0.1959$ ):
- With DeepSeek-Coder-6.7B, source-guided transfer reached 0.7880, while the best non-source candidate reached 0.2254 (+0.5626 advantage).
- A fresh independent repeat confirmed these results (0.8069 vs. 0.2509).
- Ablation Insight: Directly copying the source recipe (hp copy) resulted in 0.1959 (matching the original weak target), whereas the LLM-adapted recipe (hp transfer) reached 0.7880. This indicates a recipe-adaptation regime where the LLM successfully adjusted the source's high-performance learning rate (too small for the weak target) to a suitable value.
Family-Level Performance:
- AlexNet: 6/8 wins across SVHN, Imagenette, and CelebA-Gender.
- alt nn1: 8/10 wins on CIFAR-10.
- Other Architectures: Results were mixed or negative for BagNet, DPN68, and AirNext on SVHN, and gains were small or unstable on CIFAR-100 and Imagenette (non-AlexNet).
Failure Modes:
- The primary failure mode was validity (e.g., spatial dimension collapse in small-image architectures, unsupported hyperparameter keys) rather than just low accuracy.
- Datasets like MNIST were too easy (saturation), and CelebA-Gender saturated near 0.96 accuracy for all arms.

5. Significance and Claims

The paper claims that source-guided candidate generation is a viable search strategy that can outperform non-source baselines, but its effectiveness is highly dependent on the specific architecture family, dataset, and LLM used.

Disentangling Mechanisms: The study demonstrates that source guidance does not merely copy recipes. In some cases (CIFAR-10), the source recipe is transferable, and the LLM adds marginal gains. In others (SVHN AlexNet), the source recipe is incompatible with the weak target, and the LLM's ability to adapt the recipe is the primary driver of success.
Practical Utility: The most practical application is a hybrid approach: evaluate both source-guided and target-only arms under the same budget and retain the best valid candidate only if it improves the original target.
Scope Limitations: The authors modestly note that the strongest results are concentrated on specific regimes (CIFAR-10 and SVHN AlexNet). The paper avoids overstating architectural discovery, noting that the primary mechanism observed is hyperparameter/transform transfer rather than novel architecture design. The results are defined at the protocol level, acknowledging that exact bitwise reproducibility is not guaranteed due to non-deterministic training environments.