💻 computer science

Goal-Aware Adaptive Regulation Across Modalities: Evolutionary Architecture Search for Transformer Language Models on a Consumer AMD GPU

This paper demonstrates that the GaaR evolutionary architecture search framework, originally validated on convolutional networks, successfully transfers to autoregressive transformer language models on a consumer AMD GPU via DirectML, achieving competitive performance and revealing that regulated exploitation outperforms random search on structured landscapes while favoring wide-shallow architectures under fixed step budgets.

Original authors: Leon Sandler

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Leon Sandler

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a robot to speak, draw, or solve puzzles. To do this, you need to build a "brain" for it, which scientists call a neural network. But here's the tricky part: you don't know exactly how big that brain should be, how many layers it needs, or how to tune its settings. This is where Neural Architecture Search (NAS) comes in. Think of NAS as a super-organized, tireless robot architect that tries out thousands of different brain designs to see which one works best.

Usually, this job is done in massive, expensive data centers filled with powerful computer chips made by one specific company (NVIDIA), costing thousands of dollars in electricity and time. It's like trying to build a skyscraper using only a fleet of giant cranes. But what if you could do this on a regular laptop or a gaming computer you already own? That's the big question this paper tackles: Can we find the perfect AI brain design using just a standard, affordable graphics card from a different company (AMD), without needing those fancy, expensive data centers? The answer matters because it could let anyone, from a student in a bedroom to a researcher with a small budget, experiment with AI designs without breaking the bank or the planet.

The Story of the Budget-Friendly AI Architect

Meet GaaR (Goal-Aware Adaptive Regulation). Think of GaaR as a clever, single-file robot architect that lives on your computer. In a previous adventure, this robot successfully designed a vision system (a brain that "sees" images) using just a consumer-grade AMD graphics card. It found a design that was 94.36% accurate at recognizing images in 45 minutes, logging every single step in a simple local database.

Now, the author, Leon Sandler, asks: "Can this same robot architect handle a completely different job?" Instead of teaching the AI to see (like the previous image task), can it teach the AI to speak? This new job involves Transformer Language Models, the kind of technology that powers chatbots and text generators. These are very different from image brains; they predict the next word in a sentence rather than identifying objects in a picture.

The Experiment: A Tiny Shakespearean Play

To test this, the researcher set up a race on the same old AMD graphics card (an RX 5700 with 8 GB of memory). This card is special because it doesn't play nice with the usual "CUDA" software that most AI tools use; it needs a different path called DirectML. The robot architect had to navigate this path without any help from the usual big-data tools.

The task was simple but strict:

The Goal: Teach the AI to predict the next letter in a text of The Complete Works of William Shakespeare (specifically the "Tiny Shakespeare" version).
The Rules: The robot could only try out a design for a very short time—exactly 300 steps of learning per design. It couldn't just run forever; it had to be efficient.
The Variables: The robot could tweak six things about the AI's brain: how wide it is, how many "heads" (attention mechanisms) it has, how deep it is, how much text it can remember at once, how many examples it looks at at once, and how fast it learns.

The Results: Speed vs. Size

The robot went to work, running through eight generations of designs. In just 3 minutes and 16 seconds of real-world time, it found a champion design.

The Winner: The best design had 0.8 million parameters (a measure of brain size).
The Score: It reduced the "confusion" (validation loss) of the AI from 2.171 down to 2.021. In plain English, the AI got better at guessing the next letter in the story.
The Surprise: The robot didn't pick the biggest, deepest, most complex brain. Instead, it picked a wide but shallow brain with a short memory (context).

Why? Because of the Budget Effect. Since every design only got 300 steps to learn, the designs that could process those steps the fastest won. A giant, deep brain is slow to train; a wide, shallow brain is fast. On a budget of time, the robot learned that "fast and simple" beats "slow and complex." It's like a race where everyone has to run exactly 100 meters; the person who can sprint the fastest wins, even if they aren't the strongest runner.

The Reality Check: When Does the Robot Actually Win?

Here is the most honest part of the story. The researcher didn't just say, "The robot is a genius!" They tested it against a random guesser. They ran the race 10 times, comparing the robot's "greedy" strategy (always picking the best one so far and tweaking it) against someone just picking designs completely at random.

The Result: With only 8 tries (the small budget used in the main experiment), the robot and the random guesser performed equally well. The robot didn't beat the random guesser; they tied.
The Lesson: The robot's "smart" strategy only works when the problem is big enough and the budget is long enough to see patterns. On a tiny, smooth landscape with very few tries, being smart doesn't help much.
The Proof: To show the robot could be smart, the researcher tested it on a much bigger, complex problem (a physics simulation with 6,000 possible designs). There, the robot found the perfect solution 70% of the time, while the random guesser found it 0% of the time.

What This Means for You

This paper proves that cross-modal NAS (using the same search engine for different types of AI) works on cheap, consumer hardware. It shows that you can design a text-generating AI on a standard AMD graphics card in minutes, without needing a supercomputer.

However, it also sets a clear boundary: if you only have a tiny amount of time or money to test designs, a "smart" search strategy might not be better than just guessing randomly. You need enough room to let the strategy work its magic.

The author has shared all the code, the logs, and the database so anyone can repeat this experiment. The message is clear: AI research doesn't have to be locked behind expensive doors. With the right tools, a consumer-grade computer can be a powerful laboratory for discovering how to build smarter, more efficient AI brains.

Technical Summary: Goal-Aware Adaptive Regulation Across Modalities

Problem Statement
Neural Architecture Search (NAS) is traditionally dominated by data-center workloads requiring thousands of GPU-days and specific NVIDIA/CUDA hardware. This reliance excludes researchers without access to such infrastructure and ignores the vast installed base of capable consumer hardware, particularly AMD GPUs on Windows which lack official ROCm or CUDA support. While previous work established that evolutionary search could run on consumer hardware for computer vision (CNNs), it remained unproven whether such a lightweight, goal-aware engine could transfer to the distinct modality of autoregressive Transformer language models, which differ significantly in architecture, loss surfaces, and hardware pressure points.

Methodology
The study utilizes the GaaR (Goal-Aware Adaptive Regulation) framework, a single-file evolutionary search engine designed for local, resumable, and auditable execution on a single consumer device. The methodology involves transferring this engine to Transformer language modeling with minimal structural changes:

Engine Consistency: The core GaaR loop remains unmodified. It operates on a "exploit-best/explore-mutation" strategy: reading the highest-scoring genome as a parent, applying bounded mutations, instantiating the candidate phenotype, training it under a fixed budget, and logging results to a local SQLite database.
Phenotype & Genome: The phenotype is an unmodified, from-scratch PyTorch Transformer implementation (from the train-llm-from-scratch project). The genome consists of six genes: embedding width ( $n\_embed$ ), head count ( $n\_head$ ), block count ( $n\_blocks$ ), context length, batch size, and learning rate. A divisibility repair operator ensures $n\_embed$ remains a multiple of $n\_head$ .
Hardware & Portability: All experiments ran on a single AMD Radeon RX 5700 (8 GB) under Windows 10 using DirectML (via torch-directml). This device has no CUDA path and no official ROCm support. The setup operates in fp32 (mixed precision is unavailable) and handles operator fallbacks (e.g., AdamW's fused lerp falling back to CPU) without engine modification.
Fitness Function: Fitness is defined as the negative validation loss on the character-level Tiny Shakespeare corpus (vocabulary 65) after exactly 300 AdamW optimization steps. This "fixed step budget" was chosen to align with the original GaaR design, though the paper notes its implications for throughput.

Key Results

Search Trajectory: In an illustrative eight-generation search, the engine improved the character-level validation loss from a seed of 2.171 (perplexity 8.8) to a champion of 2.021 (perplexity 7.5) with a 0.8 M-parameter model. The entire search completed in 3 minutes and 16 seconds of wall time.
Throughput & Budget Effects: The search revealed a "budget effect" where the fixed-step constraint implicitly favored "wide-shallow, short-context" genomes. These architectures converted the fixed 300 steps into more effective learning due to higher step-throughput (up to 23.5 steps/s vs. 6.1 steps/s for deeper models).
Random Search Baseline: A controlled ten-seed comparison between GaaR's greedy rule and uniform random search at an eight-evaluation budget showed statistical parity (mean best validation loss: 2.047 for GaaR vs. 2.042 for random; 5 wins each). This indicates that on this specific landscape with such a small budget, the greedy rule offers no measurable advantage over random sampling.
Contrast with Structured Landscapes: Conversely, on a larger, structured 6,000-configuration physics-simulation landscape, the same greedy rule dominated random search, finding the exact global optimum in 70% of repeats compared to 0% for random search.

Significance and Claims
The paper claims that cross-modal NAS on commodity hardware is a practical foundation for reproducible, energy-modest architecture research. Its primary contributions are:

Modality Transfer: Demonstrating that the GaaR engine transfers from CNNs to Transformers with only a genome redefinition, running entirely on an unsupported AMD GPU via DirectML.
Portability Envelope: Defining the constraints of NAS on consumer AMD hardware (fp32-only, operator fallbacks) without requiring engine changes.
Honest Delimitation: Providing a controlled baseline that honestly defines the limits of the greedy exploitation rule. The paper argues that regulated exploitation pays off on larger, structured landscapes or with sufficient sequential depth, but offers no advantage over random search in trivially small budgets on smooth landscapes.
Reproducibility: Full release of code, generation logs, and configuration files to enable exact reproduction of the search process.

The authors emphasize that while the absolute quality of the resulting language model (perplexity 7.5) is not competitive with state-of-the-art models, the experiment successfully establishes feasibility and transferability. It proves that architecture discovery can be conducted end-to-end on a single consumer device in minutes, democratizing access to NAS for researchers without data-center resources.