Long-Horizon Autonomous Architecture Research with a Language-Model Agent: A Behavioural Case Study

The Digital Lab Rat: When AI Tries to Be a Scientist Alone

Imagine a world where computers don't just follow instructions but are asked to figure out how to solve a puzzle on their own. This is the realm of Machine Learning, a branch of science where we teach computers to learn from data instead of hard-coding every rule. Usually, when we want a computer to get better at a task—like recognizing cats in photos or predicting the weather—we use a method called Neural Architecture Search. Think of this like trying to build the perfect Lego tower. You have a box of bricks (the computer's code), and you want to stack them in a way that makes the tower stand the tallest and strongest. Traditionally, humans have designed the rules for how to pick the bricks, or we've used simple math to guess which combinations might work best.

But what if we handed the entire box of Legos to a super-smart robot and said, "You figure it out. Build the best tower you can, and don't stop until you're done"? That's the big question this paper explores. It asks: If we give a single Artificial Intelligence (AI) agent all the tools a human researcher needs—like a notebook, a library, and a computer to run tests—can it act as a lone scientist for weeks at a time? We care about this because as AI gets smarter, we want to know if it can help us discover new things faster, or if it will just get stuck in a loop of making tiny, boring changes. The answer isn't just about building better towers; it's about understanding how to build a team where humans and robots work together without the robot getting bored or confused.

---

The Experiment: A Robot Researcher on a Long Journey

In this study, the researchers set up a digital laboratory and handed the keys to a single, general-purpose AI. They didn't just ask the AI to solve a math problem; they asked it to design a new type of computer brain called a Vision Transformer. Specifically, they gave it a tricky constraint: the brain had to work using only "channel attention" (focusing on the colors and textures of an image) and was forbidden from using "spatial attention" (looking at where things are in the image). This was like asking a chef to cook a gourmet meal without ever using a knife or a stove, forcing the AI to invent a completely new way of cooking.

The AI was given a specific set of tools: a way to manage its code, a tracker to watch its experiments, a persistent memory to remember what it tried, and, halfway through, access to a digital library of scientific papers. The human researchers stepped back, only intervening to change the rules of the game or to nudge the AI when it got stuck. The AI was left to run for about 10 weeks of real-world time, submitting roughly 150 experiments and using about 2,400 GPU-hours of computer power. It was a marathon, not a sprint.

The Story of Three Phases

The experiment unfolded in three distinct chapters, each with its own mood and outcome.

Phase 1: The Golden Hour and the Wall
At the start, the AI was like a kid in a candy store. It made rapid, massive improvements. In just the first six guesses, it discovered a fundamental trick that boosted its accuracy by a huge amount. It was so effective that this single early idea accounted for about 80% of all the progress the AI ever made on the small test dataset. But then, the fun stopped. The AI hit a "saturation wall." It kept trying to tweak its settings—changing numbers here, adjusting knobs there—but nothing worked. It was stuck in a loop of making tiny, incremental changes that didn't help. It was as if the robot had run out of new ideas and was just spinning its wheels.

Phase 1b: The Library Opens
The human researchers noticed the robot was stuck. They didn't change the robot's brain; instead, they gave it a new tool: access to a library of scientific papers. Suddenly, the AI's behavior changed. It stopped just tweaking numbers and started making bold changes to the actual code, inspired by what it read. It began to propose new structures rather than just adjusting old ones. This suggests that the robot wasn't "stupid"; it just needed a different kind of key to unlock the next level of thinking.

Phase 2 and 3: Scaling Up and Stumbling
As the researchers moved the AI to harder and harder tests (from small pictures to massive, complex images), the pattern repeated. The AI would find a good solution, then get stuck trying to make tiny improvements to that specific solution. It became very "greedy," meaning it loved its current best idea so much that it refused to try anything completely different, even when that idea was clearly failing. For example, on the hardest test, the AI found a solution that worked okay, then spent 18 tries making tiny, useless tweaks to it, ignoring the fact that it was stuck. It was like a driver who found a flat tire, decided to keep driving, and then spent hours trying to polish the tire instead of changing it.

What the Robot Learned (and What It Didn't)

The AI was surprisingly good at some things. It independently "rediscovered" several well-known scientific facts. For instance, it figured out that for this specific type of computer brain, having just one "head" to focus on details was better than having many heads—a finding that matches what human scientists already knew. It also realized that some tricks that worked on small images actually hurt performance on big ones, showing it could learn from its mistakes.

However, the AI also showed some funny human-like flaws.

• Risk Aversion: After it tried a bold, risky idea that failed, it became very timid. It would stick to safe, boring ideas for a long time, even though the rules allowed it to try something crazy again.
• Success Chasing: It loved to keep doing what had worked recently. If it found a way to make the computer brain slightly faster, it would keep trying to make it even slightly faster, ignoring the possibility that a totally different approach might be better.
• The "Greedy" Trap: The biggest issue wasn't that the AI was bad at thinking; it was that the rules of the game forced it to be greedy. The system was set up so that the AI could only keep an idea if it was better than the one before it. This is like a game where you can only move forward if you take a step up a hill, but you can never step sideways or go back to try a different path. The AI followed the rules perfectly, which meant it got stuck on local hills instead of finding the mountain peak.

The Big Takeaway: It's the Workflow, Not the Brain

The most important finding of this paper is a bit of a surprise. The researchers concluded that the workflow (the rules and tools given to the AI) was just as important, if not more so, than the AI's own intelligence.

The AI wasn't failing because it was "dumb." It was failing because it was playing a game designed for short sprints, not long marathons. The "commit-or-discard" rule (keep it if it's better, throw it away if it's worse) forced the AI to act like a greedy climber who only looks at the step right in front of them. If the researchers had designed the game differently—allowing the AI to keep a few different ideas running at once, or forcing it to try a "moonshot" idea every few turns—the results might have been totally different.

In short, the paper suggests that to make AI a great research partner, we shouldn't just try to make the AI smarter. We need to build better "workflows" that encourage it to take risks, explore different paths, and not get stuck on its current favorite idea. The AI is a tireless, capable worker, but without the right instructions and tools, it can easily get stuck in a loop of doing the same thing over and over again. The future of AI research, the authors suggest, lies in designing better systems to guide these digital minds, rather than just hoping they will figure it out on their own.