Technical Summary: Towards Execution-Grounded Automated AI Research

Problem Statement

Automated AI research aims to leverage Large Language Models (LLMs) to generate, implement, and verify research ideas at scale, potentially converting compute into scientific discovery through recursive self-improvement. However, a critical bottleneck exists: frontier LLMs frequently generate ideas that appear plausible but fail upon execution by human researchers or automated systems. While "execution grounding" (learning from the actual results of running code) is a promising solution, it remains unclear whether automated execution is feasible for open-ended research problems and whether LLMs can effectively learn from execution feedback to improve idea generation.

Methodology

1. Automated Idea Executor System

The authors constructed a high-throughput automated executor system designed to implement natural language research ideas, run experiments, and return performance metrics. The system consists of three core components (Figure 1):

• Implementer: A server that takes natural language ideas and a baseline codebase, prompting a code-execution LLM to generate a code diff. It employs a self-revision loop (up to 2 attempts) to ensure the diff can be successfully patched into the baseline.
• Scheduler: A middleware layer that manages resource allocation, downloading codebases from the cloud, and preparing job configurations based on environment requirements.
• Worker: A GPU cluster that executes the experiments. Upon completion (or failure), it uploads logs, metrics, and metadata to a cloud bucket for analysis.

2. Research Environments

To test the system on realistic, GPU-intensive problems, two environments were defined:

• Pre-Training (nanoGPT): Based on the nanoGPT speedrun. The goal is to minimize the wall-clock time to reach a target validation loss (3.28) on the FineWeb corpus using a 124M Transformer. The system uses a proxy reward (reciprocal of loss) for search and RL to fix training time budgets, reporting actual time only for top solutions.
• Post-Training (GRPO): Based on the GRPO algorithm for math reasoning. The goal is to maximize validation accuracy on the MATH dataset for a 1.5B model (Qwen2.5-Math) within a fixed time budget.

3. Learning Algorithms

The study evaluates two methods for learning from execution feedback:

• Execution-Guided Evolutionary Search: An optimization method that blends exploration and exploitation. In each epoch, the system samples a batch of ideas, executes them, and uses the results to guide the next generation. It maintains a pool of "positive trajectories" (ideas outperforming the baseline) to generate variants (exploitation) while also sampling novel ideas (exploration).
• Reinforcement Learning (RL): The automated executor serves as a reward function to finetune an ideator model (Qwen3-30B) using the GRPO algorithm. The reward is the validation accuracy (post-training) or reciprocal of validation loss (pre-training).

Key Results

1. Feasibility of Automated Execution

The automated executor successfully implemented a large fraction of ideas generated by frontier LLMs.

• Execution Rates: In the pre-training environment, Claude-4.5-Opus and Claude-4.5-Sonnet achieved execution rates of 96% and 90%, respectively. In the post-training environment, rates were lower (52% for GPT-5, 84% for Claude-4.5-Sonnet) but still significant.
• Baseline Performance: Even without search or RL, the "best-of-N" (N=50) performance of these models exceeded the original baselines (e.g., Claude-4.5-Sonnet achieved 60.4% accuracy on GRPO vs. 48.0% baseline).

2. Performance of Execution-Guided Evolutionary Search

Evolutionary search proved highly sample-efficient, finding superior solutions within just ten search epochs:

• Post-Training: The search found a method achieving 69.4% accuracy on the GRPO task, significantly outperforming the 48.0% baseline and the best human expert solution (68.8%) from a Stanford graduate course.
• Pre-Training: The search discovered a recipe reducing training time to 19.7 minutes to reach the target loss, compared to the baseline's 35.9 minutes.
• Scaling Trends: Claude-4.5-Opus exhibited a clear scaling trend where more search epochs led to higher upper bounds. In contrast, Claude-4.5-Sonnet and GPT-5 tended to saturate early.
• Idea Diversity: The models generated substantial algorithmic ideas (e.g., architectural changes, new training dynamics) rather than just hyperparameter tuning. Claude-4.5-Sonnet generated significantly more hyperparameter-focused ideas, while Opus and GPT-5 focused more on algorithmic innovations.

3. Performance of Reinforcement Learning (RL)

RL from execution reward showed mixed results:

• Average Reward: The average reward of the ideator model improved significantly over training epochs (e.g., average accuracy on GRPO rose from 0.253 to 0.343).
• Upper Bound Failure: Crucially, the maximum reward did not improve. The model failed to discover breakthrough ideas that surpassed the initial best performance.
• Mode Collapse: RL caused the model to converge on a few simple, easy-to-implement ideas (e.g., replacing RMSNorm with LayerNorm, or applying Exponential Moving Average). This led to a collapse in idea diversity and a reduction in "thinking length" (the model stopped generating complex reasoning traces to maximize the probability of a successful, simple execution).

Significance and Claims

The paper claims to demonstrate the feasibility and potential of grounding LLM ideation in automated execution for open-ended AI research.

• Validation of Execution Grounding: The study confirms that automated executors can handle a large fraction of natural language research ideas and that evolutionary search can effectively leverage execution feedback to find solutions superior to human baselines and standard search methods.
• Limitations of Current RL: The work highlights a critical limitation of applying standard RL to open-ended research: while it improves average performance, it induces mode collapse, causing models to converge on safe, simple ideas and fail to push the upper bound of scientific discovery.
• Future Directions: The authors suggest that future work must address diversity collapse in RL, improve the generalizability of discovered ideas, and develop more capable execution agents to handle complex code generation tasks.

The paper concludes that while automated execution feedback is a powerful signal for improving AI research, current learning algorithms (specifically standard RL) require new interventions to maintain diversity and explore the "long tail" of potentially breakthrough ideas.