🤖 AI

AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation

The paper introduces AutoScientists, a decentralized team of AI agents that self-organize to autonomously conduct long-running scientific experiments through iterative hypothesis generation and critique, significantly outperforming existing single-agent and centralized approaches across biomedical machine learning, language model optimization, and protein fitness prediction tasks.

Original authors: Shanghua Gao, Ada Fang, Marinka Zitnik

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shanghua Gao, Ada Fang, Marinka Zitnik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, incredibly complex puzzle. In the past, you might have hired a single, very smart detective to solve it. This detective would look at a piece, guess where it goes, try it, and if it didn't fit, they would try again. They would do this one step at a time, alone.

The paper introduces AutoScientists, which changes the game entirely. Instead of one detective, you hire a team of 9 AI agents that work together like a dynamic research lab.

Here is how it works, using simple analogies:

1. The "Self-Organizing" Team

Most AI systems work like a strict military chain of command: a general (a central planner) tells the soldiers exactly what to do. If the general makes a bad plan, the whole team fails.

AutoScientists is more like a group of brilliant scientists in a coffee shop. They don't have a boss telling them what to do. Instead:

They all look at a shared whiteboard (the "Shared State") that lists the current best solution, failed attempts, and new ideas.
They spontaneously form teams around the most promising ideas.
If a team hits a dead end (like trying to fit a square peg in a round hole), they don't keep banging on it. They talk to the group, admit the idea is exhausted, and reorganize to try a completely different approach.

2. The "Critique Before You Spend" Rule

In many AI experiments, the computer tries a new idea, runs a simulation, and then realizes it was a waste of time. This is like buying a ticket to a concert, getting to the venue, and realizing the show is cancelled.

AutoScientists has a Review Board. Before any expensive computer time is spent running an experiment:

An agent posts a proposal to the "Research Forum."
The other agents act as peer reviewers. They say, "Wait, that idea looks weak," or "We tried something similar last week and it failed."
This filters out bad ideas before the computer even starts working, saving time and energy.

3. The "Dead-End Registry"

Imagine a team of explorers mapping a cave. If one explorer falls into a pit, they don't just forget it. They mark the spot on the map so no one else falls in.

AutoScientists keeps a Dead-End Registry. It remembers every failed experiment and why it failed. This prevents the team from wasting time repeating the same mistakes. It also helps them realize when they are stuck in a "local optimum" (a small hill that looks like a mountain) and encourages them to jump to a different part of the map to find a higher peak.

4. The Results: Beating the Solo Detective

The authors tested this team against the "best single detective" (a system called Autoresearch) and other AI agents across three major fields:

Biomedical Machine Learning (BioML-Bench): Think of this as a giant exam with 24 different medical puzzles (like predicting drug effectiveness or analyzing cell images).
- The Result: The AutoScientists team scored higher on average than any other AI. They were particularly good at Drug Discovery, where they found better solutions than the single-agent systems.
Training AI Models (GPT Optimization): This is like trying to make a language model smarter by tweaking its settings.
- The Result: The team reached a high level of performance 1.9 times faster than the single detective. Even more impressively, when the single detective got stuck and couldn't find any improvements, the AutoScientists team kept finding new, better tweaks.
Protein Engineering (ProteinGym): This involves predicting how changing a protein's structure affects its function.
- The Result: Starting with the best existing method (Kermut), the team discovered a new way to combine data that improved prediction accuracy by 12.5% on a specific test, and 6.5% across the entire benchmark.

The Bottom Line

The paper claims that AutoScientists is better at long-term scientific discovery because it mimics how human research teams actually work: they debate, they share failures, they pivot when stuck, and they don't rely on a single boss to tell them what to do next.

Important Note: The paper explicitly states that this system is designed to accelerate the process of discovery (finding better models and hypotheses). It does not claim that these AI agents can replace human scientists for clinical decisions, nor does it claim the models they found are ready to be used as medicines or treatments in hospitals without human review. The "Impact Statement" warns that these outputs must be validated by experts before being used in real-world biology or medicine.

Technical Summary: AUTOSCIENTISTS

Problem Statement

Scientific research is inherently iterative, involving cycles of hypothesis generation, experiment design, execution, and revision. While AI agents have begun to automate parts of this process, existing approaches typically follow a single research trajectory or rely on a central planner with fixed objectives. These systems struggle to sustain parallel exploration, adapt to changing experimental evidence, or preserve knowledge of failed directions over long-running experiments. Current multi-agent systems often coordinate through rigid structures (e.g., a planner decomposing a problem or agents converging via voting), assuming the search space can be partitioned into stable directions at the outset. However, in long-running scientific experimentation, productive directions shift as evidence accumulates, and hypotheses that initially appear promising may stop yielding improvements.

Methodology: AUTOSCIENTISTS

The authors introduce AUTOSCIENTISTS, a decentralized team of AI agents designed for long-running computational scientific experimentation. Unlike prior systems that rely on central coordinators or fixed decompositions, AUTOSCIENTISTS operates without a central orchestrator, allowing agents to self-organize based on a shared experimental state.

Core Architecture

The system deploys $n$ long-running Large Language Model (LLM) agents that maintain internal state across the search process. The system alternates between two phases:

Discussion and Self-Organization: Agents analyze the task, the current champion model ( $p^*$ ), and prior forum posts. They propose research directions, critique proposals, and organize into teams ( $T_1, \dots, T_K$ ). This process is dynamic; if a team stops producing improvements, agents trigger a new discussion phase to reorganize, merge, split, or retire teams based on emerging hypotheses.
Parallel Execution: Teams execute experiments in parallel. Each team consists of two specialized roles:
- Analyst Agents: Maintain search knowledge, audit untested parameters, rank research directions by empirical effect size, and propose experiments to a team queue. They ensure diversity in proposals and prioritize underexplored directions.
- Experiment Agents: Claim experiments from the queue, apply code changes to the champion, train models, and record outcomes.

Shared State and Coordination

Coordination occurs through a shared state $S$ accessible to all agents, comprising:

Champion ( $p^*$ ): The current best model with full hyperparameters and reproduction instructions.
Experiment Log ( $L$ ): A record of every completed experiment, including outcomes, metric deltas, and training diagnostics.
Shared Forum ( $F$ ): A structured space for debating proposals, announcing results, and sharing mechanistic analyses.
Team-Local State: Per-team experiment queues ( $Q_k$ ) and dead-end registries ( $D_k$ ) to track failed directions and avoid redundant exploration.

Noise-Aware Validation

To prevent "champion pollution" (promoting models based on stochastic noise), the system employs a noise-aware gate. Improvements within the empirically measured noise band are confirmed on a second random seed before being promoted to the champion. This ensures that only robust improvements drive the search trajectory.

Key Contributions

Self-Organizing Agent Teams: A novel framework where agents independently interpret a shared experimental state and decide which hypotheses to pursue. Teams form dynamically through interaction rather than external assignment, allowing research directions to emerge and evolve naturally.
Pre-Execution Critique: Agents post proposals to a shared forum where peers critique and filter weak ideas before compute is committed, reducing redundant exploration.
Sustained Improvement: The system is designed to continue identifying productive modifications even after single-agent baselines plateau, by retiring exhausted directions and reorganizing around new hypotheses.
Comprehensive Artifact Generation: Beyond the final model, the system produces a model card and a research findings report, documenting the evolution of hypotheses, abandoned directions, and the experimental search process.

Experimental Results

The authors evaluated AUTOSCIENTISTS across three domains under matched experimental budgets:

1. Biomedical Machine Learning (BioML-Bench)

Scope: 24 end-to-end tasks spanning biomedical imaging, drug discovery, protein engineering, and single-cell omics.
Performance: AUTOSCIENTISTS achieved a mean leaderboard percentile of 74.4% across all 24 tasks, outperforming the strongest prior agent (Autoresearch) by +8.33 percentage points.
Domain Specifics: The most significant gains were observed in drug discovery, where the mean leaderboard percentile improved from 46.16% (Autoresearch) to 64.52%. In protein engineering, AUTOSCIENTISTS matched the top performance (96.97%) but achieved a better mean rank.

2. GPT Training Optimization

Scope: Optimizing the training of a GPT nanochat model using validation bits-per-byte (val_bpb) as the metric.
Efficiency: Starting from the same baseline, AUTOSCIENTISTS reached a target val_bpb of ~0.978 in 34 experiments, whereas Autoresearch required 65 experiments (a 1.9× speedup).
Sustained Discovery: When starting from a champion found by AUTOSCIENTISTS (val_bpb = 0.9777), the system discovered 7 accepted improvements over 93 experiments. In contrast, the single-agent Autoresearch approach found 0 improvements over 100 experiments, repeatedly perturbing directions near local optima.

3. Protein Fitness Prediction (ProteinGym)

Scope: Extending the Kermut Gaussian-process method for protein variant-effect prediction.
Performance: On the ACE2–Spike binding assay, AUTOSCIENTISTS improved the Spearman correlation ( $\rho$ ) from 0.747 (Kermut) to 0.840 (+12.5%).
Generalization: The discovered method, frozen and applied without modification to the full 217-assay ProteinGym benchmark, improved the official average Spearman $\rho$ from 0.657 to 0.700 (+6.5%), outperforming all other supervised baselines.

Ablation Studies

The authors conducted ablations to isolate the contribution of system components:

No Analyst: Removing analyst agents significantly degraded performance on TDC-hERG, indicating the importance of guided proposal generation.
No Cross-Agent Feedback: Disabling comments and critique hurt performance on Human Plasma-Protein Binding, highlighting the value of collective filtering.
No Self-Organization: Fixing team structures prevented adaptation to stagnation, causing the largest degradation in GPT training optimization.
Independent Agents: Removing shared state and coordination led to the most severe drops in performance (e.g., Cell-Cell Communication), demonstrating that isolated agents duplicate work and converge to incompatible local optima.

Significance and Claims

The paper claims that AUTOSCIENTISTS addresses a critical limitation of existing AI agents: the inability to sustain experimental search as evidence accumulates and productive directions change. By enabling teams of agents to explore and collaborate over the design space without a central planner, the system improves experimental search efficiency under fixed compute budgets.

The authors emphasize that AUTOSCIENTISTS transforms long-running experimentation into a collective search process. Key advantages include:

Efficient Resource Use: Agents evaluate proposals before execution and share knowledge of failures to avoid dead ends.
Adaptability: The system can reorganize teams when progress stalls, allowing it to continue identifying productive modifications after individual agents plateau.
Transparency: The system generates a complete record of the search process, including failed directions and mechanistic analyses, facilitating reproducibility and scientific insight.

The work is presented as a step toward autonomous scientific discovery, particularly in computational biomedical and scientific ML settings where experimental iteration is costly. The authors note that while the system accelerates the discovery of modeling choices, outputs should not be treated as clinically actionable without expert review and independent validation.