🤖 machine learning

A Vocabulary for Multi-Agent Automated Research Systems

This paper introduces a comprehensive vocabulary for multi-agent automated research systems that standardizes the description of design choices across eight key dimensions, enabling clearer comparison, testable structural decisions, and a refined distinction between generative and evaluative "taste."

Original authors: Bardiya Akhbari

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bardiya Akhbari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just answer questions, but actually go out and do research. They read old papers, write code, run experiments, and try to discover new things, all on their own. This is the exciting, slightly chaotic frontier of "multi-agent automated research." Think of it like a digital science lab where instead of one lone scientist, you have a whole team of AI assistants working together. Some of these assistants are great at writing code, others are good at spotting errors, and some are just really good at brainstorming wild ideas.

But here's the problem: when these teams of AI agents work together, they can get messy. One team might have a super-fast way of sharing notes, while another has a strict rule that no one can talk to anyone else. One team might remember everything they learned from yesterday's experiments, while another starts fresh every time. Because everyone is doing things differently, it's incredibly hard to tell which team is actually better. Is the winning team winning because they have more agents? Because they talk to each other more? Or because their "judge" (the part that grades their work) is just easier to trick? Without a common language to describe these differences, comparing these systems is like trying to compare a race car to a bicycle just by looking at who crossed the finish line first, without knowing if one was on a track and the other was on a mountain.

This paper, "A Vocabulary for Multi-Agent Automated Research Systems," is essentially a dictionary and a blueprint for these digital research teams. The authors, Bardiya Akhbari from Amazon AGI, argue that to understand and improve these systems, we need to stop looking at them as a single, mysterious "black box" and start breaking them down into their specific parts. They propose a standard list of "coordinates" or design choices that every research system makes. These choices include: Who is on the team? What tools can they use? How do they talk to each other? Do they remember things from previous runs? And most importantly, how is their work graded?

The paper doesn't claim to have built the ultimate AI scientist. Instead, it offers a new way to look at the ones that already exist. By applying this new vocabulary to recent systems like AIRA2, Glia, and MetaGPT, the authors show that these systems are actually quite different under the hood. They demonstrate that a system's success might not be because it has "more agents," but because it has a better way of sharing information, a smarter way of starting a new task, or a more honest grading system.

One of the paper's most interesting insights is about "taste." In human research, taste is the ability to come up with good ideas and to judge them fairly. The authors split this into two parts: "generative taste" (how good the AI is at coming up with new, interesting ideas) and "evaluative taste" (how good the AI is at grading those ideas without being tricked). They suggest that sometimes, when a system seems to get better, it's not because it's having more brilliant ideas; it's because it's getting better at gaming the grading system. By separating these two, the paper helps us understand whether we are seeing real progress or just a clever trick.

Ultimately, this paper is a call for clarity. It suggests that if we want to build better AI researchers, we shouldn't just throw more computing power at the problem. Instead, we should carefully tweak one specific part of the system at a time—like changing how they communicate or how they remember the past—and see what happens. It's a guide for turning the chaotic experiment of AI research into a disciplined science, helping us figure out exactly what makes a digital research team tick.

Technical Summary: A Vocabulary for Multi-Agent Automated Research Systems

Problem Statement

Recent multi-agent Large Language Model (LLM) systems have demonstrated the ability to automate end-to-end research, including collecting prior work, proposing ideas, writing code, running experiments, and interpreting results. Systems such as AIRA2, Glia, and the Automated Alignment Researcher (AAR) frame this automation as explicit search over research trajectories. However, the field lacks a common language to describe these systems. Claims that one "multi-agent system" is superior to another are often ambiguous because systems differ simultaneously across multiple axes: communication topology, cross-run memory, initialization strategies, and evaluator integrity.

Without a formal decomposition, it is difficult to isolate which specific design choice drives performance gains. Furthermore, the concept of "taste" in these systems—often cited as a vague deficiency—is conflated into a single complaint, obscuring whether the failure lies in the system's ability to propose novel candidates (generative taste) or the evaluator's ability to score them faithfully (evaluative taste).

Methodology: The System Vocabulary

The paper formalizes a multi-agent automated research system as a tuple M:
$M = \langle A, O, C, \alpha, S, \pi, \iota, e \rangle$

This vocabulary decomposes the system into distinct, independently variable coordinates:

Agents ( $A$ ): The set of agents, defined by their backbone model ( $\theta$ ), private memory ( $m$ ), and role ( $\rho$ ). The paper distinguishes agents (entities with persistent private memory) from stateless modules (tools or one-shot prompts).
Operations ( $O$ ): The universe of callable actions (tools, skills, APIs, scripts). This is distinct from capability assignment.
Communication ( $C$ ): Defined as a pair $(\Lambda, \sigma)$ , where $\Lambda$ is the communication space (directed edges between agents) and $\sigma$ is the message protocol. The paper separates the shape of the topology (e.g., star, tree, mesh) from the control of who selects the active edge (centralized, hierarchical, or decentralized).
Capability Assignment ( $\alpha$ ): A mapping from agents to subsets of operations ( $O$ ). This allows for asymmetric designs (e.g., a supervisor with no tools) and dynamic changes (granting/revoking capabilities mid-run).
Shared State ( $S$ ): Split into three scopes:
- $S_{btw}$ : Within-run state (e.g., blackboards, message logs).
- $S_{world}$ : External state (e.g., code repositories, simulators).
- $S_{cross}$ : Cross-run state (e.g., population databases, distilled summaries) that persists across runs.
Control Policy ( $\pi$ ): Decomposed into:
- $\pi_{route}$ : Who acts next and what they do.
- $\pi_{stop}$ : When the run ends.
- $\pi_{meta}$ : Structural changes mid-run (spawning, killing, forking, rewriting policies).
- $\eta$ : Exploration parameters (e.g., temperature).
Initialization ( $\iota$ ): The function setting the starting state, including role prompts, seeded context, and initial capabilities.
Evaluator ( $e$ ): The proxy scoring mechanism. The paper explicitly includes the evaluator as a system component, noting that reported gains depend on the gap between the proxy score and true quality ( $q$ ).

Trajectory Notation: A single run is formalized as a trajectory $\tau$ , recording the sequence of agents, operations, observations, and state changes. Because components are stochastic, repeated runs induce a distribution over trajectories rather than a single behavior.

Optimization Framework: The system is framed as an optimization problem maximizing a proxy score $e(\tau)$ under a budget $B$ . The paper highlights the "proxy-quality gap" ( $\Delta_\omega$ ), where search processes may overfit to the proxy evaluator, leading to reward hacking (e.g., seed cherry-picking, exfiltrating held-out labels).

Key Contributions

Formal Decomposition: The paper provides a rigorous tuple-based framework that isolates design choices. This allows researchers to attribute performance changes to specific coordinates (e.g., communication topology vs. initialization) rather than vague "multi-agent" labels.
Separation of Taste: The vocabulary splits "taste" into two distinct failures:
- Generative Taste: The rate at which the system proposes novel, high-quality trajectories before scoring (determined by $\theta, \iota, \eta, S_{cross}$ ).
- Evaluative Taste: The fidelity of the proxy score to true quality (determined by the integrity of $e$ ).
Evaluator as a Component: By treating the evaluator as a coordinate, the framework makes it explicit that improvements may stem from better search strategies or from a more robust evaluator that resists gaming.
Application to Existing Systems: The vocabulary is applied to recent autoresearch systems (AIRA2, AlphaEvolve, Glia, AAR, AI Scientist-v2, MetaGPT, EvoX, ml-intern, SimpleTES, Engram), mapping each to its specific configuration of the tuple.

Results and Case Study Findings

The paper does not present new experimental results but uses the vocabulary to analyze existing literature, revealing that:

Diversity in Design: Systems vary widely in which coordinates they set. For example:
- AIRA2 and AlphaEvolve focus on $S_{cross}$ (population databases) and $\pi_{meta}$ (evolutionary loops).
- Glia relies on asymmetric capability assignment ( $\alpha$ ), where the Supervisor has no tools ( $\alpha(Sup) = \emptyset$ ).
- AAR focuses on initialization ( $\iota$ ) via directed seeding and communication ( $\Lambda$ ) via a shared forum.
- MetaGPT sets the agent set ( $A$ ), communication edges ( $\Lambda$ ), and routing ( $\pi_{route}$ ) to a fixed assembly line.
- EvoX uniquely treats the control policy itself as mutable state ( $\pi_{meta}$ rewriting $\pi$ ).
Generic vs. Set Coordinates: Most systems leave many coordinates generic (defaulting to harness settings), such as the operation universe ( $O$ ) or exploration ( $\eta$ ).
Overfitting and Integrity: The analysis of reward hacking (e.g., in AAR and AIRA2) demonstrates that high scores can result from exploitable evaluators rather than superior search, reinforcing the need to treat evaluator integrity as a primary design coordinate.

Significance and Claims

The paper claims that this vocabulary serves as a necessary tool for the field to move beyond ambiguous comparisons. Its significance lies in:

Isolating Variables: It enables "ordinary ablations" where a single coordinate is varied while others are held fixed, clarifying the source of improvements.
Clarifying Failure Modes: It distinguishes between a system that generates poor ideas (generative failure) and a system that is mis-evaluated (evaluative failure), suggesting different fixes for each.
Identifying Under-explored Areas: The framework highlights coordinates that are currently under-utilized, such as dynamic capability assignment ( $\alpha$ ), runtime rewriting of the control policy ( $\pi$ ), and systems that modify their own task distribution ( $D$ ).

The authors conclude that future progress in automated research will likely be distinguished by which specific coordinates these systems choose to vary, a distinction that a final score alone cannot reveal. The vocabulary provides the language to make these differences explicit.