Learning to Orchestrate Agents in Natural Language with the Conductor

This paper introduces a reinforcement learning-trained "Conductor" model that automatically discovers optimal coordination strategies, including dynamic communication topologies and tailored prompts, to orchestrate diverse large language models into a collaborative system that achieves state-of-the-art performance on challenging reasoning benchmarks.

The Gist

Imagine you have a massive orchestra of musicians. Some are world-class violinists, others are incredible drummers, and some are brilliant composers. But if you just tell them all to "play music" at the same time, it's going to be a chaotic mess. You need a Conductor.

This paper introduces a new kind of AI called the Conductor. It's not just another smart robot; it's a "manager of managers" designed to get the best out of a team of other AI models.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: Too Many Soloists, Not Enough Harmony

We have many powerful AI models (like the violinists and drummers). Some are great at math, some at coding, and some at writing stories.

• The Old Way: Humans try to manually tell these AIs how to work together. It's like a human trying to conduct an orchestra by shouting instructions from the balcony. It's slow, expensive, and often misses the best way to combine their talents.
• The New Way: We trained a small AI (the Conductor) to learn how to conduct the orchestra itself, using Reinforcement Learning. Think of this as giving the Conductor a magic baton and letting it learn through trial and error until it knows exactly how to get a perfect symphony out of the group.

2. How the Conductor Works

The Conductor doesn't do the heavy lifting itself. Instead, it acts as a strategic project manager. When you give it a hard problem (like "Write a complex computer program" or "Solve this physics puzzle"), it does three things:

• Breaks it Down: It slices the big problem into smaller, manageable chunks (subtasks).
• Assigns the Right Person: It looks at its team of AI workers and asks, "Who is best at this specific chunk?" Maybe it sends the math part to the "Math Genius" AI and the coding part to the "Coder" AI.
• Manages the Conversation: It decides who needs to hear what. It might tell the Coder, "Hey, here is the math solution the Math Genius just gave you; use that to write your code."

The Magic Trick: The Conductor learns to do this entirely in natural language. It doesn't use complex code to talk to the other AIs; it just writes clear, human-like instructions like, "Please check this code for errors," or "Explain this concept simply."

3. The "Secret Sauce": Learning by Doing

How did the Conductor learn to be so good?

• The Gym: The researchers put the Conductor in a "gym" with thousands of hard problems.
• The Scorecard: Every time the Conductor organized a team that solved a problem correctly, it got a "point." If the team failed, it got a "zero."
• The Evolution: Over time, the Conductor figured out the best strategies. It learned that for a coding problem, it should first ask one AI to plan the logic, then another to write the code, and a third to check for bugs. It learned these strategies on its own, without humans telling it exactly what to do.

4. Why is this a Big Deal?

The paper shows that a relatively small Conductor (only 7 billion parameters, which is small in AI terms) can beat the world's biggest, most expensive AI models when it leads a team.

• The "Swiss Army Knife" Effect: No single AI is perfect at everything. But the Conductor knows how to combine them so that their weaknesses cancel out and their strengths multiply.
• Cheaper and Smarter: Instead of paying for one super-expensive AI to do everything, you can use a small Conductor to coordinate a team of cheaper or specialized AIs to get better results for less money.
• Adaptability: If you change the team (maybe you only have open-source AIs available today), the Conductor can quickly relearn how to conduct that specific group to get the best results.

5. The "Self-Reflecting" Upgrade

The researchers also gave the Conductor a superpower: Recursion.
Imagine the Conductor finishes a project, looks at the result, and thinks, "Hmm, this isn't quite right. I need to try a different approach."

• It can then call itself to re-evaluate the plan.
• It can say, "Okay, the first plan failed. Let's try a new team structure."
• This allows the system to keep improving its answer in real-time, like a human who keeps refining their essay until it's perfect.

Summary

Think of the Conductor as the ultimate AI Team Leader.

• Before: Humans had to manually tell AIs how to work together.
• Now: A small, smart AI learns to organize a team of other AIs automatically.
• Result: By letting the AIs talk to each other in natural language and letting the Conductor figure out the best strategy, we get smarter, more accurate, and more efficient results than any single AI could achieve alone.

It's the difference between a group of talented musicians playing randomly in a room versus a world-class conductor guiding them to create a masterpiece.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) have achieved remarkable capabilities, utilizing them to their full potential remains challenging. Current approaches face several limitations:

• Specialization vs. Universality: Different models are fine-tuned for specific domains (e.g., coding vs. math), and no single model is optimal for all tasks.
• Manual Engineering: Existing multi-agent workflows often rely on manually designed scaffolds, fixed topologies (e.g., sequential chains, specific routing), and static prompt engineering, which lack flexibility and adaptability.
• Cost and Efficiency: State-of-the-art multi-agent baselines often require expensive, high-volume agent calls and complex human-designed coordination strategies that do not scale well.
• Latent Capabilities: The full potential of combining diverse models to solve complex reasoning problems is not yet unlocked through automated, end-to-end learning.

The paper addresses the need for an automated system that can dynamically design communication topologies, delegate subtasks, and engineer prompts to orchestrate a pool of heterogeneous LLM agents effectively.

2. Methodology: The RL Conductor

The authors introduce the Conductor, a small (7B parameter) language model trained via Reinforcement Learning (RL) to act as a meta-agent. Instead of solving problems directly, the Conductor generates a sequence of workflow steps that orchestrate larger, more powerful "worker" LLMs.

Core Framework

• Input/Output: The Conductor receives a user query and a list of available worker models (represented as IDs). It outputs three Python lists:
  1. model_id: The sequence of agents to be called.
  2. subtasks: Natural language instructions for each agent (prompt engineering).
  3. access_list: Defines the context visibility (which previous steps' outputs each agent can see), enabling complex topologies like trees, chains, or parallel branches.
• Training Paradigm:
  • Algorithm: The model is trained using GRPO (Group Relative Policy Optimization), an online RL algorithm.
  • Reward Function: The reward is determined by two conditions:
    1. Format Condition: A penalty (-1 or 0) if the output cannot be parsed into the required list structure.
    2. Correctness Condition: A positive reward (+1) if the final output of the executed workflow matches the ground truth; otherwise, a partial reward (0.5 or -0.5).
  • Training Data: The model is trained on a diverse set of reasoning tasks (Math, Coding, General Reasoning) using a pool of both open-source (e.g., Qwen, DeepSeek, Gemma) and closed-source (e.g., GPT-5, Claude, Gemini) models.

Key Extensions

• Adaptive Worker Selection: The Conductor is fine-tuned with randomized agent pools. At each training step, it is restricted to a random subset of the total available models. This forces the model to learn robust coordination strategies that generalize to arbitrary sets of agents, allowing users to deploy the system with specific cost constraints (e.g., open-source only).
• Recursive Topologies (Test-Time Scaling): The framework allows the Conductor to specify itself as a worker. This enables recursive workflows where the Conductor can review the output of a previous coordination strategy and generate a new, refined strategy (e.g., adding verification steps or changing agents) before returning the final answer. This introduces a new axis of test-time compute scaling.

3. Key Contributions

1. RL-Driven Orchestration: The paper demonstrates that a small 7B model, trained purely via end-to-end RL, can learn to design complex, natural language-based agentic workflows without manual scaffolding.
2. Emergent Capabilities: Through reward maximization, the Conductor naturally discovers advanced strategies such as:
   • Prompt Engineering: Crafting targeted instructions to leverage specific model strengths.
   • Dynamic Topologies: Moving beyond simple chains to tree structures, parallel execution, and iterative refinement.
   • Role Specialization: Assigning "planner," "coder," "verifier," or "format-checker" roles to specific models based on the task.
3. Generalization and Adaptability: The model generalizes to unseen tasks and arbitrary agent pools (including open-source only) through randomized training, eliminating the need for retraining when changing the underlying model pool.
4. Recursive Scaling: The introduction of self-referential recursion allows for dynamic test-time scaling, where the system can adaptively increase compute resources to solve harder problems.

4. Experimental Results

The authors evaluated the 7B Conductor against state-of-the-art (SOTA) models and multi-agent baselines across challenging benchmarks.

• Performance Benchmarks:

  • LiveCodeBench: The Conductor achieved 83.93%, surpassing GPT-5 (82.90%) and Claude Sonnet 4 (46.54%).
  • GPQA-Diamond: Achieved 87.5%, outperforming GPT-5 (82.3%) and Gemini 2.5 Pro (84.8%).
  • AIME25: Achieved 93.3%, beating GPT-5 (90.8%).
  • Overall: The Conductor attained SOTA results on average across math, coding, and science domains, outperforming individual workers and expensive multi-agent baselines.

• Comparison with Baselines:

  • vs. Self-Reflection: The Conductor significantly outperformed 5-turn self-reflection baselines for individual models.
  • vs. Multi-Agent Routing: It outperformed methods like MASRouter, Mixture-of-Agents (MoA), and RouterDC. Notably, the Conductor achieved these results with fewer API calls (average 3 steps vs. 5+ for baselines) and lower token usage, demonstrating superior efficiency.
  • Cost Efficiency: The Conductor achieved higher performance at a fraction of the cost compared to consensus-based inference scaling or expensive multi-agent pipelines.

• Ablation Studies:

  • Scale: A 3B Conductor learned similar agent selection but failed at complex prompt engineering, highlighting the importance of the base model's reasoning capabilities for instruction design.
  • Subtasks: Removing the ability to generate custom subtasks (forcing generic prompts) caused a significant performance drop, especially in coding tasks, proving the value of learned prompt engineering.
  • Recursion: Enabling recursion on BigCodeBench improved performance by allowing the Conductor to detect suboptimal initial strategies and switch agents (e.g., from GPT-5 to Claude/Gemini) dynamically.

5. Significance and Impact

• New Paradigm for Multi-Agent Systems: The paper shifts the paradigm from manually designed agent workflows to learned coordination strategies. It proves that RL can unlock emergent, powerful collaboration patterns that humans might not intuitively design.
• Democratization of SOTA Performance: By using a small 7B Conductor to orchestrate a pool of models, the system can achieve SOTA results even when restricted to cheaper or open-source models, provided the coordination strategy is optimal.
• Test-Time Scaling: The recursive extension offers a novel method for scaling inference compute dynamically, allowing models to "think harder" by iterating on their own coordination strategies.
• Future Directions: The work suggests that language models can serve as intelligent meta-agents, potentially extending beyond text to coordinate multimodal agents (e.g., robotics, biology) in the future.

In summary, the RL Conductor represents a significant leap in automated agent orchestration, demonstrating that a small, specialized model trained with reinforcement learning can effectively manage and amplify the capabilities of a diverse ecosystem of large language models, achieving state-of-the-art results with high efficiency and adaptability.