🤖 machine learning

OMAC: A Holistic Optimization Framework for LLM-Based Multi-Agent Collaboration

This paper introduces OMAC, a general framework that holistically optimizes Large Language Model-based Multi-Agent Systems by identifying five key dimensions and employing specialized algorithms for both single and joint optimization, thereby achieving superior performance in complex tasks like code generation and reasoning compared to state-of-the-art approaches.

Original authors: Shijun Li, Hilaf Hasson, Joydeep Ghosh

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shijun Li, Hilaf Hasson, Joydeep Ghosh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, incredibly difficult puzzle. You could try to do it alone, but you'd likely get stuck or make mistakes. So, you decide to hire a team of experts: a coder, a mathematician, a logician, and a creative writer. This is what Multi-Agent Systems (MAS) are: a group of AI "agents" working together to solve complex problems.

However, there's a catch. Usually, when humans build these AI teams, they act like rigid managers. They say, "You, the coder, talk to the mathematician. Then, the writer talks to the coder." They design the team structure and the rules by hand, guessing what might work best. If the team fails, they have to start over from scratch, guessing again.

OMAC is a new framework that changes the game. Instead of a human guessing the best team structure, OMAC is like a super-smart "Team Coach" that automatically figures out the perfect way to organize, train, and direct the AI team.

Here is how OMAC works, broken down into simple concepts:

1. The Five Levers of Improvement

The authors realized that to make an AI team better, you only need to tweak five specific things (like adjusting the dials on a radio). They call these the "Five Dimensions":

Fixing the Players (Agent Functionality): Maybe the "Mathematician" agent is bad at explaining its steps. OMAC rewrites its instructions to make it clearer.
Hiring New Players (New Agents): Maybe the team is missing a "Editor." OMAC creates a new agent from scratch to fill that gap.
Picking the Starting Lineup (Candidate Selection): Not every problem needs all 10 agents. OMAC learns to pick the best 3 or 4 for the specific task at hand.
Deciding Who Talks When (Dynamic Participation): In the middle of a conversation, maybe the "Coder" needs to stop talking and let the "Tester" speak. OMAC learns to switch roles dynamically.
Managing the Handoffs (Communication Flow): Who should send their notes to whom? OMAC learns to route information so the right person gets the right data at the right time.

2. The Two Magic Tools

To tweak these five levers, OMAC uses two special AI tools that work together in a loop:

The "Creative Brain" (Semantic Initializer):
Imagine a brainstorming session. This tool generates a bunch of different ideas for how the team could work. It might say, "What if the coder writes comments in the code?" or "What if we hire a 'Safety Inspector'?" It creates a wide variety of initial team setups.
The "Strict Judge" (Contrastive Comparator):
This is the critical part. The team tries out these new ideas on practice problems. The Judge then looks at the Winners (the team setup that solved the puzzle perfectly) and the Losers (the setup that failed).

The Judge doesn't just pick the winner; it asks, "Why did the winner win and the loser lose?"
- Example: "The winner included a 'Safety Inspector' who caught a bug. The loser didn't have one."
Based on this comparison, the Judge tells the Creative Brain: "Okay, next time, make sure every team has a Safety Inspector, and make the instructions for the coder more specific."

This cycle repeats: Brainstorm → Test → Compare → Refine. Over and over, the team gets smarter and the structure gets tighter.

3. The "One Thing at a Time" Rule

You might think, "Why not change the players, the lineup, and the communication rules all at once?"

The paper explains that if you change everything at once, you get confused. It's like trying to fix a car engine, change the tires, and repaint the car all in the same minute. You won't know which change actually made the car run better.

OMAC uses an Iterative Approach: It changes one thing (like the instructions for the coder), tests it, and locks it in. Then it moves to the next thing (like the lineup). By changing one thing at a time, it knows exactly what is working.

4. The Results

The researchers tested OMAC on three tough challenges:

Writing Code: Can the AI team write software that doesn't crash?
General Reasoning: Can the team answer tricky questions about history, science, and law?
Math: Can the team solve complex math problems?

The Outcome: OMAC consistently beat the best existing methods. It didn't just make the team slightly better; it made them significantly smarter. And because it figured out the most efficient way for the agents to talk, it actually used less computing power to get better results.

The Big Picture

Think of OMAC as the difference between a human manager trying to guess the best way to run a company, and an AI coach that runs thousands of simulations every day to find the perfect strategy.

Instead of us humans manually designing how AI agents should talk and work, OMAC lets the AI teach itself the best way to collaborate. It's a "self-improving" system that builds better teams, faster and smarter than we ever could by hand.

1. Problem Definition

The paper addresses the limitations of current Multi-Agent Systems (MAS) powered by Large Language Models (LLMs). While MAS have shown promise in complex tasks (e.g., code generation, reasoning), their development relies heavily on hand-crafted strategies and human expertise. Existing approaches suffer from two main issues:

Lack of Systematic Optimization: Most methods fix agent prompts and collaboration structures manually. While some works optimize single agents (via fine-tuning or prompt tuning) or search for architectures, they rarely address the joint optimization of both agent functionality and collaboration structure.
Static and Inflexible Designs: Current collaboration structures are often static or optimized only for specific scenarios (e.g., fixed team composition), lacking the flexibility to adapt dynamically to task requirements.
Limitations of Existing Optimizers: Recent attempts like DyLAN focus only on unsupervised team composition optimization, while others like AFlow or G-Designer focus solely on structural search, often neglecting the functional optimization of the agents themselves.

The authors propose OMAC (Optimization of Multi-Agent Collaboration), a general framework designed to holistically optimize LLM-based MAS by identifying and refining five key dimensions of the system.

2. Methodology

A. Five Key Optimization Dimensions

OMAC conceptualizes MAS as an information flow graph where agents are nodes and communication channels are edges. It identifies five dimensions for optimization:

Functional Dimensions (Node Capabilities):
1. Fun-1 (Optimizing Existing Agents): Refining the instruction prompts and few-shot examples of existing agents to improve task-specific performance.
2. Fun-2 (Optimizing Construction of New Agents): Generating and optimizing prompts for entirely new agents to be integrated into the MAS.
Structural Dimensions (Graph Construction):
1. Str-1 (Optimizing Candidate Agent Selection): Optimizing a controller prompt to select the most beneficial subset of agents from the available pool before collaboration begins.
2. Str-2 (Optimizing Dynamic Agent Participation): Optimizing a controller to dynamically select agents for specific steps in the multi-step process, based on previous outputs.
3. Str-3 (Optimizing Agent Communication Flows): Optimizing a controller to determine routing decisions (i.e., which agent's output serves as input for which other agent).

B. The Optimization Algorithm

OMAC employs a unified algorithm for optimizing any single dimension, utilizing two LLM-powered actors:

Semantic Initializer:
- Role: Explores the semantic space to generate an initial collection of diverse agents or controllers.
- Mechanism: Takes task context and dimension requirements as input, generating a set of candidate prompts (e.g., 3 initial variations) with controlled variations in emphasis and implementation.
Contrastive Comparator:
- Role: Performs contrastive reasoning to refine candidates.
- Mechanism:
  - Evaluates the initial collection on the training set to calculate performance scores.
  - Samples a positive-negative pair (high-performing vs. low-performing).
  - Analyzes the performance gap to identify underlying factors (e.g., "explicitly requiring comments improved code quality").
  - Generates a refined agent/controller that amplifies positive factors and mitigates negative ones.
- Iteration: This cycle (Evaluate $\to$ Sample Pair $\to$ Contrastive Reasoning $\to$ Refine) repeats until a maximum iteration count is reached.

C. Joint Multi-Dimension Optimization

To optimize multiple dimensions simultaneously without introducing confounding variables, OMAC uses an iterative joint optimization strategy:

It optimizes one dimension at a time while keeping others fixed.
Once a dimension is optimized, the best-performing configuration is retained, and the process moves to the next dimension.
This ensures that the Contrastive Comparator can clearly attribute performance gains to specific changes, avoiding the complexity of simultaneous variable interactions.
Efficiency Strategy: To manage computational costs, the framework selectively incorporates only the dimensions that show the most significant individual improvements into the joint optimization loop.

3. Key Contributions

Holistic Framework: Introduction of OMAC, the first framework to systematically define and optimize five distinct dimensions covering both agent functionality and collaboration structure in a unified, supervised manner.
Novel Algorithm: Proposal of a general optimization algorithm using Semantic Initializers and Contrastive Comparators to leverage supervised signals (performance gaps) for iterative refinement.
Iterative Joint Optimization: A strategy for jointly optimizing multiple dimensions that balances performance gains with computational feasibility.
Comprehensive Evaluation: Extensive empirical validation across diverse tasks (code generation, general reasoning, arithmetic reasoning) demonstrating superior performance over state-of-the-art baselines.

4. Experimental Results

The authors evaluated OMAC on three major benchmarks: HumanEval (Code Generation), MMLU (General Reasoning), and MATH (Arithmetic Reasoning).

Single-Dimension Optimization: Optimizing any single dimension individually led to significant performance improvements over strong baselines (e.g., DyLAN, ADAS, AFlow).
- Example: On HumanEval, optimizing structural dimensions (Str-1 to Str-3) improved Pass@1 from ~85.7% (DyLAN) to 87.55%.
- Example: On MATH, functional optimization (Fun-1) improved accuracy from ~32.35% to 35.17%.
Multi-Dimension Optimization: Jointly optimizing dimensions (specifically the top-performing functional and structural ones) yielded even greater gains.
- On MATH, joint optimization increased accuracy from 2.9% (single dim) to 9.6% improvement over the baseline.
Ablation Study: Removing the Contrastive Comparator (OMAC-C) resulted in lower performance, proving that contrastive reasoning is crucial for refining the initial semantic exploration. However, OMAC-C still outperformed the baseline DyLAN, showing the value of the Semantic Initializer alone.
Efficiency: OMAC reduces inference-time computational costs (fewer API calls and tokens) by dynamically selecting only necessary agents and communication paths, despite the higher training cost associated with the optimization process.

5. Significance and Impact

Paradigm Shift: OMAC moves MAS development from manual, heuristic design to automated, data-driven optimization. It treats the design of agents and their interaction structures as a learnable problem.
Generalizability: The framework is model-agnostic (tested on GPT-3.5, GPT-4o-mini, DeepSeek, Qwen) and task-agnostic, applicable to reasoning, coding, and tool-use scenarios.
Practicality: By reducing inference costs through structural optimization, OMAC addresses the scalability and commercial viability of deploying complex multi-agent systems.
Future Direction: The paper lays the groundwork for future research into gradient-based optimization for MAS and the integration of advanced tool-use capabilities within this holistic framework.

In conclusion, OMAC demonstrates that supervised contrastive reasoning applied to both agent prompts and collaboration topologies can significantly outperform existing hand-crafted or single-dimension optimization methods, establishing a new standard for designing robust LLM-based Multi-Agent Systems.