SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs
The paper introduces SRPO (Setwise Relative Policy Optimization), a novel reinforcement learning framework for multi-agent LLMs that unifies division of labor and joint co-evolution by treating the minimal set of outputs consumed in a single state transition as a unified action, thereby enabling stable and effective training across diverse workflows and model scales.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, large language models have learned to act as agents, capable of reasoning, searching for information, and using tools to solve complex problems. When these models work alone, they follow a single path of thought. However, to tackle harder challenges, researchers often deploy multiple agents that collaborate, much like a team of specialists. Sometimes, these teams divide a task so that each member handles a specific role, while at other times, they generate several different ideas simultaneously to refine a solution together. The central difficulty in training such teams has been figuring out how to reward them. Traditional methods often treat each agent's output as a separate event, even when several agents are working together to produce a single result. This creates a mismatch: the system learns from individual pieces of the puzzle rather than the complete picture they form together, making it hard to train teams that switch between working alone and working jointly.
A team of researchers has addressed this mismatch by proposing a new training method called Setwise Relative Policy Optimization, or SRPO. Instead of looking at individual responses, this approach treats the entire group of outputs that cause a change in the environment as a single unit of action. Imagine a team of explorers reaching a fork in the road; whether one person chooses a path, or the whole group debates and then picks a route together, the decision that moves them forward is the collective result. The researchers found that by grouping these outputs into what they call an "active set," they could train the system to understand that the team's joint effort is the true action. This method allows the same training rules to apply whether the team is working in a strict division of labor, where one person does one thing, or in a mode of joint co-evolution, where multiple people contribute to a shared idea at the same time.
The researchers tested this idea on two very different types of tasks: solving difficult mathematical problems and conducting multi-turn searches to find specific facts. In the math experiments, the system had to generate candidate solutions and then verify them, a process that sometimes required one agent to solve and another to check, and other times required several agents to propose different derivations at once. In the search experiments, the system had to decide when to ask for more information, with multiple agents potentially issuing search queries simultaneously before an aggregator combined the results. Across four different sizes of language models, the new method consistently outperformed existing approaches. On the mathematical benchmarks, the system achieved an average accuracy where roughly 61 to 62 percent of its generated solutions were correct, and it found at least one correct answer for about 78 percent of the problems. In the search tasks, the improvement was even more pronounced, with the new method finding correct answers for over 60 percent of queries on average, a significant jump from previous methods.
A key part of the discovery was understanding how to balance the contributions of multiple agents. If the system simply added up the changes made by each agent, a larger team would appear to have a much bigger impact than a smaller one, simply because there were more voices. The researchers solved this by normalizing the group's contribution, ensuring that a team of five agents was not unfairly weighted against a single agent just because of its size. They also demonstrated that the system could learn to switch between these modes dynamically. In some instances, the system learned that a single specialized agent was the best choice, while in others, it activated a small group of agents to work together. This flexibility meant the training process did not need to be rewritten for different team structures; the same underlying logic handled both scenarios seamlessly.
The study also looked closely at the cost of these improvements. The researchers were careful to ensure that the better results were not simply because the new method was generating more text or using more computing power. They measured the number of tokens generated and the number of tool calls, finding that the improvements came from better coordination rather than brute force. In fact, the training process often led to shorter, more efficient responses over time. The researchers noted that while the results were strong, they were based on specific benchmarks and comparisons with other published methods, and future work would need to explore how these gains hold up in real-world, long-term deployments. Nevertheless, the core finding stands: by defining the team's collective output as the fundamental unit of action, it is possible to train multi-agent systems that are more stable, efficient, and effective at solving complex problems, regardless of whether they are working alone or together.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.