🤖 AI

AgentPSO: Evolving Agent Reasoning Skill via Multi-agent Particle Swarm Optimization

AgentPSO is a novel framework that evolves multi-agent reasoning skills by treating agents as particles in a swarm optimization process, iteratively updating their natural-language reasoning strategies through a combination of personal and global best experiences to improve problem-solving performance without modifying the underlying language model parameters.

Original authors: Hyunmin Hwang, Jaemin Kim, Choonghan Kim, Hangeol Chang, Jong Chul Ye

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hyunmin Hwang, Jaemin Kim, Choonghan Kim, Hangeol Chang, Jong Chul Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of four brilliant detectives trying to solve a very tricky mystery. In the past, if they got stuck, they would either argue with each other in real-time (debating back and forth) or just try to think harder on their own. Both methods had problems: arguing took too long and sometimes they just agreed on the wrong answer because they were afraid to disagree, while thinking alone meant they kept making the same mistakes.

The paper introduces a new way to train these detectives called AgentPSO. Instead of arguing during the investigation, they train together before the case starts, learning from each other to become permanently better at solving problems.

Here is how it works, using a simple analogy:

The "Swarm of Bees" Analogy

Think of the four detectives as bees in a swarm. In nature, bees find the best flowers by sharing information. If one bee finds a great patch of flowers, the others learn from it.

In AgentPSO, each detective (agent) is like a bee carrying a "backpack" of instructions on how to solve problems. This backpack is their Skill.

The Goal: They want to upgrade their backpacks so they can solve math and logic puzzles perfectly.
The Process: They don't change their backpacks by arguing. Instead, they go through a training loop where they try to solve a batch of problems, look at what the others did, and then tweak their instructions.

The Three Magic Ingredients

Every time the team trains, each detective updates their backpack using three specific sources of advice, similar to how a particle swarm optimization (PSO) algorithm works in computer science:

The "Personal Best" (Looking in the Mirror):
The detective looks at their own history. "Hey, last time I tried this specific way of checking my math, I got it right. I should keep doing that." This is their Personal Best skill.
The "Global Best" (Looking at the Star Player):
The detective looks at the whole team. "Wow, Detective B solved the last problem perfectly using a special trick. I should try to learn that trick." This is the Global Best skill found by the entire group.
The "Self-Reflective Direction" (The Critical Coach):
This is the paper's special secret sauce. Instead of just copying the Star Player's instructions word-for-word (which might not make sense for this specific detective), the detective acts like a coach. They look at the Star Player's thought process and ask: "Why did they succeed? What did I do wrong?"
- Example: If the Star Player checked for "edge cases" (weird numbers that break rules) and the detective didn't, the coach tells the detective: "You need to add a step to check for weird numbers."
- This creates a Self-Reflective Direction, a specific instruction on how to improve, rather than just copying the answer.

The Training Loop

The team runs through this cycle 10 times:

Try: Everyone solves a set of practice problems using their current backpack instructions.
Watch: They swap their work. They see who got it right and how they did it.
Reflect: Each detective writes a note to themselves (the Self-Reflective Direction) on what to change.
Update: They combine their old notes, their personal bests, the team's best, and their new reflection to rewrite their backpack instructions.
Repeat: They try again with the new instructions.

Why This is a Big Deal

The paper shows that this method is better than the old ways for two main reasons:

No More Endless Debates: In the old "Multi-Agent Debate" methods, the detectives had to talk to each other for every single problem, which was slow and expensive. With AgentPSO, they do all the learning during training. When it's time to solve the real problem, they just work independently and vote on the answer. It's fast and efficient.
They Actually Learn, Not Just Memorize: The paper proves that the detectives didn't just memorize the answers to the practice problems. When the researchers gave them new types of puzzles (or even asked a different AI model to use the same instructions), the detectives still performed well. This means they learned general reasoning skills (like "always check your edge cases") rather than just memorizing specific answers.

The Result

By the end of the training, the team of detectives has evolved. They are no longer just four random thinkers; they are a highly tuned team where each member has a refined, reusable set of instructions that makes them smarter than they were at the start, and smarter than teams that just argue in real-time.

In short: AgentPSO is a way to teach AI agents to learn from each other's mistakes and successes before they start working, turning a group of average thinkers into a super-team of problem solvers without needing to change the AI's brain, just its "instruction manual."

Technical Summary: AgentPSO

Problem Statement

While Large Language Models (LLMs) have demonstrated significant progress in complex reasoning, their performance remains highly sensitive to prompt engineering and the specific reasoning skills they are instructed to use. Existing approaches to improve reasoning generally fall into two categories, both of which have limitations:

Single-Agent Self-Evolution: Methods that refine a single agent's prompt or policy through self-reflection often suffer from limited exploratory diversity and can become trapped in local optima constrained by the model's inherent biases.
Inference-Time Multi-Agent Collaboration: Approaches like multi-agent debate or aggregation allow agents to critique one another during inference. However, these methods are vulnerable to incorrect peer influence and biased consensus. Crucially, they are static; the agents' underlying reasoning skills do not evolve across tasks, and the computational overhead of multi-round debate at test time is substantial.

The central question addressed by this work is: Can multi-agent systems iteratively evolve their underlying reasoning skills through collective interaction, rather than merely refining the output for a single instance?

Methodology: AgentPSO

The authors propose AgentPSO, a framework that treats multi-agent reasoning skill evolution as a dynamic optimization problem inspired by Particle Swarm Optimization (PSO). Unlike standard PSO which operates on continuous numerical vectors, AgentPSO operates in a natural-language skill space.

Core Concepts

Agents as Particles: Each agent $A_i$ in a population of $N$ agents is treated as a particle.
State ( $s_i^t$ ): The "position" of the particle is the agent's current reasoning skill, defined as a natural-language instruction.
Velocity ( $v_i^t$ ): The "velocity" is a semantic update direction, representing the textual revision to be applied to the skill.
Personal Best ( $p_i^t$ ) and Global Best ( $g^t$ ): These represent the highest-performing skill states discovered by the individual agent and the entire population, respectively, based on validation accuracy.

The Evolutionary Loop

The framework operates over $T$ training iterations, optimizing only the skill instructions without updating the backbone LLM parameters. Each iteration follows four steps:

Independent Solving: All agents solve the same training batch independently using their current skills ( $s_i^t$ ). No inter-agent communication occurs during this phase to ensure clean attribution of performance to specific skills.
Peer Observation & Self-Reflection: Each agent observes the reasoning traces and correctness labels of the entire population (excluding their own skill instructions). Based on this, the agent generates a self-reflective direction ( $d_i^t$ ) using a Reflect function. This direction identifies reusable lessons from peer trajectories (e.g., checking assumptions, decomposing problems) rather than simply copying a peer's prompt.
PSO-Inspired Update: The agent updates its semantic velocity ( $v_i^{t+1}$ $v_{i}^{t + 1}$ ) by combining:
- The previous velocity ( $v_i^t$ ).
- The self-reflective direction ( $d_i^t$ ).
- The personal-best skill ( $p_i^t$ ).
- The global-best skill ( $g^t$ ).
  This is performed via a VelocityUpdate function where the LLM synthesizes a new textual instruction describing how to revise the current skill.
Skill Rewriting: The new velocity is applied to the current skill via a SkillUpdate function, generating the next skill state ( $s_i^{t+1}$ ).
Validation and Selection: Updated skills are evaluated on a validation batch. Personal-best and global-best skills are updated only if the new performance exceeds the previous best by a margin $\epsilon$ , preventing noise from destabilizing the search.

At test time, the evolved agents solve problems independently, and their answers are aggregated via majority voting, eliminating the need for costly multi-round debate.

Key Contributions

AgentPSO Framework: The introduction of a PSO-inspired framework that evolves multi-agent reasoning skills by updating agents through personal-best, global-best, and self-reflective directions in natural-language space.
Persistent Skill Evolution: Demonstration that AgentPSO yields persistent evolution of agent capabilities, achieving stronger performance than static single-agent prompting and debate-based multi-agent baselines without requiring costly test-time interaction.
Transferability: Evidence that the evolved skills transfer across different reasoning benchmarks and even to different backbone LLMs, suggesting the method learns general reasoning behaviors rather than memorizing task-specific prompts.

Experimental Results

The authors evaluated AgentPSO on mathematical reasoning benchmarks (DeepMath, MATH, AIME25, Minerva) and general reasoning benchmarks (BigBenchHard).

Performance: AgentPSO achieved the highest average accuracy (71.98%) across all benchmarks, outperforming vanilla single-agent methods (e.g., Chain-of-Thought at 64.50%), advanced single-agent methods (e.g., Self-Refine at 66.50%), and static multi-agent baselines like MAD (67.68%) and DMAD (68.79%).
Progressive Improvement: Analysis showed that both personal-best and global-best skills improved progressively over iterations, moving the population toward more effective skill configurations.
Cross-Benchmark Transfer: Skills optimized on DeepMath performed well on BigBenchHard (and vice versa), outperforming static baselines, indicating the learning of generalizable reasoning procedures.
Cross-Model Transfer: Skills optimized using GPT were successfully applied to Claude without further training, improving Claude's average accuracy from ~52-55% (with standard prompts) to 58.37%.
Ablation Studies: Removing the self-reflective direction ( $d_i$ ) caused a substantial performance drop, confirming that standard PSO guidance (personal/global best) alone is insufficient for optimizing complex reasoning skills. The self-reflective component provides the necessary behavioral guidance to extract reusable lessons.
Efficiency: While AgentPSO requires a training phase, it significantly reduces inference-time latency compared to debate-based methods (MAD, MoA) because evolved agents solve problems independently and aggregate answers via voting, avoiding multi-round communication overhead.

Significance and Claims

The paper claims that AgentPSO demonstrates that multi-agent collaboration can serve not only as an inference-time mechanism for answer refinement but also as a model-parameter-free process for evolving reusable reasoning skills at the population level.

By treating agents as particles in a swarm that learn from each other's reasoning trajectories, the framework enables agents to acquire reusable reasoning behaviors (such as problem classification, edge-case verification, and format compliance) without updating the underlying model weights. The authors posit that this approach improves the reliability and efficiency of reasoning systems by allowing agents to internalize general reasoning procedures that transfer across tasks and models, moving beyond the limitations of static prompts or inference-time debate.

The work acknowledges limitations, noting that it focuses on reasoning benchmarks rather than knowledge-intensive domains (where domain knowledge retrieval might be more critical than reasoning skill evolution) and that the training process requires additional LLM calls. However, it establishes a new paradigm for evolving agent capabilities through collective, iterative interaction.