💻 computer science

SEMAG: Self-Evolutionary Multi-Agent Code Generation

SEMAG is a self-evolutionary multi-agent framework that mimics human coding practices by dynamically decomposing tasks and automatically upgrading its backbone models, achieving state-of-the-art performance on benchmarks like CodeContests.

Original authors: Yulin Peng, Haowen Hou, Xinxin Zhu, Ying Tiffany He, F. Richard Yu

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yulin Peng, Haowen Hou, Xinxin Zhu, Ying Tiffany He, F. Richard Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a complex piece of furniture, like a grand piano, but you don't have a single master carpenter. Instead, you have a team of AI assistants.

Most current AI coding systems are like a rigid assembly line. They have a fixed set of instructions: "Cut wood, glue parts, sand surface." If the task is simple (like building a birdhouse), this rigid line is overkill and wastes time. If the task is hard (like the piano), the line doesn't have enough steps to get it right, and the piano falls apart.

SEMAG (Self-Evolutionary Multi-Agent Code Generation) is like a dynamic, self-improving workshop that mimics how a real human team of expert engineers works. It doesn't just follow a script; it adapts, argues, and upgrades itself in real-time.

Here is how SEMAG works, broken down into simple concepts:

1. The "Smart Manager" (Adaptive Workflow)

Imagine a project manager who looks at your job and asks, "How hard is this?"

If the job is easy (like "write a function to add two numbers"), the manager says, "Okay, just one person, go ahead and do it." This saves time and money.
If the job is hard (like "build a self-driving car algorithm"), the manager says, "Okay, we need a full team. Let's plan it, build a prototype, test it, argue about the design, and fix it."

This is the Hierarchical Framework. Unlike other AIs that use the same heavy process for every task, SEMAG scales its effort up or down depending on the difficulty.

2. The "Specialized Crew" (Multi-Agent Collaboration)

When the job gets tough, SEMAG doesn't just ask one AI to "fix it." It deploys a team of specialists, each with a specific job:

The Planner: Draws the blueprints before any code is written.
The Verifier: Checks the blueprints to make sure they make sense before building starts.
The Coder: Actually writes the code based on the plan.
The Debugger: If the code breaks, this agent reads the "error logs" (like a mechanic listening to a car engine) to find the exact problem.
The Explainer: Translates the confusing error logs into plain English for the team.
The Debaters: If the debugger gets stuck, a group of agents holds a "town hall meeting." They propose different solutions, argue their pros and cons, and vote on the best one.

This prevents the AI from getting stuck in a loop of making the same mistake over and over.

3. The "Self-Upgrading Team" (Self-Evolution)

This is the most magical part. Imagine your team of carpenters is working in 2024, but a new, super-powerful tool is released in 2025. A normal team would keep using the old tools until you manually told them to switch.

SEMAG is different. It has a Research Agent that constantly scans the internet, news, and tech forums.

It sees a new, smarter AI model has been released.
It reads reviews and tests it.
It says, "Hey team, this new model is better at math problems! Let's switch our main brain to this new one immediately."

It does this automatically. It doesn't wait for a human to tell it to upgrade. It evolves its own "brain" to stay at the cutting edge.

The Results: Why It Matters

Think of coding benchmarks (tests) as different types of puzzles:

Easy Puzzles: Simple logic.
Hard Puzzles: Competitive programming contests (like the Olympics of coding).

Previous AI methods were like a student who studies hard but gets stuck on the hardest questions. SEMAG is like a super-student who knows when to ask for help, when to debate with classmates, and when to buy a new textbook.

On standard tests, SEMAG solved 98.8% of the problems (nearly perfect).
On the hardest "Olympics" level tests, it improved the success rate by 3.3% over the previous best method.
But when it used its "Self-Evolution" feature to pick the absolute best AI model available at that moment, it jumped to 52.6% success on the hardest tasks—a massive leap that no other system has achieved.

In Summary

SEMAG is not just a better coder; it's a better organization.

It adapts its effort (don't use a sledgehammer to crack a nut).
It collaborates (uses a team to solve problems instead of one lonely brain).
It evolves (automatically finds and uses the smartest tools available).

It turns code generation from a rigid machine process into a flexible, human-like problem-solving session.

1. Problem Statement

While Large Language Models (LLMs) have achieved significant progress in code generation, current multi-agent frameworks suffer from three critical limitations:

Fixed Reasoning Depth: Existing methods (e.g., Self-Debugging, LDB) use static workflows. They waste computational resources on simple tasks by over-processing and fail on complex tasks due to insufficient reasoning depth.
Local Optima Traps: Most pipelines rely on a single debugging iteration. When initial outputs diverge significantly from the target, the system often gets stuck in local minima without mechanisms to aggregate diverse reasoning trajectories or escape these traps.
Static Backbone Models: Current frameworks are tightly coupled to a single LLM backbone. They lack the ability to adapt to evolving model capabilities or switch to newer, more capable models dynamically without manual intervention.

2. Methodology: The SEMAG Framework

SEMAG (Self-Evolutionary Multi-Agent code Generation) addresses these issues through a hierarchical, adaptive, and self-evolving architecture.

A. Hierarchical Adaptive Workflow

Instead of a fixed pipeline, SEMAG employs a four-level hierarchy that scales computational effort based on task complexity and feedback:

Level 1 (Direct Generation): Attempts a one-shot code synthesis. If successful, the process terminates immediately to save resources.
Level 2 (Planning & Verification): If Level 1 fails, a Planning Agent decomposes the task into a structured plan. A Plan Verifying Agent simulates Input/Output (I/O) to validate the plan before code generation.
Level 3 (Trace-Guided Debugging): If the generated code fails, the system enters an iterative debugging loop. Specialized agents collaborate:
- Embedding Trace: Adds print statements to capture runtime variable states.
- Code Explaining: Analyzes the logic of the code.
- Suggesting Agent: Compares error traces with the correct plan to identify discrepancies.
- Debugging Agent: Refines the code based on suggestions.
Level 4 (Multi-Agent Debate): If iterative debugging stalls, the system triggers a Discussion–Decision phase. Multiple Debating Agents propose alternative algorithms or parameters. A Discriminating Agent aggregates these proposals using weighted consensus to select the optimal configuration, allowing the system to escape local optima.

Adaptive Transition Mechanism: The framework uses an adaptive threshold based on execution trace similarity ( $\rho$ ) to decide when to transition between levels. If the trace similarity between iterations drops below a dynamic threshold $\delta(t, T)$ , the system escalates to a higher complexity level.

B. Self-Evolution Mechanism

SEMAG introduces a Self-Evolution Agent module that dynamically selects the optimal backbone LLM for a specific task:

Information Retrieval: Agents use search tools to gather real-time data on recent model releases, benchmarks, and community discussions.
Candidate Selection: Multiple selector agents filter links, summarize content, and propose candidate models with confidence scores based on sampled performance.
Consensus Decision: A weighted voting mechanism selects the best model ( $m^*$ ) from the candidates. This allows SEMAG to automatically upgrade its "brain" as new, more capable models emerge.

3. Key Contributions

Adaptive Hierarchical Prompting: A dynamic strategy that adjusts reasoning depth (from one-shot to multi-step planning/debate) based on real-time task complexity, optimizing the trade-off between accuracy and token usage.
Collaborative Self-Evolution: A novel discussion–decision module that aggregates diverse reasoning paths to avoid local optima, coupled with an automated model selector that adapts to the evolving LLM landscape.
State-of-the-Art Performance: The framework achieves new benchmarks across diverse datasets, demonstrating superior resource efficiency and adaptability.

4. Experimental Results

SEMAG was evaluated on seven benchmarks: four foundational datasets (HumanEval, MBPP, HumanEval-ET, MBPP-ET) and three competition-level datasets (APPS, LiveCode, CodeContests).

Performance with Fixed Backbone (GPT-4o):
- HumanEval: 98.8% Pass@1 (New SOTA).
- MBPP: 87.6% Pass@1.
- CodeContests: 38.0% Pass@1. This represents a 3.3% improvement over the previous best method (LPW) using the same backbone model.
- LiveCode: 65.0% Pass@1.
Performance with Self-Evolution:
- When the Self-Evolution mechanism was enabled to select the optimal backbone automatically (identifying models like Claude-3.7-Sonnet), SEMAG achieved 52.6% Pass@1 on CodeContests, significantly outperforming the fixed GPT-4o baseline.
Efficiency:
- SEMAG reduces token consumption by 15–20% on simpler tasks compared to fixed-depth baselines (like LPW) by avoiding unnecessary complex workflows.
- Ablation studies confirmed that removing any component (Planning, Debugging, or Debate) significantly degrades performance, highlighting the synergistic value of the full framework.

5. Significance

SEMAG represents a paradigm shift in automated code generation by moving from static, rigid pipelines to dynamic, self-adapting systems.

Resource Efficiency: It proves that "more reasoning" is not always better; adaptive depth saves significant compute on easy tasks while applying deep reasoning only where necessary.
Future-Proofing: The self-evolution capability decouples the framework from specific model versions, allowing it to leverage the latest SOTA models automatically without re-engineering the workflow.
Robustness: The multi-agent debate mechanism effectively handles complex, open-ended problems where single-agent or single-iteration approaches typically fail.

In conclusion, SEMAG demonstrates that combining hierarchical task decomposition with real-time model selection creates a highly efficient and robust system for solving complex programming challenges, setting a new standard for LLM-based code generation.