An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding
This paper introduces MSEval, a rigorous benchmark for from-scratch multi-agent coding that utilizes real-world full-stack projects and an automated evaluation framework to demonstrate that organizational topology significantly influences the speed, cost, and quality trade-offs, often rivaling model capability in importance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where building software isn't a lonely job for a single genius coder, but a bustling construction site run by a team of digital workers. This is the realm of Multi-Agent Systems, where artificial intelligence (AI) doesn't just write a single line of code but collaborates to build entire applications from scratch. Think of it like a group of friends trying to build a treehouse: one might design the blueprints, another cuts the wood, and a third paints the walls. But here's the catch: if they don't talk to each other, or if they argue over who owns the hammer, the treehouse might end up crooked, unfinished, or collapse entirely. For a long time, scientists tested these AI teams by asking, "Can one agent fix a broken toy?" But the real world is messier. It asks, "Can a whole team build a skyscraper, handle disagreements, and finish on time without running out of money?" This is the big question researchers are now tackling: how do we measure if a team of AI agents can actually work together to create real software, rather than just pretending to?
Enter MSEval, a new "gym" for testing AI teams, created by researchers from Tsinghua University and Zhongguancun Laboratory. Instead of giving the AI teams a pre-made blueprint with every single screw specified, MSEval hands them a rough idea—like a teacher's assignment sheet—and watches them build a full-stack web app from the ground up. The researchers set up a massive experiment with 100 different scenarios, mixing 10 real-world projects (like a live-streaming teaching platform or an instant messaging app) with 10 different ways the AI team could organize itself. Some teams worked like a strict assembly line, passing work down a chain; others acted like a swarm of bees, grabbing tasks as they saw fit; some even had a "manager" AI watching over everyone, while others let the agents compete against each other.
The team didn't just look at whether the final app worked; they measured everything. They tracked how long it took (wall-clock time), how much it cost in digital currency (tokens), and how many times the AI had to fix its mistakes. They used a special automated grader called TAgent, which acts like a super-strict teaching assistant. TAgent doesn't just read the code; it actually visits the live website, clicks buttons, checks if the login works, and scans the code for hidden security flaws. It then gives the team specific feedback, like "Your chat feature works, but you forgot to save the messages," allowing the team to try again.
The results were eye-opening. The study found that how the team is organized matters just as much as how smart the AI models are. In fact, changing the team's "topology" (their organizational structure) could swing the final score by over 30 points and double the time it took to finish. For example, a structured pipeline where agents hand off work in clear stages often produced the highest quality results fastest. In contrast, teams with heavy managerial oversight or chaotic "swarming" often got bogged down, wasting time and money on arguments or duplicate work. Interestingly, the researchers discovered that simply having more powerful AI models didn't guarantee success; a slightly less powerful model with a better team structure often outperformed a super-smart model with a confused team.
The paper also revealed that these AI teams rarely "crash" completely. Instead, they usually build a working but incomplete system. The most common failures weren't catastrophic errors, but missing features or logic gaps—like a door that opens but doesn't lock. The study suggests that for AI teams to be truly useful in the future, we need to treat their collaboration style as a flexible tool. We shouldn't just pick the "smartest" AI and hope for the best; we need to carefully design how they talk, who owns which part of the project, and when to stop and fix things. MSEval proves that in the race to build software, the secret sauce isn't just raw intelligence; it's the art of coordination.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.