Can Vibe Coding Beat Graduate CS Students? An LLM vs. Human Coding Tournament on Market-driven Strategic Planning

Technical Summary: Can Vibe Coding Beat Graduate CS Students?

Problem Definition
The paper addresses a critical gap in the evaluation of Large Language Models (LLMs) for code generation. While prevailing benchmarks (e.g., HumanEval, APPS) focus on syntactic correctness and unit-test pass rates for well-defined problems, they fail to capture the complexity of real-world software engineering. These real-world scenarios often require multi-agent strategic interaction, long-horizon planning, constraint optimization, and advanced algorithm design.

To bridge this gap, the authors introduce the Auction, Pickup, and Delivery Problem (APDP). This is a multi-agent, reasoning-driven benchmark based on a logistics optimization scenario. In the APDP:

• Context: Multiple transportation companies (agents) compete in a market where tasks (parcels) are sold via a reverse first-price sealed-bid auction.
• Mechanism: Agents must bid on tasks without knowing future task distributions, then solve a static Pickup and Delivery Problem (PDP) for their won tasks under vehicle capacity constraints.
• Objective: The goal is to maximize profit, defined as the sum of winning bids minus transportation costs (distance × cost per km).
• Complexity: The problem is NP-hard and involves non-cooperative multi-agent systems (auction strategy), cooperative multi-agent systems (fleet routing), and constraint optimization. It requires agents to model opponents, calculate marginal costs, and make strategic decisions (e.g., underbidding to secure future route synergies) under strict time limits.

Methodology
The study evaluates the capabilities of state-of-the-art LLMs against human-coded agents in a controlled tournament setting.

1. Participants:
   • Human Agents: 17 agents developed before the advent of modern LLMs. These include 12 agents created by graduate students in an Intelligent Agents course at EPFL (given 2–3 weeks to solve the problem) and 5 baseline agents developed by AI laboratory researchers.
   • LLM Agents: 40 agents generated by four leading LLMs: OpenAI's GPT-5 Thinking, Google's Gemini 2.5 Pro, Anthropic's Claude Opus 4.1, and DeepSeek's DeepThink R1.
2. Prompting Strategies: Each LLM was tested with five distinct prompting methodologies to simulate various levels of "vibe coding" (natural language to code):
   • Two author-generated prompts (A1, A2) providing the same instructions given to the students.
   • Iterative Refinement (IR): Self-play feedback loops.
   • LLM as a Critic (CR): Using a stronger model to critique and improve a generated agent.
   • GPT-5 Generated Prompt (GEN): Using GPT-5 to optimize the prompt itself.
3. Experimental Setup:
   • Tournaments: 12 double all-play-all tournaments were conducted across four different network topologies (Switzerland, France, Great Britain, Netherlands).
   • Scale: Approximately 40,000 matches were played. In each match, agents competed 1v1, swapping control of different company fleets to ensure fairness.
   • Debugging: Unlike the human agents (which were evaluated as-is), LLM agents underwent rigorous debugging cycles to fix syntax and semantic errors until they were bug-free, ensuring the evaluation focused on algorithmic competence rather than implementation bugs.

Key Results
The results demonstrate a clear superiority of human-coded agents over LLM-coded agents in this complex, strategic domain:

1. Human Dominance: The top 5 positions in the tournament rankings were consistently occupied by human-coded agents (specifically the student agents).
2. LLM Underperformance: The majority of LLM-coded agents (33 out of 40) were defeated by very simple baseline human agents (e.g., "ExpCostFixedBid," which uses a fixed average cost for bidding).
3. Reasoning Deficits:
   • In preliminary tests involving simpler variants (e.g., A* search), LLMs frequently failed to implement fundamental requirements, such as admissible heuristics, despite explicit instructions.
   • LLMs struggled with semantic bugs, such as violating time limits, failing to deliver won tasks, or ignoring capacity constraints, often requiring significant manual intervention to resolve.
4. Degradation of Existing Solutions: In a counter-intuitive finding, when the best-performing LLM (GPT-5) was given the code of the winning human agent and asked to "improve" it, the resulting agent performed significantly worse, dropping from 1st place to 10th. This suggests LLMs struggle to effectively utilize in-context learning to refine complex, working solutions.

Significance and Claims
The paper claims that high performance on existing coding benchmarks (which prioritize syntax and unit tests) does not translate to the ability to solve real-world problems requiring strategic reasoning and optimization.

• Limitations of Current Benchmarks: The authors argue that current metrics understate the difficulty of real-world engineering tasks. The APDP benchmark highlights that LLMs can generate code that runs but often fails to generate code that competes.
• Vibe Coding Gap: The study questions the efficacy of "vibe coding" (using natural language prompts to generate complex software) for graduate-level or professional engineering tasks involving multi-agent strategy.
• Call to Action: The work motivates a shift in the field toward "reasoning-driven code synthesis" benchmarks. The authors advocate for evaluations that stress planning, opponent modeling, and strategic interaction in open-ended, real-world scenarios, rather than just functional correctness.

The authors remain modest regarding their claims, noting that they are not creating a leaderboard of specific models or claiming to have found the optimal prompting strategy. Instead, they present a representative sample of current capabilities, concluding that while LLMs are impressive tools, they currently lack the strategic depth to outperform experienced human engineers in complex, competitive optimization problems.