🤖 AI

ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenas

The paper introduces ProxyWar, a novel framework that evaluates large language model code generation by embedding agents in competitive game environments to reveal significant discrepancies between static benchmark scores and actual dynamic performance, thereby advocating for richer, competition-based assessment methods.

Original authors: Wenjun Peng, Xinyu Wang, Qi Wu

Published 2026-02-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenjun Peng, Xinyu Wang, Qi Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of chefs to cook a meal. Currently, the way we test these chefs (which represent AI language models) is by giving them a recipe and asking, "Did you follow the instructions?" If they hand you a dish that looks like the picture in the book, we give them a passing grade. This is like the current "static benchmarks" used to test AI code generators.

But here's the problem: Just because a chef followed the recipe doesn't mean the food tastes good, doesn't mean they cooked it fast enough, and doesn't mean they can handle it if the oven breaks or if a customer sends the dish back.

ProxyWar is a new way of testing these AI chefs. Instead of just checking if they followed the recipe, ProxyWar puts them in a competitive kitchen arena where they have to cook against each other in real-time.

Here is how the paper explains this new framework using simple concepts:

1. The Arena: A Game Board

Instead of a quiet test kitchen, ProxyWar sets up a game board. The AI models are asked to write code that creates a "player" (an agent) to play various games, like Sudoku, Tic-Tac-Toe, or Poker.

The Setup: The AI is given the rules of the game and asked to write the code for a player.
The Twist: It's not just about playing the game; it's about playing better than the other AI players.

2. The Three Rounds of Testing

The paper describes a three-step process to see if the AI is actually good, not just lucky:

Round 1: The Safety Check (Unit Testing)
Before the game starts, the code is checked to make sure it doesn't crash. Did the chef remember to turn on the stove? Does the code have the right structure? If the code fails here, it gets sent back to the AI with a note saying, "Fix this error," and the AI tries again. This tests the AI's ability to debug itself.
Round 2: The Tournament (The Fight)
Once the code passes the safety check, the AI players are thrown into the arena. They play head-to-head against other AIs.
- Why this matters: In a normal test, two AIs might both "pass" because they both followed the rules. But in the arena, one might be slow, make silly mistakes when the game gets hard, or get confused by a tricky opponent. ProxyWar measures who actually wins, not just who followed the rules.
Round 3: The Scoreboard (TrueSkill)
Instead of a simple "Pass/Fail," the system uses a ranking system (like in video games) to give every AI a skill rating. This rating tells us not just if the code works, but how efficient, stable, and smart it is compared to the others.

3. What They Discovered

The researchers tested many different AI models (some made by big tech companies, some open-source) using this new method. They found some surprising things that old tests missed:

The "Fast but Dumb" vs. "Slow but Smart" Trap: Some AIs wrote code that was theoretically perfect but so slow that they lost the game because they took too long to think. Old tests would have said, "Great job, perfect code!" but ProxyWar said, "You lost because you were too slow."
Specialists vs. Generalists: Some AIs were great at writing code for simple tasks but fell apart when the game got complex or tricky. Others were good at strategy but bad at fixing their own mistakes.
The "Memorization" Problem: Some AIs seemed to have memorized the answers to simple puzzles. But when the game changed slightly or they had to play against a tricky opponent, those memorized answers failed. ProxyWar exposed this weakness because the games were dynamic and unpredictable.

The Big Picture

The paper argues that we can't just ask an AI, "Can you write code that passes a test?" We need to ask, "Can you write code that survives and wins in a messy, competitive, real-world situation?"

ProxyWar is like a gladiator arena for code. It doesn't just check if the code is alive; it checks if the code is tough, fast, and smart enough to beat the competition. The authors believe this gives us a much clearer picture of which AI is actually ready for the real world.

Technical Summary: ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenas

1. Problem Statement

The evaluation of Large Language Model (LLM) code generation has largely stagnated behind static benchmarks and simplistic metrics. Current approaches, such as HumanEval and MBPP, rely heavily on execution-based "pass@k" metrics or text-based similarity scores (e.g., BLEU, CodeBLEU). These methods suffer from several critical limitations:

Static Nature: They evaluate code in isolation, failing to capture dynamic execution constraints, runtime stability, and robustness under shared resource budgets.
Simplicity Bias: Benchmarks often consist of standalone functions with limited test cases, ignoring the complexity of real-world software engineering (e.g., dependencies, multi-file coordination).
Lack of Comparative Context: They do not measure how code performs against alternative solutions in competitive or adversarial settings, missing nuances in algorithmic efficiency and strategic decision-making.
Overlooked Dimensions: Metrics focus almost exclusively on functional correctness, neglecting operational characteristics such as efficiency, maintainability, and adaptability to edge cases.

Consequently, there is a significant gap between benchmark scores and the actual viability of LLM-generated code in dynamic, real-world scenarios.

2. Methodology: The ProxyWar Framework

ProxyWar is a novel framework designed to assess LLM code generation through competitive, controlled execution within diverse game environments. It orchestrates a pipeline from code generation to competitive evaluation, moving beyond binary pass/fail tests to holistic performance assessment.

2.1 System Architecture

The framework operates through five distinct layers:

Game Environment Layer: Defines the game as a tuple $G = \langle S, A, T, R, O, s_0 \rangle$ , managing state transitions, rules, and observations. It supports both perfect and imperfect information scenarios.
Agent Layer: Enforces a minimal interface contract ( $\pi: O_{space} \times \{0, 1\}^{|A|} \to A$ ) that all generated agents must implement, ensuring fair comparison under identical constraints.
Code Generation Layer: Transforms natural language specifications into executable agent code. It includes a Repair Function that iteratively refines code based on error feedback, simulating real-world debugging workflows.
Testing Layer: Validates agents through a hierarchical suite of tests covering syntax, interface compliance, basic functionality, game logic, and robustness (edge cases). Failed tests trigger the repair loop.
Tournament Management Layer: Orchestrates automated tournaments where agents compete head-to-head. It utilizes TrueSkill rating systems to compute skill rankings based on match outcomes, providing a continuous performance metric rather than a binary score.

2.2 Evaluation Methodology

ProxyWar employs a multi-dimensional assessment strategy:

Competitive Performance: Uses TrueSkill ratings to rank agents based on win/loss/draw outcomes in tournaments. It reports conservative skill estimates ( $\mu - 3\sigma$ ) to ensure robustness.
Static Code Quality: Measures Pass@1 (initial correctness), Repair Rate (ability to self-correct), and performance across hierarchical test layers (Structure, Function, Logic, Robustness).
Operational Characteristics: Tracks runtime metrics including decision speed (latency), error rates (timeouts, exceptions), and participation rates.

2.3 Experimental Setup

The framework was implemented using Python and evaluated across 18 state-of-the-art LLMs (including general-purpose, reasoning-enhanced, and code-specialized models) and 10 diverse game environments. These environments range from single-player puzzles (Sudoku, 2048) to two-player board games (Tic-Tac-Toe, Reversi) and multiplayer card games (Texas Hold'em). Each model participated in over 10,000 matches to ensure statistical significance.

3. Key Contributions

The paper makes four primary contributions:

ProxyWar Framework: The first competitive, execution-based evaluation framework for LLM code generation that integrates automated generation, iterative repair, and tournament-based ranking.
Multi-Dimensional Assessment: A methodology that captures operational characteristics (efficiency, robustness, adaptability) beyond traditional pass/fail metrics.
Scalable Platform: An implementation supporting diverse game environments with varying complexity and information structures, enabling systematic assessment across different programming paradigms.
Empirical Evaluation: A comprehensive study revealing previously hidden limitations in LLM code generation, particularly regarding algorithmic creativity, optimization strategies, and the gap between functional correctness and practical deployability.

4. Key Results and Findings

The empirical evaluation yielded several critical insights:

Discriminative Power (RQ1): ProxyWar reveals substantial performance gaps invisible to static metrics. For instance, while DeepSeek-R1 and Qwen2.5-Coder may have comparable Pass@k scores, ProxyWar showed a nearly threefold difference in tournament win rates (39.6% vs. 13.4%). Models with 100% participation rates (perfect test passing) still exhibited wide variations in competitive performance.
Static vs. Dynamic Performance (RQ2): There is a weak correlation (Spearman's $\rho = 0.23$ $ρ = 0.23$ ) between Pass@1 and competitive ranking.
- Efficiency vs. Sophistication: In Sudoku, DeepSeek-R1 generated a minimalist backtracking agent that outperformed GPT-4.1's more complex heuristic-based approach (MRV/LCV). GPT-4.1's code was theoretically superior but nearly 28x slower in practice due to Python interpretation overhead, leading to worse tournament outcomes.
- Debugging Limits: While some models (e.g., Claude3.5-Sonnet) achieved perfect repair rates, this did not guarantee strong tournament performance, indicating limits in current self-correction capabilities.
Environmental Sensitivity (RQ3): Model performance is highly context-dependent.
- Complexity: Performance gaps widen significantly as game complexity increases (e.g., Reversi vs. Tic-Tac-Toe).
- Information Structure: Perfect information games favor search/planning capabilities, while imperfect information games (e.g., Poker) favor probabilistic reasoning. No single model excels universally across all game types.
- Specialization: Code-specialized models often underperform in competitive, context-rich scenarios compared to general-purpose or reasoning-enhanced models.

5. Significance and Claims

The authors position ProxyWar as a foundational step toward more rigorous, reproducible, and realistic evaluation of program synthesis systems.

Bridging the Gap: The framework addresses the disconnect between benchmark performance and practical utility by evaluating code under shared resource constraints and adversarial conditions.
Beyond Correctness: It highlights that "correct" code is insufficient; operational characteristics like latency, stability, and strategic adaptability are crucial for real-world deployment.
Future Research: ProxyWar lays the groundwork for research into LLM-driven algorithm discovery and adaptive problem solving. The authors suggest that future work could explore whether LLM coders can discover novel strategies or outperform hand-crafted agents in complex games.
Modest Scope: The paper acknowledges that game environments, while diverse, may not fully capture the complexity of all real-world software engineering tasks (e.g., large-scale collaborative development). It views the current environment suite as a starting point for broader, non-game benchmarks.

In conclusion, ProxyWar demonstrates that competitive, game-based evaluation provides a fine-grained differentiation of LLM capabilities, exposing operational limitations that static benchmarks fail to detect.