💬 NLP

CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments

This paper introduces CATArena, a novel framework that evaluates the evolutionary capabilities of LLM code agents through iterative tournaments and a dual-metric system, revealing that initial proficiency does not guarantee evolutionary potential and highlighting current limitations in leveraging peer-learning and self-reflection simultaneously.

Original authors: Lingyue Fu, Xin Ding, Linyue Pan, Yaoming Zhu, Shao Zhang, Lin Qiu, Weiwen Liu, Weinan Zhang, Xuezhi Cao, Xunliang Cai, Jiaxin Ding, Yong Yu

Published 2026-02-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lingyue Fu, Xin Ding, Linyue Pan, Yaoming Zhu, Shao Zhang, Lin Qiu, Weiwen Liu, Weinan Zhang, Xuezhi Cao, Xunliang Cai, Jiaxin Ding, Yong Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge how good a group of new apprentice chefs are at cooking.

The Old Way (Current Benchmarks):
Right now, most people test these chefs by giving them a recipe and asking them to cook one dish once. If the dish tastes okay, they get a passing grade. If it's burnt, they fail.

The Problem: This only tells you if they can follow instructions once. It doesn't tell you if they can taste their own food, realize it's too salty, fix it, or watch a master chef cook the same dish and steal their best techniques to get better next time. It's like judging a marathon runner by how fast they run the first 10 meters.

The New Way (CATArena):
The authors of this paper built a new testing ground called CATArena (Code Agent Tournament Arena). Instead of a one-off cooking test, they put the chefs (AI code agents) into a multi-round tournament.

Here is how it works, using simple analogies:

1. The Tournament Loop

Imagine a boxing ring or a chess tournament.

Round 1: Every agent tries to solve a problem (like playing a game of Chess or optimizing a computer program). They submit their first "move" or code.
The Feedback: The system runs the code. It doesn't just say "Pass" or "Fail." It gives them a scoreboard, a replay of the match, and the code used by the winners.
The Evolution: Now, the agents get a second chance. They have to look at the scoreboard, study the winners' code (Peer-Learning), and look at their own mistakes (Self-Reflection). They then rewrite their code to be better.
Repeat: This happens for several rounds. The goal isn't just to be good at the start; it's to see how much they can improve over time.

2. Two Types of Challenges

The paper tests the agents in two different "arenas":

The Objective Arena (The Solo Runner): Imagine a race where the goal is to run as fast as possible against a stopwatch. The agents try to make their code run faster and use less memory. They are competing against a fixed target, but they can see how fast everyone else is running and try to beat that speed.
The Competitive Arena (The Strategy Game): Imagine a poker table or a game of Go. Here, there is no fixed "perfect" speed. The only way to win is to be better than the other players. As the other players get smarter, the game gets harder. The agents have to adapt their strategies to beat the new, smarter opponents.

3. The Big Discovery

The researchers found something surprising: Being good at the start doesn't mean you are good at learning.

The "Star" vs. The "Growth": Some agents started out as "Star Players" (they wrote great code immediately). But when the tournament started, they got stuck. They couldn't figure out how to use the feedback to get better.
The "Underdog": Other agents started out as "Underdogs" (their first code was messy). But as the tournament went on, they studied the winners, fixed their mistakes, and eventually became the champions.
The Lesson: The paper proves that "Evolutionary Capability" (the ability to learn and adapt) is a totally different skill than "Static Capability" (the ability to just write code once).

4. The "Black Box" of Learning

The authors looked inside the agents to see how they learned. They found two main tools:

Self-Reflection: Looking in the mirror and saying, "I messed up here, I need to fix this."
Peer-Learning: Watching the winner and saying, "Oh, they did it this way. I'll try that."

The Catch: Most of the current AI agents are bad at using both tools at the same time. They tend to either ignore the winners and just keep trying to fix their own mistakes (often making them worse), or they blindly copy the winners without understanding why. Only a few top-tier agents managed to combine both strategies to truly evolve.

Summary

CATArena is a new gym for AI code agents. Instead of just testing if they can lift a weight once, it puts them in a season-long league where they have to watch their opponents, analyze their own performance, and get stronger every week. The paper shows that the agents who can learn and adapt over time are different from the ones who are just naturally talented at the start, and that current AI still struggles to master the art of continuous improvement.

Technical Summary: CATArena

Problem Statement

Current evaluation frameworks for Large Language Model (LLM) code agents predominantly rely on static, single-turn benchmarks that measure functional correctness (e.g., Pass@k) within isolated development scenarios. The authors argue that these metrics fail to capture two critical dimensions of real-world software engineering: iterative evolution and continuous optimization. Real-world development is inherently incremental and collaborative, involving self-reflection on execution logs and peer-learning from superior implementations. Existing benchmarks isolate agents from the dynamic feedback loops necessary for optimization, treating brute-force solutions as equivalent to expert-level optimizations and failing to assess an agent's capacity to refine strategies over time.

Methodology: The CATArena Framework

To address these limitations, the authors introduce CATArena (Code Agent Tournament Arena), a dynamic evaluation framework designed to assess the evolutionary capabilities of code agents through iterative tournaments.

1. Iterative Tournament Structure

Unlike static benchmarks, CATArena immerses agents in a multi-turn development loop consisting of three phases:

Initial Development (Round 1): Agents generate an initial solution $C^{(1)}$ based on task specifications without external guidance.
Execution Feedback: Submitted code is executed in a sandboxed arena. The system aggregates results into a structured feedback set containing execution logs, a global leaderboard, and peer solutions.
Iterative Evolution (Round $n > 1$ ): Agents analyze the feedback to refine their code into $C^{(n)}$ $C^{(n)}$ . This process leverages two mechanisms:
- Self-Reflection: Analyzing execution logs to identify and fix bugs or logic errors.
- Peer-Learning: Analyzing superior solutions from other agents to adopt better strategies or algorithms.

2. Task Categorization

The framework instantiates six diverse tasks categorized by their feedback signals:

Competitive Tasks (Dynamic Feedback): Agents compete in a Round-Robin tournament against each other. The difficulty is non-stationary, as the "opponent" population evolves. Tasks include Gomoku, Texas Hold'em, Chess, and Bridge. Feedback includes full tournament logs and peer strategies.
Objective Optimization Tasks (Quasi-Static Feedback): Agents optimize against fixed metrics (e.g., time complexity, throughput) where the competitive standard is raised by the population's best performance. Tasks include OIBench (algorithmic efficiency) and SWE-Perf (repository-level performance speedup).

3. Evaluation Metrics

The authors propose a dual-metric system to decouple static proficiency from evolutionary potential:

Base Performance ( $S_{base}$ ): Measures the agent's static coding ability in a zero-shot setting (Round 1 performance relative to the initial population).
Evolutionary Capability ( $S_{evo}$ ): Quantifies the rate of improvement driven by feedback. It is calculated as the slope ( $k_i$ ) of the linear fit of an agent's global performance score ( $G^n_i$ ) over $N$ rounds using Ordinary Least Squares (OLS). This metric captures the trend of adaptation while mitigating the impact of transient failures or outliers.

Key Results

Extensive experiments were conducted using minimal agents (powered by various LLMs via Google ADK) and state-of-the-art (SoTA) commercial code agents.

Orthogonality of Capabilities: The study reveals a weak correlation between $S_{base}$ and $S_{evo}$ . High initial proficiency does not guarantee strong evolutionary capabilities. For instance, while Claude-4 dominated the initial round in Chess, it suffered significant regression during evolution, whereas Gemini-2.5-Pro demonstrated a "low-start, high-growth" trajectory in Gomoku.
Task-Dependent Dynamics: Evolutionary patterns vary significantly by task. Competitive tasks (e.g., Hold'em) tend to foster convergence (agents mimicking successful strategies), while objective tasks (e.g., OIBench) drive divergence (agents pushing toward theoretical limits). In highly complex environments like Chess, many agents exhibited negative evolutionary trends, indicating a reasoning gap that feedback could not bridge.
Mechanism Limitations: Ablation studies show that current agents struggle to synergize self-reflection and peer-learning. Most agents rely predominantly on one mechanism (often self-reflection) while treating the other as noise. Only select high-capability agents (e.g., Claude, Qwen) effectively leveraged peer solutions to achieve performance breakthroughs.
Robustness: The framework demonstrates high extensibility and resistance to task variants. When rules were perturbed (e.g., inverted hand rankings in Hold'em), $S_{base}$ scores varied significantly, but $S_{evo}$ maintained its discriminative power, confirming it measures intrinsic adaptive ability rather than memorized solutions.

Significance and Claims

The paper claims to be the first to identify and formalize the evolutionary capability of code agents as a distinct dimension of intelligence, separate from static generation proficiency.

Paradigm Shift: CATArena shifts the evaluation standard from static correctness to dynamic evolutionary capability, aligning with the iterative nature of real-world engineering.
New Frontier: The results suggest that evolutionary potential is a critical frontier for future agentic system research, as current commercial scaffolding enhances static generation but fails to address the mechanisms required for continuous self-improvement.
Reliability: By validating the framework against unseen task variants and diverse environments (including multilingual and ML tracks), the authors establish CATArena as a continuous and reliable standard for assessing the next generation of intelligent agents.

The authors conclude that while iterative feedback enables agents to overcome performance plateaus typical of single-round development, significant challenges remain in complex domains where agents fail to generate robust starting points for evolution.