Technical Summary: SidConArena

Problem Statement

Current benchmarks for Large Language Model (LLM) agents often rely on static reasoning tasks or zero-sum competitive games. These environments fail to capture the complexity of real-world economic interactions, which are typically open-ended, mixed-motive, and involve positive-sum opportunities. In such settings, agents must simultaneously negotiate, create surplus through cooperation, compete for scarce assets, and execute long-horizon investment plans under partial observability. Existing evaluations provide limited insight into how agents behave in these dynamic, socially interactive, and economically grounded scenarios.

Methodology

The authors introduce SidConArena, a benchmark framework that formalizes a multi-player economy as a finite-horizon Partially Observable Stochastic Game (POSG). The framework is built upon the economic structure of the board game Sidereal Confluence and consists of three coupled phases per turn:

1. Negotiation (Unstructured Cooperative Game): Agents address inherent production deficiencies by engaging in natural-language bargaining. They exchange resources via binding trade vectors. Trades are executed only if they increase the heuristic utility for both parties, modeled as a cooperative sub-game.
2. Production (Deterministic Optimization): Agents activate production converters to transform input resources into outputs. This phase is a deterministic, Knapsack-like combinatorial optimization problem constrained by the inventory accumulated during negotiation.
3. The Confluence (Competitive Imperfect-Information Game): Agents compete for permanent assets (Colonies and Technologies) via a Simultaneous Multi-Track Priority Auction. Agents submit sealed bids using a liquid currency ("Ships"). Allocation follows a "Right-of-Choice" mechanism based on bid ranking and asset availability, introducing stochasticity and hidden information.

Framework Architecture

SidConArena employs a modular architecture designed to separate agent reasoning from environment execution:

• Observation Construction: The environment constructs structured observations from four channels: private inventory/capabilities, public board signals, market context (auction data), and interaction history.
• Phase-Aware Dispatching: A central "Brain" routes tasks to specialized reasoning modules (e.g., TurnPlanCaller, TradeCaller, EconomyCaller, BidCaller) based on the current game phase.
• Neural-Symbolic Action Interface: To reconcile free-form natural language with rigid game logic, agents generate function calls rather than direct state modifications. This interface validates syntax and ensures quantitative precision for trades, bids, and production plans.
• Asynchronous Execution: Unlike standard turn-based systems, the negotiation phase utilizes a non-blocking, event-driven pipeline allowing high-frequency, simultaneous interaction among agents.

Experimental Protocol

The evaluation utilizes two complementary settings:

1. Homogeneous Self-Play: All agents in a game are instantiated from the same LLM backbone to measure absolute economic performance.
2. Heterogeneous Elo Tournaments: Different model backbones interact in the same economy. Agents are ranked using an Elo-style rating system derived from terminal economic outcomes.

The study evaluates a diverse suite of models, including compact agents (e.g., GPT-4o-mini), general-purpose chat models (e.g., Qwen-Plus, DeepSeek-V3), and high-capability frontier models (e.g., GPT-5, Gemini-3-Flash-Preview, Claude-Opus-4).

Key Contributions

• A New Benchmark Framework: SidConArena provides a structured environment for evaluating LLM agents in open-ended, positive-sum bargaining, moving beyond static or zero-sum paradigms.
• Hybrid Interaction Design: The framework successfully combines natural-language negotiation with rule-grounded, executable game mechanics via a neural-symbolic interface.
• Comprehensive Evaluation Metrics: It introduces terminal economic value as the primary objective, alongside Elo ratings for competitive strength, and detailed trajectory analysis for behavioral diagnostics.

Results

The experiments reveal distinct patterns in agent capabilities:

• Model Differentiation: Stronger frontier models generally achieve higher terminal scores and Elo ratings. However, performance is not solely a function of model generation age; specific capabilities in bargaining and valuation are required.
• Valuation Errors: Despite producing syntactically valid actions, agents frequently misvalue resources. A recurring failure mode involves the overvaluation of "Ships" (the bidding currency), where agents treat them as intrinsically scarce strategic assets rather than adhering to their local marginal value defined by the environment.
• Passive Bargaining: Agents tend to be socially cooperative but strategically passive. They often accept reasonable offers immediately rather than leveraging scarcity, counter-offering, or inducing competition among buyers to extract surplus.
• Short-Horizon Planning: Agents struggle with long-horizon investment planning. While individual turn decisions may appear locally defensible, agents often fail to construct a coherent trajectory from early investment to compound production and late-game score conversion, resulting in weak terminal outcomes.

Significance

The paper posits that SidConArena serves as a critical testbed for identifying the gap between surface-level agent competence (instruction following, role-playing) and deeper economic reasoning. The framework demonstrates that successful performance in mixed-motive economies requires a combination of context-specific valuation, strategic bargaining, and long-horizon planning under partial observability.

The authors conclude that current LLM agents, while capable of executing the game mechanics, remain far from robust economic decision-makers. They often fail to ground decisions in the task-specific utility model, exhibit passive negotiation strategies, and lack the ability to connect local decisions into coherent long-term investment plans. SidConArena is intended to facilitate the development and evaluation of future agents capable of reliable positive-sum coordination and strategic economic planning.