Technical Summary: EcoGym – Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies

Problem Statement

Long-horizon planning is a recognized core capability of autonomous Large Language Model (LLM) agents. However, current evaluation frameworks suffer from significant limitations: they are largely episodic, confined to specific domains, or insufficiently grounded in persistent economic dynamics. Existing benchmarks often rely on closed, proprietary pipelines (e.g., Vending Bench) or fail to reflect the heterogeneous, interdependent nature of open-ended economic activity. Furthermore, there is a gap between performance in controlled virtual benchmarks and robustness under continuous, high-stakes practical deployment, where agents often exhibit fragility despite strong short-term results. The field lacks a transparent, community-driven framework to assess agents on sustained, strategic coherence over effectively unbounded time horizons.

Methodology

To address these gaps, the authors introduce EcoGym, a generalizable, open-source benchmark for continuous plan-and-execute decision-making in interactive economies.

Design Principles

EcoGym is built upon three core design principles:

1. Simple Action Space, Unbounded Horizon: Unlike benchmarks with complex, high-dimensional action spaces, EcoGym restricts agents to a small set of discrete actions (typically 4–5 primitives). However, the interaction horizon is effectively infinite (1000+ steps, simulating 365-day loops), shifting the evaluation focus to long-term strategic planning, stability, and cumulative optimization rather than short-horizon task completion.
2. Economic Environments as Evaluation Ground: The benchmark grounds evaluation in daily economic activities, moving beyond code generation or simple reasoning to assess how LLMs function as economic actors. Success is measured by tangible market outcomes (e.g., Net Worth, Income, DAU) rather than simple rewards.
3. Latent Mechanics for Exploratory Discovery: The underlying mathematical rules governing system dynamics (e.g., seasonality, elasticity, decay rates) are hidden from the agent. Agents must actively explore, hypothesize, and discover these mechanics through interaction and feedback to maximize utility.

Environments

EcoGym comprises three distinct environments unified under a single decision-making process:

• Vending: A retail management challenge adapted from the closed-source Vending-Bench (now open-sourced). Agents act as sole proprietors maximizing Net Worth through strategic procurement and dynamic pricing. The environment features hidden market parameters like seasonality and price elasticity.
• Freelance: A gig-economy challenge where agents maximize Income while balancing labor, skill development, and health management (energy/stress). Agents must avoid burnout (a hidden physiological threshold) and navigate a task pool derived from diverse datasets (coding, finance, legal, STEM).
• Operation: A platform growth challenge where agents act as operators maximizing Daily Active Users (DAU). The system exhibits "Zero-Attractor" dynamics where user activity decays to zero without intervention. Agents must manage content quality, creator incentives, and user acquisition under hidden system coefficients.

Experimental Setup

The authors evaluated 11 leading LLMs (including GPT-5.2, Gemini-3-Pro/Flash, Claude-Sonnet-4.5, Kimi-k2.6, Qwen3.5, DeepSeek-v3.2, and GLM-5) across these environments. The evaluation included:

• Standard Metrics: Net Worth, Income, and DAU over 365-day simulations.
• Diagnostic Studies: Eight specific analyses covering context window length, failure modes, temporal behavioral evolution, human baselines, memory modules (working, symbolic, episodic, Mem0), "thinking with action," and environment complexity scaling.

Key Results

The experiments reveal a systematic tension in current LLM capabilities: no single model dominates across all three scenarios.

• Performance Variance: In the Vending environment, the Gemini-3 series (particularly Pro) demonstrated dominant asset appreciation. In Freelance, Gemini-3-Pro maintained a lead, though the margin was narrower. In Operation, Claude-Sonnet-4.5 ranked first.
• Suboptimality: Models exhibited significant suboptimality in either high-level strategies or efficient action execution. For instance, top models in Operation prioritized scale (quantity) over refinement (quality), correctly identifying the dominant variable for yield, whereas others failed to align with the reward mechanism.
• Context Window Sensitivity: Extending the context window beyond the default (128 steps) did not yield consistent gains. Gemini-3-Flash showed volatile performance with longer contexts, while Gemini-3-Pro's performance degraded as the window extended to 1024 steps, highlighting instability in processing super-long contexts.
• Memory and Reasoning:
  • Memory: Adding external memory modules (working, symbolic, episodic) generally enhanced performance but was not universally beneficial; efficacy was highly model-dependent and task-specific.
  • Thinking Mode: Enabling "Thinking" (generating reasoning content integrated into the context) catalyzed a universal performance elevation across models, significantly narrowing the gap between Flash and Pro variants in the Operation task.
• Human Baseline: In the Operation environment, human experts achieved an average DAU of 1,404. Notably, Claude-Sonnet-4.5 surpassed this human baseline, suggesting SOTA LLMs have the potential for super-human performance in specific long-horizon economic planning scenarios.
• Complexity Scaling: When the Vending environment complexity was increased (more SKUs), Gemini-3-Flash showed robust adaptability with profits scaling positively, whereas Gemini-3-Pro stagnated, failing to capture additional value in larger state spaces.

Key Contributions

1. Infinite-Horizon Planning Evaluation: EcoGym provides an open, generalizable framework for assessing LLM agents under continuous, non-episodic interaction with a compact action space and unbounded temporal horizon. It isolates long-term strategic coherence and stability as first-class evaluation targets.
2. Utility-Guided Economic Assessment: The benchmark establishes an outcome-oriented paradigm grounded in economic returns (Net Worth, Income, DAU) across three widely-existing economic settings, moving evaluation beyond simple rewards to tangible market impact.
3. Multi-Dimensional Empirical Analysis: Through rigorous evaluation of state-of-the-art LLMs and eight diagnostic studies, the authors uncover critical performance gaps and bottlenecks in long-horizon planning, such as the lack of a universal winner, context window instability, and the specific benefits of "thinking" modes and memory architectures.

Significance and Claims

The paper claims that EcoGym addresses a critical void in existing benchmarks regarding the evaluation of agents' long-term economic viability. By forcing agents to navigate resource scarcity and stochastic dynamics over sustained periods, the framework reveals that while frontier models demonstrate impressive short-term reasoning, they struggle to maintain strategic coherence over long-time decisions.

The authors modestly acknowledge that the current framework relies on rule-based abstractions and may not fully capture the unpredictability of irrational human behavior. They position EcoGym as an open, extensible testbed for transparent long-horizon agent evaluation and for studying controllability–utility trade-offs in economic settings. The work aims to guide the community toward developing general-purpose agents that are not only capable of reasoning but are also robust and strategically aligned over the long haul.