StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley

Technical Summary of StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production–Living Simulations with Stardew Valley

Problem Statement
Autonomous agents aiming to navigate human society must master a complex interplay of "production-living systems," where productive activities (e.g., farming, crafting, resource generation) and living activities (e.g., social interaction, relationship building) are mutually reinforcing yet constrained by shared resources like time and energy. Existing benchmarks fail to evaluate these skills simultaneously. Traditional interactive benchmarks often focus on either low-level control and strategic gameplay with minimal textual roles, or socially situated interactions that lack structured production mechanisms and resource constraints. Consequently, there is a lack of systematic assessment for agents operating within the coupled, constraint-bound structures that characterize real-world daily life.

Methodology
To address this gap, the authors introduce StarDojo, a novel environment and benchmark built upon the open-ended simulation game Stardew Valley. The methodology involves three core components:

1. Environment Architecture: The authors developed StarDojoMod, a C# extension based on the Stardew Modding API (SMAPI). This mod facilitates a unified, user-friendly Python interface that allows agents to interact with the game engine directly via sockets. This architecture eliminates the need for manual screen captures or keyboard/mouse emulation, enabling the retrieval of internal game states (character status, tile information, global time/weather) and the execution of diverse action skills. The system supports parallel execution of multiple headless environment instances across major operating systems (Ubuntu, macOS, Windows).
2. Observation and Action Spaces: The observation space integrates visual modalities (gameplay screenshots at 720p) with structured textual data, including character status (health, energy, inventory), local tile information (7×7 grid), and global context (time, weather, season). The action space abstracts the game's core mechanics into ten fundamental skills (e.g., move, use, interact, craft, choose_item), excluding high-level shortcuts like navigate to ensure agents demonstrate realistic exploration capabilities.
3. Task Suite: The benchmark comprises 1,000 meticulously curated tasks across five domains: Farming, Crafting, Exploration, Combat, and Social Interactions. Tasks are categorized by difficulty (Easy, Medium, Hard) and require varying degrees of resource management, long-term planning, and adaptation to dynamic environmental factors (time, stamina, weather). A subset of 100 representative tasks, StarDojo-Lite, is provided for efficient evaluation. An automated evaluation system monitors incremental state changes to determine task completion.

Key Contributions

• StarDojo Benchmark: The first benchmark to simultaneously evaluate production activities and social interactions within a dynamic, constraint-bound simulation, bridging the gap between embodied AI and social reasoning.
• Unified Interface: A robust, cross-platform environment that supports parallelized, headless execution and direct internal state access, significantly improving evaluation efficiency compared to screen-based or API-dependent methods.
• Comprehensive Task Design: A diverse set of 1,000 tasks requiring agents to balance time, energy, and resources while engaging in bidirectional feedback loops between production (generating income/items) and social progress (unlocking recipes, discounts, and relationships).
• Open-Source Release: The authors release the environment, setup instructions, evaluation scripts, and baseline implementations to facilitate further research.

Experimental Results
The authors evaluated seven state-of-the-art Multimodal Large Language Models (MLLMs), including GPT-4.1, Gemini 2.5 Pro, Claude 3.7 Sonnet, and several open-source models (Llama 4, Qwen2.5-VL, Gemma 3), on the StarDojo-Lite suite.

• Performance: Even the best-performing model, GPT-4.1, achieved a success rate of only 12.7% on the Lite task set. Open-source models performed significantly worse, with success rates below 8%.
• Difficulty Scaling: While models showed some proficiency on easy tasks (requiring <30 steps), they exhibited near-zero success rates on medium and hard tasks that demand extended action sequences and strategic planning.
• Error Analysis: A detailed analysis of GPT-4.1 failures identified four primary failure modes:
  1. Visual Understanding (42%): Difficulty recognizing small objects (approx. 10x10 pixels) and determining their precise spatial relationship to the agent.
  2. Multimodal Reasoning (21%): Over-reliance on visual input despite available textual state information, leading to hallucinations or confusion.
  3. Long-Term Planning (21%): Inability to maintain consistent strategies over long horizons, often abandoning plans prematurely.
  4. Low-Level Control (16%): Instability in executing fine-grained actions, such as selecting incorrect parameters or directions.
• Ablation Studies: Experiments removing textual input (Image Only) or visual input (Text Only) confirmed that both modalities are critical. Furthermore, "Real-time" evaluations (without pausing the game during inference) revealed significant performance degradation due to inference latency, highlighting a bottleneck for API-based agents in dynamic environments.

Significance
The paper claims that StarDojo serves as a critical testbed for advancing agentic AI in complex, real-world-like environments. By exposing substantial limitations in current MLLMs regarding visual understanding, multimodal reasoning, long-term planning, and real-time inference, the benchmark identifies specific areas requiring future research. The authors position StarDojo not as a solution for immediate deployment, but as a foundational tool to facilitate the development of robust, open-ended agents capable of lifelong learning and strategic decision-making within production-living systems. The work underscores the necessity of evaluating agents in interactive environments where spatial and physical reasoning are as essential as language capabilities.