Technical Summary: Mini Amusement Parks (MAPs)

Problem Statement

Current artificial intelligence systems, despite achieving superhuman performance on narrow benchmarks, struggle with the interconnected challenges inherent to real-world decision-making, particularly in business management. These domains require agents to simultaneously:

1. Plan over long horizons in stochastic environments.
2. Learn actively from sparse interactions to infer environment dynamics.
3. Reason over spatial information and interdependencies.
4. Optimize under uncertainty where outcomes are non-deterministic.

Existing benchmarks fail to capture this integration. Abstract reasoning tasks (e.g., ARC-AGI) lack real-world semantics; long-horizon planning benchmarks (e.g., Crafter, DiscoveryWorld) often simplify entity relationships to static prerequisites or lack stochasticity; and applied benchmarks (e.g., WebShop, OSWorld) typically focus on task completion rather than open-ended optimization under uncertainty. Consequently, there is no grounded testbed to evaluate how well agents integrate these capabilities.

Methodology: The MAPs Environment

The authors introduce Mini Amusement Parks (MAPs), a simulator where an agent acts as a park manager to maximize the park's value over a fixed time horizon. The environment is a 20×20 grid containing terrain, rides, shops, staff, and guests.

Core Mechanics

• Entities: Rides (carousels, ferris wheels, roller coasters) and shops (food, drink, specialty) drive capacity and guest satisfaction. Staff (janitors, mechanics, specialists) maintain operations.
• Progression: Entities exist in tiers (yellow, blue, green, red). In "Medium" mode, higher tiers must be unlocked via research, introducing a strategic layer where agents must anticipate future needs.
• Stochasticity: Guest behavior is probabilistic, driven by latent variables (hunger, thirst, happiness) and unobservable factors. Identical actions yield varying outcomes.
• Spatial Dynamics: Guest flow is influenced by path connectivity, adjacency to water (which increases ride excitement), and the placement of specialty shops (e.g., ATMs) in high-traffic areas.

Experimental Setup

The authors evaluated state-of-the-art (SotA) models against expert human performance across five specific challenges:

1. Open-Ended Objective (CH1): Comparing frontier models (GPT-5, Grok-4, Claude Sonnet 4.5, Gemini 2.5 Pro, Qwen3-VL) and Reinforcement Learning (PPO) agents against human experts.
2. Long-Horizon Planning (CH2): Comparing performance between "Easy" (50 steps, no research) and "Medium" (100 steps, research required) modes.
3. Active World-Model Learning (CH3): Testing if agents can improve performance by utilizing a "sandbox mode" with 100 days of exploratory actions (e.g., granting funds, undoing moves) to induce world models before evaluation.
4. Spatial Reasoning (CH4): Evaluating text-only LLMs, Vision-Language Models (VLMs), and LLMs augmented with a simple spatial heuristic to determine if visual input or rule-based placement aids performance.
5. Stochasticity (CH5): Quantifying variance in outcomes and testing Model Predictive Control (MPC) using either an oracle world model or a learned world model (WALL-E) to handle uncertainty.

Key Results

1. Performance Gap (CH1)

SotA models lag significantly behind expert humans.

• Easy Mode: The best model (GPT-5) achieved only 8.80% of the human upper bound.
• Medium Mode: Performance dropped to 4.44% of the human upper bound.
• Human Superiority: Experts outperformed systems by 11.4× on Easy and 15.3× on Medium modes.
• RL vs. LLM: PPO agents with simplified observation spaces (PPOSimple) outperformed those with full spaces (PPOFull), suggesting that the large action space exacerbates credit assignment issues for RL, whereas LLMs can leverage prior knowledge to navigate the full interface.

2. Long-Horizon Planning (CH2)

Performance deteriorates as planning demands increase.

• Models exhibit myopic optimization, often prioritizing immediate gains (e.g., building rides) over long-term strategic needs (e.g., researching staff or diverse attractions).
• Research strategies were suboptimal; models frequently researched expensive, unaffordable rides or failed to research staff, leading to cascading failures in park maintenance and cleanliness.

3. Active World-Model Learning (CH3)

Current LLMs fail to effectively leverage sandbox learning.

• In most cases, allowing agents to learn in the sandbox for 100 days resulted in no improvement or a decrease in performance.
• Failure Mode: Generated "learnings" often parroted the manual, were overly specific to transient states, or misattributed causal relationships (e.g., blaming a single action for systemic trends). The added context often diluted the prompt without providing actionable insights.

4. Spatial Reasoning (CH4)

• Visual Input: Providing images of the park to VLMs reduced performance compared to text-only inputs, suggesting visual modalities introduce confusion rather than clarity in this context.
• Heuristics: A simple spatial heuristic (placing rides near water and shops at intersections) significantly improved performance, indicating that current models lack the ability to independently reason about spatial interdependencies and crowd flow.

5. Stochasticity and World Models (CH5)

• Variance: The environment exhibits high variance in revenue and budget, though park value (the final metric) is relatively stable due to the law of large numbers.
• MPC Failure: Using a learned world model (WALL-E) for MPC degraded performance and increased costs/speed issues.
• Oracle Success: An oracle world model (perfect prediction) improved performance by >3.5×, demonstrating that the bottleneck is not the planning algorithm itself, but the inability of current systems to accurately model stochastic transitions.

Significance and Claims

The paper claims that MAPs provides a unified, scalable, and diagnostically rich testbed that exposes a broad spectrum of persistent gaps in contemporary AI systems. Its primary contributions are:

1. Benchmarking Integration: It is the first benchmark to simultaneously evaluate open-ended optimization, long-horizon planning, active world-model learning, spatial reasoning, and stochastic reasoning in a grounded business context.
2. Diagnostic Insights: The results identify specific failure modes:
   • Myopia: Inability to balance short-term gains with long-term constraints (research, staffing).
   • Inefficient Learning: Failure to construct useful world models from limited interaction data.
   • Spatial Deficits: Inability to reason about spatial layouts without external heuristics.
   • Stochastic Fragility: Brittle behavior when forced to model uncertainty without perfect predictors.
3. Future Direction: The paper argues that advancing real-world decision-making agents requires moving beyond narrow objectives and deterministic assumptions. It posits that future research must focus on sample-efficient world modeling, robust spatial reasoning, and strategic planning under uncertainty.

The authors conclude that while current frontier models show promise in specific sub-tasks, they remain far from the integrated, adaptable decision-making capabilities required for complex, real-world business management.