Evaluating Large Language Models for Decision-Making in Agent-Based Urban Mobility Simulations

Technical Summary: Evaluating LLMs for Decision-Making in Agent-Based Urban Mobility Simulations

Problem Statement
Urban mobility modeling struggles to accurately represent individual decision-making within dynamic, uncertain environments. Traditional Multi-Agent Systems (MAS) simulations, while effective for analyzing emergent behaviors, often rely on fixed heuristics and predefined rules. These rule-based approaches limit behavioral adaptability, particularly in high-contextual-variability scenarios such as sudden road blockages or traffic disruptions. While Large Language Models (LLMs) offer advanced reasoning and semantic capabilities, their integration into spatially explicit MAS simulations remains underexplored, especially regarding how they function as decision layers within geospatial constraints.

Methodology
The authors propose a hybrid architecture that integrates an LLM-based decision module into the GAMA platform, a robust environment for agent-based modeling with native Geographic Information System (GIS) support. The system does not replace traditional routing algorithms; instead, it introduces a cognitive layer that determines when agents should trigger route recalculation.

The architecture consists of three primary components:

1. Simulation Platform (GAMA): Manages the spatial environment, road networks, and agent movement. Agents monitor congestion and receive alerts regarding road blockages.
2. Integration Layer: A Python-based API (using FastAPI) that facilitates asynchronous communication between GAMA and the external decision module. It manages a relational database to store agent states, environmental conditions, and decision logs, enabling traceability and persistent memory retrieval.
3. Agentic Framework (Agno): The cognitive layer where LLMs (specifically Google's Gemini 2.5 Flash-Lite and OpenAI's GPT-4o Mini) process contextual inputs. The system prompts the LLM with agent attributes (e.g., occupation, digital trust level), current travel conditions (e.g., fatigue, weather), and environmental factors (e.g., congestion, distance to blockage). The LLM returns a structured JSON decision: either to "STAY" on the current route or "RECALCULATE."

The study evaluates two configurations: a rule-based baseline using a threshold-based heuristic score, and LLM-assisted models with and without persistent semantic memory. Experiments were conducted across two scenarios (Localized and Extended blockages) and two population scales (500 and 1000 agents), measuring metrics such as replanning rates, arrival rates, stuck time ratios, and computational overhead.

Key Contributions

• Hybrid Decision Architecture: The paper presents a novel integration of LLMs as a decision filter within a GAMA-based urban mobility simulation, decoupling cognitive reasoning from spatial execution.
• Context-Aware Adaptation: Unlike static heuristics, the proposed system allows agents to interpret complex contextual information (e.g., combining weather, trust levels, and congestion) to make route decisions.
• Memory Mechanisms: The implementation of persistent memory allows agents to retrieve past experiences and decision outcomes, influencing future behavior to promote consistency and reduce excessive reactivity.
• Empirical Evaluation: The study provides a comparative analysis of LLM-assisted agents against rule-based baselines across varying network constraints and population densities, offering insights into the trade-offs between adaptability and computational cost.

Results

• Localized Blockages: In scenarios with available alternative routes, LLM-assisted agents demonstrated greater adaptability. They achieved higher arrival rates (e.g., increasing from 70% to ~90% for 500 agents with OpenAI and memory) and more balanced spatial distribution. Memory acted as a stabilizer, reducing excessive replanning and improving decision consistency, though it occasionally increased "stuck time" depending on the specific model configuration.
• Extended Blockages: In scenarios with severe network constraints (few alternative routes), the impact of LLMs on global performance (arrival rates) was limited. However, LLM-assisted agents still significantly reduced the "stuck time ratio" compared to rule-based models, indicating improved local congestion avoidance even when global optimization was structurally impossible.
• Agent Heterogeneity: LLMs induced heterogeneous responses across different agent profiles (students, workers, retirees), particularly in localized scenarios. In extended blockage scenarios, structural constraints dominated, causing behavioral convergence across profiles.
• Computational Cost: Integrating LLMs introduced substantial computational overhead, with simulation times increasing significantly (e.g., from ~22 seconds for rule-based to ~240 seconds for OpenAI with memory for 1000 agents). However, error rates remained low (<1.25%), indicating robust system stability despite the latency.

Significance and Claims
The paper claims that LLMs serve effectively as complementary cognitive layers rather than replacements for traditional routing algorithms. The primary significance lies in the ability of LLMs to enrich behavioral representations in urban mobility simulations, enabling agents to exhibit greater contextual awareness and adaptive behavior in dynamic environments.

The authors modestly conclude that while LLMs enhance the realism of decision-making and improve local adaptation, their effectiveness is bounded by the physical constraints of the road network. The study highlights a trade-off: the integration of LLMs offers richer, more realistic behavioral modeling at the cost of increased computational resources. Future work is identified as focusing on scalability, refining memory mechanisms for efficiency, and extending the evaluation to other decision-making tasks beyond route recalculation.