Second MOASEI Competition at AAMAS'2026: A Technical Report

Technical Summary: Second MOASEI Competition at AAMAS'2026

Problem Statement
The paper addresses the challenge of evaluating multi-agent decision-making systems within open agent environments. Unlike traditional closed systems, these environments are characterized by dynamic changes where agents, tasks, and agent capabilities can appear, disappear, or evolve over time. The competition specifically targets three dimensions of openness:

• Agent Openness (AO): Agents entering or leaving the system.
• Task Openness (TO): Tasks appearing or disappearing over time.
• Frame Openness (FO): Changes in the internal state or capabilities of agents (e.g., equipment degradation or state changes).

The 2026 competition aimed to refine the evaluation infrastructure established in 2025, introducing new forms of openness and more interpretable, task-centered metrics to better assess coordination, robustness, and adaptation strategies.

Methodology and Competition Design
The competition utilized the free-range-zoo environment suite, which implements benchmark domains inspired by open agent systems. The 2026 edition retained three core tracks from the previous year while introducing specific modifications:

1. Evaluation Tracks:

   • Wildfire Fighting (AO/TO): Agents collaborate with limited resources to fight spreading fires.
   • Bonus Wildfire Fighting (AO/TO/FO): A new track introducing Frame Openness, where agent equipment states (suppressant capacities and firefighting ranges) vary over time, altering their effectiveness and reach.
   • Cybersecurity (AO): Defender agents compete against heuristic attackers to protect network nodes.
   • Ride-Sharing (TO): Agents cooperatively and competitively deliver passengers, focusing on task openness where passengers enter the system dynamically.

2. Metrics:
   The evaluation shifted toward task-centered metrics to improve interpretability. Beyond domain rewards, the competition emphasized:

   • Total task completions.
   • Mean task-completion time.
   • Mean value of completed tasks.
   • For ride-sharing specifically: passenger count, wait time, ride time, and fare value.

3. Participation and Submission:
   Eight teams registered, but only one team, DLC, submitted a final solution, targeting the ride-sharing track. Consequently, the analysis focuses exclusively on this track.

4. Proposed Approach (DLC):
   The winning team employed a planning and replanning strategy. The agent solves an optimal routing problem across the fleet of agents. Crucially, the system replans whenever new passengers enter the environment, allowing the policy to adapt to task openness in real-time.

5. Baseline Comparison:
   To contextualize the DLC submission, the authors evaluated it against several non-competition baseline policies:

   • No-operation (noop)
   • Random
   • First-In-First-Out (FIFO) variants (focus and global)
   • Greedy variants (focus and global)

Key Results
The DLC policy was evaluated across six ride-sharing configurations (three for training, three for testing) over 213 episodes.

• Throughput: DLC completed a mean of 13.62 passengers, significantly outperforming FIFO (10.80) and Random (7.40) baselines. It performed comparably to the strongest greedy baseline (15.38).
• Efficiency (Wait Time): DLC achieved a mean wait time of 24.13, which was substantially lower than FIFO baselines (~36–39) and competitive with the greedy baseline (25.48).
• Efficiency (Ride Time): A notable trade-off was observed. DLC's mean ride time was 34.87, which was significantly longer than the greedy (12.14) and FIFO (14–17) baselines.
• Configuration Stability: DLC maintained stable throughput across all six configurations (ranging from 13.08 to 14.91 completed passengers), though ride times varied more, indicating sensitivity to spatial and task-arrival differences.

Key Contributions and Claims
The paper makes the following modest contributions and claims:

1. Validation of Planning in Open Systems: The primary empirical contribution is a case study demonstrating that planning-based replanning is a viable approach for task-open environments. The DLC agent successfully adapted to dynamic passenger arrivals, achieving stable completion counts and competitive wait times.
2. Identification of Objective Trade-offs: The results highlight a specific trade-off in the ride-sharing domain: the DLC policy prioritized throughput and wait time (accepting more tasks and picking them up faster) at the cost of ride duration. The authors note that the current leaderboard objective does not explicitly balance these competing goals (completions, wait time, ride time, fare value).
3. Benchmark Evolution: The competition successfully introduced Frame Openness via the bonus wildfire track and refined evaluation metrics to be more task-centered.
4. Gap Filling: By securing a submission for the ride-sharing track (which received none in 2025), the 2026 competition provides a more comprehensive view of multi-agent decision-making across different openness dimensions when combined with the 2025 results.

Significance and Limitations
The authors explicitly state that due to the limited number of final submissions (only one team), the competition does not support broad conclusions regarding the relative strengths of different learning or planning paradigms. Instead, the significance lies in:

• Providing a focused case study of a planning-based policy in a task-open environment.
• Documenting the updated benchmark design, including the new Frame Openness track.
• Highlighting the need for clearer objective functions in future competitions to balance competing metrics (e.g., throughput vs. ride time).
• Identifying the need for better onboarding materials (starter policies, documentation) to encourage participation in complex tracks like the bonus wildfire domain.

The paper concludes that while the 2026 event had low participation, the results from the ride-sharing track offer a valuable benchmark for future research into multi-agent planning under open-system conditions.