Technical Summary: PLATO (Pointer Learner for Agent and Task Openness)

Problem Statement

Open Agent Systems (OASYS) are multi-agent environments where the sets of agents and tasks change unpredictably over time. This "openness" manifests as Agent Openness (AO), where agents join or leave the system, and Task Openness (TO), where tasks appear or disappear. These dynamics fundamentally challenge standard Multi-Agent Reinforcement Learning (MARL), which typically assumes fixed state and action spaces. Existing methods address these issues only partially: padding and masking approaches introduce artificial bounds, while recent graph-based or hypergraph methods often handle only one dimension of openness or rely on restrictive assumptions (e.g., fixed agent sets or bounded task spaces). Consequently, there is a lack of frameworks that jointly handle AO and TO without requiring retraining or imposing fixed bounds on the agent-task space.

Methodology

The paper introduces PLATO (Pointer Learner for Agent and Task Openness), a Centralized Training and Decentralized Execution (CTDE) MARL algorithm designed to operate over unbounded state and action spaces.

1. Formalization: TaAgO-MG

The authors formalize the problem as a Task-and-Agent-Open Markov Game (TaAgO-MG). This extends prior Task-Open Markov Games (TaO-MG) by treating both the agent set ($N$) and task set ($X$) as dynamic variables that evolve over time. A generator function $\Psi$ adapts the base Markov game components (state, action, transition, and reward) at each timestep based on the current composition of agents and tasks.

2. Architecture

PLATO employs an actor-critic architecture with the following components:

• Pointer Actor (Decentralized):

  • Instead of classifying actions from a fixed vocabulary, the actor uses a pointer network mechanism.
  • Observation Factorization: The agent's local observation is factored into an agent-feature matrix and a task-feature matrix.
  • Query Generation: To handle AO, the agent set is aggregated into a fixed-length query vector using permutation-invariant statistics (mean, variance, min, max) processed by an MLP. This ensures the policy is invariant to agent ordering.
  • Key Generation: Each available task is encoded into a key vector via an MLP.
  • Content-Based Pointing: The actor computes attention scores between the query and task keys using additive attention. The output is a probability distribution directly over the currently available tasks (plus a NO-OP action). As tasks appear or disappear, the support of this distribution changes automatically without retraining.

• GNN Critic (Centralized):

  • The critic estimates the value of the global state during training.
  • It represents the open state as a bipartite graph where nodes are agents and tasks, and edges represent potential interactions.
  • A Graph Convolutional Network (GCN) propagates information across agent-task edges.
  • The graph structure dynamically changes with the number of agents and tasks, avoiding fixed-size concatenation. A mean pooling layer followed by an MLP produces a scalar value estimate.

3. Training

PLATO is trained using Multi-Agent Proximal Policy Optimization (MAPPO). The actor and critic are optimized jointly using Generalized Advantage Estimation (GAE). The computational complexity per timestep scales with the number of active agents ($|N_t|$) and tasks ($|X_t|$), specifically $O(|N_t|^2 + |N_t||X_t|)$, rather than a fixed bound.

Key Contributions

1. PLATO Architecture: The first MARL architecture to jointly handle AO and TO via content-based pointing, eliminating the need for fixed bounds, padding, or retraining when the environment composition changes.
2. TaAgO-MG Formalization: A rigorous extension of task-open formulations that provides formal guarantees that PLATO is well-defined and permutation-invariant over countably infinite, unbounded state and action spaces.
3. Empirical Evaluation: Comprehensive evaluation in the Wildfire Suppression domain (part of the MOASEI initiative), demonstrating performance across various grid sizes and openness configurations (S0–S3).
4. Zero-Shot Generalization: Evidence that policies trained on specific grid sizes generalize effectively to unseen, larger grid sizes without retraining, outperforming state-of-the-art baselines in consistency.

Experimental Results

The authors evaluated PLATO against smart baselines (DICG, DGN, MOHITO) and heuristic baselines (NOOP, Random, Strongest, Weakest) across 2×3, 3×3, and 4×4 grids under four setups ranging from closed (S0) to fully open (S3, combining endogenous spread, exogenous ignitions, and agent depletion).

• Performance: PLATO achieved the highest episode returns in the most demanding open setups (S2 and S3) on 3×3 and 4×4 grids with statistical significance. In the closed setup (S0), it performed on par with other smart methods.
• Efficiency: PLATO demonstrated superior reward-per-fight efficiency in most open configurations, indicating better resource allocation and task prioritization.
• Generalization: In zero-shot evaluations (training on smaller grids, testing on larger ones), PLATO showed more consistent generalization than baselines. Notably, MOHITO collapsed to negative returns in certain zero-shot open scenarios, whereas PLATO maintained positive returns.
• Ablation Studies: The authors investigated encoder types (MLP vs. LSTM) and attention scoring (additive vs. dot-product). Results indicated that the combination of LSTM + Additive attention yielded the strongest zero-shot generalization in highly open (S3) settings, while MLP + Dot-product performed best in simpler, closed (S0) settings.

Significance and Claims

The paper claims that PLATO addresses a fundamental limitation in current MARL: the inability to scale to unbounded, dynamic agent and task sets without architectural changes or retraining. By leveraging pointer networks and graph neural networks, PLATO treats the action space as a dynamic set of content rather than a fixed index.

The authors assert that PLATO provides a well-defined solution for OASYS, proving that the policy remains valid regardless of the number of agents or tasks. The empirical results suggest that this approach enables robust, scalable deployment in real-world scenarios where system composition is unpredictable, offering a significant step forward in making MARL applicable to open, dynamic environments. The work does not claim to solve all forms of openness (specifically excluding Frame Openness) but establishes a strong foundation for handling structural changes in agent and task sets.