Technical Summary: ASALT – Adaptive State Alignment for Lateral Transfer in Multi-agent Reinforcement Learning

Problem Statement

Multi-agent Reinforcement Learning (MARL) faces significant challenges when transferring knowledge from a source domain to a target domain where the environments differ structurally. While prior work has explored transfer learning in MARL, most existing approaches impose a strict constraint: the observation space and global state space dimensionalities must be identical across source and target domains.

In realistic scenarios, such as traffic management, fleet coordination, or competitive games like StarCraft II, the number of agents, adversaries, or environmental configurations often changes. These changes alter the dimensionality of the observation and state spaces. Consequently, standard lateral transfer methods fail because they cannot map heterogeneous inputs into a shared representation without forcing identical network architectures. Furthermore, existing methods often focus solely on transferring actor (policy) information, neglecting the potential benefits of transferring critic (value function) information, which is crucial for estimating advantage functions and coordinating agents. Finally, transferring policies between domains with conflicting coordination requirements often leads to "negative transfer," where prior knowledge degrades rather than improves performance.

Methodology: The ASALT Framework

The authors propose ASALT (Adaptive State Alignment for Lateral Transfer), a framework designed to accommodate mismatched state-space dimensionalities between source and target domains without imposing structural constraints on the network architecture. The core innovation lies in the use of adapter modules that map target-domain inputs into a shared embedding space compatible with pre-trained source agents.

Key Components

1. Observation Adapter:

   • Function: Maps target-domain observations ($o_t$) into a shared embedding space ($\hat{e}_o$) that aligns with the input space of frozen source actors.
   • Architecture: The adapter decomposes target observations into agent-specific features ($o_a$) and contextual features ($o_c$) describing other agents.
     • Agent-specific features are processed via a Fully Connected Network (FCN).
     • Contextual features are processed via a Hierarchical Multi-Head Attention (H-MHA) mechanism. This module captures higher-order relational dependencies across structured property groups (e.g., allies vs. enemies), producing embeddings that aggregate contextual information.
   • Output: The final embedding is a concatenation of agent-level, relational, and aggregated contextual embeddings, sized to match the source policy's input dimension.

2. State Adapter:

   • Function: Maps the global state of the target domain ($s_t$) into a shared state embedding ($\hat{e}_s$) for input to source critics.
   • Architecture: Utilizes a two-layer Transformer Encoder. The global state is projected into token embeddings and processed through Multi-Head Self-Attention (MHSA) and Feed-Forward Networks (FFN) with residual connections and layer normalization. This explicitly captures interactions among state components.
   • Purpose: Enables the transfer of coordination patterns encoded in the value functions (critics) of the source domain.

3. Transfer Module (Lateral Connections):

   • Mechanism: Implements lateral knowledge transfer from frozen source agents to the target agent.
   • Actor Transfer: The target actor attends to representations from top-$N$ source actors (selected via clustering or similarity). Layer-wise attention mechanisms ($\alpha_k$) weight and aggregate these lateral signals.
   • Critic Transfer: Similarly, lateral connections propagate weighted representations ($\beta_k$) from source critics to the target critic. To handle differing reward scales, rewards are normalized before constructing critic-side signals.
   • Training: The target agent is trained using the original target-domain inputs ($o_t, s_t$) while simultaneously receiving lateral guidance from the source agents via the adapters.

Training Paradigms

The paper investigates two training strategies:

• Independent Training: Adapters are trained first on collected trajectories from both domains, then frozen while the target agent is trained.
• Joint Training: Adapters and the target agent are optimized concurrently using target-domain trajectories. The authors find joint training to be more sample-efficient, as it induces embeddings that explicitly maximize the utility of transferred information.

Key Contributions

1. Handling Dimensionality Mismatch: ASALT is the first method to explicitly accommodate mismatched observation and global state space dimensionalities between source and target domains in MARL, removing the requirement for identical network architectures.
2. Dual-Adapter Architecture: The introduction of both Observation and State adapters allows for the transfer of both policy (actor) and value (critic) knowledge, facilitating more effective coordination transfer.
3. Mitigation of Negative Transfer: By mapping target inputs into a shared latent space and selectively transferring task-relevant information (rather than raw observations), ASALT reduces the risk of negative transfer, particularly in heterogeneous domains.
4. Sample Efficiency: The framework significantly improves sample efficiency, allowing target agents to converge faster than training from scratch or using existing baselines.

Experimental Results

The authors evaluated ASALT on three benchmark environments: StarCraft II Multi-Agent Challenge (SMAC), Google Research Football (GRF), and Multi-Particle Environments (MPE).

• Performance vs. Baselines: ASALT consistently outperformed domain adaptation methods (DANN, CORAL, CycleGAN) and state-of-the-art transfer learning baselines (LA-QTransformer, MALT, PSMARL).
• Sample Efficiency: In scenarios with observation space mismatches (e.g., transferring from 3 agents to 8 agents in SMAC), ASALT required significantly fewer environment steps to reach 80% of the asymptotic reward compared to all baselines. For instance, in the 3m to 8m transfer, ASALT required $0.54 \times 10^5$ steps versus $5.1 \times 10^5$ for the baseline.
• Negative Transfer: In scenarios with conflicting coordination requirements (e.g., 3m map vs. 3s5zvs3s6z), ASALT demonstrated superior robustness, converging faster and achieving higher final win rates than methods prone to negative transfer.
• Ablation Studies:
  • Joint vs. Independent: Joint training of adapters and the target agent yielded significantly faster convergence than independent training.
  • Actor vs. Actor-Critic: Transferring both actor and critic information generally improved stability and early learning, though in cases of massive state-space expansion, actor-only transfer sometimes achieved comparable asymptotic performance.
  • Attention Mechanisms: Attention-based observation adapters outperformed simple FCN-based adapters, particularly in up-scaling scenarios.

Significance and Claims

The paper claims that ASALT addresses a critical bottleneck in real-world MARL deployment: the inability to transfer policies when the number of agents or environmental complexity changes. By decoupling the target input dimensionality from the source policy architecture through adaptive embedding, ASALT enables lateral transfer in heterogeneous settings.

The authors emphasize that their approach:

• Reduces Negative Transfer: By learning to map target states into a shared representation, the method filters out irrelevant or conflicting information, a common failure mode in cross-domain transfer.
• Enhances Coordination: The inclusion of state adapters and critic transfer allows the system to leverage value-based coordination patterns, not just action policies.
• Maintains Computational Efficiency: While adding adapter modules incurs a marginal computational overhead (approx. 21% increase in training time in their experiments), this is offset by the substantial reduction in the number of environment interactions required for convergence.

The work concludes that while ASALT is highly effective, its performance is dependent on the degree of mismatch between domains, and future work should explore hierarchical transfer and theoretical analysis to further reduce computational costs.