🤖 AI

Offline Multi-agent Continual Cooperation via Skill Partition and Reuse

The paper proposes COMAD, a principled framework for continual offline multi-agent skill discovery that leverages skill partition and reuse to overcome catastrophic forgetting and distributional shift, enabling agents to efficiently learn and expand coordination skills across sequential tasks.

Original authors: Yuchen Xiao, Lei Yuan, Ruiqi Xue, Tieyue Yin, Yang Yu

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuchen Xiao, Lei Yuan, Ruiqi Xue, Tieyue Yin, Yang Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a team of robots to work together to solve a series of different puzzles. In the real world, you can't just let them run around, fail, and try again forever (that's "online" learning); it's too expensive and dangerous. Instead, you have a giant library of old video recordings showing how humans or other robots solved these puzzles in the past. This is "offline" learning.

The problem is that the puzzles keep changing. One day they are moving boxes, the next day they are navigating a maze, and the day after that, they are playing a complex strategy game. If you just train the robots on the new puzzle, they often forget how to do the old ones (this is called "catastrophic forgetting"). If you try to force them to remember everything at once, they get confused and their performance drops (this is "interference").

This paper introduces a new method called COMAD (Continual Offline Multi-agent Skill Discovery) to solve this. Here is how it works, using simple analogies:

1. The Problem: The "One-Size-Fits-All" Trap

Previous methods tried to teach robots a fixed set of "skills" (like a toolbox with a hammer, a screwdriver, and a wrench). They assumed this toolbox was big enough for every new puzzle.

The Flaw: When a new, weird puzzle arrives that needs a "wrench" but also a "saw," the old toolbox isn't enough. The robots try to mash the new skill into the old ones, or they forget the old skills to make room for the new ones. It's like trying to fit a giant elephant into a small car; eventually, something breaks.

2. The Solution: A Dynamic "Skill Library"

COMAD treats skills not as a fixed list, but as a growing, organized library.

Step 1: The Librarian (The Auto-Encoder): First, COMAD looks at the old video recordings (the dataset) and acts like a smart librarian. It watches the robots working together and says, "Ah, I see a pattern here where they surround an enemy. Let's call that a 'Focus Fire' skill." It extracts these patterns and turns them into reusable "skill cards."
Step 2: The Multi-Headed Architect: Instead of having one brain that tries to do everything, COMAD gives the team multiple "heads" (specialized experts).
- When a new puzzle arrives, the system checks: "Do we already have an expert who knows how to handle this?"
- If the answer is yes (the new puzzle is similar to an old one), it reuses that expert's "head." This is Reuse.
- If the answer is no (the puzzle is totally new), it builds a new head specifically for this task. This is Partition.

3. The "Reusability Estimator": The Confidence Meter

How does the system know if it should reuse an old skill or build a new one? It uses a Confidence Meter.

Imagine you are walking into a room. If the room looks exactly like a room you've been in before, you feel confident and use your old memory of how to navigate it.
COMAD measures the "familiarity" of the current situation. If the situation is familiar, it turns up the volume on the old skills. If the situation is strange, it turns down the old skills and focuses on learning the new ones. This prevents the robots from getting confused by mixing up old and new instructions.

4. The Result: Learning Without Forgetting

By separating the skills (Partition) and only using the ones that fit (Reuse), COMAD achieves two things:

Forward Transfer: It learns new tasks faster because it can borrow useful tricks from old tasks.
Backward Transfer: It doesn't forget old tasks because it keeps them in their own specialized "heads" rather than overwriting them with new data.

The Bottom Line

Think of COMAD as a smart team of specialists rather than a single generalist.

Old methods were like a single generalist trying to memorize every single move for every game, eventually getting overwhelmed and forgetting the basics.
COMAD is like a team that builds a new specialist for a unique game but keeps a "phone book" of all their previous specialists. When a new game comes up, they check the phone book, call the right specialist, and if no one fits, they hire a new one. This allows the team to keep getting better at new things without losing their ability to do the old things.

The paper proves this works across many different types of robot teamwork scenarios, from grid-world games to complex strategy battles, showing that this "grow-as-you-need" approach is much more efficient and stable than trying to force everything into a fixed box.

Technical Summary: Offline Multi-agent Continual Cooperation via Skill Partition and Reuse (COMAD)

1. Problem Formulation

The paper addresses the challenge of Continual Offline Multi-agent Reinforcement Learning (MARL). In this setting, agents must learn from a sequential stream of offline datasets, where each dataset corresponds to a different task modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP).

The core difficulty lies in the exponential growth of the coordination skill space. Existing offline skill discovery methods typically rely on fixed-size, heuristically designed skill libraries. In open-ended environments where tasks arrive sequentially, these fixed libraries struggle with:

Distributional Shift: The mismatch between the learned policy and the behavior policy is exacerbated by the joint state-action space growing with the number of agents.
Catastrophic Forgetting: Agents lose previously acquired skills when adapting to new tasks.
Plasticity Loss: Agents fail to accommodate emerging skills or new coordination patterns.
Interference: Directly fine-tuning skill representations often leads to conflicting gradients, causing the policy to collapse into a suboptimal "coupled solution" that fails to distinguish between heterogeneous coordination patterns.

The goal is to construct an expandable skill library that allows agents to continually discover new coordination skills from offline data while selectively reusing task-invariant patterns from previous tasks to maximize performance on the current task without degrading performance on past tasks.

2. Methodology: COMAD Framework

The authors propose COMAD (Continual Offline Multi-agent Skill Discovery via Skill Partition and Reuse), a framework designed to partition skills and estimate their reusability. The training cycle for each incoming task consists of two stages:

A. Skill Discovery from Offline Data

To extract skills from mixed multi-agent behavior data, COMAD utilizes a Variational Auto-Encoder (VAE) architecture:

Global Encoder: Encodes global state-action pairs into skill representations ( $z_m^i$ ) based on global coordination patterns.
Local Encoder: Infers skills using only local agent information (historical trajectories $\tau^i$ ) to ensure executability in decentralized settings.
Action Decoder: Decodes skill embeddings into actions, forming a skill-conditioned policy.
Population-Invariant Networks: To handle varying numbers of agents, the framework employs Transformer-based self-attention mechanisms to process inputs of varying lengths into fixed-length representations.

B. Skill Partition and Reusability Estimation

To handle the expanding skill space and prevent modal collapse, COMAD employs a Multi-Head (MH) Architecture:

Multi-Head Policy: Instead of a single skill encoder, the actor uses multiple heads ( $G_k^\pi, G_k^p$ ) corresponding to different tasks or skill clusters. A shared feature extractor ( $F$ ) processes inputs, while specific heads are allocated or reused.
Reusability Estimator: A density-based estimator (using Noise Contrastive Estimation, NCE) approximates the state density $d_k(s)$ of previous tasks.
Skill-Augmented Objective: The advantage function is augmented with a weighted sum of action logits from previous skills. The weights, or reusability scores ( $\Delta_m^k(s)$ ), are calculated based on the estimated state density of the current task relative to past tasks. This guides the policy to leverage relevant past skills while avoiding interference.
Regularization: L2 regularization is applied to the feature extractors of new tasks to prevent catastrophic forgetting of the base representation learned on the first task.

C. Theoretical Foundation

The paper formalizes the continual learning problem as a constrained optimization task. Theoretical analysis (Theorem 4.1) demonstrates that the optimal continual policy is a geometric mixture of single-task optimal policies. The reusability scores in COMAD act as an approximation of the Lagrange multipliers in this optimization, serving as a gating mechanism to select useful skills and filter out those causing constraint conflicts.

3. Key Contributions

Problem Formulation: The paper defines the problem of continual offline skill discovery in multi-agent systems, highlighting the limitations of fixed-size skill libraries in open-ended environments.
COMAD Framework: It proposes a principled framework that combines skill partitioning (via multi-head architectures) and selective reuse (via density-based reusability estimation) to manage the expanding skill space.
Theoretical Analysis: The authors provide a theoretical proof showing that their method approximates the optimal solution for continual skill discovery, framing the reusability score as a mechanism to balance performance constraints and skill transfer.
Empirical Validation: Extensive experiments across five diverse environments (Level-based Foraging, Cooperative Navigation, SMAC, SMACv2, and Multi-agent MuJoCo) demonstrate the framework's effectiveness.

4. Experimental Results

The authors evaluate COMAD against three categories of baselines: offline skill discovery methods (ODIS, HiSSD), continual learning methods (EWC, OWL, Rehearsal), and vanilla single-head methods (Multi-task, Fine-tune, From Scratch).

Overall Performance: COMAD achieves superior average performance (64.34) across 9 task streams, significantly outperforming all baselines.
Forward Transfer: COMAD demonstrates strong forward transfer, leveraging past skills to accelerate learning on new tasks, outperforming methods that suffer from plasticity loss (e.g., ODIS, HiSSD).
Backward Transfer: The method maintains or improves performance on previous tasks (positive backward transfer) by partitioning skills and fine-tuning shared features, avoiding the catastrophic forgetting seen in Fine-tune (FT) and Elastic Weight Consolidation (EWC) methods.
Handling Heterogeneity: In complex scenarios like SMAC (mixing symmetric and asymmetric tasks) and SMACv2 (stochasticity and large team sizes), COMAD successfully partitions heterogeneous behaviors while reusing common coordination patterns, whereas baselines often fail to learn feasible policies or suffer from severe interference.
Ablation Studies: Removing the skill encoders or the reusability estimation mechanism significantly degrades performance, confirming the necessity of both skill partitioning and the gating mechanism.

5. Significance and Claims

The paper claims that COMAD provides a new formulation for multi-agent continual learning that effectively addresses the trade-off between stability (preserving old skills) and plasticity (learning new skills) in offline settings.

Principled Approach: Unlike heuristic approaches, COMAD is grounded in theoretical analysis showing that skill reuse can be optimized via reusability estimation.
Scalability: By dynamically expanding the skill library based on state density, the method avoids the bottlenecks of fixed-size libraries, making it suitable for open-ended environments where new tasks and skills continuously emerge.
Efficiency: The framework improves sample efficiency by extracting and reusing coordination skills, which is critical for offline MARL where online interaction is prohibited.

The authors acknowledge limitations, noting that the current reusability score relies on state density and may be insufficient for complex real-world scenarios requiring fine-grained semantic understanding. They suggest future work could integrate Large Language Models (LLMs) or human feedback to refine the skill gating mechanism. However, within the scope of the presented experiments, COMAD is presented as a robust and effective solution for continual offline multi-agent cooperation.