🤖 AI

MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning

This paper introduces MEAL, the first benchmark for continual multi-agent reinforcement learning that leverages JAX and GPU acceleration to efficiently train on long sequences of 100 tasks, thereby revealing failure modes invisible in shorter, traditional studies.

Original authors: Tristan Tomilin, Luka van den Boogaard, Samuel Garcin, Constantin Ruhdorfer, Bram Grooten, Fabrice Kusters, Yali Du, Andreas Bulling, Mykola Pechenizkiy, Meng Fang

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tristan Tomilin, Luka van den Boogaard, Samuel Garcin, Constantin Ruhdorfer, Bram Grooten, Fabrice Kusters, Yali Du, Andreas Bulling, Mykola Pechenizkiy, Meng Fang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a group of robots how to run a busy kitchen. They need to learn how to chop onions, cook soup, and serve it to customers. Now, imagine that every few minutes, the kitchen layout changes completely: the stove moves, the walls shift, and the customers change their orders.

This is the problem the paper MEAL tackles. It introduces a new "training ground" (a benchmark) to see how well these robot teams can keep learning new kitchen layouts without forgetting how to cook the old ones.

Here is the breakdown of the paper in simple terms:

1. The Problem: The "Short Memory" of AI

Right now, most AI research on "lifelong learning" (learning forever) is like studying a student who only takes three or four math tests in a row. The computers used to run these tests are slow (like old CPUs), so researchers can't afford to make the students take 100 tests in a row.

Because of this, we don't know what happens when an AI has to learn a long sequence of tasks. Does it get tired? Does it start forgetting how to do the first task when it learns the tenth?

Furthermore, most of this research looks at one robot learning alone. But in the real world, robots often work in teams. When they work together, things get messy. If Robot A forgets how to chop onions, Robot B might get stuck waiting for them, and the whole team fails.

2. The Solution: MEAL (The Super-Fast Kitchen Simulator)

The authors built MEAL (Multi-agent Environments for Adaptive Learning). Think of this as a "video game engine" for robot kitchens that runs on a super-fast graphics card (GPU) instead of a slow processor.

The Speed Trick: Because they used a special tool called JAX, they can simulate thousands of kitchens at once. This means they can train robots on 100 different kitchen layouts in just a few hours on a single computer. In the past, this would have taken weeks or required a massive supercomputer.
The Infinite Kitchen: Instead of designing 100 kitchens by hand, they wrote a program that builds them on the fly. It's like a chef who can instantly conjure up a new kitchen with different wall positions and obstacles, ensuring the robots never get bored or stuck on the same map twice.

3. The Experiments: What Happened When They Tested the Robots?

The researchers tested various "learning strategies" (methods to help robots remember) on these 100-task sequences. Here is what they found:

The "Short Sequence" Trap: When they only tested the robots on 10 tasks, many strategies looked good. But when they pushed the sequence to 100 tasks, the results changed completely. Some strategies that looked great on short tests failed miserably on long ones. It's like a student who passes a pop quiz but fails the final exam because they didn't study the long-term material.
The Teamwork Struggle: The more robots they added to the kitchen, the harder it became.
- With 1 robot, it's just a solo act.
- With 2 robots, they can split the work (one chops, one cooks), and performance goes up.
- With 3 or 4 robots, it gets chaotic. They bump into each other, block the stove, and forget who is supposed to do what. The paper found that adding more agents actually made it harder for the team to remember how to cooperate.
The "Forgetting" vs. "Learning" Trade-off:
- Some methods were great at remembering old tasks but terrible at learning new ones (they got stuck in the past).
- Other methods were great at learning new things but forgot everything they knew yesterday.
- The best performer was a method called PackNet, which was like giving the robot a set of specialized tools for each specific kitchen, so it didn't mix up the instructions.

4. The "Partners" Twist

The researchers also tested what happens if the robots have to work with different partners every time. Imagine you are a chef, and every day you work with a different sous-chef who has a totally different style.

The robots had to learn to adapt to a "random" partner, then a "planner" partner, then a "human-like" partner.
They found that while the robots could learn to work with a new partner quickly, they often forgot how to work with the old partners. The "team chemistry" was the first thing to break.

5. The Big Takeaway

The main message of the paper is: Don't trust short tests.

If you only test an AI on a few tasks, you might think it's a genius at lifelong learning. But if you put it in a marathon of 100 tasks, you might see it collapse. The paper argues that to truly understand how AI learns over a lifetime, we need to run these long, fast, multi-agent simulations.

In summary: MEAL is a fast, flexible kitchen simulator that proves learning to cooperate in a team over a long period is incredibly hard, and that we need better tools to test AI before we can trust it to work with us in the real world.

Technical Summary: MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning

Problem Statement

Continual Reinforcement Learning (CRL) research is currently constrained by computational limitations. Most existing benchmarks restrict experiments to short task sequences (typically 3–10 tasks) because CPU-bound environments make training on longer sequences impractical. This limitation obscures long-horizon learning dynamics, such as the emergence of failure modes that only appear at scale. Furthermore, while CRL has seen progress in single-agent settings, Continual Multi-Agent Reinforcement Learning (CMARL) remains largely unexplored. In cooperative multi-agent settings, agents must establish implicit conventions to coordinate; as task distributions or partners change, these conventions can break down, amplifying individual forgetting into team-level failures. Current benchmarks lack the capacity to support the long task sequences and efficient evaluation required to study these phenomena at scale.

Methodology

To address these gaps, the authors introduce MEAL (Multi-agent Environments for Adaptive Learning), the first benchmark designed specifically for continual multi-agent RL.

Core Architecture and Implementation

JAX-Based Pipeline: MEAL is built entirely on JAX, leveraging its just-in-time (JIT) compilation, vectorization, and GPU execution capabilities. This allows for high-throughput simulation and training, enabling the execution of 100-task sequences on a single GPU within a few hours—a feat previously computationally infeasible with CPU-bound environments.
Procedural Generation: To facilitate long sequences without manual design bottlenecks, MEAL employs a procedural generator for Overcooked (a cooperative grid-based cooking environment). The generator creates kitchens with varying grid sizes and obstacle densities, ensuring solvability through a built-in validator that confirms agents can complete at least one cook-deliver cycle.
Scalable Difficulty: The benchmark defines three difficulty levels (Easy, Medium, Hard) by varying grid dimensions (6–11 tiles) and obstacle density (15%–35%). Higher levels introduce longer paths, tighter bottlenecks, and greater structural variability, forcing more sophisticated coordination.
Task Sequences: MEAL supports discrete task sequences with three regimes:
1. Fixed-level: Tasks sampled i.i.d. from a single difficulty generator.
2. Curriculum: Tasks ordered from easy to hard.
3. Repetition: Base sequences repeated to study plasticity loss.
Partner Adaptation: Beyond layout changes, MEAL introduces non-stationarity through changing cooperation partners. Partners include hardcoded strategies, planning-based agents, and populations trained with Best-Response Diversity (BRDiv).

Evaluation Metrics

The paper proposes three core metrics adapted for the multi-agent setting:

Average Soup Delivery ( $S$ ): The mean normalized performance across all tasks at the end of training, balancing stability and plasticity.
Forgetting ( $F$ ): Quantifies performance decline on past tasks due to interference from new tasks, using an exponentially decaying weight to penalize earlier forgetting more strongly.
Forward Transfer ($FT$): Measures how prior experience accelerates learning on new tasks by comparing the Area Under the Curve (AUC) of the continual learner against a single-task baseline.

Coordination Metrics: The authors also introduce metrics to analyze role specialization (Heterogeneity Index) and Coordination Forgetting, which measures whether cooperative strategies degrade faster than raw performance scores.

Key Contributions

First CMARL Benchmark: MEAL is the first benchmark to unify continual learning and multi-agent reinforcement learning, specifically designed to study forgetting, transfer, and partner adaptation in cooperative settings.
Hardware-Accelerated Execution: It is the first CRL benchmark to utilize hardware-accelerated (GPU) execution, scaling training on modest hardware to task sequences (up to 100 tasks) far longer than existing benchmarks allow.
Procedural Task Generation: The paper provides a scalable, procedurally generated environment suite with controllable difficulty, ensuring the benchmark remains relevant as methods improve.
Empirical Insights: The authors demonstrate that experimental conclusions can change significantly as task sequences lengthen, revealing failure modes (e.g., saturation, drift) that are invisible in short-sequence evaluations.

Experimental Results

The authors evaluated a suite of standard CRL methods (FT, EWC, Online EWC, MAS, Online MAS, AGEM, ER-ACE, PackNet) combined with MARL algorithms (IPPO, MAPPO, HAPPO) across MEAL's difficulty levels.

Method Performance:
- PackNet achieved the highest average scores across all difficulty levels while maintaining near-zero forgetting, striking the best balance between stability and plasticity.
- Online EWC and Online MAS showed strong retention and stability, particularly on longer sequences, outperforming their cumulative counterparts (EWC, MAS) which suffered from over-regularization.
- AGEM achieved the best forward transfer but at the cost of high forgetting.
- Fine-Tuning (FT) exhibited catastrophic forgetting, immediately losing performance on previous tasks.
Impact of Sequence Length: On short sequences (10 tasks), methods like EWC and Online EWC appeared similarly stable. However, on 100-task sequences, standard EWC suffered from severe over-regularization and early saturation, whereas Online EWC maintained plasticity through its decaying importance weights.
Agent Count Scaling: Performance generally peaked with 2–3 agents and declined as team sizes increased (4–5 agents). Larger teams exacerbated non-stationarity and credit assignment issues, making continual cooperation significantly harder to learn and retain.
Partial Observability: Introducing partial observability (limited field of view) degraded performance. However, switching from MLP to CNN encoders largely recovered this loss, suggesting CNNs are better suited for extracting spatial structure from egocentric views.
Coordination Dynamics: The study revealed that agents can maintain high soup delivery scores while their underlying coordination strategies (role specialization) degrade. Standard performance metrics often mask this "coordination forgetting," which is more pronounced in regularization-based methods.

Significance and Claims

The paper claims that MEAL addresses a critical bottleneck in CRL research: the inability to test methods on long-horizon sequences due to computational constraints. By enabling GPU-accelerated training of 100-task sequences, MEAL reveals that:

Short sequences are misleading: Many method rankings and conclusions drawn from 5–15 task sequences do not hold when sequences are extended to 100 tasks.
Continual cooperation is distinct: Retaining cooperative behavior is a unique challenge that worsens with more agents, harder tasks, and changing partners, distinct from single-agent retention.
Future Directions: The benchmark highlights the need for methods purpose-built for CMARL that can jointly handle partner- and environment-level non-stationarity, stabilize credit assignment under partial observability, and promote structured exploration.

The authors position MEAL as a practical foundation for studying these challenges at scale, reducing the energy footprint of experiments and broadening access for researchers without large compute clusters.