🤖 AI

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

This paper introduces *alem*, a JAX-based benchmark for evaluating open-ended multi-agent coordination in long-horizon survival tasks, revealing that while frontier LLMs excel at individual tasks, they struggle significantly with coordination—a distinct bottleneck where communication proves critical and performance varies widely across models.

Original authors: Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker, Alexander Rutherford, Davide Paglieri, Aidan Scannell, Henry Gouk, Elliot J. Crowley, Tim Rocktäschel, Amos Storkey

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker, Alexander Rutherford, Davide Paglieri, Aidan Scannell, Henry Gouk, Elliot J. Crowley, Tim Rocktäschel, Amos Storkey

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of explorers trying to survive in a vast, ever-changing wilderness. They need to chop wood, mine stone, build shelters, and fight monsters. But here's the catch: they can't just do these things alone. Sometimes, they need to chop a tree at the exact same moment to make it fall. Other times, one person must start digging a hole, and another must finish it within a few seconds, or the hole collapses.

This paper introduces a new "training ground" called ALEM (which means "world" in Amharic) to test how well AI agents can work together in this kind of chaotic, long-term survival game.

The Problem: Solo Stars vs. Team Players

Until now, most tests for AI have been like solo video games. They ask, "Can this AI solve a puzzle?" or "Can it navigate a maze?" But in the real world, AI will need to work in teams. The authors noticed that existing tests for teamwork were too simple: they were short, the rules were fixed, or the team members didn't have to talk to each other much.

They wanted to see if modern AI (Large Language Models, or LLMs) could handle long-term coordination. Can they remember a plan made 50 steps ago? Can they talk to a teammate to say, "I'll go left, you go right"? Can they adapt when the situation changes?

The Solution: ALEM (The Survival Simulator)

The authors built ALEM, a digital world based on a game called Craftax. Think of it as a high-tech version of Minecraft or Terraria, but with a twist:

Procedural Generation: Every time the game starts, the map and the specific teamwork challenges are randomly generated. This stops the AI from just "memorizing" the answers.
The "Coordination Spectrum": The game has a dial that controls how hard teamwork is.
- Easy: You can work alone mostly.
- Medium: You need to pass tools to each other (like handing a hammer to someone building a wall).
- Hard: You must act in perfect unison, like a synchronized swimming team, or the task fails.
Soft Specialization: In this world, anyone can do any job, but if you aren't the "specialist" for that job, you might fail. This forces the team to figure out who should do what on the fly.

The Experiment: 13 AI Agents vs. The World

The researchers put 13 different modern AI models into this world. They didn't teach them how to play; they just dropped them in (this is called "zero-shot"). They also trained some "robot" agents (using standard reinforcement learning) to act as a benchmark for what is possible.

The Results were surprising:

The AI Struggled: On average, the AI agents only achieved about 6% of the possible score. They were far from solving the game.
Being Smart Alone Doesn't Mean Being Smart Together: Some AI models were great at doing solo tasks (like gathering wood) but terrible at coordinating. One model, GPT-5.4, was very good at basic tasks but failed miserably when it needed to sync up with teammates. Another model, Gemini-3.1, was much better at teamwork, even beating some of the trained robot agents in the hardest scenarios.
Size Isn't Everything: Bigger AI models (with more "brain power") didn't necessarily work better as a team. Sometimes a smaller model coordinated better than a giant one.

The "Why": What Makes Teamwork Hard?

The researchers took apart the AI agents to see what was causing the failures. They found three main ingredients:

Communication is King: When they turned off the ability for agents to talk to each other, their teamwork scores crashed. The AI agents were using chat to say things like, "I'm going to the tree, you wait here," or "Give me wood." Without this, they were just guessing.
Memory is for Planning: The AI had a "scratchpad" (a private notepad). The best performers used this notepad to write down future plans (e.g., "Step 1: Mine stone. Step 2: Build furnace"). The weaker models just used the notepad to repeat what they saw right now, which didn't help them plan ahead.
Reasoning Helps: When the AI was forced to "think" before acting, it did better at both solo tasks and teamwork.

The "Mixed Team" Surprise

The researchers also tried mixing different AI models in one team (e.g., a Gemini agent working with a GPT agent). They expected the team to be as good as the smartest member. Instead, the team performed at the average of the two. It turns out, having a super-smart teammate doesn't help if the other teammate doesn't understand the plan or the communication style.

The Bottom Line

This paper shows that coordination is a unique skill that current AI hasn't mastered yet. Just because an AI is good at answering questions or solving puzzles alone doesn't mean it can work in a team.

ALEM provides a controlled way to measure this specific "teamwork bottleneck." It shows that to build AI that can truly work with humans or other AIs in the real world, we need to focus less on just making them smarter at solo tasks and more on teaching them how to communicate, plan together, and trust each other.

The code for this "survival simulator" is now open for other researchers to use and improve upon.

Technical Summary: Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

Problem Statement

As Large Language Models (LLMs) are increasingly deployed as autonomous agents, they must operate in multi-agent settings where success depends on coordinating with others over long horizons. However, existing evaluations rarely test these demands simultaneously. Current benchmarks typically focus on single-agent tasks, short interactions, or highly structured multi-agent settings (e.g., fixed roles, pre-specified goals). There is a lack of evaluations that combine long-horizon interaction, open-ended task structures, and explicit, procedurally generated coordination demands. Consequently, the mechanisms required for sustained multi-agent coordination—such as communicating intent, allocating roles, maintaining shared plans, and aligning actions over time—remain poorly understood.

Methodology: The ALEM Benchmark

To address these gaps, the authors introduce ALEM (Alem translates to "world" in Amharic), a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics.

Environment Design

ALEM is a partially observable stochastic game (POSG) featuring procedurally generated environments where agents explore, trade, craft, build, and fight. The benchmark is distinguished by four key axes:

Procedural Coordination Generation: Unlike prior work where coordination requirements are fixed, ALEM procedurally generates the coordination structure itself each episode. Tasks (mining, construction, combat, crafting) are assigned coordination types (synchronous or handover) and required agent counts (2 to N).
Coordination Spectrum: The benchmark spans a full spectrum of coordination coupling based on temporal separation ( $\Delta t$ $Δ t$ ) between coupled actions:
- Long-range: One agent gathers resources used by another later (large $\Delta t$ ).
- Handover: One agent initiates a task, and another must complete it within a short window (small $\Delta t$ ).
- Synchronous: Agents act on the same target at the same timestep ( $\Delta t = 0$ ).
Soft Specialisation: Instead of hard role gates, agents have "soft" specialisation. Specialists succeed deterministically, while non-specialists succeed with probability $1-\alpha$ . This rewards effective role allocation while allowing online adaptation.
Controllable Difficulty: A single scalar parameter $\alpha \in [0, 1]$ scales coordination difficulty without altering the underlying world layout. Higher $\alpha$ tightens execution constraints (e.g., shorter handover windows, higher N-agent requirements, lower non-specialist efficiency).

Evaluation Protocol

The authors evaluated 13 modern LLMs in zero-shot homogeneous teams across Easy, Medium, and Hard settings.

Models: Included proprietary models (GPT-5.4-High, Gemini-3.1-Pro-High) and open-weight models (Gemma-4, Llama-3.1/3.3, Qwen-3.5/3.6).
Baselines: Trained Multi-Agent Reinforcement Learning (MARL) agents (IPPO, MAPPO, PQN-VDN, HyperMARL-IPPO) trained for up to 1 billion steps served as reference points.
Interfaces: Agents interacted via a structured text interface receiving observations, teammate messages, and a scratchpad, returning structured actions.
Metrics: Performance was measured via normalised episode return, decomposed into Base% (individual task progress) and Coord.% (coordination-specific achievements).

Key Results

1. Coordination is a Distinct Bottleneck

Current LLM agents remain far from solving ALEM, averaging only ~6% normalised return across difficulties.

Base vs. Coordination Competence: High individual task competence does not imply coordination competence. For instance, GPT-5.4-High achieved strong base-task rewards but significantly lower coordination rewards compared to smaller models like Gemma-4-26B-A4B-it.
MARL Comparison: On the hardest setting, Gemini-3.1-Pro-High (17.5% Coord.%) approached the performance of MARL agents trained for one billion steps (17.6% Coord.%), demonstrating that while LLMs struggle, the best models can reach near-MARL levels of coordination in specific contexts.

2. Ablation Studies on Agent Components

The authors ablated three components of the agent harness to determine their impact on coordination:

Communication: This was the largest contributor to coordination performance. Removing communication caused the most significant drop in Coord.% (e.g., Gemini-3.1-Pro-High dropped from 17.5% to 5.3%). Agents used communication primarily to broadcast intent, assign actions, and align timing.
Memory (Scratchpad): The utility of memory was model-dependent. Gemini-3.1-Pro-High used scratchpads as forward-looking planners (multi-line, future-tense, turn-indexed plans), and removing it hurt performance. Gemma-4-31B-it used scratchpads mostly for state summaries, and the impact of removal was negligible.
Reasoning: Reducing reasoning lowered both Base% and Coord.% for both models. Qualitative analysis showed that reduced reasoning eliminated planner-like behaviors (e.g., turn-indexed plans) and led to shorter, less directive communication.

3. Heterogeneous Teams

In initial experiments with heterogeneous teams (mixing different models), performance was close to the average of the constituent homogeneous baselines, rather than matching the strongest member. This suggests that adding a stronger model does not automatically lift team performance, likely due to misalignment in communication conventions and planning horizons.

4. Failure Modes

Task Structure: Agents struggled most with Synchronous-Hard tasks requiring simultaneous action, achieving only ~18% coverage even for the best LLM. Handover tasks were comparatively more accessible.
Scale: Performance did not scale linearly with parameter count. Within model families (e.g., Gemma-4, Qwen), coordination performance varied non-monotonically across dense and mixture-of-experts variants.

Significance and Contributions

The paper makes three primary contributions:

ALEM Benchmark: A JAX-based, procedurally generated benchmark for long-horizon, open-ended multi-agent coordination with controllable difficulty and soft specialisation.
Interfaces: Provision of pixel, symbolic, and text interfaces to evaluate both MARL and zero-shot LLM agents under matched coordination demands.
Systematic Evaluation: A comprehensive evaluation of 13 LLMs and 4 MARL baselines, identifying coordination as a distinct bottleneck separate from base-task competence.

The authors conclude that coordination is a distinct bottleneck for frontier LLM agents. While current models can survive and perform individual tasks, they fail to effectively communicate, allocate roles, and execute shared plans in open-ended, long-horizon settings. ALEM provides a measurable and controlled testbed to develop agents capable of these complex cooperative behaviors. The code is available at https://github.com/alem-world/alem-env.