Technical Summary: Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork

1. Problem Statement

In-Context Reinforcement Learning (ICRL) has recently enabled foundation agents to adapt instantaneously to novel tasks by treating interaction history as a prompt for sequence models, achieving success in single-agent domains (e.g., via Algorithm Distillation and Decision-Pretrained Transformers). However, its efficacy in Ad-Hoc Teamwork (AHT)—where an agent must coordinate with unknown, heterogeneous partners without prior communication or coordination protocols—remains largely unexplored.

Existing AHT benchmarks suffer from three critical limitations:

1. Lack of Scale: They typically rely on small sets of homogeneous partners or limited task variants, failing to provide the millions of transitions required to rigorously test in-context generalization.
2. Conflated Generalization: They often fail to disentangle generalization across teammates from generalization across environmental layouts, making it difficult to distinguish genuine coordination from statistical overfitting to specific partner-environment pairs.
3. Missing Infrastructure: There is a lack of standardized, reproducible pipelines for generating learning histories (sequences of episodes showing improvement) and conducting online evaluation of test-time adaptation.

Consequently, it is unclear whether current sequence-model-based ICRL agents can perform the strategic inference required to identify a partner's hidden intent and adapt their policy purely from interaction history in multi-agent settings.

2. Methodology: The ICRL4AHT Benchmark

To address these gaps, the authors introduce ICRL4AHT, a large-scale, reproducible benchmark built upon a high-throughput JAX implementation of OvercookedV2. The benchmark is designed to rigorously probe the limits of ICRL in cooperative settings through a three-stage pipeline:

A. Benchmark Specification and Teammate Suite

• Environment: OvercookedV2 is modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP). It introduces critical complexities absent in previous versions, including partial observability (5x5 field of view), stochastic starting positions, and dynamic recipe changes.
• Teammate Diversity: The benchmark constructs a heterogeneous "Teammate Suite" comprising:
  • RL Policies: A diverse population of partners trained using algorithms known for generating varied behaviors (FCP, BRDiv, LBRDiv, CoMeDi).
  • Heuristic Policies: Four distinct families (H1–H4) ranging from "Recipe Aware" (minimal autonomy) to "Utility Greedy" (high autonomy), ordered by increasing cooperability.
• Distribution Shift: The experimental design strictly separates training and testing distributions. Agents are trained exclusively on RL-generated learning histories and tested exclusively on unseen heuristic policies, inducing a severe distribution shift that precludes simple interpolation.

B. Data Infrastructure

• Learning Histories: Unlike standard offline RL datasets containing static expert trajectories, ICRL4AHT collects "learning histories"—sequences of episodes where an ego agent (trained via PPO) improves from random to expert behavior against a fixed teammate.
• Scale and Filtering: The pipeline generates over 1.19 billion original transitions, filtered down to ~150 million high-quality transitions across 80 unique teammates. A quality-based filtering mechanism selects trajectories with high final performance and significant improvement gradients to ensure the ICRL models learn genuine adaptation dynamics.
• Storage: Data is serialized in chunked HDF5 format with a JSONL index for efficient metadata lookup and context-window batch sampling.

C. Evaluation Tracks

The benchmark defines two rigorous evaluation tracks to disentangle generalization capabilities:

1. Track 1 (Teammate Generalization): Tests adaptation to unseen partner behaviors within familiar layouts ($L_{train} \times \Pi_{test}$).
2. Track 2 (Layout Generalization): Tests robustness under simultaneous shifts in both partner behavior and environmental dynamics ($L_{test} \times \Pi_{test}$).

D. Baselines

The authors evaluate representative history-conditioned ICRL algorithms:

• Algorithm Distillation (AD): Trains a sequence model to autoregressively predict actions from learning histories, aiming to learn "how to learn" from context.
• Decision-Pretrained Transformer (DPT): Trains a Transformer to predict optimal expert actions given a task-specific context, treating the history as a prompt for zero-shot adaptation.

3. Key Results

The empirical analysis reveals notable limitations in current ICRL baselines when applied to AHT:

• Failure to Outperform Random: Contrary to their success in single-agent domains, both AD and DPT frequently fail to outperform a simple random baseline across both evaluation tracks. In several complex coordination scenarios (e.g., H1 teammates on "test wide" layouts), the ICRL agents achieve significantly negative returns, indicating they actively interfere with the partner rather than coordinating.
• Absence of In-Context Improvement: A defining characteristic of ICRL is performance improvement within a single context window as history accumulates. The evaluation curves for AD and DPT are consistently flat, showing no upward trend in returns as the number of testing episodes increases. This suggests the models fail to perform the necessary Bayesian inference to identify the partner's policy from the interaction history.
• Robustness of Failure: The failure persists across:
  • Context Length: Increasing the context window from 250 to 2,000 (and up to 10,000) transitions yields no consistent improvement.
  • Model Scale: Scaling model parameters (Small to Large configurations) results only in modest gains, failing to restore robust adaptation.
  • Teammate Action Conditioning: Providing explicit access to the partner's actions (+TA) does not resolve the issue, suggesting the bottleneck is not information availability but the model's inability to causally link observed actions to latent intent.
  • Architecture: Switching to a recurrent architecture (Hybrid-AD) or using an offline RL objective (AMAGO-Offline) does not qualitatively change the outcome.

4. Key Contributions

1. Scalable Benchmark and Pipeline: The authors provide the first reproducible, JAX-based pipeline for teammate generation, learning-history collection, and standardized evaluation of cooperative multi-agent ICRL methods with controllable diversity.
2. Diverse Teammate Suite: They construct a heterogeneous library of teammates that strictly separates training (RL-based) and testing (heuristic-based) distributions, enforcing a rigorous test of generalization that prevents overfitting to specific partner clusters.
3. Benchmarking the Limits: The paper demonstrates that representative history-conditioned sequence-model ICRL baselines, while successful in single-agent settings, struggle with the strategic inference required for partner adaptation under the OvercookedV2 AHT protocol.

5. Significance and Claims

The paper positions ICRL4AHT as a critical testbed for the next generation of coordination algorithms. The authors claim that their findings highlight a fundamental challenge: current sequence-model architectures struggle to perform the implicit Bayesian partner inference required for AHT under realistic distribution shifts.

The significance of this work is primarily diagnostic. Rather than proposing a new operational system, the authors systematically characterize the limitations of existing ICRL approaches. They argue that the failure is not merely an implementation detail or a hyperparameter tuning issue, but reflects a substantive limitation of the studied paradigms when faced with realistic partner diversity and environmental shifts. The open-source benchmark and empirical findings are intended to guide the community toward developing architectures with explicit partner inference modules and training objectives that incentivize information-seeking exploration.