🤖 AI

Evaluating XAI Support From A Hierarchical Reinforcement Learning Policy in Human-Agent Collaboration

This paper presents the first systematic evaluation of intrinsically explainable hierarchical reinforcement learning in human-agent collaboration using the Overcooked-AI benchmark, revealing that while explanations showed trends toward faster improvement, audio-delivered explanations significantly undermined the human-agent working alliance by creating unmet partnership expectations.

Original authors: Mateus Levi Simões Fernandes, Alberto Sardinha

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mateus Levi Simões Fernandes, Alberto Sardinha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a high-speed video game with a robot teammate. You both need to chop onions, cook soup, and serve it before time runs out. In the real world, when two humans play together, they talk: "I'll grab the tomato, you grab the pot!" or "Watch out, I'm coming through!" This chatting helps them build a shared understanding of what's happening. But robots are often like silent, mysterious machines; they move fast and do things, but they don't tell you why. This creates a gap. If a robot is going to be a true partner, not just a tool, it needs to explain its thoughts. This is where "Explainable AI" (XAI) comes in. It's the science of teaching robots to say, "I'm doing this because..." so humans can trust and work with them better. But here's the tricky part: if the robot talks too much, it might distract you. If it talks in the wrong way, it might annoy you. The big question is: how do we make a robot explain itself without getting in the way of the game?

This paper dives into that exact problem by setting up a digital kitchen experiment using a game called Overcooked-AI. The researchers used a special kind of robot brain called "Hierarchical Reinforcement Learning." Think of this like a robot that doesn't just think about every tiny step (like "move left, move right"), but instead thinks in bigger chunks, like "get an onion" or "put soup in a bowl." Because it thinks in these big chunks, it can naturally tell you what it's planning to do next. The team wanted to see if letting this robot talk to its human player would make them work better together. They tested two ways of talking: the robot could type messages on the screen (Text) or speak them out loud (Audio).

The results were a mix of surprises and lessons. First, the robot's explanations didn't make the team score more points immediately. In fact, the human players who didn't get any explanations actually scored slightly higher on average, though the difference wasn't huge. This suggests that for people who are already good at games, constant updates might just be a distraction. However, there was a silver lining: the players who got explanations seemed to learn how to work with the robot faster. They improved their scores more quickly over time, as if the robot's chatter helped them figure out the robot's style sooner.

But the most interesting discovery happened with the way the robot spoke. When the robot spoke out loud (Audio), something strange happened. Even though the players learned faster, they felt a much weaker connection to the robot. They didn't feel like they were on the same team. It's as if the robot's voice made them expect a real conversation and a promise of what would happen next. But because the robot was just reacting to the moment (like a reflex), it often changed its mind instantly after speaking. When the robot said, "I'm going to the pot," but then immediately ran to the counter instead, the human felt betrayed. The spoken voice made the robot seem like a partner who should keep its word, but the robot's brain was too reactive to actually keep that promise.

In contrast, when the robot just typed messages on the screen, the players didn't feel this betrayal. They could ignore the text if they wanted to, so the robot didn't feel like a "fake" partner. The study suggests that while talking out loud grabs attention and helps people learn faster, it also raises expectations that a simple, reactive robot might not be able to meet. The researchers conclude that if we want robots to talk to us, we need to make sure they are actually good at keeping the promises their voices imply, or else the partnership might feel broken.

Technical Summary: Evaluating XAI Support From a Hierarchical Reinforcement Learning Policy in Human-Agent Collaboration

Problem Statement

The transition of AI systems from tools to collaborative partners requires them to be not only effective but also comprehensible to human teammates. While Explainable AI (XAI) offers potential solutions, systematic evaluation in collaborative contexts remains limited. Existing research often relies on hand-crafted policies in custom environments (e.g., Minecraft), which limits the generalizability of findings to state-of-the-art teaming research that utilizes established benchmarks like Overcooked-AI. Furthermore, current literature lacks a systematic evaluation of XAI support generated from intrinsically explainable, learned policies (as opposed to post-hoc explanations or hand-crafted rules) within these high-performance benchmarks. There is also a gap in understanding how explanation modality (text vs. audio) affects collaboration in visually demanding, fast-paced tasks.

Methodology

The authors conducted a between-subjects experiment ( $n=38$ ) in the Overcooked-AI benchmark environment, specifically using the Counter-Circuit layout.

Agent Architecture: The study utilized the Hierarchical Ad Hoc Agents (HA2) architecture. This system employs a Manager-Worker structure where a high-level Manager policy ( $\pi_H$ ) selects semantically meaningful subtasks (e.g., "pick up onion," "place onion in pot") from a predefined set of 12 domain-specific goals. A low-level Worker policy ( $\pi_W$ ) then executes primitive actions to complete the selected subtask. This hierarchy provides intrinsic explainability, as high-level decisions are human-interpretable.
Explanation Generation: Real-time explanations were generated directly from the Manager's subtask selections. To prevent information overload, a novel trigger-based system was implemented to deliver explanations only at coordination-relevant moments. Triggers included:
- Blocking: When the human obstructs the agent's path.
- Critical Path: When the agent selects essential subtasks for task progression.
- Distance Threshold: Periodic updates during extended movement.
- Subtask Change: Announcements when the Manager switches goals.
Experimental Conditions: Participants were randomly assigned to one of three conditions:
1. Control: No explanations.
2. Text: Explanations displayed in a semi-transparent bar at the bottom of the screen.
3. Audio: Identical content delivered via browser-native text-to-speech.
Procedure: Participants completed four 80-second gameplay sessions. Metrics included game scores, learning rates (linear slopes across sessions), NASA-TLX workload assessments, and subjective evaluations using the Godspeed Questionnaire and a modified Human-Robot Fluency Assessment. The sample consisted of gaming-experienced adults (mean familiarity score 8.1/10).

Key Contributions

First Systematic Evaluation in Benchmark: This work provides the first systematic evaluation of XAI support generated from state-of-the-art learned policies (HA2) within an established human-agent teaming benchmark (Overcooked-AI), demonstrating that intrinsically explainable reinforcement learning architectures can generate authentic real-time explanations without compromising performance.
Modality Comparison: It presents the first comparison of explanation modalities (text vs. audio) in a real-time human-agent collaborative task, isolating the effects of delivery mode on performance and subjective experience.
Baseline Methodology: The authors establish a baseline trigger-based delivery system for hierarchical policy explanations, offering a methodology for evaluating intrinsically explainable RL architectures in benchmark environments.

Results

Performance (H1): There was no significant difference in immediate game scores between the Control and combined explanation conditions ( $p=0.136$ ). However, a trend suggested that participants receiving explanations (particularly audio) showed faster learning rates (improvement over sessions) compared to the Control group ( $d=0.59$ ), though this did not reach statistical significance ( $p=0.087$ ).
Cognitive Load (H2): No significant differences in cognitive workload (NASA-TLX) were found between text and audio modalities.
Subjective Experience (H3): A significant and robust finding emerged regarding the Working-Alliance Bond. Participants in the Audio condition rated their bond with the agent significantly lower than those in the Text or Control conditions ( $p=0.003$ $p = 0.003$ ). Text explanations did not negatively impact the bond compared to the Control.
- Anthropomorphism: A trend indicated that audio participants rated the agent as less anthropomorphic, though this did not survive Bonferroni correction.
- Mechanism: The authors interpret the bond reduction in the audio condition as a "claim-versus-reality" gap. Spoken explanations frame the agent as a communicative partner making commitments, but the underlying model-free, reactive policy cannot sustain these commitments (e.g., it may change subtasks instantly after speaking). This gap is less noticeable in text (which can be ignored) or silence.

Significance and Claims

The paper claims to advance the understanding of when and how XAI operates in fast-paced collaborative environments. The primary insight is that matching explanation modality to the underlying policy's capacity is crucial.

Architectural Mismatch: The study suggests that spoken explanations activate partnership expectations that reactive, model-free policies cannot meet, leading to a rupture in the working alliance. The "bond cost" is not an indictment of audio delivery itself, but of policies that lack the planning horizon or human-modeling capabilities to back up verbalized intentions.
Adaptation vs. Execution: The results suggest that XAI support may enhance human adaptation to agent behavior patterns (accelerating learning) rather than immediate task execution, particularly for expert users who may already possess efficient coordination strategies.
Future Direction: The authors argue that the path forward for effective collaborative XAI is architectural. To sustain the partnership implied by spoken explanations, future systems should incorporate longer commitment horizons (binding plans) and explicit models of human teammates to enable bidirectional, theory-of-mind-based communication, rather than merely narrating reactive outputs.

The paper concludes that while hierarchical RL offers a promising foundation for intrinsically explainable AI, the efficacy of XAI is constrained by the policy's ability to align its actual behavior with the relational expectations its delivery method (especially speech) creates.