💻 computer science

Improving Human-Robot Teamwork in Urban Search and Rescue Through Episodic Memory of Prior Collaboration

This paper demonstrates that equipping a robot with an automatically selected episodic memory of prior collaboration patterns, represented as a knowledge graph, significantly improves human-robot teamwork in Urban Search and Rescue scenarios by increasing rescue success rates and reducing task completion time.

Original authors: Taewoon Kim, Emma van Zoelen, Mark Neerincx

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Taewoon Kim, Emma van Zoelen, Mark Neerincx

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are about to play a high-stakes video game with a robot partner. The goal is to dig through a pile of rubble to save a trapped victim. In the past, you and the robot had to figure out how to work together from scratch every single time. You'd spend the first few minutes guessing what the other was thinking, maybe making mistakes, and wasting precious time.

This paper describes a new way to help the robot skip that "getting to know you" phase. Instead of starting with a blank slate, the robot is given a single, pre-loaded "cheat sheet" based on a successful teamwork session from the past.

Here is how they did it and what happened:

1. The "Cheat Sheet" (Episodic Memory)

The researchers didn't just save a video of the past; they turned a successful teamwork session into a structured knowledge graph. Think of this like a flowchart or a recipe card.

The Situation: "We are at the top of a rock pile."
The Action: "Robot picks up the big rock."
The Next Action: "Robot drops the rock to the side."
The Result: "Victim is safe."

They collected 209 of these "recipe cards" from previous experiments where humans and robots worked together.

2. Finding the Best Recipe (The AI Brain)

Having 209 recipes is overwhelming. You don't want to read all of them before the game starts. So, the team used a special type of AI (called a Graph Neural Network) to look at all 209 recipes and group them into clusters, like sorting books by genre.

They found one specific group of recipes that was:

Simple: It didn't require constant back-and-forth instructions between the human and robot.
Effective: It got the job done quickly.
Autonomous: It let the robot take the lead on the heavy lifting.

They picked the "best" recipe from this group to be the robot's starting memory.

3. The Experiment: Playing with a Head Start

They ran a simulation with 20 human participants.

Group A (No Memory): The robot started with an empty brain. It had to learn how to work with the human as they went.
Group B (With Memory): The robot started with that one "best recipe" loaded into its brain.

The Results:

More Success: When the robot had the memory, the team saved the victim 41.3% of the time, compared to only 25.7% without it. That's a huge jump.
Faster Work: The team finished the task about 283 seconds faster on average.
The "First Impression" Effect: The biggest boost happened right at the very beginning of the game. It was like the robot walked into the room knowing exactly what to do, while the human was still figuring out the controls. This allowed them to get into a rhythm immediately.

4. The Catch (It's Not Perfect)

The paper notes a trade-off. Because the "cheat sheet" was designed for a specific type of rock pile, it worked great in the early, easier rounds. However, in the hardest rounds (where a tricky "brown rock" was introduced that could make the rescue impossible), the robot sometimes tried to apply the old recipe to a new problem. This led to a few more accidents (like rocks falling on the victim) in those specific difficult scenarios.

The Bottom Line

The paper proves that giving a robot a single, human-readable memory of a past successful teamwork session helps it start a new job much better than starting from zero. It doesn't make the robot a genius that knows everything; it just gives it a solid head start, allowing the human and robot to sync up faster and save more lives in the critical early moments of a disaster response.

Important Note: This study was done entirely in a computer simulation (a virtual world). The paper does not claim this works in real-world disasters yet, nor does it claim the robot is safe to use in real life without human supervision. The "memory" is also transparent—humans can read the recipe card and change it if they need to.

Technical Summary: Improving Human-Robot Teamwork in Urban Search and Rescue Through Episodic Memory of Prior Collaboration

Problem Statement
Effective human-robot teamwork in dynamic, hazardous environments like Urban Search and Rescue (USAR) requires robots to adapt to partners and task situations immediately, rather than learning solely through trial and error during interaction. While prior work in the MATRX USAR environment has demonstrated that robots can adapt over time and that human-robot teams can externalize collaboration patterns (CPs) via chat and reflection interfaces, a gap remains in utilizing these historical patterns to initialize future interactions. The core problem addressed is how to transfer explicit, prior team experience into a new collaboration episode to improve early-stage performance, rather than relying on the robot to start with an empty memory state.

Methodology
The authors propose a framework where historical collaboration patterns are represented as knowledge-graph episodic memories and utilized to initialize a robot before a new task begins.

Ontology and Representation:
- Collaboration patterns (CPs) are modeled as property graphs using JanusGraph. The ontology includes six entity types: robot, participant, cp, situation, robot_action, and human_action.
- Relationships are captured via edge families (has_cp, has_situation, has_robot_action, has_human_action). To preserve action order, action edges are distinguished by sequence indices (e.g., has_robot_action_0, has_robot_action_1), resulting in nine distinct relation labels.
- Node properties include numerical metrics (e.g., time_elapsed, success) and string attributes (e.g., action descriptions). String attributes are encoded into vectors using Sentence-BERT, while numerical features are normalized and projected to a common dimensionality.
Graph Representation Learning:
- A Relational Graph Convolutional Network (RGCN) is employed to learn meaningful representations of the CP graphs. The model uses a node-classification objective with a single linear classifier on top of the final RGCN layer.
- The ten node classes correspond to the schema types (one cp, one situation, three human_action stages, and five robot_action stages). This supervision allows the encoder to distinguish between early and late steps in a collaboration pattern.
- After training, graph-level embeddings are generated by mean-pooling the final node embeddings of each CP graph.
Memory Selection and Initialization:
- The learned graph embeddings are clustered using K-means (with $K=5$ ).
- The authors heuristically select Cluster 1 as the source for reusable memory because it exhibits the highest success rate (42%) and structural simplicity (dominated by consecutive robot actions with minimal human intervention).
- The CP closest to the centroid of Cluster 1 is selected as the representative memory. This CP involves a situation ("Top of rock pile," "Large rock") and a two-step robot action sequence (pick up and drop a large rock) without requiring human input.
- In the experimental setup, the robot is initialized with this single, preloaded CP before the human-robot interaction begins.

Key Contributions
The paper outlines three primary contributions:

Episodic Team Memory Representation: Representing CPs from prior interactions as knowledge-graph episodic memories that preserve situational context, action structure, and task outcomes in a reusable format.
Selecting Reusable Prior Experience: Utilizing graph representation learning with node-type supervision to organize historical CPs and identify a representative pattern for initialization.
Improved Early-Team Performance: Demonstrating that initializing a robot with a selected prior memory significantly improves performance metrics, particularly at the start of collaboration.

Experimental Results
The study involved 20 participants and 160 round-level observations within the MATRX USAR simulation. The robot was tested in two conditions: without initialized memory and with the single selected CP preloaded.

Success Rate: Rescue success increased from 25.7% (without memory) to 41.3% (with memory), representing a 60.7% relative improvement.
Task Time: The average task time was reduced by 283 seconds.
Temporal Dynamics: The most significant gains occurred in Round 1, suggesting the memory helps teams orient to the task before human adaptation can occur.
Trade-offs: While success and time improved, victim harm was slightly higher on average in the memory-initialized condition, likely due to accidental rock falls when the robot executed heavy-debris removal strategies. Performance also declined in the hardest rounds (7 and 8), where the reused CP did not account for the specific risks of the "brown rock" introduced in later scenarios.

Significance and Claims
The authors position this work as a concrete mechanism for initializing a robot with inspectable prior team experience, rather than a new general theory of memory transfer.

Transparency: Unlike opaque policy updates, the reused experience remains an explicit, human-readable situation-action structure that can be inspected, discussed, and revised by operators.
Early Interaction Support: The results suggest that reusable episodic memory is most effective at the beginning of collaboration, providing a coordinated starting point when shared routines have not yet formed.
Limitations: The paper modestly acknowledges limitations, including the simulation setting, the small sample size, the heuristic selection of a single centroid-nearest CP, and the lack of baselines involving random or expert-selected memories. The authors note that the approach does not eliminate safety risks associated with aggressive debris removal but offers a deployment model where prior knowledge supports coordination under human oversight.