RFM-HRI : A Multimodal Dataset of Medical Robot Failure, User Reaction and Recovery Preferences for Item Retrieval Tasks

This paper introduces the RFM-HRI dataset, a multimodal collection of human-robot interactions in medical crash-cart settings that systematically analyzes user verbal and non-verbal reactions to various communication failures and their preferences for recovery strategies to improve safety-critical HRI systems.

The Gist

Imagine you are in a high-stakes emergency room. A doctor needs a specific medication right now, and they are asking a robot to help them find it in a rolling cart full of supplies. This is the setting for a new study called RFM-HRI.

Think of this research as a "Robot Breakup Simulator" for healthcare. The researchers wanted to understand what happens when a robot helper messes up, how humans react emotionally, and what the humans wish the robot would do to fix the mistake.

Here is the breakdown of their work in simple terms:

1. The Setup: The "Magic" Cart

The researchers built a robot that looks like a standard hospital crash cart (a rolling cabinet with drawers full of medical supplies). They put it in both a hospital and a university lab.

• The Trick: The robot wasn't actually smart enough to do this on its own yet. A human "Wizard" (like the Wizard of Oz) was hiding nearby, controlling the robot's lights and voice through a computer.
• The Goal: The human participant would ask the robot for an item (like "I need epinephrine"), and the robot would try to guide them to the right drawer using lights and speech.

2. The "Glitch" Factory

To study failure, the researchers didn't wait for the robot to break naturally. Instead, they intentionally broke it in four specific ways, like a chef testing a recipe by adding too much salt, too little heat, or the wrong ingredient:

• The "Vague" Glitch (Speech Failure): The robot says, "Open a drawer," but doesn't say which one. It's like a GPS saying, "Drive somewhere," without giving an address.
• The "Slowpoke" Glitch (Timing Failure): The robot waits 3 seconds before answering. In an emergency, that feels like an eternity.
• The "Wrong Turn" Glitch (Search Failure): The robot points to the wrong drawer and says, "It's in there!" when it's actually in a different one.
• The "Deaf" Glitch (Comprehension Failure): The robot says, "I didn't understand," and asks you to repeat yourself, even though you spoke clearly.

3. The Human Reaction: From Confusion to Frustration

The researchers recorded 41 people (doctors, nurses, and regular folks) doing these tasks. They used cameras to track facial expressions and asked people how they felt afterward.

What they found:

• The "Oops" Moment: When the robot failed, people didn't just get annoyed; they got confused. It was like trying to solve a puzzle where the pieces keep changing shape.
• The Emotional Rollercoaster:
  • Success: When the robot worked, people felt relief and confidence.
  • Failure: When the robot messed up, people felt confusion, annoyance, and frustration.
  • The Time Bomb: The longer the study went on, the more the confusion faded, but the frustration grew. It's like when you try to fix a leaky faucet; the first time you're confused, but by the tenth time, you're just angry.
• Loss of Control: When the robot failed, people felt like they lost control of the situation. They felt like passengers in a car where the driver is asleep.

4. The "Make-Up" Strategy: How to Fix It

After every mistake, the researchers asked: "If the robot could fix this, what would you want it to do?"

The Big Surprise:
Even though this was a physical task (opening drawers), people overwhelmingly wanted the robot to just talk to them.

• 64% of the time, people wanted a verbal apology and explanation. They wanted the robot to say, "I'm sorry, I got confused. The syringes are actually in Drawer 4."
• Only a tiny fraction wanted the robot to just flash a light or make a noise without speaking.
• The Lesson: When a robot mess up, humans want transparency. They want to know what went wrong and how to fix it, not just a blinking light.

Why Does This Matter?

Think of this dataset as a training manual for robot therapists.

Right now, if a robot in a hospital makes a mistake, it might just freeze or keep trying the same wrong thing. This study teaches us that:

1. Mistakes hurt trust: If a robot fails, the human feels less in control and more stressed.
2. Words matter: To fix a broken interaction, the robot needs to "speak up," admit the error, and give clear instructions.
3. We can build better robots: By using this data, engineers can teach robots to recognize when a human is getting frustrated (by looking at their face or voice) and automatically switch to a "repair mode" that involves a clear, honest explanation.

In a nutshell: This paper is about teaching robots that when they mess up, they shouldn't just hide or keep going; they should apologize, explain what happened, and help us get back on track. It's the difference between a robot that is a clumsy partner and one that is a helpful teammate.

Technical Summary

Here is a detailed technical summary of the paper "RFM-HRI: A Multimodal Dataset of Medical Robot Failure, User Reaction and Recovery Preferences for Item Retrieval Tasks."

1. Problem Statement

While autonomous robots in real-world environments inevitably experience interaction failures, the Human-Robot Interaction (HRI) community lacks comprehensive data on how users respond to these failures in safety-critical, high-stakes environments like healthcare.

• The Gap: Existing datasets often focus on social or conversational settings (e.g., chatbots, social robots) rather than physically grounded, time-critical tasks.
• The Specific Challenge: In emergency medicine, delays or errors in accessing supplies (e.g., via crash carts) can disrupt workflow and patient care. Current research lacks:
  1. Multimodal datasets capturing user reactions (verbal and non-verbal) to specific robot failure types in medical contexts.
  2. Paired annotations of failure events and user-preferred recovery strategies.
  3. Systematic analysis of how emotional responses evolve over repeated failures.

2. Methodology

A. Experimental Design

The authors conducted a Wizard-of-Oz (WoZ) study involving 41 participants (12 healthcare professionals and 29 non-experts) across two settings: a university laboratory and a hospital training area.

• Task: Participants performed a guided item retrieval task using a semi-autonomous crash cart robot. The robot provided verbal instructions and visual cues (LEDs) to guide users to specific drawers.
• Conditions: A within-subjects design with five randomized conditions:
  1. Success: Error-free interaction.
  2. Speech Failure: Under-specified instructions (e.g., "Open the drawer" without specifying which one).
  3. Timing Failure: Deliberate 3-second delay before responding.
  4. Search Failure: The robot highlighted the wrong drawer (incorrect spatial guidance).
  5. Comprehension Failure: The robot claimed it did not understand the request.

B. Robotic Platform

• Hardware: A standard crash cart retrofitted with addressable LED strips, a text-to-speech (TTS) engine, an onboard microphone, and an egocentric camera.
• Control: A custom GUI allowed a WoZ operator to trigger specific failure modes and control multimodal cues (LEDs, speech) in real-time while maintaining the illusion of autonomy.

C. Data Collection & Modalities

The study generated 214 interaction samples. For each trial, the following multimodal data were captured:

1. Visual: Egocentric video processed via Google MediaPipe to extract:
   • 468 3D facial landmarks.
   • Head pose (pitch, yaw, roll).
   • Gaze estimation.
   • Facial Action Units (AUs) intensity.
2. Audio/Text: Speech transcripts generated via Whisper (ASR), capturing user-robot dialogue.
3. Self-Reports: Post-trial surveys assessing:
   • Emotional State: Dominant emotion (e.g., confusion, frustration) and Self-Assessment Manikin (SAM) ratings (Valence, Arousal, Perceived Control).
   • Recovery Preferences: Preferred robot recovery strategies (e.g., verbal apology, visual redirection, multimodal cues).
4. Metadata: Trial boundaries, failure type labels, and participant demographics.

3. Key Contributions

1. RFM-HRI Dataset: A publicly available, multimodal dataset containing 214 samples with synchronized facial action units, head pose, speech transcripts, and self-reports. It is the first dataset to pair structured failure injection in a medical item-retrieval scenario with user recovery preferences.
2. Failure Taxonomy & Analysis: The authors defined and analyzed four specific communication failure types (Speech, Timing, Search, Comprehension) derived from three years of field observations, providing a structured framework for studying robot errors in healthcare.
3. Empirical Characterization of User Responses: The study provides quantitative evidence of how affective valence, perceived control, and specific emotions (confusion, frustration) change based on failure type and trial order.
4. Recovery Strategy Insights: The paper documents user preferences for recovery strategies, revealing a strong bias toward verbal transparency and guidance over non-verbal or purely physical recovery cues.

4. Key Results

A. Emotional Responses to Failure

• Dominant Emotions: Failure trials were dominated by negative cognitive-evaluative states: Confusion (25.9%), Annoyance (22.2%), and Frustration (17.5%).
• Success vs. Failure:
  • Failure: Associated with significantly lower Valence (unpleasantness), lower Perceived Control, and higher Arousal.
  • Success: Characterized by Surprise, Relief, and Confidence.
• Temporal Evolution: As participants experienced repeated failures, Confusion decreased (likely due to familiarity with the error pattern), while Frustration increased significantly over time.

B. Affective Dimensions (SAM Ratings)

• Perceived Control: Failed interactions resulted in a significant drop in perceived control (Median: 2.5) compared to successful ones (Median: 4.0).
• Valence: Successful interactions were rated as pleasant (Median: 1.0 on a scale where 1=pleasant, 5=unpleasant), while failures were rated as unpleasant (Median: 4.0).
• Correlation: There was a strong negative correlation between Valence and Perceived Control ($\rho = -0.436$), indicating that loss of agency is a primary driver of negative affect in robot failures.

C. Recovery Preferences

• Modality Preference: Users overwhelmingly preferred Verbal recovery strategies (63.9% of mentions). Non-verbal strategies accounted for 32.5%, and Multimodal strategies only 3.6%.
• Specific Strategies: The most preferred verbal strategies were:
  1. Clarifying prompts (e.g., asking for repetition).
  2. Self-correction & acknowledgment (e.g., "Sorry, I made a mistake").
  3. Specific re-direction (explicitly stating the correct drawer).
• Failure Type Independence: Recovery preferences did not vary significantly by failure type, suggesting that universal principles (transparency, acknowledgment) are more critical than failure-specific tactics.

5. Significance and Future Impact

• Foundation for Adaptive Systems: The RFM-HRI dataset enables the training of machine learning models to detect robot failures in real-time using multimodal signals (facial expressions, speech patterns) and to select appropriate recovery strategies based on user state.
• Design Implications: The findings challenge the assumption that physical robots primarily need physical recovery cues. Instead, verbal transparency and actionable guidance are paramount in safety-critical settings. Robots should explicitly acknowledge errors and provide clear next steps.
• Safety-Critical HRI: By focusing on the crash cart scenario, the work bridges the gap between social HRI research and the rigorous demands of healthcare, offering a blueprint for designing robots that maintain trust even when they fail.
• Future Directions: The authors propose using this dataset to develop autonomous failure management systems that can infer user reactions and execute context-appropriate recovery policies, moving from Wizard-of-Oz simulations to fully autonomous clinical deployments.

In summary, RFM-HRI provides a critical resource for understanding the human side of robot failure in high-stakes environments, demonstrating that confusion and frustration are immediate reactions to errors, but transparent verbal communication is the most effective path to recovery.