Technical Summary: REPAIR-Bench

Problem Statement

While robots are increasingly deployed in high-stakes environments like emergency departments, they remain prone to failures caused by perception errors, time pressure, or environmental uncertainty. Characterizing these failures and user responses is critical for maintaining trust and task completion. However, existing Human-Robot Interaction (HRI) benchmarks suffer from three primary limitations:

1. Independence Assumption: Prior work often treats failures as independent events, failing to model cross-trial history or the accumulation of user adaptation over repeated failures.
2. Binary Limitations: Most datasets support only binary failure detection (success/failure) rather than multi-class failure categorization.
3. Rule-Based Recovery: Recovery strategies are typically researcher-defined rather than predicted from user behavior, lacking a framework to infer user-preferred recovery methods from interaction context.

There is a lack of integrated benchmark frameworks that connect failure detection, understanding, and recovery decision-making within a single evaluation setting.

Methodology

The authors introduce REPAIR-Bench, a multimodal benchmark built upon the RFM-HRI dataset, which captures interactions between users and a semi-autonomous medical crash-cart robot in both laboratory and hospital settings.

Dataset and Preprocessing

• Data Source: 214 interaction trials from 41 participants involving four induced failure types (speech, timing, comprehension, search) and one success condition.
• Modalities: The dataset includes synchronized video, audio, and task transcripts.
• Feature Extraction:
  • Visual: Facial Action Units (AUs), 3D head pose, gaze estimates, and facial landmarks extracted via Google MediaPipe.
  • Audio/Text: Time-stamped speech transcripts via Whisper ASR.
  • Ground Truth: Post-interaction surveys capturing emotional reactions and preferred recovery strategies.
• Preprocessing: The authors standardized failure annotations with precise onset/offset timestamps. Visual features were filtered (Butterworth filter) and normalized. Transcripts were cleaned for recovery modeling, removing timestamps to focus on dialogue content.
• Final Set: After excluding 30 samples with inconsistent transcripts or timestamps, the analysis set comprises 184 samples.

Evaluation Tasks

REPAIR-Bench defines three novel evaluation tasks:

1. Task 1: Temporal Failure Detection

   • Goal: Detect the moment a robot's response fails based on user behavioral reactions, accounting for cross-trial dependencies.
   • Input: Sequential visual features (AUs, landmarks, pose, gaze) in sliding windows.
   • Architecture: A two-level Hierarchical Recurrent Neural Network (HRNN) using Gated Recurrent Unit (GRU) cells. The model processes window-level features and maintains a session-level history vector updated after each trial to capture longitudinal user adaptation.
   • Output: Binary pre/post-failure labels per window and a temporal transition moment estimation.

2. Task 2: Failure Classification

   • Goal: Distinguish between specific failure categories (comprehension, speech, timing, search) and success using visual cues.
   • Architecture: A session-aware temporal model combining a bidirectional LSTM for frame-wise encoding and a unidirectional GRU for processing trial embeddings in chronological order.
   • Output: Multi-class failure type label.

3. Task 3: Recovery Strategy Prediction

   • Goal: Predict user-preferred recovery strategies (e.g., apology, step-by-step guidance) based on interaction transcripts and failure context.
   • Architecture: Fine-tuned Small Language Models (SLMs) using Quantized Low-Rank Adaptation (QLoRA). Models evaluated include Mistral-7B and Gemma-9B.
   • Formulation: A constrained generative ranking problem where the model outputs a comma-separated list of strategies from a predefined candidate set.

Key Results

Task 1: Failure Detection

• The two-level HRNN with 3-second windows achieved a strict macro F1 of 0.80, outperforming single-session GRU baselines (0.68) and flat baselines.
• Localization: The model achieved a median absolute error of 2.97 seconds and a mean signed error of -0.51 seconds (indicating a slight tendency to predict slightly early).
• The hierarchical approach improved performance by modeling cross-session history, demonstrating that user adaptation across repeated failures is a significant factor.

Task 2: Failure Classification

• The HRNN achieved a macro F1 of 0.44, outperforming a BiLSTM-GRU baseline (0.38) with the same parameter count.
• Performance was strongest for the Success class (F1 = 0.93) and Speech failures, while Comprehension and Timing failures remained more difficult to distinguish.
• Statistical analysis confirmed a modest but significant association between self-reported emotion and specific failure conditions.

Task 3: Recovery Strategy Prediction

• Base Models: Base Mistral-7B outperformed base Gemma-9B, highlighting the necessity of instruction tuning for structured output. Base Gemma-9B performed near chance levels.
• Fine-Tuning: QLoRA fine-tuning significantly improved performance. For Mistral-7B, Hit@5 increased from 0.68 to 0.76, and F1@5 increased from 0.30 to 0.32.
• Fine-tuned Gemma-9B achieved performance comparable to fine-tuned Mistral-7B, demonstrating that parameter-efficient fine-tuning is critical for aligning generalist LLMs with specific recovery prediction tasks.

Significance and Contributions

The paper claims REPAIR-Bench is the first HRI benchmark to jointly address failure classification, localization, and recovery prediction. Its primary contributions are:

1. Unified Framework: It provides a standardized framework for evaluating the full lifecycle of failure in HRI, moving beyond isolated detection to include recovery decision-making.
2. Longitudinal Modeling: It demonstrates that modeling cross-trial history (via hierarchical architectures) improves failure detection accuracy compared to treating sessions as independent.
3. User-Centered Recovery: It shifts recovery strategy selection from rule-based systems to data-driven predictions of user preferences, which is essential for building trustworthy systems in safety-critical domains like healthcare.
4. Community Resource: It offers the HRI and Medical HRI communities a dataset with synchronized multimodal signals and annotated ground truth for failure and recovery, facilitating research into adaptive and transparent robot behaviors.

The authors note limitations, including the relatively small sample size for recovery prediction (184 samples) and the use of a predefined set of recovery strategies, suggesting future work should expand the dataset and incorporate richer multimodal signals.