Technical Summary: Signal or 'Noise': Human Reactions to Robot Errors in the Wild

Problem Statement

While social signals (verbal and nonverbal cues) have been established as reliable indicators of robot errors in controlled laboratory settings, their efficacy and manifestation in real-world ("in-the-wild") environments remain under-explored. Prior research has largely focused on single-user, single-interaction scenarios with social robots. It is unclear how these signals function in public spaces with non-social robots, particularly during group interactions, repeated interactions, and when errors propagate or occur successively. The core challenge is determining whether the rich but potentially "noisy" social signals observed in the wild can be effectively utilized for error management in deployed robotic systems.

Methodology

To address these gaps, the authors conducted a three-week public field deployment of an autonomous coffee-making robot in a university building hallway.

System Design:

• Hardware/Software: The system utilized a Kinova Gen3 robot arm controlled via ROS and a Kinova API. The user interface was built in React. The system included a capsule coffee machine, custom dispensers, and a turntable for delivery.
• Sensing: Data collection relied on three Kinect cameras (two for recording, one for proximity detection) and a webcam (for intrusion detection). A microphone captured audio.
• Autonomy: The system operated without experimenter supervision. It handled ordering via touch screen, coffee preparation, and safety protocols (e.g., pausing if a user entered the operating space).
• Error Manipulation: The system was programmed to trigger four specific pre-defined errors randomly (e.g., failing to grab a capsule, not pressing the brew button) at a maximum rate of one per coffee. Additionally, the system captured unexpected, unprogrammed errors. All triggered errors were eventually resolved by the robot to ensure users received their drinks.

Study Procedure:

• Participants: 49 unique participants engaged in 39 complete interactions (26 single-user, 13 group interactions). Fourteen interactions involved repeat participants.
• Data Collection: Video and audio were recorded upon user consent (indicated by stepping over a taped line and signing a form).
• Analysis: Researchers performed a thematic analysis using inductive and deductive coding. Two coders annotated user behaviors (verbal, facial, body movement) and speech relevance. Metrics included reaction time, reaction duration, and rates of volunteered vs. irrelevant speech.

Key Contributions

1. Autonomous Deployment Platform: The development of a fully autonomous coffee-making robot system capable of long-term public deployment, data collection, and safe operation without human supervision.
2. Empirical Understanding of Wild Interactions: An empirical analysis of individual and group social responses to robot errors in unconstrained environments, covering repeated interactions, successive errors, and error propagation.
3. Lessons on Social Signal Utility: A discussion on the opportunities and challenges of using social signals for error management in real-world Human-Robot Interaction (HRI), highlighting the dual nature of these signals as both information-rich and "noisy."

Results

1. Prevalence and Variety of Reactions:
Social signals were reliably exhibited in response to errors (97.67% of errors elicited a reaction). However, the modality range was broader than in lab studies, including walking away, talking to passersby, and using phones. Common reactions included smiling, talking, leaning in, and physically assisting the robot.

2. Volunteered Information vs. Noise:

• Information Richness: Participants frequently volunteered information beyond simple error acknowledgment. 58.14% of robot errors elicited speech providing additional details (e.g., diagnosing the cause: "I think it didn't press hard enough").
• Noise: A significant portion of speech was irrelevant to the task. In group interactions, the rate of irrelevant speech was notably higher (4.18 sentences/min) compared to single-user interactions (0.10 sentences/min).
• Contextual Ambiguity: Signals were triggered not only by the robot but also by environmental factors, phones, and other people, making it difficult to isolate the source of a reaction.

3. Impact of Interaction Type:

• Group Interactions: Groups produced a higher density of social signals. Every member in a group interaction expressed signals, whereas this was not consistent in single-user interactions. Groups engaged in "joint sensemaking," often debugging errors aloud and elaborating on each other's reactions.
• Repeated Interactions: Over time, participants tended to disengage (e.g., looking at phones, walking away) and stopped recording the robot. However, they also reacted to the absence of errors in subsequent interactions if they had experienced errors previously, indicating shifting mental models.
• Successive and Propagated Errors: Multiple errors led to shifts in engagement and valence. Successive errors often resulted in negative valence speech (e.g., calling the robot "stupid") or disengagement. Propagated errors (where one error led to a chain of incorrect robot actions) elicited increasingly explicit and diagnostic reactions.

Significance and Claims

The paper claims that social signals in the wild are rich but "noisy."

• Richness: These signals provide deep insights into user mental models, intent, and perception of the robot, offering information that goes beyond immediate error detection (e.g., user trust, environmental context).
• Noisiness: The signals are often ambiguous, context-dependent, and triggered by factors unrelated to the robot's performance (e.g., novelty effects, phone usage, group dynamics).

The authors conclude that while social signals hold promise for making robots more "human-aware" and adaptive, their utility in real-world systems is complicated by the lack of control found in lab settings. Effective implementation requires context-aware interpretation that can distinguish useful cues from noise, potentially by weighting signals based on group dynamics or scene understanding. The paper does not claim that current models can simply be transferred from the lab to the wild; rather, it emphasizes the need for new approaches to handle the complexity of public, unconstrained interactions.