💻 computer science

Achieving Interaction Fluidity in a Wizard-of-Oz Robotic System: A Prototype for Fluid Error-Correction

This paper proposes key criteria for achieving fluid error correction in Wizard-of-Oz robotic systems and presents a Virtual Reality simulation environment for mobile manipulators designed to meet these standards.

Original authors: Carlos Baptista De Lima, Julian Hough, Frank Förster, Patrick Holthaus, Yongjun Zheng

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Carlos Baptista De Lima, Julian Hough, Frank Förster, Patrick Holthaus, Yongjun Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to have a conversation with a robot. You say, "Pick up that cup," and the robot starts moving. But then you realize, "Wait, I meant the blue cup, not the red one!" In a perfect world, the robot would instantly stop, look at you, and grab the blue cup.

In the real world of robotics today, it's usually much clunkier. The robot might finish grabbing the red cup, or it might freeze, or it might take a long time to figure out what you changed your mind about. It feels like talking to someone who is constantly lagging on a bad video call.

This paper is about building a better "practice ground" for robots so they can learn to be smoother, faster, and more human-like in how they handle mistakes.

The Problem: The "Wizard" is Too Slow

To teach robots how to talk and move, researchers often use a method called Wizard-of-Oz (WoZ). Imagine a movie set where the "robot" is actually being controlled by a human hidden in a control room (the "Wizard"). The human watches the user and presses buttons to make the robot move and speak, pretending to be the robot's brain.

The problem is that most of these "control rooms" (software systems) are messy.

The Lag: By the time the Wizard sees what the user said, clicks a button, and the robot moves, seconds have passed. It's like playing a video game with terrible internet lag.
The "Can't Stop" Button: If the Wizard makes a mistake, or if the user changes their mind, many systems don't have a quick way to say, "Stop! Do something else!" The robot just keeps plodding along.
The Black Box: When the Wizard is done, it's hard to replay exactly what happened to study it later. It's like trying to learn to drive by watching a blurry, unrecorded video.

The Solution: A Virtual Reality "Flight Simulator"

The authors built a new, high-tech training environment using Virtual Reality (VR). Think of it as a flight simulator for robots, but instead of a plane, it's a robot arm on a mobile base (like a robot on wheels).

Here is how their new system fixes the problems, using some simple analogies:

1. The "Instant-Edit" Button (Interruptibility & Correction)

In old systems, if the Wizard told the robot to walk to the kitchen, and then realized the user wanted the living room, the Wizard might have to stop the whole program and restart it.

The New Way: Their system is like a GPS app on your phone. If you are driving to the store and decide to go to the bank instead, you just tap the new destination, and the GPS instantly recalculates. The robot doesn't stop; it just smoothly turns toward the new goal. The Wizard can also hit a big "Emergency Stop" button to cancel everything immediately for safety.

2. The "Live Status Bar" (Pollability)

Sometimes a Wizard doesn't know exactly where the robot is in its task. Is it halfway there? Is it stuck?

The New Way: The Wizard has a dashboard that acts like a "progress bar" for the robot's brain. They can see exactly how far the robot has moved or how close it is to picking up an object. It's like having a live tracker for a delivery driver so you know exactly when they will arrive.

3. The "Stopwatch" (Latency Measurement)

In the old days, if the interaction felt slow, nobody knew why. Was it the computer? The internet? The human thinking too slowly?

The New Way: This system is like a race car with sensors on every part of the engine. It measures exactly how long it takes for a sound to go from the user's mouth, to the Wizard's ear, to the button click, and finally to the robot's movement. By knowing exactly where the time is lost, they can fix the specific bottleneck.

4. The "Perfect Replay" (Reproducibility)

If a researcher wants to study a mistake, they used to have to guess what happened.

The New Way: The system records everything in perfect detail: where the robot was, what the user saw, what the Wizard clicked, and exactly when everything happened. It's like having a "Save Game" feature. They can replay the exact same interaction over and over again to test new ideas, just like a video game player replaying a level to beat a high score.

Why Does This Matter?

The goal is to create robots that feel "fluid." Just like when you and a friend are passing a ball, you don't stop and think about every move; you just flow.

By using this VR "flight simulator," researchers can train robots to handle interruptions and corrections naturally. Once the robot learns these skills in the virtual world (using data collected by the Wizard), it can eventually do them on its own in the real world.

In short: This paper presents a new, high-tech "training gym" for robots. It gives the human controllers (the Wizards) the superpowers they need to make robots stop, think, and adapt instantly, paving the way for robots that feel less like clunky machines and more like helpful, fluid partners.

1. Problem Statement

Current Human-Robot Interaction (HRI) systems, particularly those utilizing speech interfaces, suffer from a lack of "fluidity." Interactions are often described as labored, slow, and fragile. While robots have advanced in vision and motion, the transition from speech processing to physical action often involves noticeable delays and an inability to handle real-time corrections (e.g., a user changing an instruction mid-action).

The core problem identified is the lack of a suitable development platform for Wizard-of-Oz (WoZ) systems. WoZ is a methodology where a human operator (the "Wizard") tele-operates a robot to simulate autonomous behavior for data collection and prototyping. However, existing WoZ systems often fail to support the specific properties required for fluid interaction, such as:

High Latency: Delays between user commands and robot execution.
Rigidity: Inability to interrupt or correct ongoing actions in real-time.
Poor Data Fidelity: Inability to accurately log interaction timings for training future autonomous systems.
Complex Interfaces: Overly complicated GUIs that increase the cognitive load on the Wizard, leading to further delays.

2. Methodology

The authors propose a new framework and a prototype system designed to address these gaps.

A. Defining Criteria for Fluidity

Based on literature review and direct experience, the authors define four core criteria that a WoZ system must meet to support fluid interaction:

Interruptibility and Correction (IaC): The ability for the Wizard to update a robot's current action (e.g., changing a target destination) or cancel all ongoing actions immediately for safety and adaptability.
Pollability: The system must allow the Wizard to query and visualize the exact state and progress of the robot's current action at any given moment to make informed decisions.
Latency Measurement and Optimization: The system must measure and minimize latency across the entire pipeline (User speech $\to$ Wizard perception $\to$ Action execution $\to$ User perception).
Reproducibility of Action Timings: The system must log all interaction data with high temporal accuracy to allow for the reconstruction of interactions and the training of autonomous models.

B. Literature Review

The authors conducted a review of 14 recent WoZ studies. The analysis (summarized in Table 1 of the paper) revealed that:

Most systems use custom GUIs that are not generalizable.
Very few systems explicitly support Interruptibility (only 2 papers) or Pollability (only 1 paper).
Latency is rarely measured or optimized systematically.
Reproducibility is often compromised by synchronization issues between different data streams.

C. Prototype System Architecture

The authors developed a Virtual Reality (VR) HRI simulation environment using the following stack:

Robot Model: A mobile manipulator ("Fetch" robot) simulated in Gazebo and controlled via ROS (Robot Operating System) and MoveIt for planning.
Visualization: A Unity-based Visualizer running in a Docker container, communicating with the ROS backend via ROS-TCP Connectors.
User Interface: The user interacts via a VR headset (Meta Quest 3) and hand controllers, speaking commands into a microphone.
Wizard Interface: The Wizard views the scene on a separate screen, sees the robot's camera feed, and controls the robot via mouse clicks.
Control Module: A Python-based module handles action control and logging. It receives ROS messages from the Unity visualizer and sends execution commands back to the simulator.

3. Key Contributions

The paper's primary contribution is the design and implementation of a WoZ prototype that satisfies the four fluidity criteria, which existing systems fail to do comprehensively.

Implementation of Interruptibility & Correction:
- Goal Update: The Wizard can override a current action by clicking a new target (object or location) while the robot is moving.
- Emergency Cancel: A dedicated "Cancel all actions" button instantly revokes current goals, bringing the robot to a safe default state.
Enhanced Pollability:
- The Wizard has a continuous, synchronous view of the robot's state and the environment.
- Visual cues (red cone for target, green highlight for object, blue spot for destination) provide immediate feedback on the robot's current intent and progress.
Latency Optimization & Measurement:
- The system decomposes latency into four distinct phases: (1) User speech to Wizard initiation, (2) Wizard perception to action start, (3) Action start to user perception, and (4) Interruption/repair delay.
- Optimization strategies include full audio stream access for the Wizard and tuning MoveIt recovery behaviors to be less conservative, reducing execution lag.
High-Fidelity Reproducibility:
- All data is logged in JSON format with precise timestamps and ID tags.
- Logged Data Includes: Wizard mouse-click vectors, robot base/joint states, interactable object positions/rotations, user camera view, and user controller inputs.
- This allows for the "playback" of interactions, enabling the reconstruction of the scene for analysis or training autonomous agents.

4. Results

While the paper presents a prototype rather than a completed user study (which is planned for future work), the technical evaluation demonstrates:

System Integration: The successful integration of ROS, Gazebo, Unity, and VR hardware via TCP connectors, ensuring synchronous updates between the Wizard and User views.
Criteria Satisfaction: The prototype explicitly addresses the gaps found in the literature review. Unlike previous systems, it offers a unified interface that supports real-time correction, granular state polling, and precise temporal logging.
Scalability: The architecture is designed to be generalizable beyond the Fetch robot to any ROS-based simulation and Unity environment.

5. Significance

This work is significant for the HRI community for several reasons:

Bridging the Gap: It addresses the "fluidity" gap in HRI, moving beyond simple command execution to dynamic, human-like interaction where errors can be corrected seamlessly.
Data-Driven Development: By providing a system that logs high-fidelity, time-accurate data, it enables the training of better autonomous control algorithms. Future autonomous robots can be trained on data collected from these fluid WoZ interactions.
Standardization: It proposes a set of criteria (IaC, Pollability, Latency, Reproducibility) that should become standard requirements for future WoZ research platforms.
Safety and Robustness: The emphasis on interruptibility and immediate cancellation aligns with safety standards for personal care robots, ensuring that human operators can intervene instantly if the robot behaves unpredictably.

In conclusion, the authors present a robust, VR-based WoZ platform that serves as a foundational tool for developing the next generation of fluid, responsive, and safe human-robot interactions.