💻 computer science

A Multi-Operator Mixed-Reality Interface for Multi-Robot Control and Coordination: Co-Located and Private Workspace Collaboration

This paper presents a mixed-reality interface for multi-operator multi-robot control that supports both co-located shared and private workspace modes, demonstrating through a user study that while both modes enable effective mission execution, the co-located shared workspace significantly enhances perceived collaboration, shared understanding, and handoff clarity.

Original authors: Omotoye Shamsudeen Adekoya, Antonio Sgorbissa, Carmine Tommaso Recchiuto

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Omotoye Shamsudeen Adekoya, Antonio Sgorbissa, Carmine Tommaso Recchiuto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pilot a team of three tiny, remote-controlled robots through a complex building to find hidden treasures. Now, imagine you have a partner helping you. The challenge isn't just controlling the robots; it's figuring out how to work together without stepping on each other's toes or giving the robots conflicting orders.

This paper describes a new "control room" built inside a Mixed Reality (MR) headset. Think of it not as a flat computer screen, but as a floating, miniature 3D diorama of the real world where the robots are moving. You can reach out, grab a tiny robot, and tell it where to go.

The researchers wanted to answer one big question: Does it matter if you and your partner are standing in the same room looking at the same floating map, or if you are in different rooms looking at your own separate maps?

Here is how they tested it and what they found, using simple analogies:

The Two Ways to Play

The system offered two different "modes" for the two operators:

The "Shared Table" Mode (Co-located):
Imagine you and your partner are standing in the same room. You both put on headsets and see the exact same miniature map floating in the air between you. You can point at the same spot, look at the same robot, and see each other's hand gestures. It's like two people huddled over a single physical map on a table.
The "Private Desk" Mode (Private-workspace):
Imagine you and your partner are in different rooms. You both have your own headsets, and you each see your own floating map. You are working on the same mission, but you can't see where your partner is looking or what they are pointing at. You have to guess what they are doing based on the robot's movements.

The Rules of the Game (Conflict Prevention)

A major problem in team robot control is "traffic jams." If you tell a robot to go left and your partner tells it to go right at the same time, the robot gets confused.

To fix this, the system uses a "Lease" system, like a "talking stick" or a "hot potato" game:

When you want to control a specific robot, you "lease" it.
The system turns that robot Green for you (you own it) and Red for your partner (they can't touch it).
You can only give orders to the robot you are holding. This prevents the robots from getting conflicting commands, no matter which mode you are in.

The Experiment

The researchers put 36 people (18 pairs) into a simulation. They had to find hidden digital tags in two different virtual environments: a hospital and an office.

Goal: Find as many tags as possible in 10 minutes.
Task: They had to switch between watching the whole team and taking direct control of individual robots to navigate tight corners.

The Results: What Worked Best?

Here is the surprising part: Both modes worked equally well for the actual job.

The Scoreboard: The pairs found roughly the same number of tags whether they were in the same room or different rooms. The "Private Desk" mode was reliable enough that you didn't need to be in the same room to get the job done.
The Feeling: However, the experience was very different. The pairs in the "Shared Table" mode felt much more connected. They reported:
- Understanding each other's intentions better.
- Feeling like they were truly "in the same space."
- Finding it much easier to hand off control of a robot to their partner.
- Having less mental stress trying to guess what their partner was doing.

The Takeaway

Think of it like this: If you just need to get a package delivered, you can do it alone or with a friend on the phone (Private Mode). But if you are trying to build a complex piece of furniture together, it is much easier and less frustrating if you are standing in the same room looking at the same instructions (Shared Mode).

The paper concludes that while you can control robots effectively from separate rooms, physically being in the same room and sharing the same virtual view makes the teamwork feel smoother, clearer, and more natural. It doesn't make the robots move faster, but it makes the humans work better together.

Technical Summary: A Multi-Operator Mixed-Reality Interface for Multi-Robot Control and Coordination

Problem Statement

Scaling multi-robot systems from single-operator supervision to team-level collaboration presents significant challenges. Conventional 2D Ground Control Stations (GCS) require operators to mentally integrate disparate data streams (maps, camera feeds, robot states), a cognitive burden that degrades situational awareness and increases workload as mission complexity grows. While Mixed Reality (MR) interfaces have shown promise in reducing this burden for single operators by providing spatially embodied views, extending these systems to support multi-operator collaboration introduces new complexities. Specifically, effective collaboration requires mechanisms for mutual awareness, division of labor, and, crucially, the prevention of conflicting commands when multiple humans attempt to control the same robot team. Furthermore, it remains unclear whether physically co-locating operators in a shared MR workspace offers tangible benefits over a private-workspace mode where operators share mission data but view it through independently placed local interfaces.

Methodology

The authors extended their previous HORUS (Holistic Operational Reality for Unified Systems) interface to support multi-operator control. The system architecture and experimental design are detailed below:

System Architecture

The proposed system utilizes a three-layer architecture:

SDK Layer: Publishes robot metadata, maps, and task capabilities.
ROS 2 Bridge: Handles command, state, and map communication between the simulation/robots and the interface.
Unity MR Runtime: Instantiates the workspace, visualizations, and coordination logic.

Key technical features include:

Registration-Driven Construction: The scene is dynamically constructed from SDK payloads rather than assuming a fixed set of robots.
Dual Collaborative Modes:
- Co-located Shared Workspace: Operators share a common physical location and a single, spatially anchored virtual workspace.
- Private Workspace: Operators share the same mission state but interact via independently placed local workspaces.
Conflict Prevention (Per-Robot Leases): To prevent conflicting interventions, the system employs a lease-based arbitration mechanism. A robot is "leased" to an operator when they open its control panel, author a task, or initiate teleoperation. The lease state (free, held locally, held remotely) is visually encoded (white, green, red) and functionally enforced, blocking conflicting actions on the same robot.
Synchronization: Low-rate collaborative state (workspace snapshots, registration boundaries) is synchronized via Unity Netcode for GameObjects (NGO), while heavy robotics data remains in the ROS transport layer.

Experimental Design

A human-subject study was conducted with 36 participants (18 pairs) controlling a team of three simulated Nova Carter mobile robots in two environments (an office and a hospital).

Conditions: Each pair performed trials in both the Co-located and Private modes using a counterbalanced crossover design.
Task: Participants had 10 minutes per trial to locate 10 hidden AprilTags (points of interest) using a combination of supervisory planning (waypoint assignment) and teleoperation.
Constraints: In the private mode, participants were in separate rooms with no verbal communication; in the co-located mode, they were in the same room with natural face-to-face communication allowed.
Metrics: Objective performance (tags found), System Usability Scale (SUS), NASA-TLX (workload), and a custom 8-item Collaboration and Shared Workspace questionnaire.

Key Contributions

Multi-Operator MR Architecture: A system supporting both co-located and private-workspace collaboration via shared-state synchronization and per-robot control leases.
Workspace-Centric Interaction Design: An interface featuring explicit mechanisms for shared tasking, awareness cues (lease states), and conflict prevention.
Dynamic Control Arbitration: A lease-based approach allowing parallel work on different robots while preventing simultaneous control of the same robot.
Empirical Evaluation: A controlled study isolating the effect of physical co-location, comparing it against a private-workspace alternative in a realistic multi-robot scenario.

Results

The study yielded the following findings:

Task Performance: There was no significant difference in objective task performance between the two modes. Pairs found an average of 3.94 tags in the co-located mode and 4.22 in the private mode ( $p = .633$ ). This indicates that the private-workspace mode is sufficient for effective mission execution even without direct verbal communication.
Usability and Workload: Perceived usability (SUS) and overall workload (NASA-TLX) were statistically comparable across both conditions.
Collaboration and Awareness: The co-located shared workspace significantly outperformed the private mode in subjective collaboration metrics ( $p < .001$ $p < .001$ ). Participants reported:
- Higher awareness of teammate actions.
- Better common understanding of the task state.
- Greater ease in coordinating actions and executing control handoffs.
- A stronger feeling of sharing a physical space.
Preference: A significant majority of participants (21/29) preferred the co-located mode for joint operation, citing clearer handoffs and better mission awareness.
Experience Effects: Prior teleoperation experience was a strong predictor of task success, particularly in the co-located mode. However, the interface was learnable for novices within a short tutorial (approx. 15 minutes), and training time was more correlated with general control proficiency than prior VR exposure.

Significance and Claims

The paper claims that physically co-locating the MR workspace improves how operators coordinate, even when the underlying control tools remain unchanged. The primary value of the co-located mode is not an increase in raw task throughput (which was comparable in both modes) but rather an enhancement in collaborative fluency, shared understanding, and the clarity of control handoffs.

The authors conclude that while private-workspace modes are viable for distributed collaboration, shared spatial references are a critical design principle for tightly coordinated local teams. The results suggest that future multi-operator MR systems should support both modes: co-located shared workspaces for high-fidelity local collaboration and private workspaces for distributed participation. The study validates that explicit arbitration mechanisms (leases) are necessary to enable safe parallelism, but the physical alignment of the workspace significantly reduces the cognitive load required to maintain mutual awareness.