💻 computer science

SHARE: Towards Head-Mounted AR with User-Centric SLAM in Shared Human-Robot Workspaces

This paper presents SHARE, a user-centric SLAM system for shared human-robot workspaces that prioritizes AR user experience by adaptively adjusting transmission priorities and leveraging visual feature redundancy to significantly reduce latency while maintaining high tracking accuracy.

Original authors: Tianyuan Du, Tianyi Hu, Hanting Ye, Maria Gorlatova

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianyuan Du, Tianyi Hu, Hanting Ye, Maria Gorlatova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are wearing a pair of magical glasses that can paint digital pictures onto the real world. You might see a virtual robot arm helping you build a LEGO castle, or a glowing arrow pointing you to your lost keys. This is Augmented Reality (AR). But for these pictures to feel real, they have to be perfect. If you turn your head and the virtual picture lags behind even a tiny bit, your brain gets confused, the picture looks like it's swimming, and you might even feel sick. To make this magic work, the glasses need to know exactly where you are and what the room looks like, instantly. This job is called SLAM (Simultaneous Localization and Mapping). It's like a super-fast brain that is constantly drawing a map of the room while figuring out where you are standing inside it.

Now, imagine you aren't alone in the room. There is also a real robot, like a vacuum cleaner or a helper bot, working alongside you. Both you and the robot are trying to draw the same map at the same time. Usually, to save energy, these devices send their drawings to a powerful computer nearby (called an "edge server") to combine them into one giant, perfect map. The problem is that this computer often gets overwhelmed. It treats you and the robot exactly the same, sending data back and forth at the same speed. But you and the robot have very different needs: your magical glasses need to be lightning-fast to keep the illusion alive, while the robot can be a little slower and still do its job. If the computer doesn't know the difference, your virtual pictures might start to lag, breaking the magic.

This is where a new system called SHARE comes in. The researchers behind SHARE realized that treating a human and a robot exactly the same is a recipe for disaster in a shared workspace. They built a system that acts like a smart traffic controller for data. Instead of giving everyone the same amount of road space, SHARE looks at who is driving and adjusts the speed limits. It knows that the human wearing the AR headset is in a "speed zone" where every millisecond counts, while the robot is in a "cruising zone" where it can handle a bit more waiting.

To do this, SHARE uses a few clever tricks. First, it creates a "happiness score" (called Quality of Experience, or QoE) for each agent. For the human, the score drops instantly if there is any delay. For the robot, the score only drops if the robot gets lost or makes a mistake. The system constantly checks these scores and uses a special math tool (called a PID scheduler) to decide who gets to talk to the server first. If the human is moving fast or looking at a boring, empty wall where it's hard to find landmarks, the system shouts, "Human first!" and speeds up their data. If the robot is just cruising, it waits its turn.

Second, SHARE is a master of spotting redundancy. In a small room, you and the robot might both be looking at the same corner of the wall at the same time. Old systems would send the picture of that corner twice, wasting time and energy. SHARE calculates exactly how much your view overlaps with the robot's view. If you are both looking at the same spot, it only sends the data once, saving precious time. It's like two friends describing a movie to a third person; instead of both talking over each other, they take turns or combine their stories so no one gets bored waiting.

The researchers tested this system in the real world using actual AR headsets (Meta Quest 3) and a ground robot (TurtleBot 4). They found that SHARE was incredibly effective. For the human users, the system reduced the delay (latency) to an average of 13.22 milliseconds. This is a huge improvement, cutting the lag by 43.3% compared to the standard system, and it kept the delay well below the 20-millisecond limit needed to prevent the "swimming" effect. Crucially, the robot didn't get lost; it stayed accurate to within less than 2 centimeters.

In a user study with 20 people, the results were even more impressive. When people used SHARE to control the robot through their AR glasses, they felt the robot was much more predictable and responsive. In fact, 65% of the participants said SHARE was their favorite system, and no one picked it as their least favorite. The study suggests that by treating humans and robots differently based on what they actually need, we can make shared workspaces feel much smoother and more magical, without needing super-expensive computers to do the heavy lifting.

Technical Summary: SHARE: Towards Head-Mounted AR with User-Centric SLAM in Shared Human–Robot Workspaces

1. Problem Statement

The paper addresses a critical latency bottleneck in Augmented Reality (AR)-enabled Human-Robot Collaboration (HRC) systems that rely on multi-agent Simultaneous Localization and Mapping (SLAM). While existing multi-agent SLAM frameworks (e.g., COVINS-G) effectively coordinate heterogeneous agents (robots and AR headsets) via edge servers, they treat all agents uniformly. This uniformity fails to account for the fundamentally different latency requirements of HRC agents:

AR Headsets: Require strict motion-to-photon latency (typically < 20 ms) to prevent visual drift and user discomfort.
Robots: Can tolerate significantly higher latency (up to 750 ms–1 s) as their control loops are less sensitive to immediate visual feedback.

Current edge-assisted SLAM systems often exceed the 20 ms threshold for AR users due to uniform resource allocation, redundant computations in shared indoor spaces, and the lack of differentiation between agent types. The authors identify three specific challenges:

Heterogeneous Latency Tolerance: How to prioritize AR latency while exploiting the robot's looser constraints.
Dynamic Visual Environments: How to balance latency and accuracy when visual feature density fluctuates (e.g., featureless walls vs. rich textures).
Map Redundancy: How to eliminate inefficient, redundant map merging operations in small-scale indoor environments where agents frequently share overlapping lines of sight.

2. Methodology: The SHARE System

The authors propose SHARE, a user-centric SLAM system designed to dynamically allocate resources based on agent-specific Quality of Experience (QoE) and visual environment dynamics. The system architecture integrates three core components:

A. Agent-Specific QoE Modeling

SHARE introduces a QoE model that quantifies tracking performance differently for AR users and robots, moving beyond uniform latency metrics.

Formulation: The QoE is defined as a function of latency ( $d$ $d$ ) and accuracy ( $a$ $a$ ), utilizing an exponential relationship to capture diminishing marginal returns.
- For AR Users: Latency is the dominant penalty factor ( $P_i = d - d_0$ , where $d_0 = 20$ ms). Accuracy provides marginal benefits.
- For Robots: Accuracy is the dominant penalty factor ( $P_i = a_0 - a$ , where $a_0 = 1$ cm). Latency has diminishing marginal returns as robots can buffer delays.
Real-time Accuracy Estimation: Since ground-truth trajectories are unavailable in real-time, SHARE deploys a lightweight Deep Neural Network (DNN) based on MobileNetV2 on mobile agents. This network estimates localization error in real-time using three input modalities: visual keyframe features, Point Spatial Distribution (PSD) matrices, and scalar tracking statistics.

B. PID Scheduling Pipeline

To address dynamic visual conditions, SHARE employs a Proportional-Integral-Derivative (PID) scheduler to adjust frame transmission rates.

Mechanism: The scheduler treats the deviation between the target QoE and current QoE as an error signal. It dynamically modulates the frame transmission rate ( $R$ ) for each agent.
Adaptation: Agents in feature-poor environments (requiring higher frame rates for accuracy) receive increased transmission priority, while agents in feature-rich environments (where redundancy is high) have their rates reduced to lower latency. This replaces static or rule-based thresholding with continuous, feedback-driven optimization.

C. Visual Overlap Analysis for Redundancy Reduction

To mitigate computational overhead in shared indoor workspaces, SHARE implements a selective map merging strategy.

Pose-Based Overlap Calculation: Instead of processing full feature sets to detect overlaps, the system calculates the "overlap volume" between agents' view cones using only 6-DoF pose data (approx. 200 bytes vs. 200 KB of features).
Probabilistic Merging: Map merging is triggered based on the overlap volume. The system uses a probabilistic threshold to skip redundant merges when agents are in close proximity with high visual overlap, retaining merges only when they offer significant new geometric information (i.e., at the peak of visual overlap).

3. Key Contributions

User-Centric QoE Framework: The first QoE model for HRC that explicitly differentiates between latency-sensitive AR users and accuracy-critical robots, enabling differentiated resource allocation.
Real-Time Accuracy Estimation: A lightweight DNN architecture that estimates SLAM localization error without ground-truth data, enabling the QoE model to function in real-world deployments.
Adaptive PID Scheduling: A control-theory-based scheduling pipeline that dynamically adjusts transmission priorities based on real-time QoE deviations and environmental dynamics.
Redundancy-Aware Map Merging: A pose-based methodology to identify and trim redundant map merging operations in confined indoor spaces, reducing edge computation and communication overhead.
System Implementation: A full implementation of SHARE using Meta Quest 3 headsets and a TurtleBot 4 robot, integrated with the COVINS-G framework.

4. Results and Evaluation

The system was evaluated using real-world field studies (5 trajectories, 483.93 meters) and synthesized datasets (Warehouse and Firestation environments).

Latency Performance: SHARE achieved an average tracking latency of 13.22 ms for AR users, a 43.3% reduction compared to the baseline COVINS-G system. Crucially, SHARE maintained latency below the 20 ms AR threshold across all trajectories, whereas COVINS-G frequently exceeded this limit.
Accuracy Performance: SHARE maintained sub-2 cm Absolute Trajectory Error (ATE) and sub-1 cm Relative Pose Error (RPE) for all agents, comparable to state-of-the-art SLAM systems. In feature-sparse environments (Firestation), SHARE outperformed baselines by increasing frame rates via the PID scheduler.
Efficiency: The visual overlap calculation reduced map merging operations by an average of 43.72% compared to COVINS-G without significant accuracy loss.
User Study: A study with 20 participants revealed statistically significant improvements in robot predictability (+25.3% over COVINS-G), responsiveness (+12.1%), and reachability (+24.6%). 65% of participants rated SHARE as their most preferred solution.

5. Significance and Claims

The paper claims that SHARE represents a significant step forward in making AR-powered HRC viable for real-world applications by resolving the latency-accuracy trade-off inherent in heterogeneous multi-agent systems.

Differentiation is Key: The work demonstrates that treating heterogeneous agents (humans vs. robots) with uniform SLAM policies is suboptimal. By prioritizing the AR user's strict latency requirements while relaxing constraints for robots, the system achieves a superior overall experience.
Context-Aware Optimization: The integration of QoE modeling with PID scheduling and visual redundancy analysis allows the system to adapt to both agent heterogeneity and environmental dynamics, ensuring robust performance in feature-sparse or crowded indoor spaces.
Practical Viability: The system operates effectively on modest hardware (Intel N100 on AR headsets, 4-core CPU on edge servers), proving that high-performance, low-latency collaborative SLAM does not require expensive GPU acceleration.

The authors conclude that these insights extend beyond SLAM, suggesting that any edge-based multi-agent system serving heterogeneous clients can benefit from quality-of-experience differentiation and context-aware data fusion.