💻 computer science

ARTOO-DARTU: Studying AR-HRC With AR Obstruction Mitigation During a Warehouse Task

This paper introduces ARTOO-DARTU, an augmented reality system for warehouse human-robot collaboration that integrates an obstruction detection and mitigation pipeline to prevent visual occlusion, thereby significantly improving task efficiency and real-world visibility as demonstrated in a 34-participant user study.

Original authors: Christian Fronk, Hanting Ye, Zhehan Qu, Maria Gorlatova

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Christian Fronk, Hanting Ye, Zhehan Qu, Maria Gorlatova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are working in a busy warehouse, but instead of just moving boxes, you are teaming up with a robot friend to find specific items. To help you work together, you are wearing special glasses (Augmented Reality, or AR) that show you floating digital notes, arrows, and "thought bubbles" coming from the robot. These digital hints tell you where the robot is going, what it's thinking, and what it needs you to do.

This sounds helpful, right? But here's the problem: The digital notes can get in the way.

The Problem: Digital Clutter Blocking Real Life

The researchers found that if you have too many floating digital notes, they can accidentally cover up the real-world labels on the boxes you need to read. It's like trying to read a street sign while someone is holding a giant, semi-transparent poster right in front of your face. You might miss the sign, get confused, or even bump into things.

In their study, they simulated a warehouse and found that as more robots started working, the digital "noise" got so bad that it frequently blocked the workers' view of the important labels they needed to do their jobs.

The Solution: ARTOO-DARTU

To fix this, the team at Duke University built a system called ARTOO-DARTU. Think of it as a "smart traffic cop" for your digital glasses.

Here is how it works:

The Digital Hints: The system shows you the robot's "thoughts" (like "I'm tired" or "I'm going left") and its location.
The Smart Filter (ODM): This is the magic part. The system constantly scans what you are looking at. If it sees that a digital note is covering up a real-world box label or an object, it instantly makes that digital note completely invisible (transparent).
The Result: You can still see the robot and its location, but the part of the digital note that was blocking your view of the real world disappears. Once the robot moves away, the note pops back into view.

It's like wearing glasses that automatically turn the digital overlays into ghosts whenever they block your view of a real person or object.

The Game: Pocket MonstARs

To test this, they didn't just ask people to move boxes; they turned it into a video game called Pocket MonstARs.

The Goal: You and the robot have to "catch" 15 monsters.
The Twist: Some monsters are floating in the air (easy to see). Others are hiding inside real cardboard boxes. To catch the ones in the boxes, you have to read the real labels on the boxes to find the right one.
The Test: They had people play the game with the "smart filter" turned on and off.

What They Found

The results were clear and surprising:

Without the filter: The digital notes actually made people slower. The notes blocked the box labels, so people had to struggle to find the right boxes.
With the filter: People were 46% faster at the whole task. When the filter was on, people were 61% faster specifically at finding the boxes with hidden monsters.
The Feeling: People felt more confident and less frustrated when the filter was active. They felt like they were actually good at the job.

The Takeaway

The paper concludes that Augmented Reality is a great tool for helping humans and robots work together, but only if it doesn't blind you to the real world.

If you put digital information on top of a real-world task, you have to make sure that information doesn't hide the things you actually need to see. The "smart filter" (ODM) proved that by automatically hiding the digital clutter when it gets in the way, you can make the whole team faster, safer, and happier.

In short: AR is like a helpful co-pilot, but if the co-pilot keeps waving a giant flag in front of your windshield, you'll crash. ARTOO-DARTU is the system that makes the flag disappear the moment it blocks your view of the road.

Technical Summary: ARTOO-DARTU: Studying AR-HRC With AR Obstruction Mitigation During a Warehouse Task

Problem Statement
Human-robot collaboration (HRC) in warehouse environments increasingly relies on Augmented Reality (AR) to convey robot intentions, internal states, and spatial awareness. However, a critical challenge arises when AR situated analytics—virtual content anchored to mobile robots—obstructs the user's view of real-world elements essential for task completion, such as inventory labels, physical objects, and safety signs. In dynamic warehouse settings where both workers and robots move unpredictably, static AR view management techniques (e.g., moving content to the ceiling) are insufficient because they break the spatial coupling between the analytics and the robot. Furthermore, existing obstruction detection methods often rely on predefined 3D models or cloud-based vision-language models, which introduce latency, privacy risks, and fail to generalize to unseen real-world elements like specific inventory labels. This paper addresses the need for an AR system that provides rich situated analytics while actively mitigating real-world obstructions without disrupting the spatial alignment of the content or the robot's task execution.

Methodology
To address these challenges, the authors developed ARTOO-DARTU (Augmented Reality Technique Of Obstruction Deterrence while Aiding Robotic Teaming for Users), an AR system for the Microsoft HoloLens 2, and evaluated it through a controlled user study using a gamified task called Pocket MonstARs.

System Architecture: The system integrates a HoloLens 2 headset, a TurtleBot 4, and an edge server (NVIDIA RTX 3060). The headset runs a Unity application that displays situated analytics and allows user control via a virtual map marker. The edge server hosts a FastAPI-based communication layer and performs heavy computation for the Obstruction Detection and Mitigation (ODM) pipeline.
Situated Analytics: The system visualizes robot position, internal state (battery, heat, Wi-Fi, maintenance), command progress, and intended trajectory. Key indicators include a spherical location indicator, a "thought bubble" for personified state messages, a command status screen, and trajectory arrows.
Obstruction Detection and Mitigation (ODM): The core innovation is the ODM pipeline, designed to operate in near real-time without cloud dependency.
1. Projection: The HoloLens captures a raw image of the real world (without virtual content) and computes 2D bounding box projections for all active virtual items.
2. Detection: The image and projected boxes are sent to the edge server, which runs YOLOv5 for object detection and EasyOCR for text detection.
3. Mitigation: The system calculates the overlap between virtual item projections and detected real-world objects/text. If a virtual item overlaps a real-world element by more than 30%, that item is flagged. The HoloLens app then renders the flagged item as fully transparent until it moves more than 20 cm away from the obstruction point, preserving spatial alignment while restoring visibility.
Experimental Task (Pocket MonstARs): A 2.4×2.4 m task area containing 12 labeled boxes and 4 boxes with YOLO-recognizable objects. Users act as warehouse workers, directing the robot to "catch" 15 monsters. Ten monsters are visible virtual targets; five are "box monsters" hidden inside real-world boxes, requiring users to identify specific labels or objects to locate the target.
User Study: A 2 × 3 mixed-factorial study with 34 participants. The between-subjects factor was Group (ODM Active vs. ODM Inactive). The within-subjects factor was Trial (No Analytics, Situated Analytics, Situated Analytics with Arrows). Performance was measured by catch times, and subjective feedback was gathered via surveys (GEQ, Ease-of-Use, Trust).

Key Contributions

ARTOO-DARTU System: An integrated AR-HRC system delivering situated analytics and robot control specifically designed for mobile warehouse robots, with 83% of participants preferring it over a baseline with no analytics.
ODM Pipeline: The first obstruction detection and mitigation pipeline specifically designed for robot-coupled situated analytics. Unlike prior work, it operates on previously unseen real-world elements (labels and objects) in dynamic environments without requiring cloud infrastructure or predefined 3D models.
Pocket MonstARs: A controlled, gamified experimental task that abstracts warehouse inventory picking. It preserves the critical demands of real-world target identification (reading labels, recognizing objects) while enabling systematic evaluation of AR obstruction and analytics.

Results

Efficiency: The presence of situated analytics increased overall task efficiency by 46%, but only when the ODM was active. Without ODM, analytics did not improve efficiency and sometimes hindered performance due to obstructions.
Real-World Subtasks: Participants with the ODM active were 61% faster on subtasks requiring real-world visibility (catching "box monsters") compared to those without ODM. Specifically, in the Situated Analytics (SA) trial, the ODM group caught box monsters in 32.7s versus 57.9s for the non-ODM group ( $p=0.002$ ).
Overall Performance: Participants with the ODM active were 39% faster overall in the SA trial compared to the non-ODM group.
Subjective Feedback: Users in the ODM group reported significantly lower perceived difficulty and higher feelings of competence. 83% of all participants ranked a trial with AR analytics as their favorite. Users with ODM inactive frequently cited frustration with obstructed views, a sentiment absent in the ODM group.
Arrows: The addition of trajectory arrows (SA+ARR) did not significantly improve performance over SA alone and, in some cases, led to slower catch times due to navigation errors where users placed the robot marker inside boxes based on arrow placement.

Significance and Claims
The paper claims that AR situated analytics can significantly enhance efficiency and user experience in warehouse HRC scenarios, but only when paired with a robust obstruction mitigation strategy that preserves real-world visibility. The authors emphasize that obstruction management should be treated as a "first-class concern" and a prerequisite for effective AR assistance, rather than a secondary refinement. The study demonstrates that the ODM successfully balances the need for rich robot state information with the safety and usability requirement of maintaining visibility of task-relevant physical elements. The authors note that while their study used a gamified abstraction, the findings highlight a fundamental trade-off in AR-HRC: analytics improve interaction only if the system prevents the virtual content from obscuring the physical world. They modestly conclude that future work is needed to validate these findings in real warehouse environments with actual workers and safety-critical elements.

The Problem: Digital Clutter Blocking Real Life

The Solution: ARTOO-DARTU

The Game: Pocket MonstARs

What They Found

The Takeaway

More like this