💻 computer science

AssistDLO: Assistive Teleoperation for Deformable Linear Object Manipulation

AssistDLO is an adaptive assistive teleoperation framework that combines real-time state estimation, visual assistance, and a geometry-aware shared-autonomy controller to significantly improve Deformable Linear Object manipulation, particularly by equalizing success rates for novice users while demonstrating that optimal assistance strategies must dynamically account for both operator expertise and material properties.

Original authors: Berk Guler, Simon Manschitz, Kay Pompetzki, Jan Peters

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Berk Guler, Simon Manschitz, Kay Pompetzki, Jan Peters

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to untangle a knot in a long, floppy piece of string while wearing thick gloves and looking at the string through a flat, 2D TV screen. You can't see how deep the knot is, and if you pull the wrong part, the whole mess tightens or jumps out of your view. This is the daily struggle for robots trying to handle "Deformable Linear Objects" (DLOs)—things like ropes, cables, and wires that bend and twist in infinite ways.

The paper introduces AssistDLO, a new "co-pilot" system designed to help human operators control robots to untangle these tricky ropes. Instead of letting the robot do the whole job alone (which is hard because the rope's physics are too complex to predict perfectly) or letting the human struggle blindly, AssistDLO offers two types of help: Visual Assistance and Action Assistance.

Here is how it works, broken down with simple analogies:

1. The Problem: The "Flat Screen" and the "Floppy Snake"

When humans control robots remotely, they usually see a 2D video feed. This makes it hard to judge depth (how far away something is). With a stiff stick, this is annoying but manageable. With a floppy rope, it's a disaster. If you try to grab a spot on the rope but miss, your robot arm might accidentally sweep across the table, knocking other parts of the rope around and making the knot worse.

2. The Solution: A Smart Co-Pilot

The AssistDLO system uses two cameras on the robot's wrists to build a real-time 3D map of the rope. It then offers two ways to help:

Visual Assistance (The "Ghost Rope"): Imagine looking through your VR headset and seeing a glowing, transparent 3D outline of the rope floating exactly where it is in the real world. This doesn't move the robot for you; it just gives you "X-ray vision" so you can see exactly where the rope is in 3D space, helping you judge depth and avoid collisions.
Action Assistance (The "Safety Funnel"): This is the paper's big innovation. Instead of just showing you the rope, the system gently nudges the robot arm to help you grab it.
- Old Way (The Magnet): Previous systems acted like a magnet. If you aimed near the rope, the robot would just pull your hand straight toward the target. The problem? If you aimed slightly off, the robot would drag your hand through the rest of the rope, knocking it around before it even grabbed the target.
- New Way (The Funnel): The new system, called SA-CBF, acts like a funnel. Imagine the rope is sitting at the bottom of a smooth, invisible slide. If you try to grab the rope, the system guides your hand down the slide directly to the target. But if you try to move your hand sideways (where the rest of the rope is), the "walls" of the funnel gently push you back. It lets you grab the rope precisely without accidentally sweeping the rest of the mess around.

3. The Experiment: Who Benefits?

The researchers tested this with 22 people trying to untangle overhand knots using different types of ropes (some short and stiff, some long and floppy). They found that one size does not fit all:

For Beginners (The "Novices"): The "Safety Funnel" (Action Assistance) was a game-changer. It acted like a training wheel, increasing their success rate from about 71% to 88%. It helped them grab the rope without messing up the knot.
For Experts (The "Pros"): Experienced users didn't like the "Safety Funnel" as much. They felt it was getting in their way. Instead, they preferred the "Ghost Rope" (Visual Assistance). They already knew how to grab the rope; they just needed better eyes to see it clearly in 3D.
For Stiff Ropes: The "Safety Funnel" worked best here because the rope didn't move much, making the 3D map accurate.
For Long, Floppy Ropes: The "Safety Funnel" struggled because the rope would swing out of the camera's view. In these cases, the "Ghost Rope" (Visual Assistance) was better because it helped the user see the big picture.

The Bottom Line

The paper concludes that you can't just pick one "best" way to help a robot untangle a rope.

If the operator is new or the rope is stiff, a gentle physical nudge (the Funnel) helps the most.
If the operator is expert or the rope is long and floppy, better eyes (the Ghost Rope) are more helpful than physical nudges.

The system proves that the best robot assistant isn't a robot that takes over, but one that adapts its help based on who is holding the controller and what kind of rope they are fighting with.

Technical Summary: AssistDLO – Assistive Teleoperation for Deformable Linear Object Manipulation

Problem Statement

Manipulating Deformable Linear Objects (DLOs), such as ropes and cables, presents significant challenges in robotics due to their infinite-dimensional configuration space, complex nonlinear dynamics, and inherent underactuation. While fully autonomous solutions have advanced in specialized tasks, they often fail in unstructured environments due to unmodeled physical interactions and complex self-intersections. Consequently, teleoperation remains a critical approach for safe execution. However, standard teleoperation suffers from embodiment mismatches, degraded depth perception (especially with 2D feeds), and high cognitive loads required for bimanual coordination. Traditional assistive teleoperation methods, particularly those relying on simple linear blending of human and autonomous commands, often act as "geometric magnets" that ignore local DLO geometry. This can lead to unintended displacements of ungrasped rope segments, altering topology and worsening entanglement before a grasp is established. Furthermore, there is a lack of analysis regarding which assistance strategies are most effective for varying levels of operator expertise and different physical properties of DLOs (e.g., stiffness, length).

Methodology: The AssistDLO Framework

The authors propose AssistDLO, a bimanual assistive teleoperation framework designed to augment, rather than replace, the operator's capabilities. The system integrates real-time multi-view state estimation, visual assistance (VA), and a geometry-aware shared-autonomy controller based on Control Barrier Functions (SA-CBF).

1. Perception Pipeline

The framework utilizes a model-free visual perception pipeline driven by dual wrist-mounted RGB-D cameras ( $C_l, C_r$ ).

Trace Extraction: The system employs SAM 2 for instance segmentation, refined by HSV color thresholding to generate a robust 2D binary mask. A Voronoi-based skeletonization method extracts a sparse contour set, which is pruned to form a clean 1D skeletal graph representing the DLO.
Multi-View Fusion: 3D point sets from both cameras are transformed into a common task frame using robot forward kinematics. To resolve depth ambiguities and occlusions, the system fuses these point sets via a union operation, followed by voxel-grid downsampling to create a non-parametric, dense 3D state representation ( $R_t$ ) of the DLO.

2. Teleoperation Interface

Operators interact via a Virtual Reality (VR) environment using an HMD and hand controllers.

Visual Assistance (VA): The system renders the regularized 3D DLO point set ( $R_t$ ) directly into the operator's VR workspace. This provides stereoscopic depth cues and motion parallax, enhancing situational awareness without altering the robot's control commands.
Action Assistance: The system infers the operator's intent to select a target grasp point on the DLO. It then applies a shared-autonomy controller to guide the end-effector.

3. Shared Autonomy: SA-CBF vs. SA-LB

The core innovation is the SA-CBF controller, contrasted against a baseline SA-LB (Linear Blending).

Intent Estimation: A spatiotemporal hysteresis filter selects a target point on the DLO based on the operator's hand position, ensuring topological continuity and preventing erratic jumps between rope strands.
SA-LB (Baseline): Uses linear interpolation between the human input and the autonomous target. The paper notes this acts as a context-agnostic attractor that may intersect ungrasped rope, causing displacement.
SA-CBF (Proposed): Constructs a dynamic, geometry-aware "safety funnel" using Control Barrier Functions (CBF). A height-field barrier ( $z_b$ ) is defined over the workspace, allowing free movement above the table but restricting downward motion except through narrow funnels centered on valid rope points. This ensures the robot approaches the rope vertically without sweeping or displacing surrounding segments. The controller solves a Quadratic Program (QP) to find the optimal safe velocity that tracks the human's intent while satisfying the CBF constraint.

Experimental Evaluation

A user study was conducted with 22 participants (12 naive, 10 expert) performing a bimanual knot-untangling task. The study evaluated four control modes: Pure Teleoperation (PT), Visual Assistance (VA), SA-LB, and SA-CBF. Four distinct ropes with varying lengths and flexural rigidities (stiffness) were used to test robustness against physical variations.

Key Results

Overall Performance: SA-CBF achieved the highest overall success rate (85.23%), though the improvement over PT (76.14%) was not statistically significant at the population level. However, SA-LB performed slightly worse than PT (72.73%), highlighting the risks of context-blind linear blending.
Expertise-Dependent Effects:
- Naive Users: SA-CBF significantly improved success rates from 70.83% (PT) to 87.50%. It acted as a "skill equalizer," drastically reducing task completion time and physical demand.
- Expert Users: Experts maintained high performance (>80%) across most conditions without assistance. They preferred Visual Assistance (VA) over action assistance, reporting a significant loss of authority and increased conflict when using SA-CBF or SA-LB.
DLO Property-Dependent Effects:
- Stiff Ropes: SA-CBF was most effective for stiffer ropes (Red and Orange), where precise grasping is difficult due to tension and friction. It achieved a 100% success rate for naive users with the stiffest rope.
- Compliant/Long Ropes: For long, highly compliant ropes (Blue), action assistance showed limited benefit due to the rope frequently moving out of the wrist-camera field of view. In these cases, VA was more effective, providing the necessary global visual context.
Subjective Metrics: Naive users rated SA-CBF as highly helpful and safe. Experts rated VA highest for understandability and control, while finding physical interventions frustrating.

Significance and Contributions

The paper claims that effective DLO teleoperation cannot rely on a fixed assistance strategy. The primary contributions are:

A Novel Framework: AssistDLO integrates multi-view state estimation with a CBF-based shared autonomy controller that preserves operator authority while preventing geometric violations.
Context-Aware Control: The SA-CBF method demonstrates that geometry-aware constraints (funnels) are superior to simple attractors (magnets) for DLOs, preventing unintended topological changes.
Empirical Insights: The study provides evidence-based design guidelines showing that assistance must be adaptive:
- Action assistance (SA-CBF) is critical for naive users and stiffer objects, acting as a safety net and skill equalizer.
- Visual assistance (VA) is preferred by expert users and is more effective for long, compliant objects where local perception is insufficient.
Limitations and Future Work: The authors modestly acknowledge that the system assumes a single distinguishable DLO and is limited by the field of view of wrist cameras. They suggest future work should focus on adaptive arbitration between assistance modes based on real-time estimates of operator proficiency and object behavior.

In conclusion, the paper argues that shared autonomy for deformable objects must be user-aware and material-aware, moving beyond static blending strategies to dynamic, context-sensitive interventions.