⚡ electrical engineering

A Distributed Stereo Visual Feedback Framework for VR- based Telesurgical Robotic Teleoperation: Design and Latency Characterization

This paper presents and characterizes a distributed stereo visual feedback framework for VR-based telesurgical teleoperation, revealing that while network transmission delay is minimal, the majority of the total system latency (approximately 78 ms) stems from image processing and display rendering stages.

Original authors: Haniyeh Altafi, Rayan Shakeel, Kourosh Zareinia

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Haniyeh Altafi, Rayan Shakeel, Kourosh Zareinia

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to play a video game where you control a robot arm in a room across the world. You move your hand, and the robot moves, but you can only see what the robot sees through a screen. If the picture on the screen is even a tiny bit late, your brain gets confused. It's like trying to catch a ball while wearing glasses that show the ball where it was a second ago, not where it is now. This is the big problem in "teleoperation," which is just a fancy word for controlling a robot from far away.

To make this work well, especially for delicate tasks like surgery, you need two things: a clear, 3D picture (so you can judge depth, like how far away a needle is) and speed (so the picture updates instantly). Scientists call this "stereo visual feedback" and "low latency." If the delay is too long, the operator feels clumsy, the robot might crash, or in a medical setting, a mistake could happen. The big question researchers ask is: "Where does the time go?" Is it the internet connection slowing things down, or is it the computer taking too long to process the picture?

This paper dives into that mystery by building a special "time machine" for robot vision. The researchers created a system where a robot on one side of the room sends a 3D video feed to a person wearing a Virtual Reality (VR) headset on the other side. They didn't just measure the total time; they broke the journey down into three distinct legs, like timing a relay race to see which runner is the slowest. They found that while the internet connection was incredibly fast—like a bullet train—the computer processing the video and the VR headset showing it were the real bottlenecks, acting more like a slow-moving snail.

The Robot's 3D Eyes and the VR Headset

The researchers set up a system that acts like a remote pair of eyes. On the "follower" side (where the robot is), there is a special stereo camera that takes two pictures at once, just like human eyes do, to create a 3D effect. These pictures are sent over a network to a "master" side, where a human operator is waiting. The human wears a Meta Quest 3 VR headset, which is like a high-tech window that lets them feel like they are standing right next to the robot.

The journey of the video starts at the camera, travels through a middle computer (which acts like a post office sorting mail), arrives at the main computer, gets processed by a game engine called Unity, and finally pops up inside the VR headset. The team wanted to know exactly how long each step of this journey took.

The Relay Race of Time

To figure out where the time was hiding, the team split the video's journey into three stages, like a relay race with three runners:

The Network Runner: This is the time it takes for the video to travel over the wires.
The Unity Runner: This is the time the computer takes to unpack the video and get it ready for the VR headset.
The VR Runner: This is the time the headset takes to actually show the picture to your eyes.

They measured the Network Runner first. They sent 10,000 frames of video and found it was blazing fast. The average time was just 5.59 ± 1.04 ms. That is incredibly quick! They even tested if the middle "post office" computer slowed things down, and it added less than 1 millisecond. This proved that the internet connection itself was not the problem; it was the fastest part of the whole system.

However, when they looked at the Unity Runner, the race slowed down significantly. The computer took an average of 40.06 ± 10.12 ms just to process the video and get it ready. This was the biggest chunk of the delay. The final VR Runner (the headset) took another 32.30 ± 0.09 ms to display the image.

When you add all three runners together, the total time from the moment the camera sees something to the moment the human sees it in the VR headset was 77.95 ± 10.17 ms.

The Big Surprise: It's Not the Internet!

The most important discovery in this paper is a bit of a plot twist. Many people assume that if you want a faster robot, you just need a faster internet connection. But this study showed that even with a super-fast network (the 5.59 ms part), the total delay was still nearly 78 milliseconds. Why? Because the computer processing the video and the headset showing it were taking up most of the time.

It's like having a Ferrari (the internet) stuck in a traffic jam caused by a slow toll booth (the computer processing) and a slow driver (the headset rendering). No matter how fast the car is, the trip takes a long time because of the stops. The researchers found that the network was only responsible for a tiny fraction of the delay. The real "traffic jam" was happening inside the software and the display hardware.

Does It Actually Work?

To see if this system was actually useful, the researchers asked four people to play a game. The game was a "pick-and-place" task: using the robot to move small blocks from one spot to another. They tried doing it with the VR 3D view and without it (using a normal 2D screen).

The results were promising. When the participants used the VR system, they finished the task faster (averaging 373.7 seconds with VR vs. 527.2 seconds without). They also moved the robot in shorter, more direct paths and dropped fewer blocks. Every single participant said the VR version gave them a better sense of depth and felt more convenient.

However, the authors are careful to say this was just a "preliminary" test with only four people. It suggests the system works well, but it's not a final proof that it will work for everyone in every situation. They need to test it with more people and in more difficult conditions before declaring it a perfect solution.

The Takeaway

This paper builds a solid foundation for future robot surgery and remote work. It proves that you can send high-quality 3D video to a VR headset with very little delay from the internet. But it also sends a clear message to engineers: if you want to make these systems faster, don't just buy a faster internet cable. You need to fix the software that processes the video and the hardware that displays it. The internet is ready; now the computers and headsets need to catch up.

Technical Summary: A Distributed Stereo Visual Feedback Framework for VR-based Telesurgical Robotic Teleoperation

Problem Statement

In robot-assisted telesurgery and high-precision teleoperation, effective depth perception and hand-eye coordination rely heavily on low-latency stereo visual feedback. While Virtual Reality (VR) interfaces offer immersive visualization that can improve situational awareness, the total end-to-end latency in these systems is often a critical bottleneck. Previous research has highlighted that visual delays can degrade task performance, increase operator workload, and compromise safety. However, many existing studies focus primarily on network transmission delays or general interaction interfaces, failing to characterize the full "camera-to-headset" visual feedback pipeline. Specifically, there is a lack of detailed decomposition of latency sources—distinguishing between network transmission, software processing (e.g., game engine rendering), and display rendering—which is essential for identifying true bottlenecks in VR-based telesurgical systems.

Methodology

The authors developed and evaluated a distributed stereo visual feedback pipeline designed for VR-based robotic teleoperation. The system architecture consists of five primary components:

Stereo Camera: A GXIVISION 3D Stereo VR USB Camera Module capturing synchronized side-by-side images (2560 × 720 pixels) at 30 fps.
Sender-Side Computer: Runs Windows 11 with an NVIDIA RTX 3060 Ti. It encodes video using H.264 via GStreamer with hardware acceleration (NVENC) and transmits the stream via UDP/RTP.
Intermediate Network Computer: An Ubuntu-based bridge used for UDP packet forwarding. In this study, impairment functions (delay, jitter, packet loss) were disabled to establish a baseline, though the setup allows for future network simulation.
Master-Side Computer: Runs Windows 11 with an NVIDIA RTX 4060 Ti. It decodes the H.264 stream (using NVDEC), processes the side-by-side image in Unity (separating left/right views via custom shaders), and prepares the frame for display.
VR Headset: A Meta Quest 3 connected via Meta Quest Link/OpenXR to display the immersive stereo view.

Latency Characterization Approach:
The study decomposed the total latency ( $L_{total}$ ) into three distinct stages:

$L_{network}$ : Transmission and decoding time (measured via GStreamer latency tracer).
$L_{Unity}$ : Frame update and processing time within the Unity engine (measured via Time.realtimeSinceStartup).
$L_{VR}$ : Rendering and display latency from Unity submission to the headset.

The authors compared multiple implementation strategies, including a baseline Python socket approach versus an optimized GStreamer pipeline, and CPU-based encoding versus GPU-accelerated (NVENC/NVDEC) encoding. Additionally, a preliminary pick-and-place task was conducted with four participants to compare VR-based stereo visualization against a non-VR condition, measuring task completion time, path length, and dropped pieces.

Key Contributions

Pipeline Implementation: The creation of a distributed stereo visual feedback system that captures, transmits, processes, and displays synchronized stereo video in a Meta Quest 3 headset, specifically tailored for future telesurgical applications.
Latency Decomposition: A novel characterization method that isolates network transmission, Unity processing, and VR rendering delays. This approach moves beyond simple computer-to-computer latency measurements to identify the specific stages contributing to end-to-end delay.
Optimization Validation: Demonstration that hardware-accelerated encoding/decoding (GPU) and structured media frameworks (GStreamer) significantly outperform basic Python socket implementations in terms of latency, jitter, and frame stability.
Empirical Baseline: Provision of quantitative latency data for a VR-based teleoperation setup, establishing a reference for future studies on stereo perception and operator performance.

Experimental Results

Network Transmission: The optimized GStreamer pipeline achieved an average network latency of 5.59 ± 1.04 ms with 0.45 ms jitter and zero frame loss over 10,000 frames. The intermediate bridge computer introduced negligible additional delay (<1 ms) compared to a direct connection.
Optimization Impact: Switching from a Python socket baseline to the GStreamer pipeline reduced average latency by approximately 44.7% (from 10.11 ms to 5.59 ms) and eliminated packet loss. Furthermore, GPU-accelerated encoding/decoding reduced mean pipeline latency by 38.1% and significantly improved frame timing stability (reducing inter-frame standard deviation from ~44 ms to ~8 ms) compared to CPU-based processing.
End-to-End Latency: The total camera-to-display latency was measured at 77.95 ± 10.17 ms. The breakdown revealed that network transmission contributed only a small fraction (~5.6 ms). The dominant sources of latency were:
- Unity frame update: ~40.06 ms
- VR display rendering: ~32.30 ms
Preliminary Task Evaluation: In a pick-and-place task with four participants, the VR condition resulted in faster completion times (mean 373.7 s vs. 527.2 s without VR), shorter Cartesian path lengths, and fewer dropped pieces (mean 1.25 vs. 4.00). All participants subjectively reported better depth perception and preferred the VR condition.

Significance and Claims

The paper claims that its primary significance lies in demonstrating that network transmission is not the dominant source of latency in VR-based teleoperation systems. Instead, the visualization and display stages (Unity processing and VR rendering) introduce the majority of the delay.

Consequently, the authors argue that optimizing network protocols alone is insufficient for achieving low-latency telesurgical feedback. Future system design must prioritize the optimization of the visualization pipeline, including camera frame rates, texture upload overhead in game engines, and VR rendering paths.

The study positions its proposed pipeline as a practical foundation for future research into stereo perception, visual delay, and operator performance in VR-based telesurgery. The authors explicitly state that the preliminary task results are descriptive observations due to the small sample size and that the system has not yet been integrated with haptic feedback or physical robotic control loops. The work serves as a necessary step toward characterizing the full visual feedback loop before more complex, integrated teleoperation studies can be conducted.