💻 computer science

A 3DGS-Driven Dynamic Viewpoint and Vibrotactile Framework for Subsea Teleoperation Validated via fNIRS

This paper presents a multimodal teleoperation framework combining a 3DGS-driven dynamic viewpoint system and vibrotactile feedback, which a human-subject study validated as significantly enhancing operator performance and cognitive endurance under subsea communication latency by providing occlusion-free exocentric visualization and intuitive haptic cues.

Original authors: Fang Xu, Tianyu Zhou, Ruitong Tian, Md Jahidul Islam, Jing Du

Published 2026-07-16

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Fang Xu, Tianyu Zhou, Ruitong Tian, Md Jahidul Islam, Jing Du

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to drive a car through a maze made of giant, foggy pipes, but you can only see a tiny, shaky square of the road directly in front of your bumper. Now, imagine that every time you turn the steering wheel, the car doesn't move for a full second. That is the nightmare scenario for operators of underwater robots, known as ROVs (Remotely Operated Vehicles). These machines are the divers of the deep, sent to inspect sunken ships, oil rigs, and flooded tunnels where humans can't go. But controlling them is incredibly hard. The water is often murky, the spaces are tight, and the signal from the surface takes a long time to travel. This creates a "perceptual bottleneck": the human brain has to guess where the robot is and what's around it, leading to mental exhaustion and mistakes. Scientists have long tried to fix this by giving operators better cameras or vibration suits, but they haven't been sure which method actually keeps the human brain from crashing under the pressure.

This paper dives into that exact problem, testing a new way to help humans control these underwater robots. The researchers built a system that combines two tricks: a "magic eye" that automatically finds the best angle to see the robot, and a "vibrating vest" that buzzes when the robot gets too close to a wall. They tested this setup in a simulated underwater maze with 30 volunteers, introducing delays in the control signal ranging from zero seconds up to a full second. They didn't just watch how well the robots did; they also strapped the volunteers into a special helmet that measured blood flow in their brains to see how hard their minds were working.

The results were surprising. When the signal was fast, the vibrating vest was great at keeping the robot on a straight path, like a gentle nudge from a friend. But when the signal was slow (0.5 to 1.0 seconds), the "magic eye" system won hands down. This system used a cutting-edge technology called 3D Gaussian Splatting to build a real-time, 3D model of the underwater world and then automatically moved a virtual camera to show the operator a clear, third-person view of the robot, even if the robot was hiding behind a pipe. The study found that this proactive view prevented the operator's brain from "checking out." In fact, when the delay was high, operators using the standard camera view showed signs of cognitive disengagement—their brains essentially gave up trying to keep track of the space. The new system, however, kept their brains actively engaged and in control, allowing them to navigate safely even when the signal was lagging.

The Problem: Driving Blind in a Foggy Pipe

Controlling a robot underwater is like trying to thread a needle while wearing boxing gloves and looking through a straw. The operator sits on a ship or a dock, looking at a screen that shows what the robot's camera sees. But that camera is stuck on the robot's nose. If the robot is in a narrow tunnel, the operator can't see the walls until they are right next to them. To make it worse, sound waves travel slowly through water, so there is often a delay between when the operator pushes a button and when the robot actually moves. If the delay is 1.0 second, the operator is essentially driving a car where the steering wheel and the brakes are disconnected by a full second.

In these situations, the human brain has to do a lot of heavy lifting. It has to imagine the 3D shape of the tunnel, guess where the robot is, and predict where it will be in a second. This is called "mental rotation," and it is exhausting. When the delay gets too long, the brain can get overwhelmed. The researchers suspected that if they could offload some of this mental work to a computer, the operator could stay calm and focused longer.

The Solution: A Magic Camera and a Buzzing Vest

The team created a new system to help the operator, built on a framework that connects a robot control system (ROS) with a game engine (Unity). They split the job into two parts: one for planning ahead and one for reacting quickly.

1. The Magic Camera (DAVS):
Instead of just showing the robot's nose-cam view, the system builds a live, 3D map of the underwater world using a technology called 3D Gaussian Splatting. Think of this as a cloud of millions of tiny, colorful, glowing dots that represent the walls, pipes, and the robot itself. Because the computer has this 3D map, it can instantly generate a new view from any angle. The system uses a "Dynamic Adaptive Viewpoint System" (DAVS) to constantly calculate the best place to put a virtual camera. It looks for a spot where the robot is visible and the path ahead is clear, then it renders that view for the operator. It's like having a drone that automatically flies around the robot to show you the best angle, but it happens instantly on your screen.

2. The Buzzing Vest (Haptics):
While the magic camera helps you plan your route, the second part helps you react. The operator wears a vest with small motors (vibrators) on their torso. If the robot gets too close to a wall on the left, the left side of the vest buzzes. If it's close on the right, the right side buzzes. This gives the operator a "sixth sense" for immediate danger without them having to look away from the screen.

The Experiment: Testing the Brain and the Robot

To see if this new system actually worked, the researchers set up a controlled experiment. They invited 30 people to play a video game where they had to pilot a simulated BlueROV2 through a complex, flooded industrial facility filled with pipes and narrow gaps.

They tested three different ways of controlling the robot:

Egocentric (The Baseline): Just the standard camera view from the robot's nose.
Haptic (The Reactive): The standard view plus the buzzing vest.
Exocentric (The Proactive): The magic 3D camera view (DAVS) without the buzzing vest.

They also introduced four different levels of "lag" or delay in the control signal: 0.0 seconds (instant), 0.2 seconds, 0.5 seconds, and a full 1.0 second.

While the participants played, they wore a special cap that used light to measure blood flow in their prefrontal cortex—the part of the brain responsible for planning, decision-making, and working memory. This allowed the researchers to see exactly how hard the brain was working, rather than just asking the participants how tired they felt.

The Findings: When the Brain Gives Up

The results revealed a clear pattern depending on how fast the signal was.

When the signal was fast (0.0 to 0.2 seconds):
The buzzing vest was the winner. Because the robot moved almost instantly, the operators could use the vibrations to make tiny, precise adjustments. They stayed on the perfect path better than with the other methods. The magic camera didn't offer much advantage here because the standard view was already good enough.

When the signal was slow (0.5 to 1.0 seconds):
The magic camera took over. At a 0.5-second delay, the standard view started to fail. Operators made more mistakes and crashed more often. But the most interesting finding came from the brain scans. When using the standard view with a 0.5-second delay, the blood flow in the prefrontal cortex dropped. The researchers call this "cognitive disengagement." It means the delay was so confusing that the operators' brains stopped trying to build a mental map of the space; they essentially gave up, leading to poor performance.

In contrast, the operators using the magic 3D camera kept their brains fully active. The clear, third-person view helped them maintain their mental map of the environment, even with the delay. Their brains showed high activity in the planning centers, and they navigated the maze much faster and safer than the others. At the extreme 1.0-second delay, the magic camera was the only system that allowed operators to complete the task significantly faster and with fewer collisions than the other two methods.

What This Means

The study suggests that for underwater robots, the type of help you need changes depending on how fast the connection is. If the connection is instant, simple vibrations are great for keeping the robot on track. But if the connection is slow and laggy, you need a system that gives you a clear, global picture of the world.

The most important discovery is that a good interface doesn't just make the robot move better; it keeps the human operator's brain from shutting down. By using the 3D magic camera to handle the hard work of figuring out "where am I?" and "what's around me?", the system frees up the operator's brain to focus on the actual task. This could be a game-changer for inspecting underwater infrastructure, where delays are common and mistakes can be costly. While the system was tested in a simulation and not yet in the real ocean, the results show a promising path toward making underwater robots easier and safer for humans to control.

Technical Summary: A 3DGS-Driven Dynamic Viewpoint and Vibrotactile Framework for Subsea Teleoperation Validated via fNIRS

Problem Statement
Teleoperating Remotely Operated Vehicles (ROVs) in flooded, highly cluttered infrastructure presents a severe perceptual bottleneck. Operators must infer global spatial clearance and complex geometry from narrow, 2D egocentric camera views, a process that demands heavy mental rotation and spatial updating. This cognitive burden is exacerbated by subsea communication latency, which creates a temporal disconnect between operator input and visual feedback. In low-texture, turbid underwater environments, standard autonomy often fails, necessitating human-in-the-loop control. However, existing teleoperation interfaces often rely on static visual augmentations or single sensory modalities, which can suffer from structural occlusion or fail to distribute the cognitive load effectively. Furthermore, prior evaluations rarely utilize objective physiological metrics to determine if these aids genuinely reduce cognitive demand or merely shift it to different mental processes.

Methodology
The authors propose a multimodal teleoperation architecture built on a ROS-Unity framework that decouples proactive spatial planning from reactive boundary avoidance. The system integrates two primary components:

Dynamic Adaptive Viewpoint System (DAVS) with 3D Gaussian Splatting (3DGS):
- Real-time Reconstruction: The system utilizes Monocular Gaussian Splatting SLAM (MonoGS) to ingest onboard RGB streams, continuously estimating the ROV's 6-DoF pose and reconstructing the environment as a volumetric 3DGS map without offline pre-training.
- Viewpoint Optimization: A DAVS optimizer evaluates candidate viewpoints on a sphere surrounding the vehicle. It minimizes a multi-objective cost function balancing occlusion (ray-cylinder intersection tests), spatial entropy (information richness), temporal smoothness, and distance constraints.
- Rendering: The system synthesizes a photorealistic, occlusion-free, exocentric (third-person) perspective at 10 Hz by rendering the active 3DGS map from the optimized virtual camera pose.
Reactive Vibrotactile Feedback:
- A torso-mounted vibrotactile suit provides haptic proximity cues.
- The system discretizes the ROV's collision geometry into sectors and computes minimal Euclidean clearance to the surrounding mesh.
- A non-linear activation function maps this clearance to vibrotactile intensity, providing immediate, localized safety warnings without requiring the operator to avert their gaze.

Experimental Design
The efficacy of the architecture was validated through a controlled human-subject study ( $N=30$ ) using a BlueROV2 in a simulated, complex underwater facility. The study employed a $3 \times 4$ repeated-measures design:

Modalities: Egocentric (baseline 2D feed), Haptic (Egocentric + vibrotactile cues), and Exocentric (3DGS-DAVS third-person view). A combined Exocentric + Haptic condition was excluded to isolate the neural correlates of proactive vs. reactive offloading.
Delays: Four communication latency levels (0.0 s, 0.2 s, 0.5 s, and 1.0 s) were injected to simulate subsea acoustic link conditions.
Metrics:
- Behavioral: Task completion time, average speed, collision rate, and trajectory deviation (RMSE).
- Neurophysiological: Functional Near-Infrared Spectroscopy (fNIRS) was used to measure hemodynamic responses ( $\Delta HbO$ ) in the Anterior Prefrontal Cortex (APFC) and Dorsolateral Prefrontal Cortex (DLPFC), regions associated with executive control and working memory.

Key Results

Behavioral Performance:
- At minimal delays (0.0–0.2 s), all modalities performed similarly, though Haptic feedback yielded the lowest trajectory deviation, indicating effective path adherence when telemetry is instantaneous.
- At severe delays (0.5–1.0 s), the Exocentric (3DGS-DAVS) modality significantly outperformed both Egocentric and Haptic modes in completion time and collision avoidance. Specifically, at 1.0 s delay, Exocentric users completed tasks faster and with fewer collisions than the other groups.
- Haptic feedback alone improved path adherence at low delays but failed to prevent performance degradation at high latencies, as the cognitive burden of temporal delay overwhelmed the reactive cues.
Neurophysiological Findings (fNIRS):
- Cognitive Disengagement: At the critical 0.5 s delay, standard Egocentric teleoperation resulted in a significant drop in prefrontal activation (specifically in the APFC and RDLPFC). The authors interpret this as "cognitive disengagement," where the perceptual bottleneck overwhelmed working memory, causing the operator's executive control to collapse.
- Sustained Executive Control: Conversely, the Exocentric 3DGS-DAVS modality maintained significantly higher prefrontal activation at 0.5 s delay while simultaneously delivering superior behavioral performance. This suggests the system successfully sustained the operator's executive function, allowing them to actively parse delayed telemetry rather than disengaging.
- At 1.0 s delay, modality-specific differences in prefrontal activation vanished, suggesting a universal cognitive threshold where latency overwhelms human executive function regardless of the interface.

Significance and Contributions
The paper makes three primary contributions:

A 3DGS-Driven Multimodal Architecture: It presents a novel ROS-Unity framework that couples real-time 3D Gaussian Splatting with a dynamic viewpoint system to prevent environmental occlusion, integrated with reactive haptic cues.
Evaluation of Latency Resilience: It isolates the distinct impacts of proactive visual spatial offloading versus reactive somatosensory boundary avoidance, demonstrating that global geometric context (Exocentric) is more critical than localized alerts (Haptic) under severe latency.
Neurophysiological Discovery: It provides objective fNIRS evidence of a "cognitive disengagement" phenomenon in standard teleoperation at critical delays. The study demonstrates that spatially grounded assistance (3DGS-DAVS) preserves operator endurance and executive control, preventing the collapse of working memory that occurs with traditional interfaces.

The authors conclude that while pure autonomy remains brittle in degraded underwater environments, this multimodal architecture offers a viable path to extend mission duration and operator stamina by computationally offloading the burden of 3D spatial updating, thereby optimizing the human-in-the-loop for high-latency subsea operations.

The Problem: Driving Blind in a Foggy Pipe

The Solution: A Magic Camera and a Buzzing Vest

The Experiment: Testing the Brain and the Robot

The Findings: When the Brain Gives Up

What This Means

More like this