eStonefish-Scenes: A Sim-to-Real Validated and Robot-Centric Event-based Optical Flow Dataset for Underwater Vehicles

This paper introduces eStonefish-Scenes, a synthetic event-based optical flow dataset for underwater vehicles generated via the Stonefish simulator, along with the eWiz processing library, and validates its sim-to-real transferability by demonstrating that a model trained exclusively on this synthetic data achieves high-accuracy optical flow estimation on real-world underwater sequences without fine-tuning.

The Gist

The Big Problem: Teaching Robots to "See" Underwater

Imagine you are trying to teach a robot submarine how to swim through a coral reef without crashing into anything. To do this, the robot needs to understand how things are moving around it (a concept called optical flow).

Usually, robots use standard cameras that take pictures like a human does—snapshots of the world. But underwater is tricky: the water is murky, the light changes fast, and things move quickly. Standard cameras get blurry and confused in these conditions.

Enter Event Cameras. Think of these not as cameras, but as super-fast, hyper-sensitive eyes. Instead of taking full pictures, they only "blink" when something changes (like a fish swimming by or a shadow passing). They are incredibly fast, don't get blurry, and work great in the dark.

The Catch: To teach a robot how to use these "blink-eyes," you need a massive library of practice data. But getting real underwater data is a nightmare. It's expensive, dangerous, and you can't easily know exactly where the robot is or how fast it's moving to check if it's doing the right thing. It's like trying to learn to drive a car in a blizzard without ever seeing the road or having a GPS.

The Solution: A "Video Game" for Robots

The authors of this paper decided to build a perfect video game to solve this problem. They created a synthetic dataset called eStonefish-Scenes.

Think of this dataset as a high-tech flight simulator for underwater robots. Instead of going to the ocean, they built a digital ocean inside a computer using a simulator called Stonefish.

Here is what makes their "game" special:

1. The World: They didn't just make a flat floor. They built a vibrant, 3D coral reef with thousands of different corals and plants.
2. The Wildlife: They added schools of fish that swim naturally. They used a "Boids" algorithm (think of it as a digital version of how birds flock) so the fish move together, avoid obstacles, and react to the robot, just like real fish.
3. The Robot: They simulated a real robot submarine (BlueROV2) equipped with the special "blink-eye" camera.
4. The Secret Sauce: Because this is a computer simulation, they know the perfect truth. They know exactly how fast the robot is moving and exactly how the water is flowing. This gives them the "answer key" that is impossible to get in the real ocean.

The Toolkit: "eWiz"

Building a dataset is hard, but using it should be easy. The authors also built a software toolbox called eWiz.

• Analogy: If the dataset is a giant warehouse of raw ingredients, eWiz is the kitchen, the recipe book, and the chef's knife all in one. It helps researchers load the data, chop it up (augment it), cook it (train their AI), and taste-test it (evaluate the results). It handles all the messy technical details so scientists can focus on the cooking.

The Big Test: Does the Game Teach Real Skills?

The biggest question is: If a robot learns in a video game, can it actually drive a real car?

To answer this, the team did a "Sim-to-Real" test:

1. Training: They taught a neural network (an AI brain) to predict motion using only the fake data from their video game. They never showed it a single real underwater photo.
2. The Real World Test: They took a real robot submarine, put a real "blink-eye" camera on it, and drove it around a swimming pool in a lab.
3. The Ground Truth: To know if the robot was right, they put a giant, high-resolution poster of a coral reef on the bottom of the pool. They used math to calculate exactly how the robot moved relative to that poster.
4. The Twist (Uncertainty): Real life is messy. Sometimes the poster is hard to see, or the water is cloudy. The team developed a special way to measure confidence. Imagine the robot saying, "I'm 99% sure I'm moving left here, but I'm only 50% sure about that blurry spot over there." They used this "confidence score" to judge the robot's performance fairly.

The Results: A Home Run

The results were impressive. The AI, which had never seen a real underwater scene, was able to navigate the real pool with high accuracy.

• It successfully predicted how the robot was moving.
• It handled the "blurry" and uncertain parts of the real world gracefully.
• It proved that you don't need to spend millions of dollars and risk expensive equipment to train underwater robots. You can train them in a digital coral reef first.

Summary

In short, this paper is about building a realistic underwater video game to teach robots how to swim.

• The Problem: Real underwater data is too hard to get.
• The Fix: A synthetic dataset (eStonefish-Scenes) with realistic fish and coral, plus a software toolkit (eWiz) to make it easy to use.
• The Proof: They trained an AI on the fake data, tested it in a real pool, and it worked almost perfectly.

This is a huge step forward because it means we can build smarter, safer underwater robots without needing to dive into the deep ocean just to collect practice data.

Technical Summary

1. Problem Statement

Event-based cameras (EBCs), or Dynamic Vision Sensors (DVS), offer significant advantages for underwater robotics, including high temporal resolution, low latency, and immunity to motion blur and low-light conditions. However, the development of robust optical flow algorithms for underwater EBCs is severely hindered by a critical lack of labelled datasets.

• Data Scarcity: Real-world underwater event-based datasets are rare, expensive to collect, and difficult to annotate with ground truth due to water turbidity, light scattering, and the complexity of obtaining precise motion data.
• Sim-to-Real Gap: Existing synthetic datasets often fail to capture the specific hydrodynamic coupling between underwater vehicles and sensors, or they rely on post-processing frame-based videos (e.g., video-to-event conversion) rather than native simulation, leading to domain shifts.
• Evaluation Limitations: Current benchmarks often lack uncertainty-aware evaluation metrics, treating all pixels equally despite the varying reliability of ground truth in real-world scenarios.

2. Methodology

The authors propose a comprehensive framework comprising a synthetic dataset generation pipeline, a data processing library, and a rigorous real-world validation protocol.

A. Synthetic Dataset Generation (eStonefish-Scenes)

The dataset is generated using the Stonefish simulator, a physics-based underwater robotics simulator.

• Environment Generation:
  • Stonefish-SceneGen: A tool to procedurally generate diverse underwater environments featuring rocky seabeds and vibrant coral reefs with varied poses and sizes.
  • Stonefish-Boids: A package implementing Boids algorithms to simulate realistic, reactive schools of fish that interact with obstacles and the ROV, adding dynamic complexity.
• Sensor & Vehicle Configuration:
  • Simulates a BlueROV2 equipped with a DAVIS346 event camera (346×260 resolution, 70° FOV).
  • Captures data in two configurations: Forward-looking (for obstacle avoidance/inspection) and Down-looking (for odometry).
  • Includes both static scenes (static corals) and dynamic scenes (moving fish schools).
• Data Output: Generates synchronized streams of events, grayscale images, and dense ground-truth optical flow derived directly from the 3D environment simulation.

B. Data Processing Library (eWiz)

To facilitate research, the authors developed eWiz, a Python library for event-based data handling.

• Storage: Uses an optimized HDF5 format with Blosc compression and pre-computed timestamp mappings to handle high-throughput event data efficiently.
• Functionality: Supports data loading, augmentation (time warping, noise injection), encoding (Gaussian, event counting), visualization, and training utilities.
• Algorithms: Includes implementations of contrast maximization, various loss functions (photometric, smoothness), and evaluation metrics.

C. Real-World Validation Protocol

To validate the sim-to-real transfer, a dedicated real-world dataset was collected using a BlueROV2 with a DAVIS346 camera in an indoor pool at CIRS (Girona).

• Ground Truth Acquisition: A high-resolution textured poster on the pool floor served as a planar reference. Ground-truth optical flow was computed via homography-based registration between consecutive frames and the poster.
• Uncertainty Estimation: A Monte Carlo perturbation method was applied to keypoint correspondences to estimate per-pixel uncertainty (covariance maps). This allows for uncertainty-weighted evaluation, where unreliable regions (e.g., low texture) contribute less to the error metric.

D. Network Architecture & Training

• Model: A ConvGRU-based recurrent neural network (inspired by Hagenaars et al.) designed for event-based optical flow.
• Training Strategy: The network was trained exclusively on the synthetic eStonefish-Scenes dataset using self-supervised photometric and smoothness losses.
• Evaluation: Tested on the real-world dataset without fine-tuning to assess pure sim-to-real generalization.

3. Key Contributions

1. eStonefish-Scenes Dataset: The first event-based optical flow dataset specifically tailored for underwater environments, featuring native DAVIS346 resolution and robot-centric dynamics.
2. Open Data Pipeline: Release of Stonefish-SceneGen and Stonefish-Boids, enabling the community to easily generate custom underwater scenes with realistic flora and fauna.
3. eWiz Library: A comprehensive, open-source framework for event-based data processing, storage, and evaluation, bridging the gap between raw data and deep learning pipelines.
4. Uncertainty-Aware Evaluation: Introduction of a novel evaluation framework that incorporates per-pixel uncertainty into performance metrics (AEE/AAE), providing a more reliable assessment for real-world data where ground truth is imperfect.
5. Sim-to-Real Validation: Demonstration that a model trained solely on synthetic data can achieve high performance on real underwater sequences without fine-tuning.

4. Results

The study evaluated a ConvGRU network trained on 10 synthetic sequences (approx. 6,400 samples) and tested on real-world data.

• Synthetic Performance: On the held-out synthetic test set, the model achieved an Average Endpoint Error (AEE) of 0.636 pixels and an Average Angular Error (AAE) of 0.173 radians.
• Real-World Performance: Without any fine-tuning, the model achieved an uncertainty-weighted AEE of 0.79 pixels and an AAE of 0.2 radians on the real-world dataset.
• Qualitative Analysis: Visual results showed strong agreement between predicted and ground-truth flow in regions with sufficient texture and event activity. Discrepancies were primarily observed in textureless areas or regions with high uncertainty, validating the effectiveness of the uncertainty-weighted metrics.
• Significance of Results: The small performance gap between synthetic and real data (especially when accounting for uncertainty) confirms that the proposed simulation accurately captures the essential dynamics of underwater event-based vision.

5. Significance

This work addresses a major bottleneck in underwater robotics research by providing a scalable, high-fidelity alternative to costly real-world data collection.

• Accelerated Research: By providing a ready-to-use dataset and processing tools, it lowers the barrier to entry for developing event-based vision algorithms for Autonomous Underwater Vehicles (AUVs).
• Robustness: The demonstration of successful sim-to-real transfer suggests that synthetic data can replace or significantly reduce the need for large-scale real-world annotation campaigns.
• Methodological Advancement: The integration of uncertainty estimation into evaluation metrics sets a new standard for assessing optical flow in challenging, real-world environments where ground truth is inherently noisy.
• Future Impact: The framework supports the transition from frame-based to event-based processing in underwater applications, enabling faster, more energy-efficient, and robust navigation systems for AUVs.