SLAM&Render: A Benchmark for the Intersection Between Neural Rendering, Gaussian Splatting and SLAM

1. Problem Statement

The field of robotics and computer vision is witnessing a convergence between Simultaneous Localization and Mapping (SLAM) and Neural Rendering (specifically Neural Radiance Fields - NeRFs, and Gaussian Splatting). While these technologies are increasingly being combined to create high-fidelity 3D maps for autonomous robots, existing benchmarks fail to address the specific challenges of this intersection:

• Lack of Sequential Operations: Most Neural Rendering datasets (e.g., DTU, Mip-NeRF360) are designed for offline Novel View Synthesis (NVS) with static camera trajectories, lacking the sequential, real-time constraints of SLAM.
• Sensor Motion Inaccuracy: Many datasets are collected via handheld devices, drones, or mobile robots, making it difficult to reproduce exact sensor motions for rigorous algorithm comparison.
• Missing Multimodal Data: Existing datasets often lack synchronized robot kinematic data (joint positions, flange poses), which is crucial for integrating SLAM with robotic control systems.
• Environmental Variability: Few datasets offer controlled variations in lighting and object rearrangement (occlusions), which are critical for testing the robustness of neural rendering methods against real-world illumination shifts.

2. Methodology and Dataset Design

The authors introduce SLAM&Render, a fully controlled benchmark recorded using a KUKA LBR iisy 3 R760 robotic manipulator equipped with an Intel RealSense D435i camera in an eye-to-hand configuration.

Key Data Acquisition Features:

• Scale: 40 static sequences across 5 distinct scene setups.
• Sensors: Time-synchronized RGB-D images (848×480 @ 30Hz), IMU data (210Hz), and robot kinematic data (joint angles @ 25Hz).
• Ground Truth: High-precision camera poses obtained via an OptiTrack Motion Capture System (MCS) with 12 cameras, achieving a mean calibration error of 0.4 mm.
• Experimental Conditions:
  • Lighting: Four controlled conditions: Natural, Cold Artificial, Warm Artificial, and Dark.
  • Scenes: Five setups featuring consumer goods (supermarket items), industrial objects, transparent/opaque materials, and varying levels of clutter/occlusion.
  • Trajectories: Each sequence includes separate Train and Test trajectories. Train trajectories ensure full visibility, while Test trajectories include partial occlusions and viewpoint extrapolation to test generalization.
• Calibration: Extensive hand-eye calibration was performed using a chessboard, providing both factory and estimated intrinsic/extrinsic parameters to ensure high accuracy.

3. Key Contributions

1. First Integrated Benchmark: SLAM&Render is the first dataset to simultaneously provide multimodal sensor data (RGB-D, IMU), robot kinematics, and ground-truth poses in a fully controlled environment with reproducible motions.
2. Decoupled Train/Test Trajectories: Unlike standard NVS datasets where test frames are often sampled from the training trajectory, SLAM&Render provides completely independent test trajectories. This allows for the rigorous evaluation of overfitting in neural rendering models.
3. Controlled Environmental Variables: The dataset systematically varies lighting and object arrangements, enabling researchers to isolate the impact of illumination changes and occlusions on SLAM and rendering performance.
4. Robotics Integration: By releasing robot kinematic data (joint configurations and flange poses), the dataset enables the development and assessment of SLAM methods that leverage robot kinematics for drift correction and pose initialization.

4. Experimental Results

The authors evaluated state-of-the-art baselines to validate the dataset's utility:

A. Novel View Synthesis (NVS) Evaluation:

• Baselines: Gaussian Splatting (GSPLAT) and FeatSplat (FSPLAT).
• Finding: When evaluated on frames sampled from the training trajectory, both models showed improved metrics with more iterations. However, when evaluated on the independent test trajectory, performance plateaued or declined after 7k iterations.
• Conclusion: This confirms that standard evaluation methods (sampling from training data) mask trajectory overfitting. The independent test trajectory in SLAM&Render reveals that models often generate floating artifacts to fit training views rather than reconstructing true 3D geometry.

B. SLAM / Camera Tracking Evaluation:

• Baselines: MonoGS (Gaussian Splatting SLAM) tested in three configurations: Default, Kinematics Integration, and Kinematics as Initial Seed.
• Finding:
  • Kinematics Integration: Continuously integrating robot kinematic data into the optimization process significantly reduced Absolute Trajectory Error (ATE RMSE), allowing for effective drift correction (e.g., reducing error from ~3.9cm to ~0.4cm in specific setups).
  • Kinematics as Seed: Using kinematics only as an initial seed for pose estimation actually decreased accuracy compared to the default. This suggests that early pose biases propagate through subsequent photometric optimization if not continuously corrected.
• Conclusion: The dataset proves the necessity of continuous multimodal fusion rather than simple initialization for robust robotic SLAM.

5. Significance

SLAM&Render addresses a critical gap in the research community by providing a standardized, high-fidelity benchmark for the emerging field of Neural SLAM. Its significance lies in:

• Rigorous Evaluation: It forces researchers to move beyond "training-set testing" by providing independent test trajectories that expose overfitting in neural rendering.
• Robotic Applicability: It bridges the gap between theoretical rendering and practical robotics by including kinematic data, enabling the study of how SLAM can be tightly coupled with robot control loops.
• Robustness Testing: The controlled lighting and occlusion variations provide a stress test for algorithms, ensuring that future methods are robust to real-world environmental changes before deployment.

The dataset is released under a CC-BY 4.0 license, including tools for data association, ROS2 bag extraction, and synchronization, facilitating immediate adoption by the research community.