RoboArmGS: High-Quality Robotic Arm Splatting via Bézier Curve Refinement

Technical Summary: RoboArmGS

Problem Statement
Constructing photorealistic and controllable digital assets of robotic arms from real-world observations is critical for applications in robot control, policy learning, and simulation. However, current approaches face a fundamental discrepancy between idealized kinematic models (URDF) and actual physical execution. Existing methods typically bind static 3D Gaussians to URDF links, forcing them to passively follow rigid kinematic motion. This approach fails to account for real-world dynamic deviations—such as control latency, joint friction, and calibration errors—leading to severe rendering artifacts and misalignment between the digital asset and visual observations. Furthermore, many existing pipelines rely on complex, multi-stage processes requiring multi-view data and labor-intensive manual segmentation, limiting their scalability and applicability outside controlled laboratory settings.

Methodology: RoboArmGS
The authors propose RoboArmGS, a novel hybrid representation designed to harmonize rigid kinematic priors with actual visual observations using monocular video. The framework consists of two core modules:

1. Structured Gaussian Binding (SGB):
   Instead of treating 3D Gaussians as an unstructured collection, SGB anchors each Gaussian to a specific face on the robot's URDF mesh within its local coordinate frame. This enforces topological consistency by ensuring all Gaussians on a single link undergo a collective rigid transformation.

   • Physics-driven Local Parameterization: The motion is driven by forward kinematics (FK) from a MuJoCo engine to maintain piecewise rigidity. Gaussian properties are decomposed into dynamic, pose-dependent states (determined by link poses) and learnable static attributes (local offsets for position, rotation, scale, color, and opacity).
   • Binding-Aware Adaptive Densification: To capture high-frequency details without breaking structural integrity, the method implements binding inheritance (new primitives inherit the face index of their progenitor) and a structural preservation constraint (preventing the pruning of Gaussians associated with any mesh face).

2. Bézier-based Motion Refiner (BMR):
   To address the mismatch between idealized FK and real-world dynamics, BMR acts as a lightweight motion residual field. It learns a smooth, continuous-time residual transformation to correct the rigid model.

   • Learnable Bézier Correction: A global, time-varying correction is modeled using a learnable Bézier curve in a 9-dimensional parameter space (3D translation + 6D rotation). This captures complex motion patterns and temporal drifts.
   • Per-Joint Static Correction: To handle hierarchical dependencies, the model includes static, learnable embeddings for each joint. These correct intrinsic structural discrepancies like link length errors or joint calibration drift.
   • Pose Composition: The final pose for each joint is computed by composing the nominal FK pose, the global Bézier correction, and the per-joint static offset.

The entire system is optimized via a rendering loss (L1 + D-SSIM) combined with geometric regularization (penalizing large local offsets) and Bézier velocity regularization (ensuring temporal smoothness of the correction).

Key Contributions

• RoboArmGS Framework: A hybrid representation that synergizes Structured Gaussian Binding and a Bézier-based Motion Refiner. It enforces geometric consistency while explicitly correcting dynamic discrepancies, reconciling rigid kinematic priors with real-world observations.
• RoboArm4D Dataset: A carefully collected benchmark featuring widely used industrial robotic arms (Franka FR3, UR-5e, ABB IRB 120). It provides synchronized monocular videos, calibrated camera parameters, time-stamped joint trajectories, and URDF files to facilitate the evaluation of high-fidelity digital asset creation.
• State-of-the-Art Performance: The method achieves superior results in both novel-view synthesis (static reconstruction) and novel-pose synthesis (dynamic rendering of unseen joint configurations) compared to existing baselines.

Results
Evaluated on the RoboArm4D dataset, RoboArmGS demonstrates significant improvements over state-of-the-art methods:

• Novel-View Synthesis: Achieved a PSNR of 28.669, SSIM of 0.938, and LPIPS of 0.065, outperforming methods like Robo-GS, 2DGS, and SuGaR. The structured binding prevents floating artifacts and maintains structural coherence.
• Novel-Pose Synthesis: Achieved a PSNR of 31.704, SSIM of 0.967, and LPIPS of 0.039. Unlike generic 4D reconstruction methods that fail to render specific joint angles, or simple FK-driven baselines that suffer from motion misalignment, RoboArmGS produces sharp, kinematically accurate images for unseen poses.
• Ablation Studies: Confirm that removing the SGB module severely degrades static reconstruction quality, while removing the BMR module results in significant artifacts in highly articulated regions (e.g., the gripper), validating the necessity of both components.

Significance
The paper positions RoboArmGS as a foundational step toward creating robust simulation platforms for robotics. By bridging the gap between idealized kinematics and physical reality using monocular video, the method enables the generation of digital assets that are both visually photorealistic and kinematically accurate. This capability is essential for bridging the sim-to-real gap in robot control and policy learning. The authors emphasize that their approach offers a scalable alternative to complex multi-view pipelines, requiring only casual monocular video inputs while explicitly modeling the dynamic deviations inherent in real-world robotic execution. The release of the RoboArm4D dataset and code aims to support future research in high-fidelity robotic digitization.