Technical Summary: NVS-HO: A Benchmark for Novel View Synthesis of Handheld Objects

Problem Statement

Novel View Synthesis (NVS) for handheld objects in real-world environments remains a largely unexplored challenge. While existing datasets have advanced 6-DoF pose estimation for hand-object interactions, they often rely on depth sensors, pre-scanned 3D object models, or MANO hand models, making them unsuitable for RGB-only NVS tasks. Furthermore, current NVS-oriented datasets (e.g., CO3D, WildRGB-D) either lack ground-truth poses for objects or focus on static scenes rather than dynamic handheld manipulation. The specific difficulty lies in capturing the full appearance of an object while it is manipulated by a hand, which introduces partial occlusions and motion blur, all while the camera remains static. Existing methods struggle to estimate accurate poses from RGB images alone under these unconstrained conditions, leading to significant performance gaps in rendering quality.

Methodology

Dataset Creation (NVS-HO)

The authors introduce NVS-HO, the first benchmark for RGB-only NVS of handheld objects. The dataset comprises 67 objects (grocery items and toys) recorded in two complementary sequences:

1. Handheld Sequence (HS): The object is freely manipulated by a human in front of a static camera. This sequence simulates natural interaction and is used for training NVS models. It includes frames with motion blur to reflect real-world conditions.
2. Board Sequence (BS): The same object is fixed on a ChArUco board while the camera moves around it. Marker detection on the board provides accurate ground-truth 6-DoF camera poses. This sequence is used exclusively for evaluation.

Data Processing Pipeline:

• Segmentation: Grounded SAM2 is used to generate binary foreground masks for both sequences, removing the background and the hand (in the HS).
• Pose Alignment: To evaluate NVS methods fairly, a Combined Sequence (CS) is created by merging masked HS and BS images. COLMAP is run on the CS with the BS poses injected as hard constraints. This forces the HS poses to be estimated in the same metric scale and coordinate frame as the BS.
• Evaluation Protocol: A method-agnostic procedure aligns the test poses (BS) into the coordinate frame of any NVS method's estimated HS poses using a Sim(3) transformation (Kabsch–Umeyama algorithm). This allows for quantitative comparison between synthesized and ground-truth images.

Baselines and Experiments

The authors established baselines using two pose estimators and two NVS architectures:

• Pose Estimators:
  • COLMAP: A classical Structure-from-Motion (SfM) pipeline using a sequential matcher.
  • VGGT: A state-of-the-art Vision Transformer-based model. The authors modified VGGT to prune background tokens and focus attention on the foreground object to handle the large uniform backgrounds common in their setup.
• NVS Models:
  • Nerfacto: A NeRF-based implementation from Nerfstudio.
  • Splatfacto: A 3D Gaussian Splatting implementation.
• Training Strategy: Models were trained on the HS using poses estimated by either COLMAP or VGGT. Experiments included training with and without pose optimization (refinement) during the NVS training phase. Alpha transparency carving was used to preserve object boundaries.

Metrics

Evaluation is performed on masked images to focus on the object and handle background rendering variations:

• Foreground Evaluation: PSNR, SSIM, and LPIPS computed strictly on the object region.
• Background Evaluation: Metrics computed on the background region, with ground-truth background colors normalized to match the specific NVS method's output to ensure fair comparison.

Key Results

Experiments on the NVS-HO dataset reveal significant performance gaps under unconstrained handheld conditions:

1. Pose Estimation: COLMAP significantly outperforms VGGT across all metrics (PSNR, SSIM, LPIPS), exhibiting better median values and lower variance. This suggests that traditional SfM remains preferable for reliable pose estimation in handheld scenarios compared to current feed-forward transformer approaches.
2. NVS Performance: Splatfacto (Gaussian Splatting) consistently outperforms Nerfacto (NeRF) across all metrics.
3. Absolute Performance: Despite the relative superiority of Splatfacto with COLMAP, absolute performance remains low (e.g., median PSNR ~16).
4. Pose Refinement: Activating pose optimization during NVS training provided only marginal improvements, indicating that the primary bottleneck is the quality of the initial pose estimation rather than the refinement capability of the NVS model.
5. Qualitative Analysis: Renderings often fail to fully reconstruct the object from novel viewpoints, particularly when using VGGT poses, highlighting the difficulty of learning full object appearance from occluded, handheld sequences.

Significance and Contributions

The paper claims the following contributions:

• First RGB-Only Benchmark: Introduces the first benchmark specifically designed for NVS of handheld objects in real-world environments without relying on MANO hand models or pre-scanned 3D object models.
• New Dataset: Releases a dataset of 67 objects featuring paired Handheld (training) and Board (evaluation) sequences with ground-truth poses.
• Method-Agnostic Protocol: Proposes a standardized evaluation protocol that aligns ground-truth viewpoints into the reference frame of any NVS method, facilitating fair comparison across different pose estimation strategies.
• Identification of Challenges: The benchmark reveals that current state-of-the-art pipelines struggle significantly in unconstrained handheld conditions, particularly due to the difficulty of pose estimation and handling hand-induced occlusions.

The authors conclude that NVS-HO provides a challenging, practical foundation to drive future research toward more robust approaches for RGB-based novel view synthesis of handheld objects.