DF3DV-1K: A Large-Scale Dataset and Benchmark for Distractor-Free Novel View Synthesis

1. Problem Statement

Radiance fields (e.g., NeRF, 3D Gaussian Splatting) have achieved photorealistic novel view synthesis. However, existing methods often struggle with visual distractors (e.g., moving people, passing cars, shadows, reflections) commonly found in casually captured, short-span image sequences.

While "distractor-free" radiance field methods aim to synthesize clean novel views from these cluttered inputs, the field faces two critical bottlenecks:

1. Lack of Large-Scale Data: Existing datasets (e.g., RobustNeRF, On-the-go) are small-scale (5–12 scenes) and lack sufficient diversity in distractor types and scene themes. They often fail to challenge state-of-the-art (SOTA) methods, leading to performance saturation where differences are only visible via subtle zoom-ins.
2. Absence of Systematic Benchmarks: Without a large-scale, challenging benchmark containing paired clean and cluttered images per scene, it is difficult to rigorously evaluate the robustness of new methods or transition from scene-specific optimization to generalizable solutions.

2. Methodology

A. Dataset Construction (DF3DV-1K)

The authors introduced DF3DV-1K, a large-scale real-world dataset designed specifically for distractor-free novel view synthesis.

• Scale: 1,048 scenes (726 indoor, 322 outdoor) with a total of 89,924 images.
• Data Collection: Captured over 9 months using 12 different consumer cameras (e.g., iPhone, Samsung) to mimic casual capture conditions.
• Paired Data: Every scene includes both clean (distractor-free) and cluttered (with distractors) image sets.
• Diversity:
  • 128 Distractor Types: Ranging from common objects (cars, people) to complex categories like fluid (water), shadows, semi-transparent objects, and semantically similar distractors.
  • 161 Scene Themes: Covering diverse indoor and outdoor environments.
  • Resolution: 4K images captured with automatic exposure/focus.
• DF3DV-41: A curated subset of 41 scenes systematically designed to test specific challenging scenarios (e.g., nighttime, fluid dynamics, semantic similarity) to evaluate method robustness in extreme conditions.

B. Data Curation Pipeline

1. Capture: Operators capture clean and cluttered views. For controllable scenes, they are captured separately; for uncontrolled scenes, they are captured simultaneously and separated manually.
2. Quality Control: Low-quality images (motion blur in static regions) are removed. Clean images are verified to be free of distractors.
3. Pose Estimation: COLMAP is used to estimate camera poses jointly from clean and cluttered images to ensure a shared coordinate system.
4. Verification: Scenes are reconstructed using instant-ngp. Failed reconstructions (degenerate geometry) are re-tuned or discarded.

C. Benchmarking & Application (DI2FIX)

• Benchmark: The authors evaluated 10 methods (9 recent distractor-free radiance field methods + vanilla 3DGS) across the full dataset and the DF3DV-41 subset.
• DI2FIX (Distractor-Free DIFIX): To demonstrate the dataset's utility beyond benchmarking, the authors fine-tuned DIFIX (a diffusion-based 2D enhancer) on the DF3DV-1K* subset (1,007 scenes). This created a plug-and-play enhancer that takes a radiance field rendering and removes distractor artifacts without modifying the underlying 3D model.

3. Key Contributions

1. DF3DV-1K Dataset: The first large-scale, real-world dataset for distractor-free synthesis, offering ~100x more scenes and ~10x more distractor types than previous benchmarks. It provides paired clean/cluttered images essential for supervised learning and rigorous evaluation.
2. Comprehensive Benchmark: A systematic evaluation of 10 methods across 1,048 scenes, identifying AsymGS and RobustSplat as the most robust current methods, while highlighting that existing methods still struggle with semantically similar and fluid distractors.
3. DI2FIX Application: A demonstration that large-scale data enables generalizable improvements. Fine-tuning a 2D enhancer on DF3DV-1K significantly improved rendering quality across multiple methods, proving the dataset's value for training robust vision models.

4. Results

Benchmark Performance

• Difficulty: DF3DV-1K and DF3DV-41 are significantly more challenging than RobustNeRF and On-the-go. On DF3DV-1K, even the best methods achieve lower PSNR (20.5 dB) compared to RobustNeRF (29 dB), indicating the dataset effectively differentiates method capabilities.
• Top Performers: AsymGS and RobustSplat showed the highest robustness, followed by OCSplats and DeGauss. The ranking generally correlates with publication dates, suggesting steady progress in the field.
• Failure Cases: Methods struggle most with:
  • Semantically similar distractors: Hard to distinguish from static background.
  • Fluid distractors: Semi-transparent and scattered nature confuses masking strategies.
  • Nighttime scenes: Low light makes threshold-based filtering unreliable.

DI2FIX Improvements

• Quantitative Gains: Fine-tuning DIFIX on DF3DV-1K (creating DI2FIX) yielded average improvements of +0.96 dB PSNR and -0.057 LPIPS on held-out sets (DF3DV-41 and On-the-go).
• Qualitative Gains: DI2FIX successfully removed artifacts (e.g., floating chess pieces, vegetable clutter) and inpainted occluded regions, whereas the base DIFIX (trained on smaller datasets) often failed or introduced noise.
• Scale Sensitivity: Performance improved consistently as the training set size increased from 250 to 1,000 scenes, confirming the importance of large-scale data diversity.

5. Significance

• Advancing the Field: DF3DV-1K addresses the critical gap in data availability, moving the field from small-scale, scene-specific evaluations to large-scale, generalizable research.
• Standardization: It provides a unified, challenging benchmark that prevents "overfitting" to small datasets and forces methods to handle real-world complexity.
• Generalization: The success of DI2FIX demonstrates that large-scale distractor-free datasets can be used to train post-processing enhancers that boost the performance of any underlying radiance field method, paving the way for more robust, plug-and-play 3D vision solutions.
• Future Direction: The dataset highlights specific failure modes (fluids, semantics, night), guiding future research toward more sophisticated masking and semantic understanding mechanisms.