Technical Summary: OV3D-Bench

Problem Statement

Open-vocabulary monocular 3D detectors have demonstrated strong performance in controlled settings, yet the field lacks a unified, diagnostic evaluation protocol. Current benchmarks suffer from three critical flaws:

1. Inconsistent Protocols: Models are evaluated under different conditions, often partitioning categories into disjoint "base" and "novel" sets, which makes cross-paper comparisons unreliable.
2. Unrealistic Oracles: Many recent methods (e.g., DetAny3D, OVMono3D) utilize "target-aware" metrics. These protocols prompt models with the ground-truth category names present in each specific test image. This assumption is deployment-incorrect (a deployed detector does not know the ground-truth classes of a scene) and silently suppresses cross-class false positives (hallucinations), inflating performance scores.
3. Entangled Metrics: The standard metric, $AP_{3D}$, conflates geometric localization quality with semantic correctness. A low score cannot distinguish whether a model failed to find an object or merely mislabeled a correctly localized box.

Consequently, existing protocols fail to diagnose specific failure modes or reflect realistic deployment conditions where the label space is revealed only at test time, not per-image.

Methodology: OV3D-Bench

The authors introduce OV3D-Bench, a diagnostic benchmark designed to evaluate open-vocabulary monocular 3D detectors under deployment-realistic conditions across seven indoor and outdoor datasets (KITTI, nuScenes, Argoverse 2, SUNRGBD, ScanNet-200, ARKitScenes, and Hypersim).

1. Evaluation Protocol

• Dataset-Level Prompts: Instead of per-image ground-truth oracles, OV3D-Bench prompts models with the full vocabulary of the dataset at test time. This removes per-image privileged information while retaining scene-level priors (e.g., urban scenes contain cars).
• Decoupled Metrics: The benchmark separates performance into three distinct axes:
  • Geometric Accuracy: Measured via 3D Class-Agnostic Recall. This metric credits a prediction if it matches a ground-truth box with IoU $\ge$ 0.15, regardless of the predicted category. This isolates localization capability from semantic errors.
  • Semantic Robustness: Analyzed via confusion matrices and prompt sensitivity. The authors test stability against prompt variations (underdetailed, standard, and overdetailed phrasing) using the coefficient of variation (CV) of $AP_{3D}$.
  • Cross-Domain Generalization: Performance is evaluated on unseen datasets to assess open-set generalization, distinguishing between in-domain and out-of-domain transfer.

2. Training-Free Baseline

To isolate the gap between localization and semantics, the authors propose a simple, training-free baseline:

• Mechanism: A frozen closed-vocabulary detector (e.g., Cube R-CNN) generates 3D bounding boxes. These boxes are projected to 2D, cropped, and encoded using a vision-language model (VLM) encoder (specifically SigLIPv2).
• Remapping: The visual features are compared against text embeddings of the dataset-level vocabulary. Detections are remapped to the category with the highest similarity above a threshold.
• Purpose: This approach holds geometric localization fixed while varying only the vocabulary, acting as a probe to determine if failures are geometric or semantic.

Key Results

The authors evaluated seven representative detectors (including OV-3Det, ImOV3D, DetAny3D, OVMono3D, WildDet3D, and closed-vocabulary baselines remapped with SigLIPv2).

1. Geometry is Mature; Semantics is the Bottleneck:

   • Detectors consistently localize objects well (high 3D Class-Agnostic Recall) but frequently mislabel them. For example, five different detectors correctly localized a vehicle in Argoverse 2 but disagreed on its category (labeling it as a truck, truck cab, or large vehicle).
   • The gap between Class-Agnostic Recall and standard $AP_{3D}$ indicates that semantic recognition, not geometric localization, is the primary failure mode.

2. Prompt Sensitivity:

   • Performance is highly sensitive to prompt phrasing. For instance, WildDet3D's $AP_{3D}$ on KITTI collapsed from 18.6 to 5.4 when prompts were changed from "car" to "a detailed high-resolution photo of a car."
   • In contrast, the SigLIPv2-based remapping baseline remained stable (within 1.7 $AP_{3D}$) across prompt variations, suggesting that conditioning detection directly on prompts introduces instability.

3. Target-Aware Protocols Inflate Scores:

   • The target-aware protocol significantly overestimates performance by discarding hallucinated classes. On ScanNet-200, DetAny3D's score inflated by 1.9$\times$ (from 2.85 to 5.47 $AP_{3D}$) when switching from dataset-level to target-aware prompting.
   • This inflation is most severe for datasets with large vocabularies and for models that suffer from cross-class confusion.

4. Competitiveness of Simple Baselines:

   • The training-free SigLIPv2 remapping baseline (applied to Cube R-CNN) matches or surpasses purpose-built open-vocabulary methods on unseen outdoor domains.
   • In terms of runtime, the Cube R-CNN* pipeline is 2.6$\times$ faster than DetAny3D (223.9 ms vs. 400 ms per image) while remaining competitive in accuracy.

5. Domain Transfer Asymmetry:

   • Geometric localization transfers well across domains, but semantic recognition does not. Models trained on nuScenes generalize well to KITTI and Argoverse 2, but models trained on KITTI perform poorly on nuScenes.
   • Indoor-to-indoor transfer is particularly difficult, with ScanNet-trained models failing to generalize to ARKitScenes and Hypersim.

Significance and Claims

The paper claims that the community has settled on evaluation practices that are "optimistic rather than diagnostic." By introducing OV3D-Bench, the authors provide a framework that:

• Exposes Hidden Failures: It reveals that current "state-of-the-art" models often fail due to semantic confusion rather than geometric inability.
• Redefines the Bottleneck: The results indicate that geometric localization is a solved or mature problem, while open-vocabulary semantics remains the primary unresolved gap.
• Offers a Practical Alternative: The simple, training-free remapping strategy demonstrates that complex, purpose-built architectures may not be necessary to achieve competitive open-vocabulary performance, provided the localization backbone is strong.

The authors conclude that future progress should prioritize closing the semantic gap and that evaluation protocols must move away from target-aware oracles to reflect realistic deployment constraints.