💻 computer science

WildDet3D: Scaling Promptable 3D Detection in the Wild

This paper introduces WildDet3D, a unified geometry-aware architecture capable of handling diverse prompt modalities and auxiliary depth cues, alongside the creation of WildDet3D-Data, the largest open 3D detection dataset to date, together establishing new state-of-the-art performance in open-world monocular 3D object detection.

Original authors: Weikai Huang, Jieyu Zhang, Sijun Li, Taoyang Jia, Jiafei Duan, Yunqian Cheng, Jaemin Cho, Mattew Wallingford, Rustin Soraki, Chris Dongjoo Kim, Donovan Clay, Taira Anderson, Winson Han, Ali Farhadi, B

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weikai Huang, Jieyu Zhang, Sijun Li, Taoyang Jia, Jiafei Duan, Yunqian Cheng, Jaemin Cho, Mattew Wallingford, Rustin Soraki, Chris Dongjoo Kim, Donovan Clay, Taira Anderson, Winson Han, Ali Farhadi, Bharath Hariharan, Zhongzheng Ren, Ranjay Krishna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a photograph of a messy living room. You can see a coffee table, a cat, and a lamp. A normal computer program might tell you, "That's a cat" and "That's a lamp." But it doesn't really know how far away the cat is, how big the table actually is, or which way the lamp is facing. It sees a flat picture, not a 3D world.

WildDet3D is a new AI system designed to fix this. It's like giving a computer "spatial vision" so it can understand the world in 3D, just like you and I do, but with the superpower of understanding thousands of different objects instantly.

Here is a simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "Flat World" Trap

Most current AI detectors are like people wearing blinders. They can only look at the world in one specific way:

The Text-Only Detective: Can only find things if you ask, "Where is the dog?"
The Box-Only Detective: Can only find things if you draw a square around them first.
The 2D-Only Detective: Can tell you what something is, but not where it is in 3D space (how deep, how big, or how it's rotated).

Real life is messy. Sometimes you want to ask a robot, "Pick up the red mug." Sometimes you want to tap a screen to point at a chair. Sometimes you have a depth sensor (like on an iPhone), and sometimes you don't. Existing AI systems usually fail if you change the rules.

2. The Solution: The "Swiss Army Knife" AI

The researchers built WildDet3D, which is like a Swiss Army Knife for 3D vision. It doesn't care how you ask it to look; it just adapts.

Talk to it: You can say, "Find the chair."
Point at it: You can tap a spot on the screen, and it finds the object there.
Box it: You can draw a square around it, and it lifts that square into 3D space.

It's the first system that can do all three of these things at once, making it perfect for robots, Augmented Reality (AR) glasses, and mobile apps.

3. The Secret Sauce: "Optional Glasses"

One of the coolest features is how it handles depth (how far away things are).

Without depth: It's like looking at a painting. It has to guess how far away things are based on experience.
With depth: It's like putting on 3D glasses. If the camera provides a depth map (like a LiDAR sensor on a phone), WildDet3D uses it to get incredibly accurate measurements.

The genius part? If you don't have those 3D glasses, the AI doesn't crash or quit. It just switches to "guessing mode" and still works, though not quite as perfectly. It degrades gracefully, like a car that can drive on both highways and dirt roads.

4. The Training Data: The "Massive Library"

To teach an AI to recognize 3D objects, you need millions of examples. But labeling 3D data is hard and expensive—it's like trying to measure every single brick in a city with a ruler.

The team created WildDet3D-Data, a massive new library of over 1 million images covering 13,500 different categories (from "toasters" to "wild bears").

How they did it: They didn't just hire humans to measure everything. They used a "team of experts" approach. They had five different AI models try to guess the 3D shape of objects in 2D photos. Then, they used a super-smart AI (a Vision-Language Model) to filter out the bad guesses. Finally, human workers checked the best candidates to make sure they were perfect.
The Result: A dataset 138 times larger than previous ones, covering almost every object you can imagine in the real world.

5. What Can It Do? (Real-World Magic)

The paper shows off some fun demos:

On your iPhone: You can open an app, point your camera at your office, and ask, "Where is the stapler?" The app draws a 3D box around it in your screen.
In Augmented Reality (AR): Put on Meta Quest glasses, look at a messy desk, and the AI draws 3D boxes around everything, helping you see the "volume" of the room.
For Robots: A robot arm can be told, "Grab the green chips," and WildDet3D tells the robot exactly where the chips are in 3D space so it can pick them up without knocking them over.
The "Brain" Combo: They paired it with a smart language AI. You can ask, "What is the most expensive thing in this room?" The language AI figures out it's the laptop, and WildDet3D instantly draws a 3D box around it.

The Bottom Line

WildDet3D is a giant leap forward. It takes the flexibility of modern chatbots (which understand text) and combines it with the spatial awareness of a human. It's a general-purpose tool that can be plugged into robots, phones, and glasses to help them understand the physical world, not just the pictures of it.

Think of it as teaching computers to stop looking at the world as a flat movie screen and start seeing it as a real, three-dimensional place they can interact with.

1. Problem Statement

Monocular 3D object detection (recovering 3D extent, location, and orientation from a single RGB image) is a cornerstone of spatial intelligence for robotics, AR/VR, and autonomous driving. However, existing methods face three critical bottlenecks preventing open-world deployment:

Lack of Generalization: Most models are trained on closed-set categories in controlled environments (e.g., autonomous driving datasets) and fail to generalize to the long-tailed, diverse categories found in the real world.
Rigid Prompting: Current systems typically support only a single interaction modality (either text queries or fixed 2D bounding boxes), failing to unify the diverse ways users specify targets (e.g., language commands, point clicks, or box selections).
Inability to Leverage Geometric Cues: Existing methods often cannot gracefully incorporate auxiliary geometric signals (like sparse LiDAR or partial depth) when available, nor do they handle the inherent scale ambiguity of monocular vision effectively without them.
Data Scarcity: High-quality 3D datasets are limited in scale and category coverage compared to 2D datasets, hindering the training of open-vocabulary 3D models.

2. Methodology: WildDet3D

The authors propose WildDet3D, a unified, geometry-aware architecture designed for open-vocabulary monocular 3D detection.

A. Architecture Overview

The model consists of three main components (Figure 3):

Dual-Vision Encoders:
- Image Encoder: A ViT-H backbone (pretrained on segmentation) that extracts high-resolution semantic features.
- RGBD Encoder: A DINOv2-based backbone that accepts an optional depth map (or zero-filled input if unavailable). It produces "depth latents" representing geometric cues.
- Depth Fusion Module: A ControlNet-style residual block that injects depth latents into the image features. This allows the model to utilize metric scale information when depth is available while degrading gracefully to pure monocular mode when it is not.
Promptable Detector:
- Unifies four prompt modalities: Text (category names), 2D Points, 2D Bounding Boxes, and Exemplar Boxes (visual similarity).
- Uses a SAM3-inspired prompt encoding mechanism where text and geometric prompts are encoded into a unified query sequence.
- Employs per-prompt batching to handle arbitrary numbers of categories and instances without padding.
Deeply-Supervised 3D Detection Head:
- Lifts 2D query features to 3D predictions using Transformer decoder layers.
- Multi-source Aggregation: Uses cross-attention to fuse camera ray embeddings (for spatial context) and depth latents.
- Unambiguous Rotation Normalization: Resolves the 4-fold rotation ambiguity in 3D boxes by enforcing dimension ordering ( $w \le l$ ) and yaw folding ( $[0, \pi)$ ), ensuring a one-to-one mapping between geometry and regression targets.
- 3D Confidence Head: Predicts a quality score combining 2D objectness and 3D depth accuracy to re-rank detections.

B. Training Strategy

Multi-task Learning: The model is trained with auxiliary heads for 2D detection and depth estimation to provide dense supervision.
One-to-Many (O2M) Matching: Each ground truth object is paired with its top- $k$ predictions to accelerate convergence.
Ignore-Region Suppression: Handles non-exhaustive annotations (common in open-world data) by suppressing negative loss for predictions overlapping with "ignore" regions, ensuring the model isn't penalized for detecting objects lacking valid 3D ground truth.

3. Key Contributions: WildDet3D-Data

To address the data bottleneck, the authors introduce WildDet3D-Data, the largest open 3D detection dataset to date.

Scale & Diversity: Contains ~1M images across 13.5K categories (a 138× increase over Omni3D), covering indoor, urban, and nature scenes.
Construction Pipeline:
1. Candidate Generation: Five complementary models (including SAM-3D, DetAny3D, and geometric fitting methods) generate candidate 3D boxes from existing 2D annotations.
2. Rule-Based Filtering: Removes implausible candidates based on occlusion, edge contact, and size ratios.
3. Selection: Uses a two-stage selection process:
  - VLM Filtering: A fine-tuned Vision-Language Model (Molmo2) scores candidates on perceptual criteria (scale, shape, rotation).
  - Human Verification: Crowdsourced annotators verify a subset, selecting the best candidate and rating quality.
Result: A curated dataset of human-verified 3D boxes that enables robust open-world generalization.

4. Experimental Results

WildDet3D establishes new state-of-the-art (SOTA) performance across multiple benchmarks:

WildDet3D-Bench (In-the-Wild):
- Achieves 22.6 AP3D (text prompt) and 24.8 AP3D (box prompt) on their new benchmark with 700+ categories.
- Outperforms the previous best (3D-MOOD) by a massive margin (2.3 AP vs. 22.6 AP).
- With ground-truth depth, performance jumps to 41.6 AP3D, demonstrating the model's ability to leverage geometric cues.
Omni3D Benchmark:
- Achieves 34.2 AP3D (text) and 36.4 AP3D (box/oracle).
- Notably, it achieves this in only 12 training epochs, whereas competitors require 80–120 epochs.
Zero-Shot Transfer:
- Trained on Omni3D, it achieves 40.3 ODS on Argoverse 2 and 48.9 ODS on ScanNet without fine-tuning, significantly outperforming baselines.
Stereo4D (Real Depth):
- When provided with real stereo depth, performance improves by 2.8× (7.5 AP $\to$ 27.7 AP), validating the architecture's flexibility.

5. Significance and Applications

The paper demonstrates that open-vocabulary 3D perception is achievable at scale through unified architectures and large-scale data curation.

Versatility: The model supports diverse interaction modes (text, click, box) and gracefully handles missing depth data.
Real-World Deployment: The authors demonstrate practical applications including:
- Mobile: An iPhone app for real-time 3D detection via ARKit.
- AR/VR: Integration with Meta Quest 3 for spatial understanding.
- Robotics: Enabling a Franka Emika Panda arm to grasp objects specified by open-vocabulary text prompts.
- VLM Integration: A "WildDet3D-Agent" that combines VLM reasoning (e.g., "find the most expensive object") with 3D localization.

Conclusion

WildDet3D bridges the gap between 2D open-vocabulary perception and 3D spatial understanding. By introducing a geometry-aware architecture that unifies multiple prompt modalities and a massive, human-verified dataset, the authors enable robust, open-world 3D detection that generalizes to unseen categories and leverages available geometric cues, paving the way for more capable embodied AI and spatial computing systems.