WildBox: A Dataset and Benchmark for Aerial Monocular 3D Detection of African Savanna Wildlife

Technical Summary: WildBox – A Dataset and Benchmark for Aerial Monocular 3D Detection of African Savanna Wildlife

Problem Statement
While drone-based monitoring is increasingly vital for wildlife ecology and conservation, existing computer vision pipelines for aerial video remain predominantly two-dimensional. Although 2D detection, counting, and tracking are established, they are insufficient for downstream tasks requiring real-world geometric reasoning, such as viewpoint-aware re-identification and drone autonomy. Current monocular 3D detection benchmarks (e.g., KITTI, nuScenes, Omni3D) are dominated by ground-level automotive or indoor imagery. Conversely, existing aerial wildlife datasets provide only 2D supervision. Synthetic datasets exist but lack real-world counterparts. This gap raises the critical question: Is monocular 3D perception feasible on real drone-based wildlife video, and how well do existing open-vocabulary 3D detectors transfer to this domain?

Methodology and Dataset Construction
The authors introduce WildBox, a dataset and benchmark comprising 237,505 3D bounding box annotations across seven African savanna species (grouped into six benchmark classes: reticulated giraffe, plains zebra, Grévy's zebra, African bush elephant, Thomson's gazelle, and rhinoceros).

• Data Source: The dataset aggregates oblique drone footage from DJI drones collected between 2023 and 2026 at Ol Pejeta Conservancy (Kenya), Bristol Zoological Society (UK), and the Mpala Research Centre.
• Annotation Pipeline: A model-in-the-loop approach was used. Raw footage was segmented into clips (max 200 frames) with stable viewpoints. The WildLIFT system proposed 3D oriented bounding boxes, which were then refined and confirmed frame-by-frame by human reviewers.
• Annotation Format: Annotations follow a KITTI/Omni3D-compatible format within a per-segment scale-normalized camera frame. Boxes include position, extent, full 3×3 rotation matrices, and consistent instance IDs within segments. Absolute metric scale is not asserted across flights; instead, coordinates are normalized by the median absolute depth of the segment's reconstruction.
• Benchmark Setup: The dataset is split into 51 training and 13 validation videos (345 segments total), with stratification by species to prevent leakage.

Evaluation Protocol
The paper evaluates two structurally distinct open-vocabulary monocular 3D architectures:

1. OVMono3D-LIFT: A Cube R-CNN architecture with a CLIP-text-prompted classifier.
2. DetAny3D: A promptable foundation model fusing frozen SAM and UniDepth-pretrained DINOv2 backbones.

Four evaluation settings were tested:

1. Closed-vocabulary Zero-Shot: Using the architecture's native 2D stage trained on Omni3D (filtered for non-wildlife classes).
2. Open-Vocabulary Zero-Shot: Using GroundingDINO for 2D proposals.
3. Ground-Truth 2D Prompt Zero-Shot: Feeding perfect 2D boxes and class labels to the 3D head to isolate 3D-stage failure.
4. Supervised Fine-Tuning: Training on the WildBox training split.

Additionally, a curriculum-initialization strategy was tested for OVMono3D-LIFT, pretraining on a merged "zebra" class (plains + Grévy's) before fine-tuning on the split classes.

Key Results

• Zero-Shot Collapse: Open-vocabulary 2D foundation models successfully localize wildlife (50.55 AP@50 for OVMono3D-LIFT, 60.89 for DetAny3D). However, zero-shot 3D lifting collapses to 0.00 AP across all architectures and 2D-input conditions, including when ground-truth 2D boxes are provided. This isolates the failure strictly to the 3D stage (depth estimation and box lifting) rather than 2D detection or specific head designs.
• Recovery via Fine-Tuning: Supervised fine-tuning on WildBox recovers measurable performance. The best configuration (OVMono3D-LIFT with curriculum initialization) achieves 8.68 ± 0.47 AP-BEV@0.50 and 13.17 ± 0.69 AP3D.
• Error Decomposition: A disentangled analysis of the Normalized Hausdorff Distance (NHD) reveals that depth (z-axis) accounts for 84–99% of the total error. In zero-shot settings, depth contributes over 99% of the error. Even with perfect 2D inputs, the depth error remains two orders of magnitude larger than other components, identifying monocular aerial depth estimation as the dominant bottleneck.
• Curriculum Learning: Pretraining on a merged superclass (zebras) before fine-tuning on subclasses improved macro performance with fewer total iterations compared to direct fine-tuning. Gains were most significant for the rarer subclass (Grévy's zebra) and the head class (plains zebra).
• Video Diversity vs. Instance Count: Performance correlates more strongly with the number of training videos (viewpoint/scene diversity) than with the raw number of annotated instances. For example, gazelles (3 videos, 43k instances) performed worse than rhinos (15 videos, 28k instances).

Significance and Claims
The paper claims that WildBox is the first dataset to combine real-world drone imagery, multiple wildlife species, and standard-format tracked 3D bounding boxes. Its primary contribution is establishing a benchmark that reveals the current limitations of open-vocabulary 3D perception in aerial domains.

The authors modestly conclude that while 2D foundation models are effective for wildlife localization, monocular aerial depth estimation remains the dominant open problem preventing successful 3D detection. The dataset and baselines are released to enable progress in 3D wildlife perception, tracked 3D perception, viewpoint-aware re-identification, and wildlife-aware drone autonomy, bridging the gap between ecological monitoring and monocular 3D scene understanding. The benchmark explicitly avoids absolute metric scale claims, focusing instead on relative-scale 3D detection within video segments.