Technical Summary: MVDGC – Joint 3D and 2D Multi-view Pedestrian Detection via Dual Geometric Constraints

1. Problem Definition

The core challenge in Multi-View Pedestrian Detection (MVPD) is the effective aggregation of visual features from different viewpoints to reason about occlusions in crowded scenes. Existing approaches generally fall into two categories, both of which suffer from significant limitations:

• Anchor-based methods: These perform 2D detection independently in each view and then associate detections based on geometric proximity and appearance. While efficient, they are vulnerable to occlusion, missed detections, and weak cross-view association due to a lack of early feature alignment.
• Centralized-based methods: These project image features onto a shared Bird's Eye View (BEV) map to perform dense occupancy detection. However, the perspective transformation required for this projection induces severe geometric distortion, stretching features non-uniformly. This leads to spatial structure breaks, blurred features, and quantization errors from discretized BEV heatmaps, degrading localization precision, particularly in dense crowds.

Furthermore, current methods often fail to capitalize on the strong mutual relationship between BEV ground points and 2D image bounding boxes, treating 2D detections merely as auxiliary signals rather than jointly optimizing them with the primary BEV localization task.

2. Methodology: MVDGC

The authors propose MVDGC, a unified, end-to-end query-based framework that jointly estimates pedestrian locations on the BEV plane and 2D bounding boxes in image views without relying on dense BEV feature maps. The framework is built upon a 3D cylindrical query representation and consists of the following key components:

2.1 3D Cylindrical Query Formulation

Instead of regressing full 3D bounding boxes (which require unavailable full 3D annotations) or 2D points alone, MVDGC represents each pedestrian as a learnable 3D vertical cylinder.

• Representation: A query $Q_{3D}$ is defined by a position component $P_{3D} = \{x, y, h, r\}$ (ground-plane coordinates, height, and radius) and a feature component $F_{3D}$.
• Geometric Bridge: The cylinder's center lies on the BEV plane, and its projection onto image views naturally forms a rectangular bounding box. This unifies BEV localization and ImV detection into a single task: refining the cylinder's position and shape.
• Advantage: Unlike 3D cuboids which project to complex polygons, cylinders project to clean rectangular boxes, directly supporting BEV localization and weakly supervising the 3D shape parameters.

2.2 Network Architecture

The decoding stage processes these queries through three specialized modules:

1. Multi-view Feature Sampling: 3D cylindrical queries are projected into 2D reference points across all camera views using camera projection matrices. Instead of warping entire feature maps into BEV, the model uses deformable cross-attention to sample local features directly from the raw image domains. This eliminates projection-induced distortion and resolution loss.
2. Intra-Inter Query Interaction: This module facilitates information exchange in two steps:
   • Intra-view: Allows communication among queries within the same camera view to establish spatial coherence.
   • Inter-view: Reinforces the homogeneity of appearance features for the same object across different views.
3. Multi-view Adaptive Fusion: To handle occlusions and varying visibility, the model learns instance-specific attention weights for every object query in every camera view. This allows the network to prioritize the most informative views for each individual pedestrian, rather than applying uniform view weights.

2.3 Training Objectives

MVDGC employs dual geometric constraints to supervise the refinement of the 3D cylinders:

• BEV Supervision: Direct regression of the cylinder's ground-plane coordinates $(x, y)$ using a KL-based regression loss to model uncertainty and a focal-style classification loss.
• ImV Supervision: The projected cylinder is constrained to align with 2D bounding boxes in the image views using Smooth L1 and Generalized IoU (GIoU) losses.
• Joint Optimization: The cylinder's shape ($r, h$) is weakly supervised by the ImV bounding box constraints, ensuring geometric consistency across domains.
• Matching Strategy: An Adaptive Hungarian Matching strategy is used, allowing one-to-many assignment (up to $K=3$) to improve recall in crowded scenes.

3. Key Contributions

1. Novel Framework (MVDGC): A query-based framework that directly regresses BEV coordinates and 2D bounding boxes, avoiding the quantization errors and projection distortions inherent in dense BEV feature maps.
2. 3D Cylindrical Representation: A unified geometric representation that bridges BEV ground points and ImV bounding boxes. This establishes a shared global identity across spatial domains, enabling joint detection and robust tracking even under severe occlusion.
3. Dual-Space Consistency: The enforcement of geometric constraints across both BEV and image views leads to state-of-the-art localization precision.
4. Generalization: Extensive validation on real-world (WildTrack) and synthetic (MultiViewX, GMVD) datasets demonstrates robustness to unseen scene configurations and camera layouts.

4. Experimental Results

The authors evaluated MVDGC on WildTrack, MultiViewX, and the generalized GMVD dataset.

• MVPD Performance: MVDGC achieved State-of-the-Art (SOTA) performance across multiple metrics. On WildTrack, it achieved a MODP of 83.8 and MODA of 94.2. On MultiViewX, it achieved a MODP of 93.0 and MODA of 95.7. It outperformed centralized methods (e.g., MVDeTr, 3DROM, EarlyBird) and methods utilizing extra information (e.g., temporal cues, visual hulls).
• Tracking Performance: By transferring global IDs from the BEV domain to image views, MVDGC significantly outperformed single-view tracking baselines.
  • On WildTrack, it achieved an ImV tracking HOTA of 65.4 and MOTA of 72.9, surpassing strong CNN and Transformer-based trackers (e.g., MeMOTR, StrongSORT) by large margins.
  • On MultiViewX, it achieved an ImV tracking HOTA of 48.4 and MOTA of 40.0.
• Ablation Studies:
  • Modules: The combination of Intra-Inter interaction and Adaptive Fusion yielded the highest gains.
  • Losses: Joint BEV-ImV supervision significantly outperformed BEV-only supervision. The KL loss for point regression proved superior for handling annotation ambiguity.
  • Query Type: The 3D cylindrical query significantly outperformed simple 3D reference points, validating the importance of geometric constraints.
  • Generalization: On the GMVD dataset (distinct training/testing scenes), MVDGC maintained high performance, demonstrating strong cross-scene generalization.

5. Significance and Claims

The paper claims that MVDGC addresses the fundamental limitations of existing MVPD methods by bypassing the need for perspective projection into a dense BEV grid. By treating pedestrians as 3D cylinders and enforcing dual constraints, the method:

• Eliminates Distortion: Directly samples features from raw images, preserving spatial relationships and avoiding the geometric distortion caused by perspective warping.
• Enables Robust Tracking: The unified 3D representation provides a consistent global identity that can be propagated across views and time, solving the disconnect between BEV localization and 2D tracking.
• Handles Occlusion: The object-centric design allows the model to reason about occluded pedestrians by leveraging features from visible body parts (torso, head) rather than relying solely on potentially occluded foot points.

The authors note a limitation in recall for extremely dense crowds where severe occlusion persists even with multiple views, suggesting future work could incorporate depth-generation models to synthesize point-cloud-like representations. However, the current work establishes a new benchmark for multi-view pedestrian detection by unifying 2D and 3D reasoning through geometric constraints.