Getting the Numbers Right$\unicode{x2014}$Modelling Multi-Class Object Counting in Dense and Varied Scenes

1. Problem Statement

Object counting is a critical computer vision task with applications in urban planning, biodiversity monitoring, and medicine. While detection-based methods (e.g., YOLO) work well in sparse scenes, they fail in dense, heavily occluded, and crowded environments where objects overlap significantly.

Existing solutions fall into two categories:

• Detection-based: Struggle with high density and occlusion.
• Density Estimation: Predicts a heatmap where the integral equals the object count. While effective for single classes, multi-class density estimation (counting multiple object types simultaneously) is under-researched.
  • Current Limitations: Existing multi-class density estimators (e.g., DSACA) often degrade in less-dense scenes. Furthermore, many rely on auxiliary tasks (like segmentation or detection) that require specific assumptions or post-processing at inference time, which can introduce errors or limit the model's ability to handle continuous density in highly congested scenarios.
  • Domain Gap: Most prior work focuses on urban datasets (VisDrone, iSAID) with uniform class distributions. They struggle in biodiversity datasets (e.g., Hicks) where samples often contain only one class or highly skewed distributions, and where objects may never naturally co-occur.

2. Methodology

The authors propose the first Vision Transformer (ViT)-based approach for multi-class density estimation, designed to handle a wide range of densities and class distributions.

A. Architecture Overview

The model follows an encoder-decoder structure:

1. Backbone (Encoder): Uses Twins-SVT (a Pyramid Vision Transformer). Unlike flat transformers, the pyramid structure extracts multi-scale features, which is crucial for handling objects of varying sizes and densities.
2. Decoder: A Multiscale CNN Decoder that fuses hierarchical features from the ViT backbone. This allows the model to leverage the global context of the transformer while maintaining the spatial precision of convolutional networks.
3. Activation Function: The density head uses Softplus activation ($\log(1 + e^{\beta x})$). Unlike ReLU, which truncates negative values (potentially losing gradient information), Softplus guarantees a numerically stable, strictly positive output while maintaining differentiability, ensuring robust learning of positive counts.

B. Key Innovation: Category Focus Module (CFM)

To address inter-category interference (where the presence of one class confuses the count of another), the authors introduce the Category Focus Module (CFM):

• Function: It acts as a lightweight auxiliary masking task during training. It extracts features from multiple backbone layers and applies dilated convolutions to increase the receptive field.
• Training: It predicts a segmentation mask for each class (non-exclusive, allowing pixels to belong to multiple classes).
• Inference: Crucially, the CFM is discarded at inference time. The model relies solely on the density estimation head. This avoids the "discreteness assumptions" of detection-augmented methods or the "thresholding assumptions" of segmentation-augmented methods, allowing for purely continuous density estimation even in complex scenes.

C. Loss Functions

The model is trained using a combined loss function:

• $L_2$ Loss: Applied to the density maps to minimize pixel-wise squared error, incorporating spatial, class-localization, and count errors.
• Cross-Entropy Loss ($L_{mask}$): Applied to the auxiliary CFM masking task.
• Total Loss: $L = (\omega \times L_{mask}) + L_2$. The weighting factor $\omega$ balances the tasks, allowing the shared parameters to learn robust features beneficial to both density estimation and class awareness.

3. Key Contributions

1. First ViT-based Multi-Class Density Estimator: Introduces a novel architecture combining Twins-SVT and multiscale CNNs, outperforming previous state-of-the-art (SOTA) methods by an average 46.7% reduction in Mean Absolute Error (MAE).
2. Robustness Across Densities: The method excels in both sparse and extremely dense scenes. In the most crowded test samples, it outperforms the detection-based YOLO11 by an order of magnitude.
3. Inference-Time Purity: The Category Focus Module (CFM) improves class awareness during training but is removed during inference. This is the first approach to treat multi-class counting entirely as a continuous density estimation problem at inference, avoiding the limitations of hybrid detection/segmentation pipelines.
4. Cross-Domain Applicability: Successfully demonstrates performance on diverse datasets, including urban scenes (VisDrone, iSAID) and biodiversity monitoring (Hicks dataset), proving adaptability to skewed class distributions.

4. Experimental Results

The method was evaluated on three benchmarks: VisDrone-DET (10 and 8 classes), iSAID (4 classes), and Hicks (8 flower species).

• VisDrone-DET (8-class):
  • Achieved an MAE of 2.29 compared to DSACA's 3.43 and YOLO11's 2.65.
  • In low-density ranges (0–10 objects), the proposed method achieved an MAE of 0.42, significantly outperforming YOLO11 (1.51) and DSACA (1.85).
  • In high-density ranges (101–1000 objects), it achieved an MAE of 8.61, vastly superior to YOLO11 (84.94).
• VisDrone-DET (10-class):
  • Reduced MAE from 3.76 (Michel et al.) to 1.99.
• iSAID (4-class):
  • Achieved an MAE of 2.84 compared to 7.85 (Michel et al.) and 10.95 (YOLO11).
  • In the 101–10,000 object range, the method reduced MAE from 50.31 to 22.11.
• Hicks (Biodiversity):
  • Demonstrated a 64% reduction in MAE compared to DSACA (0.38 vs 0.92), proving effectiveness in sparse, single-class-dominant natural environments.

5. Significance

• Bridging the Gap: The paper successfully bridges the performance gap between detection-based methods (good for sparse scenes) and density estimation (good for dense scenes), creating a unified model that works well across the entire spectrum of object density.
• Ecological Monitoring: By validating on the Hicks biodiversity dataset, the work expands the utility of automated counting beyond urban surveillance to agricultural and ecological monitoring, where object occlusion and class scarcity are common.
• Methodological Advancement: The removal of auxiliary tasks at inference time represents a significant theoretical and practical step forward, simplifying deployment while maintaining high accuracy through better feature learning during training.

In conclusion, this work establishes a new state-of-the-art for multi-class object counting, offering a scalable, robust, and purely continuous solution that outperforms both traditional density estimators and modern object detectors in diverse and challenging environments.