💻 computer science

Bootstrapping MLLM for Weakly-Supervised Class-Agnostic Object Counting

This paper introduces WS-COC, the first MLLM-driven weakly-supervised framework for class-agnostic object counting that utilizes divide-and-discern dialogue tuning, compare-and-rank optimization, and global-and-local enhancement strategies to achieve performance comparable to fully-supervised methods while significantly reducing annotation costs.

Original authors: Xiaowen Zhang, Zijie Yue, Yong Luo, Cairong Zhao, Qijun Chen, Miaojing Shi

Published 2026-02-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaowen Zhang, Zijie Yue, Yong Luo, Cairong Zhao, Qijun Chen, Miaojing Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, well-read librarian (the MLLM) how to count objects in a picture. This librarian has read millions of books and knows what a "car," a "person," or a "fish" looks like, but they have never been explicitly taught to count them in a crowd.

If you just hand them a photo of a crowded stadium and ask, "How many people are here?", the librarian might guess "about 50" because that's a number they've seen in stories, even if there are actually 5,000 people. They are great at understanding what things are, but bad at how many there are, especially when things are packed tight.

This paper introduces a new method called WS-COC to fix this problem. Instead of hiring an army of humans to draw a tiny dot on every single person in every photo (which is expensive and slow), the authors teach the librarian a few clever tricks using only the total number of people written on the back of the photo (a "weak" hint).

Here is how they do it, using three simple strategies:

1. The "Guess the Range" Game (Divide-and-Discern)

The Problem: Asking the librarian to guess the exact number (e.g., "Is it 1,243?") is too hard. It's like asking someone to guess a specific second on a clock without looking at the hands.
The Solution: Instead of asking for the exact number, the authors teach the librarian to play a game of "Higher or Lower."

Round 1: "Are there more than 1,000 people?" (Librarian says: "Yes.")
Round 2: "Are there more than 500?" (Librarian says: "Yes.")
Round 3: "Are there more than 250?" (Librarian says: "No.")
Round 4: "Are there more than 375?" ... and so on.

By breaking the big, scary number down into smaller, easier questions, the librarian gets better and better at narrowing down the answer until they can give a very accurate count. It's like finding a hidden treasure by checking if it's in the north half, then the south half, then the east quarter, etc.

2. The "Who Has More?" Race (Compare-and-Rank)

The Problem: Sometimes, the librarian gets confused because "counting" is a math concept, but "seeing" is a visual concept. It's hard to translate a picture directly into a number.
The Solution: Instead of asking "How many?", the authors ask the librarian to compare pictures.

They show the librarian four different photos of cars.
They ask: "Put these photos in order from the one with the fewest cars to the one with the most."
The librarian is really good at seeing which pile looks bigger or smaller. By practicing this "ranking" game, the librarian learns the relationship between the visual crowd and the number, which helps them get better at guessing the actual numbers later.

3. The "Zoom-In" Safety Net (Global-and-Local)

The Problem: When a crowd is super dense (like a mosh pit), the librarian tends to underestimate the number. They see a "blur of people" and think, "Oh, that's just a big group," rather than counting the individuals.
The Solution: The authors give the librarian a two-step safety net.

Step A (The Big Picture): Ask for the total count of the whole image.
Step B (The Zoom-In): If the librarian thinks the crowd is huge, cut the image into four smaller squares (like a pizza). Ask the librarian to count the people in each small square separately.
The Magic Mix: Sometimes the librarian underestimates the whole picture, but overestimates the small squares (because they see the edges of people twice). The authors simply take the average of the "Big Picture" guess and the "Small Squares" guess. It's like taking two different maps and averaging them to find the true location.

The Result

By using these three tricks, the authors taught the librarian to count objects in any category (cars, people, fish, birds) without needing a human to draw dots on every single object.

Before: The librarian was okay with sparse crowds but terrible with dense ones.
After: The librarian is now almost as good as the most expensive, fully-trained experts (who required thousands of hours of human dot-drawing), but they learned it using much cheaper, simpler hints.

In short: They didn't force the AI to memorize the answer; they taught it how to think about counting by breaking problems down, comparing scenes, and double-checking its work.

1. Problem Statement

Object counting is a fundamental computer vision task with applications in traffic monitoring, crowd analysis, and industrial inspection.

Limitations of Fully-Supervised Methods: State-of-the-art methods typically rely on point-level annotations (marking every individual object). This is labor-intensive, expensive, and impractical for dense scenes where objects occlude one another.
Limitations of Existing Weakly-Supervised Methods: Current weakly-supervised approaches use only image-level counts (total number of objects) as supervision. However, they are largely restricted to single-category counting (e.g., counting only people) and struggle with class-agnostic counting (counting arbitrary categories specified by text).
Limitations of Multimodal Large Language Models (MLLMs): While MLLMs offer a text-promptable interface for class-agnostic counting, they suffer from a modality gap. They perform reasonably well in sparse scenes but significantly underestimate counts in dense scenes due to a lack of exposure to dense object distributions during pre-training. Directly fine-tuning MLLMs to regress exact counts often fails to bridge this gap.

2. Methodology: WS-COC

The authors propose WS-COC, the first MLLM-driven framework for weakly-supervised class-agnostic object counting. Instead of directly fine-tuning the MLLM to predict a scalar count (which is difficult), the framework "bootstraps" the MLLM using three novel strategies:

A. Divide-and-Discern Dialogue Tuning (D3T)

Concept: Instead of asking the model to predict the exact number in one shot, the task is reformulated into a series of range-judgment questions via multi-round dialogue.
Mechanism:
1. Start with a broad range (e.g., $[1, 2000]$ ).
2. Ask the MLLM: "Are there more than $\tau$ objects?" where $\tau$ is the midpoint.
3. Based on the "Yes/No" response, narrow the range (e.g., if "Yes", new range is $[\tau+1, 2000]$ ).
4. Repeat until the range is sufficiently small, then ask for the final count.
Benefit: This curriculum learning approach guides the MLLM from easy (coarse) reasoning to hard (fine-grained) reasoning, mitigating the difficulty of direct regression.

B. Compare-and-Rank Count Optimization (CRCO)

Concept: To address the modality gap between high-dimensional visual features and discrete text tokens, the model is trained to understand relative ordering rather than absolute values.
Mechanism:
1. Stratified Sampling: Images are clustered into groups based on their ground-truth counts (e.g., sparse, medium, dense) to ensure diverse sampling.
2. Ranking Task: The model is presented with a set of images (e.g., 4 images) and asked to rank them in ascending order of object counts.
3. Optimization: The model is trained using language modeling loss to predict the correct ranking sequence.
Benefit: Learning relative differences is more visually intuitive for MLLMs than predicting absolute numbers, helping the model internalize quantity awareness.

C. Global-and-Local Counting Enhancement (GLCE)

Concept: A test-time inference strategy to correct the systematic underestimation in dense scenes.
Mechanism:
1. Global Prediction: Query the MLLM for the total count of the full image ( $c_g$ ).
2. Thresholding: If $c_g$ is below a threshold (e.g., 100), accept it. If above, the scene is considered "dense."
3. Local Partitioning: Split the image into a grid (e.g., $2 \times 2$ ) and query the MLLM for the count of each sub-image ( $c_k$ ).
4. Fusion: Sum the local counts ( $c_l$ ). Due to edge effects, $c_l$ often overestimates, while $c_g$ underestimates. The final prediction is the average of $c_g$ and $c_l$ .
Benefit: This leverages the complementary nature of global context (underestimation) and local detail (overestimation due to edge splitting) to achieve higher accuracy in dense crowds.

3. Key Contributions

First Weakly-Supervised Class-Agnostic Framework: WS-COC is the first method to achieve class-agnostic counting using only image-level supervision, eliminating the need for costly point-level annotations.
Novel Bootstrapping Strategies: The introduction of D3T (dialogue-based range reduction), CRCO (ranking-based optimization), and GLCE (global-local fusion) effectively bridges the gap between MLLM capabilities and dense counting requirements.
State-of-the-Art Performance: The method achieves performance comparable to or better than fully-supervised methods while using significantly less annotation data.
Efficiency: By using LoRA (Low-Rank Adaptation) for fine-tuning, the training cost is kept low despite using large MLLM backbones.

4. Experimental Results

The method was evaluated on four benchmarks: FSC-147, CARPK, PUCPR+, and ShanghaiTech.

Performance on FSC-147:
- WS-COC achieved a Test MAE of 13.91, outperforming the baseline MLLM-Zero (38.19) and the direct fine-tuning baseline WS-COC-Base (21.08).
- It surpassed several fully-supervised state-of-the-art methods (e.g., CountGD, T2ICount) and significantly outperformed other weakly-supervised methods (e.g., GCNet).
- Dense Scene Performance: In scenes with $>100$ objects, WS-COC reduced the MAE from 149.69 (MLLM-Zero) to 54.37, a massive improvement.
Cross-Dataset Generalization:
- WS-COC achieved the best results on PUCPR+ and ShanghaiTech (SHA) and competitive results on CARPK and SHB, demonstrating strong generalization to unseen domains.
Ablation Studies:
- Removing D3T increased MAE by ~3.2.
- Removing CRCO increased MAE by ~2.8.
- Removing GLCE increased MAE by ~1.8.
- The strategies are mutually reinforcing and essential for the final performance.
Cost Efficiency:
- Training time was only 3.44 hours on an NVIDIA L20 GPU, significantly lower than fully-supervised diffusion-based methods (e.g., CountGD took 11.73 hours).

5. Significance

This paper represents a significant step forward in efficient computer vision. By leveraging the reasoning capabilities of MLLMs and designing specific training/inference strategies (D3T, CRCO, GLCE), the authors demonstrate that weak supervision (image-level counts) is sufficient to train models that rival fully-supervised (point-level) methods.

Practical Impact: This drastically reduces the cost and time required to deploy object counting systems in real-world scenarios (e.g., counting cars in parking lots or people in crowds) where obtaining pixel-perfect annotations is infeasible.
Theoretical Insight: It highlights that MLLMs possess latent counting abilities that can be unlocked not by brute-force regression, but by aligning their reasoning processes (dialogue, ranking, decomposition) with the nature of the counting task.