🤖 machine learning

The MixCount Dataset: Bridging the Data Gap for Open-Vocabulary Object Counting

The paper introduces MixCount, a large-scale dataset and benchmark for mixed-object counting generated via an automatic pipeline that synthesizes diverse images with pixel-perfect annotations, demonstrating that training on this synthetic data significantly improves state-of-the-art models' performance on real-world counting tasks by addressing the limitations of existing noisy or scarce datasets.

Original authors: Corentin Dumery, Niki Amini-Naieni, Shervin Naini, Pascal Fua

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Corentin Dumery, Niki Amini-Naieni, Shervin Naini, Pascal Fua

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to count items in a messy room. If you only show the robot pictures of a single type of item—like a bowl full of identical red apples—it learns to count apples very well. But the moment you throw in a mix of red apples, green pears, and yellow lemons all jumbled together, the robot gets confused. It might count the pears as apples, or get distracted by the pattern on the tablecloth and start counting the tablecloth instead of the fruit.

This is the problem the MixCount paper addresses. The authors argue that current "counting robots" (AI models) are failing in real-world scenarios because they haven't been trained on the right kind of messy, mixed-up data.

Here is a breakdown of their solution, using simple analogies:

1. The Problem: The "Toy Box" vs. The "Real World"

For years, researchers trained these AI models using datasets that were like perfectly organized toy boxes.

Real-world data is expensive and noisy: Taking photos of real factories or markets and having humans count every single item is slow, expensive, and humans make mistakes (like missing a hidden item).
Old synthetic data was too simple: Previous attempts to create fake data were like drawing simple stick figures. They lacked the complexity, lighting, and clutter of the real world.

Because of this, the AI models are like students who only studied for a test using a single practice sheet. When the actual exam (the real world) has a mix of questions, they fail.

2. The Solution: The "Infinite Simulator"

The authors built a digital factory (a data generator) that acts like a hyper-realistic video game engine. Instead of taking photos, they build 3D scenes on a computer.

The Ingredients: They use real-world 3D scans of objects (like a specific teapot or a toy dinosaur) and real-world textures (like wood grain or metal).
The Process: They drop these objects into a virtual room, turn on realistic lights, and let physics simulate how they fall, roll, and pile up. Some objects might be partially hidden behind others, just like in real life.
The Magic: Because it's a computer simulation, they know exactly how many objects are in the scene, where they are, and what they look like. There are no human counting errors. They can generate 58,000 unique scenes with 4 million objects automatically.

Think of it like a chef who can instantly cook 50,000 different variations of a complex stew, knowing exactly how much salt and pepper is in every single bowl, without ever tasting a spoonful.

3. The "MixCount" Dataset

The result of this factory is the MixCount dataset. It's a massive library of images where:

Everything is mixed: You see different types of objects together (e.g., teapots next to marbles).
It's tricky: There are repetitive backgrounds (like a patterned rug) that try to trick the AI.
It has "cheat sheets": For every object, the dataset provides:
- Text descriptions: Ranging from short ("blue teapot") to very detailed ("shiny blue cast iron teapot with a curved handle").
- Visual examples: A picture of the object to show the AI exactly what to look for.

4. The Results: Training the Robot

The authors tested this by taking the best existing counting robots and giving them a crash course using MixCount.

Before training: The robots struggled to tell the difference between similar-looking items or ignored the background clutter.
After training: The robots got significantly smarter.
- On a standard test for counting sunglasses, the error rate dropped by 18%.
- On a test for counting various household items, the error rate dropped by 20%.

Essentially, by training on this "perfectly labeled, infinitely diverse" synthetic data, the robots became much better at handling the messy, mixed-up reality of the real world.

Summary

The paper claims that the biggest bottleneck in teaching computers to count mixed objects isn't the algorithm itself, but the lack of good training data. By building a machine that can generate unlimited, perfectly accurate, and photorealistic "messy" scenes, they were able to teach existing AI models to count much more accurately than ever before. They call this new dataset MixCount.

Technical Summary: The MixCount Dataset

Problem Statement

Despite over a decade of research, state-of-the-art visual object counting models systematically fail in mixed-object settings, a scenario dominant in real-world applications like industrial inspection and product sorting. Current models struggle to distinguish between visually similar objects, identify self-similar components as single entities, or ignore repetitive background patterns.

The authors attribute this performance gap primarily to limitations in existing training and evaluation data:

Real-world datasets (e.g., FSC-147, PairTally) are prohibitively expensive to annotate, suffer from labeling noise, and often lack images with multiple object types annotated simultaneously.
Existing synthetic datasets (e.g., MCAC, SITUATE) lack diversity and photorealism, creating a significant domain gap between training samples and real-world images.
Diffusion-based generation is ineffective for counting tasks due to the inability to synthesize images with a controlled, precise number of objects.

Methodology

To bridge this data gap, the authors introduce MixCount, a large-scale, photorealistic dataset and a corresponding automatic data generation pipeline.

The Data Generation Pipeline

The pipeline synthesizes training samples by combining real-world captures with physically-based rendering (PBR) in Blender. Key components include:

Assets: Objects are sampled from the Digital Twin Catalog (DTC), a dataset of high-quality real-world 3D captures.
Scene Construction: The system procedurally generates scenes by sampling 2–4 object classes per scene. Objects are spawned at random 3D positions with slight scale modifiers to ensure visual similarity within classes but distinctness between instances.
Physical Simulation: Objects are simulated using rigid-body dynamics to naturally form realistic scenes with touching and partial occlusions.
Visibility Filtering: Unlike previous synthetic datasets, MixCount explicitly measures object visibility. It renders both occluded and unoccluded masks to calculate a visibility ratio ( $v_i$ ). Scenes with objects falling below a visibility threshold ( $v_t$ ) are rejected, ensuring pixel-perfect ground truth where all counted objects are visible.
Environment & Lighting: The pipeline uses real-world environment maps (from LavalIndoor and Polyhaven) and PBR materials to create complex lighting and repetitive textures that challenge counting models. Distractor objects are added to increase scene complexity.
Prompts & Annotations:
- Text: Each object is annotated with three levels of text descriptions (short, concise, detailed) generated by an LLM (Gemini 2.5), ensuring unique identifiers.
- Exemplars: The dataset provides both internal exemplars (bounding boxes within the image) and external exemplars (canonical views on white backgrounds). Internal exemplars are assigned a visibility/size score.
- Dense Ground Truth: Annotations include bounding boxes (COCO format), instance and class segmentation masks, depth maps, and normal maps.

The MixCount Dataset

The released dataset contains 58,000 images with approximately 4 million objects to count.

Scale & Diversity: It features 1,522 different object classes spanning various categories.
Composition: On average, scenes contain 68 objects from 2–4 different classes.
Splits: The dataset is divided into train (56,000), validation (1,000), and test (1,000) sets. Crucially, object classes are split such that no object in the test set appears in the training set, ensuring a rigorous evaluation of generalization.

Key Contributions

MixCount Dataset: The first large-scale, photorealistic, and diverse synthetic dataset designed specifically for training and benchmarking open-vocabulary, mixed-object counting models.
Data Generation Framework: A fully automated pipeline that synthesizes effectively unlimited, pixel-perfect labeled data, capable of being tailored to specific applications (e.g., industrial inspection) by swapping 3D assets.
Rigorous Benchmarking: A comprehensive evaluation of state-of-the-art models (CountSE, CountGD, CountGD++) on the MixCount test set, exposing severe performance degradation in mixed-object settings.

Results

The authors evaluated the utility of MixCount through two primary experiments:

1. Training Value (Transfer to Real Data)

Training state-of-the-art models on MixCount significantly improved performance on real-world benchmarks:

FSC-147: Training with MixCount reduced the Mean Absolute Error (MAE) by 20.14%.
PairTally: Training with MixCount reduced the MAE by 18.3%.
The authors note that even adding a small number of MixCount images (1,000) to the training set yielded substantial benefits, highlighting the quality of the synthetic supervision.

2. Benchmarking (Performance on MixCount)

When evaluated on the MixCount test set, existing models trained only on FSC-147 performed poorly, with the best baseline (CountGD++) achieving an Accuracy within 10% (Acc10%) of only 0.33.

Models trained on MixCount showed significant improvement, with CountGD++ (+ MixCount) achieving an Acc10% of 0.69 and reducing Normalized Absolute Error (NAE) to 0.14.
The results highlight that current models struggle without targeted training data for mixed-object scenarios.

Sensitivity Analysis

The paper also analyzes model sensitivity to prompt types:

Exemplars: Performance drops significantly when switching from internal exemplars (context-aware) to external exemplars (context-independent).
Text Prompts: Models performed best with "concise" prompts, suggesting that overly detailed descriptions may deviate from training distributions or that "short" prompts were the primary training signal.

Significance and Claims

The paper positions MixCount as a solution to a long-standing bottleneck in visual counting: the lack of scalable, high-quality, mixed-object training data.

Bridging the Gap: The authors claim MixCount effectively bridges the gap between synthetic scalability and real-world complexity, enabling models to learn fine-grained distinctions that were previously impossible to teach at scale.
Future Utility: The dataset is presented not just as a benchmark, but as a training resource that allows researchers to explore prompt sensitivity, exemplar robustness, and fine-grained counting capabilities.
Accessibility: By providing a generator that can produce "effectively unlimited" labeled data, the work aims to lower the barrier for developing robust counting systems for specific, data-scarce domains like industrial inspection.

The authors conclude that while synthetic data has distributional differences from real data, the MixCount pipeline successfully addresses the specific failure modes of current models, leading to measurable gains in real-world performance.