🤖 AI

SITUATE -- Synthetic Object Counting Dataset for VLM training

The paper introduces SITUATE, a novel synthetic dataset designed to train Vision Language Models on spatially constrained counting tasks, demonstrating that fine-tuning on this controlled data significantly improves generalization on out-of-distribution benchmarks compared to existing real-world datasets.

Original authors: René Peinl, Vincent Tischler, Patrick Schröder, Christian Groth

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: René Peinl, Vincent Tischler, Patrick Schröder, Christian Groth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant that is great at describing pictures. If you show it a photo of a sunset, it can tell you about the orange sky and the clouds. But if you ask it, "How many birds are in that sky?" it often starts guessing, getting the number wrong, or making up facts. It's like a student who is great at writing essays but terrible at basic math.

This paper introduces a new training tool called SITUATE to help these robots get better at counting, especially when objects are hidden, colored differently, or placed in specific spots.

Here is the breakdown of their work using simple analogies:

The Problem: Robots Can't Count Well

Current AI models are like people who can recognize a face instantly but struggle to count a crowd.

The "Pattern" Trap: Some existing training data is like a trick quiz. If a picture always shows nine items arranged in a specific shape, the robot learns to recognize the shape of "nine" rather than actually counting the items. It's like a student memorizing the answer key instead of learning the math.
The "Real World" Mess: Other datasets use real photos, but they are too messy. Objects are hidden behind others (occlusion), or the questions are vague. It's like asking someone to count the apples in a fruit bowl while someone else is constantly moving them around.
The Result: The robots guess. They might say a hen has three legs because they "remember" seeing a weird hen once, or they fail to count green berries if the picture is slightly blurry.

The Solution: A "Training Gym" Called SITUATE

The authors built a new dataset called SITUATE (Synthetic Object Counting Dataset). Think of this as a virtual training gym for robots, built inside a 3D computer program (Blender).

Instead of using messy real photos, they created perfect, clean 3D scenes:

The Setup: Imagine a room with a table in the middle.
The Objects: They place geometric shapes (cubes, spheres, cones) on, under, or around the table.
The Rules: They control everything. They know exactly how many red cones are on the left, how many blue spheres are under the table, and they ensure the objects don't get too hidden.
The Questions: They ask the robot specific questions like, "How many green cones are on the floor to the right of the table?"

This is like giving a student a math workbook where the problems are perfectly clear, so they can actually learn the concept of counting rather than just memorizing pictures.

The Experiment: Teaching the Robot

The researchers took a popular AI model (Qwen 2.5 VL) and gave it a "crash course" using their new SITUATE gym. They tried different teaching styles:

The "Verbose" Style: The robot was taught to talk through its thinking step-by-step (e.g., "I see two cones here, three there...").
The "Non-Verbose" Style: The robot was taught to just give the final number.
The "Mixed" Style: A combination of their new gym and an existing dataset (Pixmo).

The Results: What Worked?

The "Verbose" Style was a double-edged sword: When the robot was asked to count large numbers (5 or more), talking through the steps helped it get the answer right. However, for small numbers, it started "hallucinating" (making things up) to fill in the gaps. It was like a student who tries so hard to explain their work that they accidentally invent extra numbers.
The "Mixed" Approach Won: The best result came from combining the new SITUATE gym with the existing Pixmo dataset. This created a robot that could handle both simple and complex counting tasks better than before.
Generalization: The most important finding is that training on this clean, synthetic gym actually helped the robot get better at counting in messy, real-world photos (like the TallyQA dataset). It's like practicing with a perfect basketball hoop in a gym and then suddenly becoming better at shooting hoops in a windy park.

The Conclusion

The paper argues that to teach robots to count accurately, we need a middle ground between "too simple" (2D cartoons) and "too messy" (real photos). The SITUATE dataset provides that middle ground.

By training on these controlled 3D scenes, the robots learned to actually count rather than just guess based on patterns. The authors plan to make the gym even more realistic in the future by adding objects like cups, balls, and candles, bridging the gap between their perfect 3D world and the real world even further.

In short: They built a clean, controlled 3D playground to teach AI how to count properly, and it turned out that practicing in this playground made the AI smarter at counting in the real world, too.

Technical Summary: SITUATE - Synthetic Object Counting Dataset for VLM Training

Problem Statement

Vision-Language Models (VLMs) have achieved significant success in image captioning and visual question answering but continue to struggle with precise quantitative reasoning, specifically object counting and spatial relationship understanding. Current models often fail to translate visual recognition into accurate numbers, particularly when counts exceed five or when objects are occluded. Existing datasets present a dichotomy: simple 2D benchmarks like CountBench allow models to learn visual patterns rather than true counting (e.g., recognizing a specific pattern for the number nine), while real-world datasets like TallyQA and Pixmo count suffer from ambiguity, heavy occlusion, and a lack of control over spatial composition. Furthermore, models trained on these datasets often exhibit biases, failing to generalize when object counts deviate from training distributions or when specific attributes (like color) are queried. There is currently no unified approach that addresses counting and spatial understanding simultaneously in a controlled yet challenging environment.

Methodology

Dataset Construction (SITUATE)

The authors introduce SITUATE, a novel synthetic dataset designed to bridge the gap between artificial 2D graphics and uncontrolled real-world photos.

Generation Pipeline: The dataset was generated using Blender and BlenderProc, leveraging a Python API for procedural generation.
Scene Composition: Scenes consist of a table in a room with varied backgrounds and floor textures. Objects (cubes, spheres, cones, cylinders) are placed on, under, in front of, or to the left/right of the table.
Controlled Complexity: To ensure clear visual understanding while maintaining challenge, the table's horizontal extent is divided into bins to prevent object overlap and excessive occlusion.
Quality Control: A validation process ensures sufficient contrast between objects and backgrounds using the $\Delta E$ color difference metric in Lab-colorspace. Images with $\Delta E < 12.5$ are re-rendered.
Data Scale: The dataset contains 23,252 image-question-answer triplets (derived from 6,875 unique images rendered from 5 camera perspectives). It focuses on object counts ranging from 0 to 15, with a specific emphasis on the 5–15 range.
Question Types: Six question types are generated per image: color, shape, location, object, composite, and adversarial (asking for non-existent objects).
Ground Truth: Three ground truth formats are provided: numeric, short, and verbose (step-by-step reasoning).

Experimental Setup

To evaluate the dataset's efficacy, the authors fine-tuned the Qwen 2.5 VL 7B model using LoRA (Low-Rank Adaptation) with the UnSloth framework.

Training Variants: Four variants were created:
1. Verbose: Trained on SITUATE with verbose ground truths.
2. Non-verbose: Trained on SITUATE with short ground truths.
3. Pixmo: Trained on a subset of the Pixmo count dataset (matched for ground truth distribution).
4. Mixed: A combination of SITUATE (verbose) and Pixmo.
Training Parameters: Models were trained for one epoch on 23,252 items with a batch size of 4, using AdamW 8-bit optimizer, a learning rate of 1e-4, and a warmup of 300 steps.
Evaluation: Models were tested on held-out SITUATE data, Pixmo count, CountBench, and a filtered subset of TallyQA (focusing on counts 5–15).

Key Results

Performance on SITUATE

Baseline: The untrained Qwen 2.5 VL 7B achieved 36.3% accuracy on the SITUATE test set.
Fine-tuning Impact:
- Verbose SITUATE: Improved accuracy to 50.3%.
- Mixed (SITUATE + Pixmo): Achieved the highest accuracy at 52.7%.
- Non-verbose & Pixmo-only: Both variants resulted in lower accuracy on SITUATE (29.5% and 27.3%, respectively) compared to the baseline, suggesting that training solely on real-world data or non-verbose synthetic data does not transfer well to the specific spatial constraints of SITUATE.

Generalization and Cross-Dataset Performance

Pixmo Count: The Mixed variant achieved 60.0% accuracy, a significant improvement over the baseline (51.8%) and the SITUATE-only variants. This indicates that SITUATE training helps generalize to real-world data.
CountBench: The baseline performed well (81.9%). The Mixed variant maintained this performance (81.9%), while the Pixmo variant improved it slightly to 85.1%.
TallyQA: All models struggled due to the dataset's inherent ambiguity and occlusion. The Mixed variant achieved 27.5%, a slight improvement over the baseline (25.5%).
Comparison with Other Models:
- Molmo 7B-D: Performed exceptionally well on Pixmo (87.1%) and CountBench (92.7%) but was outperformed by the Qwen 3 VL 32B on SITUATE (79.2% vs. 47.4% for Molmo).
- Qwen 3 VL 32B: Showed the best generalization capabilities across all benchmarks, particularly on SITUATE, though it was not fine-tuned on the specific datasets used in this study.

Qualitative Observations

Verbose vs. Non-Verbose: The verbose training style improved performance for counts of 5 and above but led to hallucinations for smaller numbers (e.g., inventing objects to sum up to a target).
The "Six" Anomaly: The baseline model showed a distinct drop in accuracy for the number six (19%) compared to five (39%) and seven (35%). The Mixed dataset successfully mitigated this, raising accuracy for six to 61%.
Adversarial Questions: The non-verbose model overfit to adversarial questions, frequently claiming zero objects were present, whereas the baseline handled these correctly.

Significance and Claims

The paper claims that SITUATE fills a critical gap in VLM training by providing a controlled environment that enforces spatial reasoning and precise counting without the ambiguity of real-world photos.

Generalization: The primary finding is that fine-tuning on SITUATE (specifically the Mixed variant) improves a model's ability to generalize to out-of-distribution real-world data (Pixmo count), whereas fine-tuning on Pixmo alone does not improve performance on SITUATE.
Learning Mechanism: The study suggests that VLMs can learn to count more effectively when trained on data that explicitly separates object placement and attributes, rather than relying on statistical biases found in real-world datasets.
Future Utility: The authors posit that SITUATE serves as a valuable benchmark for evaluating the "true" counting capabilities of VLMs, distinguishing between pattern recognition and actual quantitative reasoning.

Limitations

The authors acknowledge several limitations:

Synthetic Nature: The dataset is limited to four geometric object classes and a single scene layout (table in a room), though camera angles and textures vary.
Model Scope: Experiments were limited to fine-tuning a single model (Qwen 2.5 VL 7B) for a single epoch; results on other architectures or longer training schedules were not explicitly tested.
Human Baseline: No human baseline was established, though the authors estimate human accuracy would be 90–95% given the controlled nature of the images.
Scope of Objects: Current objects are abstract; future work aims to introduce photorealistic textures and more complex object relations (e.g., cups, balls).