🤖 AI

NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models

The paper introduces NumerosityVLM, a cognitively inspired benchmark with 10,800 controlled synthetic images that reveals Vision-Language Models' numerosity perception is primarily determined by their architecture and language components rather than visual conditions, with linearly separable numerosity signals emerging early in the vision encoder.

Original authors: Yiming Fu, Fangjun Li, Xiujin Liu, Ruidong Ma, Hang Yu, Zhichen Lu, Kanwei He, Alessandro Di Nuovo, Angelo Cangelosi, Zhegong Shangguan

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiming Fu, Fangjun Li, Xiujin Liu, Ruidong Ma, Hang Yu, Zhichen Lu, Kanwei He, Alessandro Di Nuovo, Angelo Cangelosi, Zhegong Shangguan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Human beings possess a remarkable, innate ability to recognize how many items are in a group without counting them one by one. A baby can tell the difference between a pile of three crackers and a pile of four before they ever learn to speak, relying on a sense of number that emerges naturally in the developing brain. This skill, known as numerosity perception, is a fundamental part of how we understand the world. In recent years, scientists have turned their attention to artificial intelligence, specifically a type of computer system called a vision-language model. These systems are designed to look at images and answer questions about them, often performing with human-like fluency on complex tasks. However, a lingering question remains: do these machines truly understand the concept of "how many," or are they simply guessing based on visual patterns they have memorized?

To answer this, a team of researchers created a new testing ground called NumerosityVLM. They built a set of 10,800 computer-generated images designed to strip away the tricks that might fool a machine. In the real world, counting is often confused by other visual clues. For instance, a group of large objects might take up more space than a group of small objects, leading a system to think there are more of the large ones simply because they cover more area. To prevent this, the researchers controlled every detail of their images. They created scenarios where the number of items changed while the total space they occupied stayed the same, and vice versa. They also removed texture, shape, and color in stages, leaving behind simple dots, to see if the models relied on the look of the objects or the actual count. They tested seven different open-source models, ranging from smaller, older systems to larger, more advanced ones, asking them to count the items in each image.

The results revealed a clear divide between the models. The more advanced systems, which had larger internal structures, performed significantly better, achieving high accuracy even when the visual clues were misleading. The smaller, older models struggled, often failing to distinguish between different quantities and producing the same limited set of answers regardless of what was in the picture. The researchers found that the biggest factor determining success was not the visual conditions of the image, but the architecture of the model itself. The way the machine was built mattered far more than the specific visual tricks used in the test. This suggests that the ability to count is not just a matter of seeing clearly, but of having the right internal structure to process that information.

Digging deeper into how these models worked, the team looked at the different stages of the machine's processing pipeline. They discovered that the ability to distinguish numbers actually appears very early, right when the machine first looks at the image. Even the less successful models could detect the number of items in their initial visual layers. The problem, it turned out, happened later. The difference between a model that could count well and one that could not lay in how it translated those visual signals into words. The more successful models were better at taking the numerical information they had already seen and converting it into a correct text answer. The less successful models seemed to lose that information as they moved from seeing to speaking.

The study also examined how these machines handle the scale of numbers. The best-performing models were nearly perfect when counting small groups of up to four items, a range humans can recognize instantly. As the numbers grew larger, their accuracy dropped, and they began to consistently guess lower than the actual count. This pattern of underestimation grew stronger as the numbers increased, mirroring a known limitation in human perception where estimating large quantities becomes less precise. However, the most advanced models showed this bias much later than the weaker ones, suggesting they had a more robust grasp of larger numbers.

Ultimately, the research indicates that current vision-language models do possess a form of numerical understanding, but it is fragile and heavily dependent on their design. The ability to see "how many" is present in the early visual processing of these systems, but the final step of turning that perception into a reliable answer is where the machines often fail. The study concludes that the gap in performance is not due to a lack of visual data, but rather to how different models are built to interpret and express that data. By isolating these factors, the researchers have provided a clearer map of where artificial intelligence stands in its journey toward genuine numerical cognition, showing that while the eyes of the machine can see the number, the mind of the machine must still learn to count.

Technical Summary: NumerosityVLM

Problem Statement

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in high-level multimodal tasks such as visual question answering and scene understanding. However, their fundamental perceptual abilities, specifically numerosity perception (the ability to estimate or count quantities), remain poorly understood. While human infants develop numerosity perception before language acquisition, current VLMs struggle with this task. Existing counting benchmarks suffer from two primary limitations:

Limited Range: General-purpose benchmarks often cover only small numerosity ranges, failing to test the Approximate Number System (ANS).
Confounded Variables: Real-world datasets entangle numerosity with correlated visual factors such as object size, density, and total surface area. This makes it difficult to attribute counting failures to specific deficiencies in visual perception, language decoding, or reliance on shortcut cues.

Consequently, current benchmarks cannot reliably distinguish whether a model fails due to an inability to perceive visual quantities or an inability to map those perceptions to textual outputs.

Methodology

To address these limitations, the authors introduce NumerosityVLM, a cognitively inspired diagnostic benchmark comprising 10,800 synthetic images generated under strictly controlled conditions.

Benchmark Design

The dataset is constructed on a 1024 × 1024 pixel canvas with non-overlapping objects. It features three object categories (apples, butterflies, human figures) and 12 numerosity levels ranging from 1 to 100 (covering both subitizing and ANS regimes). The benchmark employs six controlled experimental conditions:

Orthogonal Control Conditions:
- Baseline: Preserves natural covariance between numerosity and total area.
- Size-Incongruent: Total surface area is fixed; individual object size varies inversely with numerosity.
- Space-Incongruent: Individual object size is fixed; spatial density increases with numerosity within a fixed convex hull.
Visual Cue Ablation Conditions:
- Texture-Ablated: Objects replaced by simplified shape templates.
- Shape-Ablated: Objects replaced by solid-colored dots.
- Color-Ablated: Colored dots converted to grayscale.

Experimental Setup

The study evaluates seven representative open-source VLMs from four architectural families (CLIP, LLaVA, InternVL, Qwen3-VL) in a zero-shot setting. The evaluation utilizes both open-ended and closed-ended prompting strategies.

Analysis Techniques

Multi-Factor Attribution: A three-way ANOVA is performed on image-level accuracy to quantify the contribution of Model, Condition, and Subset to performance variance using partial $\omega^2$ .
Systematic Bias Analysis: Hierarchical comparisons assess robustness against visual factors (size, space, texture, shape, color) and magnitude-dependent biases (residual analysis).
Layer-wise Probing: Linear Support Vector Machines (SVMs) are trained on hidden representations at every stage of the VLM pipeline (vision encoder, alignment module, language model) to determine where numerosity signals emerge and remain linearly separable.
Numerical Geometry Analysis: Principal Component Analysis (PCA) is applied to language backbone representations to examine the geometric structure of numerical encoding (linear vs. logarithmic compression).

Key Results

1. Performance Variance and Architecture

Multi-factor analysis reveals that model architecture is the dominant factor explaining performance variance (partial $\omega^2 = 0.325$ ), far exceeding the influence of visual conditions (partial $\omega^2 = 0.005$ ) or object subsets.

Tier-1 Models: Recent, larger architectures (e.g., Qwen3VL-8B, InternVL3-8B) achieve high accuracy (~77–84%) and low error magnitudes.
Tier-2 Models: Older or smaller architectures (e.g., CLIP, LLaVA-1.5) perform near random baseline levels (~21%) with high error magnitudes.

2. Visual Robustness and Bias

Size vs. Space: Models perform significantly better under the Size-Incongruent condition (fixed total area) compared to the baseline, suggesting they rely heavily on total area cues. However, performance does not significantly change under the Space-Incongruent condition (fixed convex hull), indicating a lack of sensitivity to spatial density changes in the tested models.
Cue Ablation: Removing texture significantly degrades performance for Tier-1 models relative to the baseline (dropping to 72.05% accuracy). However, further removing shape and color cues progressively improves their accuracy, reaching 77.71% under shape ablation and 80.92% under color ablation. Tier-2 models show no significant improvement across ablation conditions, suggesting they fail to utilize these cues effectively regardless of complexity.
Magnitude Bias: All models exhibit systematic underestimation for numerosities greater than 50. Tier-1 models maintain high accuracy in the subitizing range (1–4) and the lower ANS range (5–20) but degrade as magnitude increases.

3. Mechanistic Insights (Layer-wise Probing)

Vision Encoder: Linearly separable numerosity signals consistently emerge in the early layers of the vision encoder. By the end of the vision encoder, both high-performing and low-performing models encode robust numerosity representations. This indicates that visual feature extraction is not the primary source of performance disparity.
Language Model: The critical performance gap arises during the language modeling and decoding stages. Tier-1 models retain structured numerosity information through the decoding process, whereas Tier-2 models degrade rapidly.
Numerical Geometry: Tier-1 language backbones exhibit a highly ordered, logarithmically compressed numerical geometry (high Spearman correlation and Log $R^2$ ), aligning with human ANS characteristics. Tier-2 backbones show weaker ordering and less consistent scaling.

Key Contributions

NumerosityVLM Benchmark: A cognitively inspired diagnostic tool with orthogonal control over object size, spatial arrangement, and numerosity, alongside progressive cue ablation, enabling systematic diagnosis of VLM perception.
Attribution of Performance Variance: Demonstration that architectural design and multimodal integration strategies account for the majority of counting performance differences, rather than visual conditions or object categories.
Mechanistic Localization: Evidence that numerosity signals are linearly decodable in early vision layers across all models, while performance differences are primarily associated with the language model's ability to map latent representations to textual outputs.

Significance and Limitations

The paper claims that NumerosityVLM provides a necessary framework for moving beyond general performance metrics to a mechanistic understanding of VLM limitations. The findings suggest that the "counting failure" in many VLMs is not a failure of visual perception but a failure of representation-to-output alignment in the language component.

The authors acknowledge limitations:

The reliance on synthetic stimuli with strict factor isolation may limit generalization to complex real-world scenes.
The probing results provide correlational rather than causal evidence regarding the internal mechanisms.

Future work is proposed to extend the framework to real-world settings and explore causal intervention methods.