Quantification and object perception in Multimodal Large Language Models and human linguistic cognition

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have demonstrated proficiency in various tasks, they struggle significantly with quantification (e.g., understanding terms like "some," "many," "most"). The exact nature of this failure remains unclear. Previous research has identified issues but has not systematically explored:

• Cognitive Mechanisms: Whether failures stem from poor object counting (numerosity estimation), incorrect mapping of proportions to linguistic terms, or a lack of understanding of scalar relationships.
• Model Architecture: Whether "thinking" models (reasoning models trained with Reinforcement Learning on verifiable problems) differ from standard "instruct" (general-purpose) models in their handling of quantification.
• Cross-Linguistic Robustness: Whether these limitations are specific to English or persist across typologically diverse languages (Greek, Russian, Spanish, Italian, Catalan).

The paper posits that human quantification relies on a complex interplay of Generalized Quantifier Theory (GQT), scalar implicatures (pragmatic enrichment), and the Approximate Number System (ANS) (human ability to estimate quantities). It investigates whether MLLMs replicate these human cognitive processes or rely on superficial statistical patterns.

2. Methodology

The study employed a controlled production task comparing human participants against five MLLMs across six languages.

• Participants & Models:
  • Humans: 240 participants (40 per language: English, Greek, Russian, Spanish, Italian, Catalan).
  • MLLMs: Five models representing different architectures and training paradigms:
    • Instruct (General Purpose): GPT-4o, Gemma 3, Molmo 2.
    • Thinking (Reasoning): o4-mini, GLM-4.6V.
• Task Design:
  • Stimuli: Images containing black squares and white circles (varying proportions of squares from 11% to 90%).
  • Procedure: Participants/Models were asked to:
    1. Select the best quantifier describing the image (e.g., "a few," "some," "many," "most").
    2. Estimate the approximate proportion of squares (e.g., "21%-30%").
  • Controls: Randomized image ordering to prevent positional bias; object size and color standardized; temperature set to 0 for consistency.
• Analysis:
  • Numerosity: Linear-mixed effects regression to compare estimated proportions against real proportions.
  • Quantifier Mapping: Multinomial logistic regression to analyze how quantifiers map to reported proportions.
  • Metrics: Focus on ordering (lexical scales), ranges of use (Interquartile Ranges - IQR), and prototypicality (Mode of usage).

3. Key Contributions

• Decoupling Numerosity from Semantics: The study uniquely separates the ability to count/estimate (ANS) from the ability to map estimates to language (semantic/pragmatic competence).
• Cross-Linguistic Scope: It is one of the first studies to test quantification in MLLMs across six languages, challenging the assumption that English performance generalizes globally.
• Architectural Comparison: It provides empirical evidence comparing "Thinking" (Reasoning) models against standard "Instruct" models, isolating the impact of Reinforcement Learning (RL) on semantic tasks.
• Theoretical Integration: It bridges Generalized Quantifier Theory (logic/set theory) with cognitive biases (ANS) to diagnose specific failure modes in MLLMs.

4. Key Results

A. Approximate Number System (Numerosity Estimation)

• Humans: Showed high accuracy in estimating the proportion of squares.
• Thinking Models (o4-mini, GLM-4.6V): Significantly outperformed instruct models and were closer to human performance. They showed a slight tendency to underestimate high proportions (>50%).
• Instruct Models (GPT-4o, Gemma 3, Molmo 2): Systematically underestimated the proportion of squares, particularly when the actual proportion was high.
• Conclusion: Reasoning training improves numerical estimation, but a gap remains between all models and humans.

B. Quantifier Production and Mapping

• Lexical Scales (Ordering):
  • Humans: Consistently ordered quantifiers as <a few, some, many, most>.
  • Thinking Models: Generally followed the human scale but exhibited cross-linguistic variations not seen in humans (e.g., GLM-4.6V in Greek/Catalan).
  • Instruct Models: Failed to consistently follow the human scale. For example, in GPT-4o (Spanish), "many" was used for lower proportions than "some," violating logical entailment.
• Ranges of Use & Prototypicality (The "Threshold" Problem):
  • Humans: "Most" was prototypically used for 91–100% of objects. "Many" had a broad range (IQR ~40%).
  • Models: Both model types showed compressed ranges and shifted thresholds.
    • Models used "most" for much lower proportions (51–70%).
    • Models failed to adhere to set-theoretic definitions (e.g., "most" should strictly imply >50%, but models used it inconsistently).
  • Statistical Significance: Multinomial regressions confirmed significant differences between humans and models across all languages, particularly for "many" and "some."

C. Cross-Linguistic Findings

• No Centralized Representation: Models did not exhibit a single, language-independent conceptual map for quantifiers. Performance varied by language, suggesting language-specific internal representations rather than a unified multilingual semantic core.
• Unexpected Difficulty: Contrary to linguistic acquisition theories (where "some" is learned early), "many" was the most difficult quantifier for models across all languages.
• English is Not Special: English was not the best-performing language for the models, contradicting the assumption that English is the "gold standard" for LLM semantic capabilities.

5. Significance and Discussion

• Limitations of Distributional Semantics: The results suggest that MLLMs, which rely on statistical co-occurrence, struggle with topic-independent words (like quantifiers and numerals) that require logical and numerical reasoning rather than just contextual correlation.
• Reasoning vs. Understanding: While "Thinking" models improve numerical accuracy via RL, they still fail to capture the pragmatic nuances (scalar implicatures) and prototypical ranges of human language. They can count better but still do not "understand" the meaning of "most" in the same way humans do.
• Semantic Agents: The paper argues that current MLLMs are not robust semantic or pragmatic agents. Their quantification abilities are fragile, model-dependent, and language-dependent, lacking the cognitive grounding found in human ANS and pragmatic reasoning.
• Future Directions: The findings highlight the need for training paradigms that incorporate intra-speaker variability and explicit logical constraints, rather than relying solely on next-token prediction or RL on verifiable math problems.

In summary, the paper demonstrates that while MLLMs are improving at counting, they fundamentally lack the human-like cognitive architecture required to map numerical estimates to linguistic quantifiers with the necessary precision, logical consistency, and pragmatic flexibility.