Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

This paper argues that the reasoning limitations of Vision-Language Models stem from a reporting bias in training data that omits tacit information, demonstrating that simply scaling model or data size cannot overcome this issue and that intentional curation of annotations capturing such details is necessary for improvement.

The Gist

The Big Idea: The "Lazy Describer" Problem

Imagine you are trying to teach a robot how to understand the world by showing it millions of photos and the captions people wrote for them. You might think, "If I show the robot enough photos, it will eventually figure out everything."

This paper argues that quantity isn't enough. Even if you give the robot a billion photos, it will still be bad at certain tasks (like counting, understanding space, or knowing what didn't happen).

Why? Because of Reporting Bias.

Think of it like this: When humans describe a photo, we are naturally "lazy" or "efficient." We only say what is necessary to get the point across. We don't usually say, "There are exactly 17 people standing behind the field," unless it's a crime scene. We just say, "A game today!"

The robot learns from these lazy descriptions. So, it never learns that "17 people" is a specific fact, or that "behind" is a specific spatial relationship. It just learns the vibe.

The Four Skills the Robot Missed

The researchers found that human captions naturally skip over four specific types of thinking:

1. Counting: We rarely say "three dogs." We say "a pack of dogs." The robot never learns to count because it's rarely asked to.
2. Spatial Reasoning: We rarely say "the cup is to the left of the plate." We just say "a cup and a plate." The robot doesn't learn the difference between left and right.
3. Negation: We rarely say "There are no parrots in this picture." We just describe what is there. The robot struggles to understand what is missing.
4. Time: We rarely say "The ball will fall after the throw." We just describe the current moment. The robot gets confused about cause and effect.

The "More Data" Myth (Scaling)

For a long time, AI researchers believed in the "Scaling Law": If you just make the model bigger and feed it more data, it will magically become smart.

The paper says: No.

• The Analogy: Imagine you are trying to teach a student how to solve a math problem, but you only give them textbooks that say "The answer is 42" without showing the steps.
• The Result: If you give that student 1,000 textbooks, they still won't know how to get to 42. They just memorize the answer.
• The Finding: The researchers tried feeding the AI more data, bigger models, and even data in different languages. The AI got slightly better at recognizing things (like "is this a cat?"), but it did not get better at reasoning (like "how many cats?"). The "lazy" way humans write captions didn't change just because the dataset got bigger.

The Solution: Changing the Instructions

If the problem is that humans are too lazy to write detailed captions, the solution is to tell them to stop being lazy.

The researchers ran an experiment where they gave human annotators specific instructions, like:

• "Please count exactly how many objects are in this picture."
• "Describe exactly where the objects are relative to each other."
• "Mention what is not in the picture."

The Result:
When people were explicitly told to include these details, the number of "reasoning" captions skyrocketed.

• Without instructions: 2% of captions had counting details.
• With instructions: 39% of captions had counting details.

When they took this new, "intentional" data and taught the AI, the AI got much better at reasoning.

The "Synthetic" Trap

The paper also found a funny twist: Even when AI (like GPT-4) writes captions for other AI, it copies the same bad habits! Since the AI was trained on human text, it also learned to be "lazy" and skip the details. So, just using AI to generate more data doesn't fix the problem unless you give the AI very strict instructions.

The Takeaway

You can't just "scale" your way to intelligence.

If you want an AI to be good at reasoning, you can't just throw more data at it. You have to be intentional. You have to curate the data carefully and tell the people (or other AIs) writing the descriptions exactly what details to include.

In short: Don't just feed the robot a billion photos of "a game." Tell the robot, "Look at this photo, count the players, tell me who is on the left, and tell me who is missing." That's how you teach it to think.

Technical Summary

1. Problem Statement

Despite impressive performance on standardized benchmarks, Vision-Language Models (VLMs) consistently fail at specific reasoning tasks, including counting, spatial reasoning, negation, and temporal reasoning. The prevailing hypothesis in the field is that these failures stem from insufficient model scale or architecture.

The authors posit a different cause: Reporting Bias. This is a phenomenon rooted in linguistics and pragmatics where human annotators systematically omit "tacit" information when describing images. For example, people naturally say "a cat and a dog" rather than "a cat to the left of a dog" because the spatial relationship is often considered obvious or irrelevant to the conversational context (Gricean Maxims). Consequently, even web-scale datasets (e.g., LAION, LLaVA-1.5) lack sufficient training examples for these reasoning skills, leading models to fail at them despite massive data sizes.

2. Methodology

The research follows a four-step methodology to validate the hypothesis and propose a solution:

A. Theoretical Analysis & Data Audit

The authors analyzed three major open-source image-text corpora (LAION-2B, LLaVA-1.5, and Molmo) to quantify the frequency of reasoning-related keywords (e.g., "left of," "before," "no," "five").

• Metric: They calculated the raw occurrence of keywords and manually verified the "True Occurrence" (instances where the keyword actually operationalizes the reasoning logic in the image).
• Finding: Reasoning concepts are extremely rare. For instance, spatial reasoning keywords constitute only 0.1% of LAION-2B, and true negation instances are even lower.

B. Benchmark Creation

To isolate these reasoning skills, the authors curated four specific evaluation benchmarks using multiple-choice or direct-answer formats:

1. Spatial: Identifying relationships (on, under, left of, right of) between two objects.
2. Counting: Determining the exact number of objects (2–10).
3. Negation: Identifying what an object is not (e.g., "a bear that is not flying").
4. Temporal: Distinguishing between "before" and "after" states of an event.

C. Scaling Law Experiments

The authors evaluated a wide range of models (Contrastive: OpenCLIP variants; Generative: LLaVA-1.5, Molmo, Qwen, GPT-4o, etc.) on these benchmarks.

• Scale Analysis: They tested OpenCLIP models with varying parameter sizes (ViT-B to ViT-H) and training data sizes (80M to 2B images).
• Multilingual Analysis: They tested if adding translated non-English captions (to increase diversity) mitigated the bias.

D. Annotator Instruction Study (User Study)

To prove that the lack of reasoning data is due to omission rather than impossibility, the authors conducted a controlled user study:

• Setup: 100 annotators captioned the same 100 images from COCO under four different instruction sets:
  1. Standard COCO instructions (no specific reasoning prompts).
  2. LLaVA-1.5 instructions (prompted for counts/positions).
  3. PixMo instructions (prompted for counts/positions).
  4. Custom Instructions: Explicitly prompting for spatial, temporal, counting, and negation.
• Verification: They measured the "True Occurrence" of reasoning concepts in the resulting captions.

E. Fine-tuning Validation

To test if increasing reasoning data density improves performance, they fine-tuned LLaVA-1.5-13B on a dataset where they artificially increased the proportion of counting data (from ~7% to 39%) using their custom instructions.

3. Key Results

A. Prevalence of Reporting Bias

• Scarcity: In LAION-2B, spatial reasoning occurs at ~0.1%, counting at ~1.7%, and negation at ~0.1%.
• Synthetic Data: Even LLM-generated data (LLaVA-1.5) suffers from this bias, as the underlying LLMs are trained on human data that exhibits the same reporting bias.

B. Failure of Scaling

• Performance Gap: Open-source VLMs perform significantly worse than humans (e.g., ~54 points lower on average).
• No Emergence: Increasing model size (parameters) or data size (scaling laws) does not lead to the emergence of these reasoning skills.
  • Spatial/Temporal/Negation: Performance remains flat or improves negligibly with scale.
  • Counting: Shows slight improvement but requires an "intractable" amount of data to reach human performance.
• Multilingual Failure: Adding multilingual diversity (translating captions) did not improve performance, proving the bias is universal across languages, not specific to English.

C. Effectiveness of Instructions

• User Study: When annotators were explicitly instructed to include reasoning concepts, the "True Occurrence" of these concepts increased dramatically.
  • Negation/Temporal: Jumped from near 0% (in standard COCO) to 52% and 44% respectively with custom instructions.
  • Spatial/Counting: Increased from ~8-23% to ~40-50%.
• Fine-tuning: Fine-tuning LLaVA-1.5 on data with 39% counting density (vs. 6.9% in standard data) improved counting performance from 49.8% to 54.4%, proving that the architecture can learn these skills if the data is present.

4. Key Contributions

1. Identification of Reporting Bias: The paper formally identifies "reporting bias" (rooted in Gricean Maxims) as the primary bottleneck for VLM reasoning, rather than model capacity.
2. Empirical Evidence of Scaling Limits: Demonstrates that scaling data and parameters alone cannot overcome pragmatic omissions in human language; an intractable amount of data would be required to learn these skills by chance.
3. Benchmarking Suite: Introduced a rigorous set of benchmarks specifically targeting spatial, temporal, counting, and negation reasoning to evaluate VLMs beyond standard perception tasks.
4. Instructional Mitigation: Proved that annotator instructions are a critical lever. By explicitly asking for "tacit" information (e.g., "describe the position," "count the objects"), data collection can be made "reasoning-aware," significantly increasing the density of these concepts in training corpora.

5. Significance and Implications

• Paradigm Shift: The paper challenges the "bigger is better" dogma in VLM development. It argues that intentional data curation is more effective than blind scaling.
• Data Collection Strategy: Future datasets must move beyond passive scraping or generic captioning. Data collection pipelines must include specific prompts designed to elicit reasoning (e.g., asking "What is happening before/after?" or "What is not present?").
• LLM Synthesis: The findings warn that synthetic data generation is not a silver bullet; LLMs inherit human reporting biases. Instructions given to LLMs for data synthesis must be just as specific as those given to human annotators.
• Pragmatics in AI: It bridges the gap between linguistic pragmatics and AI training, suggesting that to build models that "think" like humans, we must train them on data that reflects human intent to communicate complex relationships, not just human habit of omitting them.

In conclusion, the paper asserts that Scale cannot overcome Pragmatics. To achieve human-level reasoning in VLMs, the community must shift from relying on data volume to relying on data quality and intentional instruction design.