Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling

This paper argues that the primary bottleneck in scaling multimodal large language models is not task format diversity but rather the knowledge density of training data, demonstrating that enriching image captions with structured knowledge yields more consistent performance improvements than adding task-specific supervision like Visual Question Answering.

The Gist

The Big Idea: It's Not About the Game, It's About the Library

Imagine you are trying to teach a brilliant student (the AI) how to understand the world.

For a long time, researchers thought the best way to do this was to give the student more and more different types of tests. They thought, "If we make the student practice answering riddles, solving puzzles, and playing 'guess the object' games (these are called VQA or Visual Question Answering), they will get smarter."

This paper argues that this is wrong.

The authors say: "Stop worrying about the type of test. The problem isn't that the student hasn't played enough games. The problem is that the library the student is studying from is too thin."

They call this missing ingredient Knowledge Density.

---

Analogy 1: The "Rephrasing" Trick (Why VQA Doesn't Help Much)

Imagine you have a photo of a dog running on grass.

• The Old Way (VQA): You show the photo and ask, "What animal is running?" The student answers, "A dog."
• The New Way (Caption): You just write a sentence: "A dog is running on the grass."

The paper's experiments showed something surprising: The student learns the exact same thing from the sentence as they do from the question-and-answer game.

Think of it like this:

• The Caption is the book containing the facts.
• The VQA is just highlighting a specific sentence in that book and asking, "What did I just highlight?"

If you only have a thin book, highlighting different sentences (changing the question) doesn't give you more knowledge. You still only have the same thin book. The "game" format (VQA) just rearranges the information you already had; it doesn't add anything new.

The Takeaway: Adding more question-and-answer formats is like rearranging the furniture in a small room. It looks different, but the room is still small.

---

Analogy 2: The "Super-Book" (Why Knowledge Density Wins)

So, if changing the game format doesn't work, what does? Adding more facts to the book.

The authors tried a new method: Knowledge Injection.

Instead of just showing one picture of a dog, they showed two pictures side-by-side and wrote a description that compared them.

• Old Caption: "Here is a dog."
• New "Dense" Caption: "Here is a Shiba Inu dog running on green grass, while next to it is a Golden Retriever sleeping on a red rug. Notice how the Shiba has pointy ears and the Golden has floppy ears."

The Magic:
By forcing the AI to look at two things and describe the differences, relationships, and details, they packed way more "knowledge" into a single training example.

• The Old Way: Feeding the AI 100 pictures with simple labels.
• The New Way: Feeding the AI 100 pictures with rich, comparative stories that explain how things relate to each other.

The Result:
When they trained the AI on these "Super-Books" (dense knowledge), the AI got significantly smarter at reasoning, math, and understanding complex scenes. It didn't matter that they didn't change the "game" (the task format); they just made the "library" much richer.

---

The "Diminishing Returns" Problem

You might wonder, "Why didn't we figure this out sooner?"

Think of training an AI like building a skyscraper.

• Text-only AI (like ChatGPT): We gave them a library with trillions of books. They built a skyscraper that goes straight up.
• Multimodal AI (Image + Text): We gave them a library with only billions of books. We tried to make the building taller by adding more types of windows (different tasks), but the foundation (the knowledge) was too weak. The building kept hitting a ceiling.

The paper says: Don't add more window shapes. Add more books to the foundation.

Summary in Plain English

1. The Myth: "If we give the AI more complex questions (VQA) to answer, it will get smarter."
2. The Reality: "Those questions usually just repeat what's already in the image description. They don't add new facts."
3. The Solution: "Stop focusing on the format of the question. Focus on the density of the information. Make the descriptions richer, more comparative, and more detailed."
4. The Future: To build truly smart AI that can see and understand the world, we need to stop just "labeling" images and start "teaching" them through rich, knowledge-packed stories.

In short: It's not about how many questions you ask the AI. It's about how much truth you feed it.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) have shown rapid progress but suffer from unpredictable and diminishing returns when scaling model size or increasing task diversity, unlike text-only LLMs which follow clear scaling laws.

• Current Assumption: The prevailing belief is that increasing task diversity (specifically through Visual Question Answering or VQA supervision) is the primary driver for improving multimodal reasoning and scaling behavior.
• The Bottleneck: The authors argue that current MLLMs are not limited by a lack of task formats but by a lack of knowledge density (the amount and diversity of semantic information) in the training data.
• Core Question: Does VQA supervision introduce new semantic knowledge beyond what is already present in image captions, or does it merely reformat existing information?

2. Methodology

The paper employs a two-stage experimental approach to validate the hypothesis that knowledge density, rather than task format, drives scaling.

Stage 1: Disentangling Task Format from Semantic Content

The authors conducted controlled experiments to determine if VQA adds incremental knowledge over captions.

• Setup: A 3B parameter dense multimodal model was trained under identical conditions (architecture, optimizer, token budget) with varying supervision strategies.
• Configurations:
  1. Baseline: Standard mixture of captions, VQA, pure text, and other data.
  2. Caption-Only: All VQA data replaced with model-generated captions derived from the same images.
  3. Synthetic-VQA: VQA pairs were reconstructed from the captions using a structured pipeline.
• Hypothesis: If VQA adds no new knowledge, the Caption-Only model should perform comparably to the Baseline, and the Synthetic-VQA model should match the Baseline.

Stage 2: Knowledge-Centric Training via Cross-Modal Injection

To test if increasing knowledge density improves performance, the authors introduced a data construction pipeline that enriches semantic coverage without changing the task format.

• Image Pair Construction:
  • LLM as Knowledge Base: An MLLM extracts structured semantic descriptors (categories, attributes, entities) from images.
  • Pairing Strategy: Images are paired based on coarse semantic alignment (same category) and fine-grained semantic contrast (different attributes/context).
  • LLM as Semantic Filter: A second MLLM filters pairs to ensure the relationship is coherent and informative (e.g., explainable to a child), discarding trivial co-occurrences.
• Multi-Image Interleaved Descriptions: Aggregating multiple related images into a single, coherent long-form text description to synthesize cross-instance relationships.
• Training Variants:
  • Pair-Caption-v1: Replaces standard captions/VQA with semantically paired image-caption data.
  • Interleaved: Uses multi-image interleaved descriptions.
  • Pair-Caption-v2: Uses paired captions while retaining the original VQA supervision (to test additive effects).

3. Key Contributions

1. Reframing the Scaling Bottleneck: The paper challenges the "task diversity" dogma, arguing that the primary limitation in MLLM scaling is insufficient knowledge coverage in training data, not the lack of complex task formats like VQA.
2. VQA Redundancy Demonstration: Through controlled experiments, the authors prove that VQA signals can be largely reconstructed from image captions with negligible performance loss, indicating VQA provides formatting rather than new semantic content.
3. Knowledge Density Metric & Injection: The authors define "knowledge density" as the count of distinct semantic elements (objects, relations, events) per sample and propose a Cross-Modal Knowledge Injection pipeline (semantic image pairing) to increase this density.
4. Empirical Validation of Knowledge-Centric Scaling: They demonstrate that increasing semantic coverage through structured data augmentation leads to consistent performance gains, whereas simply varying task formats does not.

4. Results

Stage 1: Task Format vs. Knowledge Content

• General Multimodal Benchmarks (MMMU, MMBench, etc.): The Caption-Only model achieved nearly identical performance to the Baseline (avg. score difference < 0.005). The Synthetic-VQA model slightly outperformed the Baseline in some tasks (e.g., AI2D, OCRBench).
• Text & Business Benchmarks: Removing VQA supervision did not degrade text reasoning or downstream business capabilities (OCR, document understanding).
• Conclusion: VQA supervision contributes little incremental semantic information; captions already encode the necessary knowledge for VQA-style reasoning.

Stage 2: Knowledge Density Interventions

• Performance Gains: The Pair-Caption-v2 configuration (paired captions + original VQA) achieved the strongest overall results:
  • General Multimodal: Improved average score from 0.593 (Baseline) to 0.602.
  • General Text: Improved average score from 0.496 to 0.528, with significant gains on GPQA-Diamond (complex reasoning).
  • Business Multimodal: Improved average score from 0.397 to 0.440, showing strong out-of-domain generalization.
• Knowledge Density Analysis: The paired-image captions contained an average of 32 knowledge elements per sample compared to 22 for standard captions/VQA—a 45% increase in semantic density.
• Interleaved Data: The "Interleaved" configuration (aggregating multiple images into one text) performed worse than the baseline, suggesting that unstructured aggregation dilutes visual-text alignment.

5. Significance and Implications

• Paradigm Shift: The paper advocates for a shift from Task-Centric to Knowledge-Centric multimodal training. Future scaling should prioritize expanding the semantic frontier (objects, relations, context) rather than inventing new task formats.
• Data Efficiency: It suggests that current MLLMs fail to scale because training data lacks sufficient knowledge coverage, not because the models are too small or the tasks too simple.
• Practical Guidance: For practitioners, this implies that investing in structured data enrichment (e.g., semantic pairing, relational grounding) is more effective than simply collecting more diverse VQA datasets.
• Theoretical Foundation: It provides a potential explanation for the weak scaling laws in multimodal systems compared to text-only LLMs: the "token count" metric is insufficient; "knowledge density" is the governing variable.

In summary, the paper establishes that captions are primary sources of semantic knowledge, while VQA is secondary (a formatting layer). To achieve scalable, robust multimodal intelligence, the focus must shift to maximizing the knowledge density within the training corpus.