💻 computer science

VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs

The paper introduces VKnowU, a comprehensive benchmark for evaluating visual knowledge understanding in Multimodal Large Language Models, and proposes VideoKnow+, a reinforcement learning-based model that significantly improves performance on this benchmark and other video understanding tasks by explicitly incorporating structured visual knowledge.

Original authors: Tianxiang Jiang, Sheng Xia, Yicheng Xu, Linquan Wu, Xiangyu Zeng, Limin Wang, Yu Qiao, Yi Wang

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianxiang Jiang, Sheng Xia, Yicheng Xu, Linquan Wu, Xiangyu Zeng, Limin Wang, Yu Qiao, Yi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot friend who can watch videos and answer questions about them. For a long time, this robot has been great at spotting things. If you show it a video of a dog chasing a ball, it can tell you, "That's a dog," "That's a ball," and "The dog is running."

But there's a problem: the robot doesn't really understand the world the way humans do. It doesn't intuitively know that if you drop a ball, it will fall down (gravity), or that a glass cup is fragile and might break if you drop it, or that a person frowning is probably upset. It sees the pixels, but it misses the "common sense" behind them.

This paper introduces a new way to test and teach robots this missing "common sense," which the authors call Visual Knowledge.

The Problem: The Robot is "Blind" to Common Sense

The authors created a giant test called VKnowU (Visual Knowledge Understanding). Think of this as a final exam for robots, but instead of math or history, the questions are about how the world works and how people think.

The test is split into two main subjects:

World-Centric (The Physics Class): Does the robot know that a heavy box is harder to lift than a feather? Does it know that a coin will fall to the ground?
Human-Centric (The Psychology Class): Does the robot understand that a boy looking at a messy room might be planning to make a bigger mess? Does it know that if adults walk in, they might be shocked?

The Results: When they gave this test to the smartest robots available (like the ones from Google, OpenAI, and open-source teams), the robots scored poorly. They were especially bad at the "Physics Class." They could describe the scene perfectly but failed to predict what would happen next or understand the materials of objects. They were "seeing" but not "understanding."

The Solution: Teaching the Robot to "See, Think, Answer"

To fix this, the authors built a new training method called VideoKnow+. They realized that robots were trying to guess answers based on their language training (like guessing the answer to a riddle without looking at the picture).

They introduced a new rule for the robot: "See, Think, Answer."

See: Before answering, the robot must first describe exactly what it sees in the video, focusing on the visual clues.
Think: Then, it uses those visual clues to reason.
Answer: Finally, it gives the answer.

They also created a special "reward system." Imagine a teacher grading the robot. If the robot tries to guess the answer using only words, the teacher gives it a low score. But if the robot writes a description that proves it saw the visual evidence (like "The box is wood, so it's heavy"), the teacher gives it a high score. This forces the robot to actually pay attention to the video, not just its internal database of facts.

The Outcome

After training with this new method and a massive new dataset of videos (called VKnowQA), the robot got much better.

It improved its score on the "Visual Knowledge" test by a significant margin.
It also got better at other general video tests, proving that learning to understand the world helps it in many areas.

The Big Picture

The paper argues that for robots to become truly intelligent, they need to stop just being "pattern matchers" (recognizing objects) and start being "world understanders." Just like a human child learns that fire is hot and glass is breakable by observing the world, robots need to learn these "visual truths" to bridge the gap between simply seeing an image and truly understanding what is happening in it.

In short: The paper built a test to show robots are bad at common sense, and then built a new training method that forces them to look at the evidence before guessing, making them much smarter and more reliable.

Technical Summary: VKnowU

Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated proficiency in recognizing objects and attributes but frequently lack an intuitive, human-like understanding of the underlying physical and social principles governing the world. This gap represents a deficiency in visual knowledge—a high-level, vision-grounded semantic layer that bridges perception and reasoning. While current models excel at fine-grained perception through multi-scale features or high-resolution inputs, they struggle with embedded, intuitive knowledge required for deeper reasoning, such as predicting physical trajectories or inferring subjective intentions. Consequently, MLLMs often rely on brittle language priors, leading to hallucinations and poor performance on tasks requiring true world understanding.

Methodology

1. VKnowU: A Comprehensive Benchmark

To quantitatively measure the gap between perceptual accuracy and cognitive reasoning, the authors introduce VKnowU, a video benchmark comprising 1,249 videos and 1,680 multiple-choice questions. The benchmark is structured around eight core dimensions of visual knowledge, categorized into two domains:

World-Centric (Physical):
- Intuitive Physics: Judging physical plausibility (e.g., gravity, object permanence).
- Object Affordance: Determining functional uses based on structural properties.
- Object Material: Identifying material composition via intrinsic or behavioral cues.
- Spatial Awareness: Inferring relative positions and navigation.
Human-Centric (Psychological/Social):
- Event Anticipation: Inferring probable subsequent events based on social norms.
- Mental State: Inferring internal psychological states and affective atmospheres.
- Social Relation: Inferring interpersonal relationships and roles.
- Subjective Intention: Reconstructing underlying goals behind observed actions.

Dataset Construction:
To ensure the benchmark strictly evaluates visual knowledge rather than language priors or audio cues, the authors employed a rigorous QA Reformulation Pipeline:

Minimize Audio Reliance: Questions solvable via audio transcripts (similarity > 0.3) were discarded.
Reduce Language Bias: Questions solvable by text-only LLMs (Pass@10 ≥ 0.5) were excluded.
Enhance Wrong Candidates: Distractors were reconstructed to be semantically plausible but incorrect, preventing elimination strategies.
Human Verification: Final manual review ensured clarity and alignment with specific visual knowledge competencies.

2. VideoKnow+: A Baseline Model for Visual Knowledge

To address the identified deficits, the authors propose VideoKnow+, a baseline model designed to explicitly integrate visual knowledge. The approach follows a See–Think–Answer paradigm:

Structured Output: The model is constrained to generate a self-contained visual description ("See") before reasoning ("Think") and answering ("Answer"). This forces the model to anchor its reasoning in perceived visual evidence.
Reinforcement Learning (RL): The model is trained using GRPO (Group Relative Policy Optimization) with a specialized Visual Knowledge Reward ( $r_v$ ). An external frozen MLLM acts as a verifier to determine if the generated visual description is sufficient to derive the answer without further reasoning.
VKnowQA: A new large-scale training corpus (VKnowQA-30K and VKnowQA-CS-12K) was constructed to support the training of VideoKnow+, ensuring no data leakage with the evaluation benchmark.

Key Results

Evaluation of SOTA MLLMs on VKnowU

The authors evaluated 28 state-of-the-art MLLMs (including open-source and proprietary models like GPT-4o, Gemini-3.0, and InternVL3.5).

Overall Gap: Even the best-performing model (Gemini-3.0-Pro) achieved 76.3%, trailing human performance (93.0%) by 16.7%.
World-Centric Deficit: The gap is most pronounced in physical tasks. Models lagged behind humans by 37.0% in Intuitive Physics and 32.7% in Spatial Awareness, with some models barely exceeding random guessing in these categories.
Human-Centric Performance: Models performed relatively better on human-centric tasks (best: 81.9%), likely due to the prevalence of social narratives in instruction tuning data.
Scaling Analysis: Increasing LLM backbone size (e.g., from 4B to 8B) yielded marginal gains on VKnowU (+1.3%) compared to general benchmarks, suggesting that performance is limited by visual grounding rather than language modeling capacity.

Performance of VideoKnow+

Training VideoKnow+ on VKnowQA with the See–Think–Answer structure and visual knowledge reward yielded significant improvements:

VKnowU: Achieved a +3.7% improvement over the baseline (Qwen2.5-VL-7B).
Generalization: Demonstrated consistent gains across other video benchmarks:
- MVBench: +5.4%
- Video-MME: +7.0%
- MMVU: +5.7%
Ablation: The combination of Supervised Fine-Tuning (SFT) for the structured format and GRPO with the visual knowledge reward ( $r_v$ ) was found to be the most effective strategy.

Significance and Claims

The paper claims that visual knowledge is a missing cornerstone for developing generalizable MLLMs. The authors argue that:

Current Limitations: Standard next-token prediction objectives struggle to capture implicit physical dynamics and social nuances solely from text-aligned supervision.
The Bridge: Visual knowledge serves as a critical intermediate layer that reduces over-reliance on language priors and mitigates hallucinations.
Learnability: The results demonstrate that visual knowledge is learnable. By explicitly structuring the reasoning process (See–Think–Answer) and rewarding the model for grounding answers in visual evidence, MLLMs can achieve more robust and human-like understanding.
Diagnostic Tool: VKnowU provides a precise diagnostic tool that disentangles physical grounding from social reasoning, revealing that these are largely independent cognitive subsystems in current models.

The work positions the explicit integration of visual knowledge as a necessary step toward advancing MLLMs from merely "seeing" to truly "understanding" the physical and social worlds.