WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata

Technical Summary: WikiVQABench

Problem Statement
Current Visual Question Answering (VQA) benchmarks predominantly emphasize perception-based tasks solvable through visual content alone, such as object recognition or scene description. However, real-world scenarios often require external knowledge not directly observable in an image to answer correctly. Existing knowledge-aware benchmarks (e.g., OK-VQA, A-OKVQA) suffer from limitations including restricted entity domains, open-ended answer formats that hinder standardized evaluation, and a lack of explicit enforcement that questions require external knowledge beyond visual inference. Furthermore, many existing datasets rely on implicit knowledge or unstructured text, lacking verifiable ground truth derived from structured, machine-readable sources.

Methodology
The authors introduce WikiVQABench, a human-curated benchmark constructed by systematically integrating Wikipedia images, article captions, and structured knowledge from Wikidata. The dataset construction pipeline follows a rigorous four-stage process:

1. Data Source and Entity Alignment: The pipeline utilizes the Wikipedia-based Image Text (WIT) dataset. For each image, the associated Wikipedia article is identified to resolve primary entities to Wikidata identifiers (QNodes).
2. Knowledge Retrieval and Filtering: SPARQL queries retrieve factual triples (subject-predicate-object) connected to the resolved entities. A filtering strategy removes low-salience metadata (e.g., IDs, URLs) and protected attributes (e.g., race, religion, gender). The remaining triples are verbalized into natural language statements.
3. LLM-Based Generation: A compact, open-source Large Language Model (Granite-3.3-8B-Instruct) generates candidate multiple-choice VQA instances. The model is prompted to synthesize questions based on the image caption and verbalized triples, ensuring the questions are directly related to the image but do not explicitly reference the source text. The output format includes one correct answer and three distractors.
4. Human Curation and Quality Control: All generated instances undergo human review via a custom UI. Annotators verify factual correctness, visual-text consistency, and, crucially, ensure that the correct answer cannot be derived from visual cues alone (enforcing knowledge necessity). Out of 2,369 generated instances, only 344 (14.5%) were accepted, reflecting a stringent quality control process.

Key Contributions

• A New Benchmark: WikiVQABench provides 344 curated multiple-choice VQA pairs grounded in structured knowledge (Wikidata) and visual content (Wikipedia), specifically designed to test knowledge-aware reasoning.
• Methodological Rigor: The paper proposes a hybrid construction methodology combining automated LLM generation with strict human curation to ensure verifiable ground truth and enforced knowledge necessity.
• Comprehensive Evaluation: The authors evaluate 15 Vision-Language Models (VLMs) ranging from 256M to 90B parameters, establishing baseline performance across different scales and architectures.
• Open Resources: The dataset and benchmarking code are publicly released to facilitate reproducible research in knowledge-grounded visual reasoning.

Results
Evaluation of 15 VLMs reveals a significant performance gap, demonstrating the benchmark's ability to discriminate model capabilities:

• Performance Range: Accuracy varies widely from 24.7% (SmolVLM-256M) to 75.6% (InternVL3-78B).
• Scaling Effects: While larger models generally perform better, there is a notable plateau in the 8B–90B parameter range (63%–66% accuracy), suggesting that scale alone is insufficient for mastering knowledge-intensive reasoning without architectural advances in knowledge integration.
• Difficulty Analysis: The benchmark maintains a balanced difficulty distribution with a mean score of 7.0 (out of 15 models failing a question), effectively challenging models across the capability spectrum.
• Baseline Performance: The smallest models (256M) perform near random chance (25% for 4-way multiple choice), indicating that current small-scale VLMs lack the capacity to integrate visual recognition with external structured knowledge.

Significance
The paper claims that WikiVQABench addresses a critical gap in VQA evaluation by explicitly enforcing the requirement for external knowledge. Unlike previous benchmarks that may allow models to succeed via surface-level perception or spurious correlations, WikiVQABench ensures that correct resolution depends on reasoning over entity properties and relationships using structured facts. The authors posit that this benchmark is essential for exposing fine-grained weaknesses in current models and guiding future development toward more robust, knowledge-grounded semantic understanding. The results highlight that while progress has been made, integrating visual evidence with structured external knowledge remains a substantial challenge for even the largest current models.