AutoViVQA: A Large-Scale Automatically Constructed Dataset for Vietnamese Visual Question Answering

This paper introduces AutoViVQA, a large-scale automatically constructed dataset for Vietnamese Visual Question Answering, and evaluates transformer-based multimodal models alongside various automatic metrics to assess their performance and alignment with human judgment in the Vietnamese context.

The Gist

Imagine you are trying to teach a robot to understand the world through its eyes and ears. You show it a picture and ask, "What is happening here?" The robot needs to look at the image, understand the question, and give a smart answer. This is called Visual Question Answering (VQA).

For a long time, we've had great teachers and textbooks for English-speaking robots. But for Vietnamese-speaking robots? The library was almost empty. The few books that existed were either too short, too simple, or written by machines that made up facts (hallucinations).

Enter AutoViVQA. Think of this paper as the blueprint for building a massive, high-quality "Vietnamese VQA Library" from scratch, but with a twist: instead of hiring thousands of humans to write every single question (which is slow and expensive), the authors built a super-smart, automated factory to do it.

Here is how they did it, explained with some everyday analogies:

1. The Problem: The "Empty Bookshelf"

Imagine trying to teach a child to read using only a few torn pages from a magazine. That's what researchers were doing with Vietnamese VQA. The existing datasets were small, repetitive, and often didn't test if the robot was actually thinking or just guessing.

2. The Solution: The "AI Assembly Line"

The authors didn't just collect data; they built a factory (a pipeline) to manufacture it.

• The Raw Materials: They took real-world photos (from a famous collection called MS COCO) and paired them with high-quality Vietnamese descriptions.

• The Foreman (The LLM): They used a powerful Large Language Model (like a super-intelligent foreman) to act as the question writer. But they didn't just say, "Write some questions." They gave the foreman a strict rulebook.

• The Rulebook (Reasoning Levels): This is the secret sauce. They told the foreman:

  • Level 1: Just name the object (e.g., "What color is the car?").
  • Level 2: Describe where things are (e.g., "Is the dog under the table?").
  • Level 3: Connect two things (e.g., "Why is the person holding an umbrella?").
  • Level 4 & 5: Get deep into cause-and-effect or reading text inside the image.

  They forced the factory to produce a balanced mix of easy, medium, and hard questions, ensuring the robot learns to reason, not just memorize.

3. The Quality Control: The "Taste-Test Panel"

In a normal factory, you might just check if the product looks okay. In this paper, they built a Quality Control Panel made of other AI models.

• The Jury: Every single question and answer generated by the factory was sent to a "jury" of 5 to 10 different AI models.
• The Vote: These models acted like judges. They asked: "Is this question confusing? Is the answer actually in the picture? Is the Vietnamese natural?"
• The Filter: If the jury didn't agree that the sample was good, it was thrown in the trash. Only the samples that passed a strict "majority vote" made it into the final dataset.

This is like a chef cooking a meal, then sending it to a panel of 10 food critics. If 8 out of 10 say, "This tastes weird," the dish is scrapped. This ensured the final dataset was clean, accurate, and free of the "robot hallucinations" that usually plague AI-generated content.

4. The Result: A New Standard

The result is AutoViVQA, a dataset with nearly 20,000 images and over 180,000 answers.

• Why it matters: When the researchers tested their new dataset on various AI models, the models got significantly smarter. It wasn't because they changed the robot's brain; it was because they finally gave it a better textbook.
• The Analogy: Imagine a student taking a test. If the practice questions are vague and full of errors, the student will fail. If the practice questions are clear, varied, and challenging, the student will ace the real exam. AutoViVQA is that perfect set of practice questions for Vietnamese AI.

The Bottom Line

This paper is about solving a "low-resource" problem (not enough data for Vietnamese) by building a self-correcting, automated system. They proved that you don't need a million human annotators to build a great dataset; you just need a smart system, a strict rulebook, and a rigorous quality-checking jury.

They didn't just build a dataset; they built a factory for intelligence that can be used to train better, more culturally aware, and more logical AI for the Vietnamese language.

Technical Summary

Here is a detailed technical summary of the paper "AUTOVIVQA: A LARGE-SCALE AUTOMATICALLY CONSTRUCTED DATASET FOR VIETNAMESE VISUAL QUESTION ANSWERING."

1. Problem Statement

Visual Question Answering (VQA) is a critical multimodal task requiring the integration of visual perception, linguistic understanding, and reasoning. While English-centric VQA has advanced significantly with large-scale models (e.g., BLIP-2, LLaVA), Vietnamese VQA remains underdeveloped due to a scarcity of high-quality, large-scale benchmarks. Existing Vietnamese datasets (e.g., ViVQA, OpenViVQA) suffer from:

• Insufficient Scale: Too small to train modern multimodal models effectively.
• Limited Reasoning Diversity: Over-reliance on simple object recognition or text extraction, lacking complex reasoning (causal, spatial, multi-step).
• Data Quality Issues: Manual annotation is expensive and hard to scale, while naive AI generation introduces hallucinations, weak visual grounding, and cultural biases.
• Evaluation Gaps: Traditional metrics (BLEU, ROUGE) often fail to align with human judgment in VQA tasks.

The core challenge is how to construct a scalable, reasoning-aware, and quality-controlled Vietnamese VQA dataset without heavy reliance on manual annotation.

2. Methodology: The AutoViVQA Pipeline

The authors propose AutoViVQA, a dataset constructed via a fully automated, LLM-driven pipeline. The process involves four key stages:

A. Data Collection & Foundation

• Visual Source: Real-world images from MS COCO to ensure visual diversity.
• Textual Source: High-quality Vietnamese captions and conversational descriptions from the VISTA corpus.
• Integration: These sources are merged to create a multimodal foundation that ensures semantic alignment between images and Vietnamese text.

B. Constraint-Guided Generation

Instead of free-form generation, the pipeline uses a controlled prompting strategy:

• Input: Merged captions provide semantic context.
• Output: For each image, the LLM generates one question and five distinct short answers (simulating multiple annotators).
• Constraints: Questions must follow natural Vietnamese syntax, remain strictly grounded in the image/caption (no hallucinations), and answers must be logically consistent.

C. Reasoning-Level Control Schema

To ensure cognitive diversity, the authors define a 5-level reasoning schema and enforce a balanced distribution (approximating a normal distribution):

1. Level 1 (Recognition): Identifying objects/attributes.
2. Level 2 (Spatial & Relational): Spatial relationships and simple comparisons.
3. Level 3 (Compositional): Multi-step reasoning involving multiple objects/actions.
4. Level 4 (Commonsense & Causal): Inferring intentions, mental states, or causality.
5. Level 5 (Text-in-Image): Reading and interpreting text within the image.

• Mechanism: The system monitors generation deficits and prioritizes underrepresented reasoning levels to prevent the dataset from collapsing into trivial questions.

D. Ensemble-Based Quality Control (No Human Annotation)

A fully automatic post-generation filtering protocol ensures high quality:

• Ensemble Evaluation: Each sample is evaluated by an ensemble of $2n+1$ independent Vision-Language Models (VLMs) and Language Models (LLMs).
• Criteria: 18 quality dimensions across four categories: Visual Quality, Contextual Complexity, Linguistic Validity, and Visual Grounding.
• Filtering: A sample is retained only if it passes a majority vote and satisfies at least 9 out of 18 criteria.
• Balancing: Distribution-aware balancing prevents dominance by frequent categories, ensuring a uniform spread of reasoning difficulties.

3. Key Contributions

1. AutoViVQA Dataset: A large-scale dataset containing 19,411 images, 37,077 questions, and 185,385 answers (5 answers per question). It covers 9 question types and 5 reasoning levels, significantly outperforming existing Vietnamese benchmarks in scale and diversity.
2. Reasoning-Aware Framework: A novel generation methodology that explicitly regulates cognitive depth (from recognition to causal inference) using a structured schema, ensuring balanced coverage of reasoning types.
3. Automated Quality Assurance: A reproducible, ensemble-based validation protocol that filters noisy data without human intervention, addressing the "hallucination" problem in synthetic data generation.
4. Benchmarking & Extensibility: The framework serves not just as a dataset but as a methodology for creating future low-resource multimodal benchmarks, adaptable to specific domains via prompt engineering.

4. Experimental Results

The authors evaluated the impact of the refined dataset on various models (Vietnamese-specific VLMs like Vintern, general LLMs like LLaMA 3.2, and commercial models like Gemini).

• Performance Gains: Training on the filtered AutoViVQA dataset led to consistent improvements across models.
  • Vintern (Finetuned) showed a 3x improvement in F1-score and an 8x increase in CIDEr compared to the base model trained on unrefined data.
  • Significant gains were observed in Precision, Recall, and Semantic Fidelity metrics (ROUGE, METEOR).
• Model Behavior:
  • Vietnamese-specific models benefited most, showing higher precision and semantic consistency.
  • General-purpose LLMs showed higher Recall but lower Precision, highlighting the trade-off between diversity and grounding, which the dataset helps mitigate.
• Human Validation: A study with 3 bilingual annotators on 1,000 samples yielded a Krippendorff's alpha of 0.72, confirming substantial agreement on fluency, visual grounding, and reasoning appropriateness.
• Error Analysis: Residual errors (e.g., vague questions, mild cultural awkwardness) accounted for less than 6% of samples, mostly filtered out by the ensemble protocol.

5. Significance and Future Work

• Significance: AutoViVQA bridges the gap for low-resource languages by demonstrating that high-quality multimodal benchmarks can be constructed automatically using LLMs, provided there is rigorous reasoning control and ensemble validation. It shifts the focus from "data collection" to "controlled data generation."
• Limitations: The dataset relies on MS COCO images, which may lack specific Vietnamese cultural contexts. It also primarily reflects standard Vietnamese, lacking regional dialect variations.
• Future Directions: Incorporating culturally diverse visual sources, enhancing bias-aware filtering, and modeling dialectal variations.

In conclusion, AutoViVQA provides a robust, scalable, and reproducible framework for advancing multimodal AI in Vietnamese, proving that automated pipelines can generate datasets that significantly improve model performance in reasoning and visual grounding tasks.