VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought

1. Problem Statement

Current Large Vision-Language Models (LVLMs) excel at holistic image understanding but struggle with precise, local region-based reasoning. Existing datasets and benchmarks face three critical limitations:

• Scalability & Cost: Most datasets requiring fine-grained reasoning rely on extensive manual annotation, making them expensive to scale and limiting data diversity.
• Lack of Grounding: Existing Multimodal Chain-of-Thought (CoT) datasets often generate reasoning steps without explicitly linking them to specific visual evidence (spatial coordinates) within the image. This makes it impossible to verify if a model's logic is based on actual image content or merely plausible text generation (hallucination).
• Evaluation Gaps: Traditional benchmarks evaluate only the final answer accuracy. They fail to assess the quality of the reasoning process, the utilization of visual evidence, or the alignment between the rationale and the final answer.

2. Methodology

The authors propose VG-CoT (Visual Grounded Chain-of-Thought), a dataset and a new evaluation framework constructed via a fully automated, three-stage pipeline.

A. Data Generation Pipeline

The pipeline constructs 13,826 high-quality samples from three source datasets (TextVQA, Visual7W, and GQA) without human annotation:

1. Stage 1: Multi-Granularity Visual Evidence Extraction:
   • Object Detection: Uses state-of-the-art models (YOLO) to extract bounding boxes for key objects.
   • OCR: Uses PaddleOCR to extract text content and locations.
   • Supplementary Data: Leverages existing scene graphs (from GQA) as initial evidence.
2. Stage 2: Visually-Grounded CoT Generation:
   • GPT-4o is prompted with the image, question, ground-truth answer, and the extracted visual evidence (bounding boxes).
   • Constraint: The model is strictly instructed to generate step-by-step reasoning that explicitly cites the spatial coordinates of the visual evidence for every logical deduction. It must also explain why incorrect options are invalid.
3. Stage 3: Rationale-Driven Grounding Refinement:
   • To capture subtle or context-dependent objects mentioned in the generated CoT but missed in Stage 1, the system extracts key nouns from the rationale.
   • These nouns serve as queries for an open-set detector (Grounding DINO) to generate precise bounding boxes.
   • The final output links every reasoning step to updated, precise visual evidence.

B. New Benchmark Dimensions

The paper introduces a benchmark that evaluates LVLMs across three complementary dimensions, moving beyond simple answer accuracy:

1. Rationale Quality (RQ): Evaluated by GPT-4o on a 1–5 scale based on:
   • Visual Evidence Utilization: Does the model use object/text locations?
   • Logical Coherence: Is the flow of reasoning logical?
   • Completeness: Are all necessary deductions present?
2. Answer Accuracy (AA): Standard binary correctness of the final prediction.
3. Reasoning-Answer Alignment (RAA): Measures the correlation between reasoning quality and answer correctness.
   • Consistency Score: Proportion of samples where high-quality reasoning leads to correct answers (and poor reasoning to incorrect ones).
   • Faithful Score: Proportion of correct answers that were derived from excellent reasoning.

3. Key Contributions

1. VG-CoT Dataset: A scalable, automated dataset of 13,826 samples that explicitly aligns multi-step reasoning with precise visual evidence (bounding boxes for objects and text).
2. Automated Construction Pipeline: A cost-effective, three-stage method that eliminates the need for manual annotation while ensuring high-quality grounding.
3. Comprehensive Benchmark: A new evaluation framework that assesses Rationale Quality and Reasoning-Answer Alignment, providing a deeper insight into model trustworthiness than accuracy alone.
4. Empirical Validation: Demonstrated that fine-tuning on VG-CoT significantly improves the reasoning capabilities of state-of-the-art LVLMs.

4. Experimental Results

The authors fine-tuned four representative LVLMs (LLaVA-1.5 7B/13B, Qwen2-VL-7B, Qwen2.5-VL-7B) using the VG-CoT dataset.

• Overall Performance: All models showed consistent improvements in Rationale Quality and Answer Accuracy.
  • Example: LLaVA-1.5-7B improved its Overall Rationale Quality from 72.2 to 83.4 and Answer Accuracy from 48.7 to 62.5.
• Task-Specific Gains:
  • Significant improvements were observed in General VQA and Relation VQA.
  • In the challenging Scene-text task, LLaVA-1.5-7B's Rationale score surged from 63.8 to 86.4.
• Alignment Analysis: Most models showed improved Consistency Scores, indicating that the models learned to derive correct answers through logical, evidence-based reasoning rather than statistical bias.
• Visual Evidence Prediction:
  • Fine-tuned models achieved an average mAP@0.5 of 44.5 (7B) and 48.7 (13B) for bounding box localization.
  • Limitation: Performance dropped significantly for Scene-text tasks (mAP@0.5 ~32.4) and under stricter IoU thresholds (mAP@0.75), indicating that precise localization of small, dense text regions remains a bottleneck.

5. Significance

• Trustworthy AI: VG-CoT addresses the "black box" nature of LVLM reasoning by forcing models to ground their logic in verifiable visual evidence, reducing hallucinations.
• Scalability: The automated pipeline proves that high-quality, grounded reasoning datasets can be constructed at scale without prohibitive human costs.
• Evaluation Paradigm Shift: The proposed benchmark shifts the focus from "Did the model get the right answer?" to "Did the model get the right answer for the right reasons?" This is crucial for safety-critical applications like autonomous driving and medical analysis.
• Future Direction: The study highlights that while logical reasoning has improved, precise spatial grounding of fine-grained text remains a challenge for future research.