DIAGRAMS: A Review Framework for Reasoning-Level Attribution in Diagram QA

1. Problem Statement

Diagram Question Answering (Diagram QA) involves answering questions based on diverse visual domains such as scientific diagrams, charts, circuit schematics, maps, and infographics. Unlike text-based reasoning, Diagram QA often requires models to interpret a chain of interconnected visual elements rather than a single region.

• The Gap: Existing annotation efforts primarily focus on answer-level grounding, where annotators mark only the region containing the final answer. This fails to capture the full reasoning path (the sequence of visual dependencies) required to derive the answer.
• The Challenge: Creating reasoning-level attribution (linking a Q&A pair to all necessary visual regions) is time-consuming and expensive. Furthermore, existing annotation tools are tightly coupled to specific dataset formats (JSON structures, coordinate systems), making them difficult to reuse across heterogeneous datasets. Researchers often have to rebuild front-end logic for every new dataset, hindering reproducibility and progress.

2. Methodology: The DIAGRAMS Framework

The authors propose DIAGRAMS, a "review-first" framework designed to decouple interface logic from dataset-specific structures. The system operates in three coordinated stages:

A. Architecture & Data Ingestion

The system uses a schema-driven approach to handle heterogeneity:

1. Schema Loader & Adapter: Ingests dataset-specific JSON records and maps them to a unified internal meta-schema. This decouples the annotation logic from the original dataset format.
2. Normalization: Standardizes heterogeneous bounding box formats (e.g., converting [x, y, w, h] to left/top/width/height) and coordinates.
3. Internal Representation: Stores images, QA pairs, candidate regions, and attribution mappings in a unified structure.

B. Evidence Proposal (Multimodal Backend)

The system leverages a multimodal model to propose evidence, operating in three modes depending on data availability:

1. Selection-based Attribution: If QA pairs and candidate regions exist, the model selects the subset of regions required for reasoning from the candidate pool.
2. Region Generation: If QA pairs exist but candidate regions are missing, the model generates new bounding boxes directly on the image.
3. QA & Region Generation: If both are missing, the system proposes new QA items and corresponding evidence regions.

C. Human-in-the-Loop Verification

Instead of drawing regions from scratch, the workflow prioritizes guided verification:

• Reviewers inspect model-proposed regions.
• They can verify, edit, remove, or add regions and QA items.
• The system tracks the provenance of every change (e.g., whether a box was proposed, edited, or newly drawn).
• Finalized attributions are exported in structured JSON and annotated images.

3. Key Contributions

1. Formalization of Reasoning-Level Attribution: The paper explicitly distinguishes reasoning-level attribution (full reasoning chain) from answer-only grounding, providing a framework to capture the former.
2. Schema-Driven Review Framework: DIAGRAMS introduces a modular architecture with dataset adapters and a meta-schema, allowing the same interface to be reused across diverse Diagram QA datasets without redesign.
3. Multimodal Proposal Pipeline: A pipeline that performs QA-conditioned evidence selection, missing-region generation, and optional QA generation to reduce manual effort.
4. Empirical Evaluation: Comprehensive evaluation across six diverse datasets (Charts, Maps, Diagrams, Circuits, Infographics) demonstrating the utility of the review-first workflow.

4. Results and Analysis

The framework was evaluated across six datasets: ChartQA, MapWise, AI2D, Circuit-VQA, InfographicsVQA, and MapIQ.

• Performance Metrics (Micro-averaged):
  • Precision: 85.39% (Reviewers retained 85.39% of model-suggested boxes).
  • Recall: 75.30% (Model suggested 75.30% of the final required boxes).
  • F1 Score: 80.03%.
• Efficiency Gains:
  • The system pre-selected a large fraction of final evidence regions, converting the task from "box creation" to "guided verification."
  • False Negative Analysis: Of the regions reviewers had to add, 80.5% were missed ground-truth boxes (selection errors), while only 19.5% were entirely new boxes drawn from scratch. This confirms that the primary bottleneck is selection, not region absence.
• Dataset Variability:
  • High Alignment: AI2D (95.19 F1) and InfographicsVQA (97.72 F1) showed strong alignment, likely due to structured layouts and clear boundaries.
  • Challenging Domains: ChartQA (47.69 F1) and Circuit-VQA (56.92 F1) showed lower recall. ChartQA suffered from missed ground-truth selection, while Circuit-VQA required significant new region creation due to fine-grained component complexity.
• Inter-Annotator Reliability: High agreement (Kappa > 0.46) was observed between human reviewers on selection-based proposals, indicating the verification task is reproducible.

5. Significance and Impact

• Reduced Annotation Cost: By shifting the workflow from manual creation to model-assisted verification, DIAGRAMS significantly reduces the time and effort required to create high-quality, reasoning-level datasets.
• Improved Model Training & Evaluation: The framework enables grounded supervision, allowing Vision-Language Models (VLMs) to be trained on full reasoning chains rather than just answer boxes. This helps mitigate hallucinations and spurious correlations.
• Reproducibility: The open-source release (MIT license) includes the framework, meta-schema, and adapters, allowing the research community to audit and build upon diverse Diagram QA benchmarks without reinventing annotation tools.
• Limitations & Future Work: The authors note that performance degrades on visually cluttered diagrams, current support is limited to bounding boxes (lacking polygons for thin lines/irregular shapes), and the system does not yet support chain-of-thought supervision. Future work aims to address these through polygon-level annotation and latency analysis.

In summary, DIAGRAMS provides a scalable, reusable infrastructure for generating high-fidelity reasoning-level attributions, bridging the gap between complex visual reasoning requirements and the practical constraints of dataset creation.