💬 NLP

DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams

The paper introduces DRAGON, a benchmark dataset and evaluation framework designed to assess the ability of vision-language models to ground their answers in specific visual evidence within diagrams, addressing the limitation of high accuracy without reliable reasoning justification.

Original authors: Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande, Manan Suri, Dinesh Manocha, Puneet Mathur, Vivek Gupta

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande, Manan Suri, Dinesh Manocha, Puneet Mathur, Vivek Gupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a test where you have to look at a complex diagram—like a map, a circuit board, or a bar chart—and answer a question about it.

In the past, if a computer program (an AI) got the answer right, we assumed it "understood" the picture. But this new paper, DRAGON, argues that getting the right answer isn't enough. The AI might be cheating. It might be guessing the answer based on the words in the question or patterns in the data, without actually looking at the specific parts of the picture that prove the answer is true.

Think of it like a student taking a math test.

The Old Way: If the student writes "42" as the answer, the teacher gives them a checkmark. We don't know if they did the math or just guessed.
The DRAGON Way: The teacher demands to see the student's work. The student must circle the specific numbers they used, draw arrows to the formulas, and highlight the part of the graph that led to "42." If they can't point to the evidence, they didn't really solve the problem, even if the final number was correct.

What is DRAGON?

DRAGON is a new "test" (benchmark) designed to catch AI models that are guessing. It asks a simple but difficult question: "Show me exactly where in the picture you found the answer."

To pass this test, the AI has to draw a box around the specific visual clues it used. These clues could be:

A specific bar on a chart.
A label on a map.
A wire on a circuit diagram.
A legend or a title.

How They Built the Test

The researchers didn't just ask the AI to guess; they built a massive library of 11,000+ questions from six different types of diagrams (charts, maps, circuits, etc.).

For every single question, humans acted as the "truth-tellers." They looked at the diagram and the correct answer, then drew the exact boxes around the visual evidence needed to prove that answer.

Analogy: Imagine a detective solving a crime. The human annotators are the ones who say, "The clue isn't just the whole room; the clue is this specific muddy footprint on the floor." The AI then has to find that same footprint.

The Three Ways They Asked the AI

The researchers tried three different "prompts" (ways of asking the AI to do the task) to see if it could learn to show its work:

EDGE (The Direct Approach): "Here is the picture and the answer. Just draw the boxes around the evidence." (Like asking a student to just write the answer).
SAGE (The Step-by-Step Approach): "First, tell me what things you need to look at (e.g., 'the red bar' and 'the year 2020'). Then, draw the boxes around them." (Like asking the student to list their steps before solving).
VERGE (The Self-Correction Approach): "Draw your boxes. Now, look at your drawing again. Did you miss anything? Is your box too big? Fix it." (Like asking the student to double-check their work).

What They Found (The Results)

The results were a bit of a wake-up call for the AI community:

AI is good at guessing, bad at proving: Many AI models got the right answer, but when asked to draw the boxes, they failed miserably. They often pointed to the wrong part of the image or missed crucial details.
The "Black Box" Problem: Even the smartest AI models (like Claude Opus or Gemini) struggled to show their work. They could say "The answer is 50," but they couldn't reliably point to the "50" on the chart.
Some models are better than others: The "closed-source" models (the big, expensive ones from companies like Anthropic and Google) were better at finding the evidence than the open-source ones, but none of them were perfect.
Asking nicely helps a little: Using the step-by-step (SAGE) or self-correction (VERGE) methods helped the AI find the right spots a bit more often, but it didn't fix the fundamental problem. The AI still often missed parts of the evidence.

Why This Matters

The paper concludes that we cannot trust AI to reason about diagrams just because it gets the right answer. If an AI is used to analyze medical charts, financial graphs, or safety diagrams, we need to know why it made a decision.

DRAGON is a tool to force AI to stop guessing and start "showing its work." It's a new standard to ensure that when an AI says it understands a picture, it can actually point to the proof.

The Limitations

The authors admit their test isn't perfect. They use simple rectangular boxes to mark evidence. This works for big bars or blocks, but it's hard to use a box to mark a thin, winding wire in a circuit diagram or a curved arrow on a map. Also, sometimes different humans might disagree on exactly which part of the picture is the "most important" clue, which adds a little bit of subjectivity to the test.

In short: DRAGON is a new rulebook that says, "Don't just give me the answer; show me the evidence in the picture." And right now, most AI students are failing that test.

1. Problem Statement

Diagram Question Answering (DQA) requires models to interpret structured visual representations (e.g., charts, maps, circuit schematics, infographics). While recent Vision-Language Models (VLMs) achieve high accuracy on standard DQA benchmarks, a critical limitation exists: correct answers do not guarantee faithful reasoning.

Current models often rely on:

Textual correlations: Guessing answers based on language patterns in the question or dataset biases.
Dataset artifacts: Learning spurious correlations rather than interpreting visual evidence.
Lack of grounding: Failing to identify the specific visual regions (labels, axes, connectors, data points) required to verify the answer.

This "black box" behavior prevents reliable evaluation of whether a model truly understands the diagram's structure or is merely hallucinating plausible answers. There is a lack of standardized benchmarks that evaluate evidence-grounded reasoning—the ability to explicitly localize the visual evidence supporting a prediction.

2. Methodology

A. The DRAGON Benchmark

The authors introduce DRAGON, a multi-domain benchmark designed to evaluate evidence-grounded reasoning.

Data Composition: The dataset aggregates 11,664 annotated question instances from six established DQA datasets:
- AI2D (Scientific diagrams)
- ChartQA (Charts/Graphs)
- Circuit-VQA (Circuit schematics)
- InfographicsVQA (Infographics)
- MapIQ & MapWise (Geographic maps)
Annotation Task: Unlike standard QA, the model's task is not just to answer, but to predict bounding boxes corresponding to the visual elements required to justify the answer.
Evidence Types: Ground truth annotations include answer-bearing regions, textual labels, legends, axes, connectors, arrows, and spatially related structures.
Construction Pipeline:
1. Region Initialization: Uses existing annotations or Template-Based Region Transfer (using homography estimation and proportional scaling) to generate candidate regions for diagrams lacking pre-existing bounding boxes.
2. Human Verification: Annotators use the DIAGRAMS tool to verify, refine, or add regions to ensure the "Gold Evidence Set" ( $B^*$ ) represents the minimal visual evidence required for the answer.
3. Quality Control: Inter-annotator agreement is measured using Complete Visual Reasoning (CVR) (full chain coverage) and Core Evidence Alignment (CEA) (primary element grounding).

B. Evaluation Framework

The authors evaluate VLMs using three distinct prompting strategies to test how reasoning structure affects grounding:

EDGE (Evidence Detection via Grounding): A direct, single-step prompting approach where the model predicts bounding boxes immediately after receiving the image, question, and answer.
SAGE (Select and Ground Evidence): A two-stage approach.
- Stage 1: The model lists relevant visual elements textually (e.g., "Bar for 2019," "Y-axis label").
- Stage 2: The model predicts bounding boxes for the listed elements.
VERGE (Verify and Refine Grounded Evidence): A self-correction approach. The model generates an initial prediction (like EDGE), then re-examines the image and its own prediction to fix coordinates, add missing evidence, or refine box tightness.

C. Metrics

Performance is measured using:

Max Pairwise IoU (MPIoU): The best overlap between any predicted box and any ground-truth box (measures ability to find some relevant region).
Grounding IoU (GIoU): The IoU between the union of all predicted boxes and the union of all ground-truth boxes (measures coverage of the entire reasoning chain).
F1 Score: Precision and recall at the box level.

3. Key Contributions

New Evaluation Task: Introduced evidence-grounded reasoning, shifting the focus from answer accuracy to the localization of visual justification.
DRAGON Dataset: A human-verified, multi-domain test set of 2,445 instances with fine-grained bounding box annotations for reasoning evidence.
Systematic Analysis: Designed a framework (EDGE, SAGE, VERGE) to dissect how prompting strategies influence a model's ability to ground visual evidence.
Empirical Findings: Demonstrated that high answer accuracy does not correlate with strong evidence grounding, revealing a fundamental gap in current VLM capabilities.

4. Experimental Results

The study evaluated 8 state-of-the-art VLMs (including Claude Opus 4.6, Kimi K2.5, Gemini 3 Pro, and open-weight models like InternVL3.5 and Qwen3.5).

Weak Grounding Performance: Even the best-performing models struggle with evidence grounding.
- Best F1 scores were low: 9.5 on ChartQA, 5.7 on Circuit-VQA, and 4.0 on InfographicsVQA.
- Only MapIQ and AI2D showed moderate success (e.g., Claude Opus 4.6 reached ~21.8 F1 on MapIQ).
Localization vs. Coverage Gap: Models are better at finding some relevant region than capturing the complete reasoning chain.
- MPIoU (coarse localization) is consistently much higher than GIoU and F1 (complete evidence).
- Example: Claude Opus 4.6 achieved 53.7 MPIoU but only 21.8 F1 on MapIQ, indicating it found relevant areas but missed critical supporting evidence.
Prompting Sensitivity:
- Prompting strategies (EDGE, SAGE, VERGE) significantly impact performance, but no single strategy dominates all domains.
- VERGE often improved localization (MPIoU) but rarely improved complete evidence coverage (F1) significantly.
- Prompting helps models find regions but does not solve the issue of missing evidence.
Model Disparity:
- Closed-source models (e.g., Claude Opus 4.6) significantly outperform open-weight models across all domains.
- Example: On MapIQ, Claude Opus 4.6 (F1: 21.8) vastly outperformed InternVL3.5-38B (F1: 2.0).
Domain Difficulty:
- Circuit-VQA and InfographicsVQA are the most difficult domains due to dense structures, relational dependencies, and visual clutter.
- MapIQ and AI2D are relatively easier, likely due to clearer spatial layouts and explicit labels.

5. Significance and Conclusion

DRAGON addresses a critical blind spot in multimodal AI evaluation. By proving that models can answer correctly without grounding their reasoning in visual evidence, the paper highlights the risk of "shortcut learning" where models exploit textual priors rather than visual understanding.

Key Implications:

Faithfulness over Accuracy: Future DQA research must prioritize reasoning faithfulness (grounding) alongside answer accuracy.
Benchmarking Standard: DRAGON provides a standardized testbed to evaluate whether models can explicitly link predictions to visual evidence, moving beyond simple answer matching.
Future Directions: The authors suggest that training methods (e.g., Reinforcement Learning) should explicitly reward models for producing evidence-grounded reasoning aligned with human-verified regions.

In summary, DRAGON reveals that while current VLMs are proficient at answering diagram questions, they largely fail to demonstrate the visual reasoning capabilities required to justify those answers, necessitating a shift in how these models are evaluated and trained.