💻 computer science

Multimodal QUD: Inquisitive Questions from Scientific Figures

This paper introduces MQUD, a novel dataset and framework that extends the linguistic theory of Questions Under Discussion (QUD) to the multimodal domain by generating inquisitive, context-aware questions from scientific figures and their accompanying text, thereby enabling Vision-Language Models to perform high-level reasoning beyond simple visual extraction.

Original authors: Yating Wu, William Rudman, Venkata S Govindarajan, Alexandros G. Dimakis, Junyi Jessy Li

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yating Wu, William Rudman, Venkata S Govindarajan, Alexandros G. Dimakis, Junyi Jessy Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Reading with a Detective's Mindset

Imagine you are reading a mystery novel. A good reader doesn't just read the words; they ask questions. When they see a character hiding a letter, they think, "Why are they hiding that?" When they see a map with a red X, they wonder, "What's at that spot?"

This paper argues that when scientists read research papers, they do the exact same thing. They look at the text and the figures (charts, graphs, diagrams) and ask deep, curious questions to understand the story.

However, current AI models are like bad detectives. They can read the text and look at the picture, but they only ask shallow questions like, "What color is this line?" or "How many bars are there?" They miss the big picture.

This paper introduces a new way to teach AI to be a better detective. They call it Multimodal QUD (Questions Under Discussion).

The Problem: The "What" vs. The "Why"

Think of a scientific paper as a courtroom trial.

The Text is the lawyer's speech.
The Figure is the evidence (a photo, a chart, a fingerprint).

Current AI benchmarks are like a quiz that only asks: "How many fingers are in the photo?" or "What time is written on the clock?" These are easy, surface-level questions.

But real scientific curiosity is different. It asks: "Why did the suspect's fingerprints appear on the clock?" or "How does this photo prove the lawyer's theory?"

The authors found that existing AI models struggle to ask these deep questions because they don't understand how the text and the picture work together to tell a story.

The Solution: A New Dataset (MQUD)

To fix this, the researchers built a new training dataset called MQUD.

Who made it? They didn't just use computers to guess. They went to the source: the original authors of the scientific papers.
How did they do it? They asked these scientists: "When you wrote this paper and drew this chart, what questions were you trying to answer? What were you curious about?"
The Result: They collected 1,250 high-quality questions. These aren't just "What is the number?" questions. They are "Why does this pattern happen?" or "How does this result change our understanding?" questions.

They also categorized these questions into two types:

The "Picture-Only" Questions: Sometimes the chart answers the question all by itself (e.g., "Which bar is the tallest?").
The "Teamwork" Questions: These are the gold mine. The chart shows a weird pattern, but you need the text to explain why it's happening. The AI must combine the visual clue with the written story to solve the mystery.

The Training: Teaching the AI to "Look"

The researchers took a standard AI model (a Vision-Language Model) and gave it a special training course using their new dataset.

Think of it like teaching a student to read a map.

Before training: If you showed the student a map and a story, they might just say, "I see a mountain."
After training: They learned to say, "The story says the river floods here, and the map shows the water level rising, so the mountain is actually a flood zone."

They tested the AI with two clever tricks to see if it was really learning:

The "Missing Map" Test: They asked the AI to generate a question without showing it the picture. The AI got much worse at its job. This proved the AI was actually using the picture, not just guessing based on the text.
The "Wrong Map" Test: This is the most important one. They swapped the correct picture with a different picture from the same paper.
- Before training: The AI didn't care. It would generate a question even with the wrong picture, because it was just looking for any visual cue.
- After training: The AI realized, "Wait, this question doesn't make sense with this picture!" It became sensitive to the specific details of the image. It learned to look at the content, not just the presence of an image.

The Results: A Smarter Detective

The paper shows that after this training:

The AI stopped asking shallow questions like "What is the value of the red bar?"
It started asking deep, "teamwork" questions like "Why does the model behave differently in this specific scenario shown in the graph?"
It became much better at connecting the dots between the visual data and the written explanation.

Summary

In short, this paper teaches AI to stop just "looking" at pictures and start "thinking" about them in the context of a story. By training on questions generated by real scientists, the AI learned to ask the kind of curious, insightful questions that humans ask when they are truly engaged in scientific discovery. It moved from being a passive observer who counts bars on a chart to an active participant who understands the story the chart is telling.

1. Problem Statement

Current Vision-Language Models (VLMs) and scientific benchmarks (e.g., ChartQA, FigureQA) primarily evaluate perceptual reasoning—the ability to extract explicit values or count elements from a figure. These benchmarks fail to capture scientific curiosity and discourse-level reasoning, where figures in research papers often trigger implicit questions that require integrating visual patterns with textual context to be resolved.

Existing datasets do not distinguish between questions that can be answered by a figure alone versus those that require cross-modal reasoning (text + figure). Consequently, models often generate generic, low-level questions or fail to "ground" their reasoning in the specific content of a figure, relying instead on visual priors or generic text patterns.

2. Methodology

A. Theoretical Framework: Multimodal QUD

The authors extend the linguistic theory of Questions Under Discussion (QUD) to the multimodal domain. In standard QUD theory, discourse is organized around implicit questions raised and resolved by assertions.

Extension: In this framework, figures act as discourse participants that raise implicit questions ( $Q_F$ ) which the surrounding text ( $C_F$ ) helps resolve.
Taxonomy: The authors define six question types adapted for multimodal discourse:
1. Cause: Why does a pattern occur?
2. Comparison: How do elements differ?
3. Extent: To what degree does a phenomenon occur?
4. Consequence: What are the implications?
5. Procedural: How is a method achieved?
6. Concept: What does a visual representation signify?
Distinction: They categorize questions into Figure-Driven (answerable by the figure alone, e.g., Comparison, Extent) and Integration (requiring text + figure, e.g., Cause, Consequence).

B. The MQUD Dataset

The authors constructed MQUD, a dataset of 1,250 validated multimodal QUDs derived from 56 scientific papers across NLP, Machine Learning, and Astronomy.

Generation: Candidate questions were generated using GPT-4o, conditioned on the figure, caption, abstract, and relevant text paragraphs.
Annotation: 17 original paper authors (domain experts) annotated the data across seven dimensions:
- Salience: Is the question relevant to the paper's core argument?
- Figure Usefulness: Is the figure necessary to resolve the question?
- Answer Source: Can the figure alone answer it, or is text required?
- Correctness & Quality: Factuality and linguistic quality.
Scale: 703 questions were human-annotated; the remaining 547 were annotated by a validated LLM judge (GPT-5-mini) with high agreement (88% precision on correctness).

C. Diagnostics for Grounding

To measure if models genuinely understand figure content, the authors introduced two reusable diagnostics:

Relative Information Gain (rIG): Measures how much harder it is to predict a question when the figure is removed ( $L_{text-only} - L_{multimodal}$ ). A high rIG indicates the model relies on the figure.
Figure Swap Diagnostic: Tests content-specific grounding. The model is given the correct text but a wrong figure from the same paper.
- Hypothesis: If a model has true grounding, a wrong figure should hurt performance more than having no figure at all (because the model expects specific visual cues). If it relies on generic visual priors, a wrong figure might still help.

D. Model Training

The authors fine-tuned Qwen3.5-9B using LoRA on the MQUD dataset. The training objective was to generate the implicit question ( $Q_F$ ) given the trigger context (Title, Abstract, Figure, Caption).

3. Key Contributions

MQUD Dataset: The first dataset explicitly targeting inquisitive questions arising from figure-text interactions in scientific papers, annotated by original authors for scientific utility.
Multimodal QUD Framework: A formal extension of QUD theory to multimodal discourse, distinguishing between figure-driven and integration-type questions.
Novel Diagnostics: Introduction of rIG and Figure Swap metrics to rigorously test whether VLMs possess content-specific grounding rather than just visual sensitivity.
Empirical Evidence: Demonstration that fine-tuning on MQUD shifts model behavior from generic visual queries to deep, context-aware scientific inquiry.

4. Results

Performance on Diagnostics

Content-Specific Grounding (Swap Test):
- Baseline (Zero-shot GPT-4o/Qwen): Showed a "generic bias." When given a wrong figure, the loss was lower than having no figure at all ( $\Delta < 0$ ), meaning they relied on generic visual cues rather than specific content. Only 18% of cases showed content-specific grounding (wrong figure hurt more than no figure).
- SFT Model: After fine-tuning, 76% of cases showed content-specific grounding ( $\Delta > 0$ ). The model was penalized for mismatched figures, proving it learned to rely on specific visual details.
Visual Information Gain (rIG):
- The rIG score increased from 0.60 (Base) to 0.97 (SFT), indicating the model became significantly more dependent on the figure for question generation.
- Ablation: A text-only SFT model achieved high swap rates (76%) but low rIG (0.27), proving that high swap rates alone do not guarantee visual grounding; the model must actually use the visual information.

Question Quality

Depth: SFT-generated questions were preferred by human experts over baselines in 75% of cases for "depth" and 78% for "diversity."
Style: The base model often produced verbose meta-commentary before asking a question. The SFT model generated concise, figure-grounded questions (e.g., asking why a pattern exists rather than what the pattern is).
Comparison with GPT-4o: While GPT-4o is sensitive to figures (rIG = 0.72), it lacks content-specific grounding (18% swap positivity). Fine-tuning closed this gap, achieving both high sensitivity and specificity.

5. Significance

Advancing Scientific AI: This work moves beyond simple "visual QA" (reading a chart) to "scientific ideation" (asking why a chart matters). This is crucial for developing AI agents that can truly assist in scientific discovery.
Auditing VLMs: The proposed diagnostics (rIG and Swap) provide a robust method to audit whether VLMs are hallucinating or genuinely grounding their reasoning in visual data, a critical step for reliable scientific applications.
Discourse Theory: It successfully bridges linguistic discourse theory (QUD) with computer vision, offering a new paradigm for understanding how humans and machines process multimodal scientific literature.
Future Applications: The framework and dataset serve as a foundation for building advanced scientific reading assistants, more rigorous QA benchmarks, and tools to audit the reasoning capabilities of large multimodal models.