💬 NLP

Helping Figures Tell their Story! Paper-Grounded Video Generation Explaining Complex Scientific Figures

This paper introduces MINARD, a novel pipeline and the FigTalk benchmark designed to generate narrated, region-grounded walkthrough videos from scientific figures and their source papers, effectively addressing the gap in current systems for explaining complex visual pipelines through step-by-step, paper-faithful storytelling.

Original authors: Ishani Mondal, Javad Baghirov, Jordan Boyd-Graber

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ishani Mondal, Javad Baghirov, Jordan Boyd-Graber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a science conference. The speaker flashes a complex diagram on the screen—a tangle of boxes, arrows, and labels representing a new computer system. They say, "The figure speaks for itself," and move to the next slide before you've even figured out where to look.

This paper argues that AI should be able to do what a good human speaker does: take that static, confusing picture and turn it into a step-by-step video tour that explains what is happening, why it matters, and where to look at every moment.

Here is the breakdown of their solution, MINARD, and their new testing ground, FigTalk, using simple analogies.

The Problem: The "Static Map" vs. The "Tour Guide"

Current AI systems treat scientific figures like a static photograph. They might say, "This is a box with an arrow," but they miss the story.

The Missing Piece: They don't know why the arrow points there, or that the orange box is the most important part because of a specific sentence in the research paper.
The Hallucination Risk: If you ask a standard video AI to explain a diagram, it might invent parts of the diagram that don't exist (like adding a red arrow where there is none) because it's trying to "guess" the story rather than reading the source material.

The Solution: MINARD (The Three-Person Team)

The authors built a system called MINARD (named after a famous 19th-century mapmaker who visualized complex data beautifully). Instead of one giant AI trying to do everything at once, MINARD acts like a small production team with three specific jobs:

The Narrator (The Scriptwriter):
- Job: Reads the entire research paper and looks at the figure.
- Analogy: Imagine a tour guide who has read the entire history book of the museum before stepping in front of the exhibit. They don't just say, "Here is a painting." They say, "This painting shows the battle of 1812, and notice how the general is pointing here because..."
- Key Feature: They use a "Critic" team to fact-check the script, ensuring the AI doesn't make up facts or skip important steps.
The Perception Team (The Spotter):
- Job: Looks only at the figure (ignoring the script for now) to find every single valid part: every text label, every box, and every arrow.
- Analogy: This is like a stage manager who has a master list of every prop on stage. They create a "whitelist" of valid things to point at. This prevents the AI from pointing at empty space or inventing a box that isn't there.
The Grounding Team (The Director):
- Job: Takes the script from the Narrator and the list of props from the Spotter, then decides exactly when to highlight which part.
- Analogy: This is the director saying, "Okay, when the guide says 'look at the engine,' the spotlight must hit only the engine, not the whole car." They ensure the highlight matches the words perfectly.

The Result: A Faithful Walkthrough

When MINARD creates a video:

It doesn't redraw the picture (which often leads to errors).
It keeps the original image and simply overlays highlights (like a laser pointer) that move in sync with the voiceover.
It explains the "why" by pulling facts from the paper, not just describing the "what" from the image.

The Test: FigTalk (The Exam)

To prove their system works, the authors created FigTalk, the first "exam" for this specific task.

The Setup: They collected 114 real scientific figures and the videos of humans explaining them in conferences.
The Challenge: They asked AI systems to recreate these explanations.
The Metric: They didn't just check if the AI sounded smart; they checked if the AI pointed at the right thing at the right time (Grounding) and if the story was factually correct based on the paper (Faithfulness).

The Findings

The "Hard" Figures Win: On simple diagrams, other AIs sometimes do okay. But on complex diagrams with many steps (the "Hard" tier), MINARD shines. It stays accurate while other systems get confused, point at the wrong things, or invent fake details.
Human Preference: When people watched the videos, they preferred MINARD's explanations 74% of the time over other methods. They found it easier to follow and more trustworthy.
The "Paper" Matters: The biggest improvement came from making the AI read the paper. Without the paper, the AI could describe the picture but couldn't explain the science behind it.

What It Is NOT (Limitations)

The authors are clear about what this system doesn't do yet:

It is designed for architecture diagrams (flowcharts, system maps). It is not currently built to explain bar charts, graphs, or statistical plots (which require a different kind of explanation).
It is slower than some other methods because it takes time to "think" and fact-check, but this slowness is what makes it accurate.

In short: MINARD is a system that turns a confusing scientific diagram into a clear, fact-checked video tour by having a team of AIs work together: one to write the script from the paper, one to find the parts, and one to direct the spotlight.

Technical Summary: Paper-Grounded Video Generation Explaining Complex Scientific Figures

Problem Statement

Scientific figures, particularly architectural diagrams and pipelines, compress complex methodologies into a single visual canvas. However, understanding these figures requires more than static interpretation; it demands a paper-grounded, step-by-step narration that aligns visual highlights with specific regions of the figure. Current video generation systems and benchmarks fail to provide this capability. Existing approaches treat figures as static images for captioning or isolated question-answering, or they generate animated explanations from abstract concepts (e.g., code or theorems) without grounding them to a specific source figure. Furthermore, presentation-generation systems often treat figures as opaque inserts, moving cursors across regions without reasoning about the logical order or scientific significance of components. Diffusion-based video generators often hallucinate unsupported details, failing to preserve the fidelity of the original scientific diagram.

The core challenge is to generate a video that:

Narrates a figure step-by-step based on the surrounding scientific paper.
Grounds each narration step to specific, validated regions of the original figure.
Maintains temporal alignment between the spoken explanation and visual highlights, adhering to multimedia learning principles (temporal contiguity, signaling, and segmentation).

Methodology: MINARD

To address these limitations, the authors introduce MINARD (Multimodal Interpretation of Narrated Architecture via Region Decomposition). MINARD is a pipeline that decomposes the task into three specialized, collaborative teams, separating the "what to say," "what is valid to point at," and "where to point" decisions.

1. The Narration Team ( $\mathcal{N}$ )

This team reads the full source paper and the target figure to generate an ordered, step-by-step narration. It employs a multi-agent workflow:

Retrieval ( $A_{ret}$ ): Assembles a figure-specific evidence bundle ( $K_F$ ) containing the caption, in-text references, relevant paragraphs describing the figure's stages, and contribution sentences. This grounds the narration in the paper's text rather than LLM priors.
Planner & Drafter ( $A_{plan}, A_{draft}$ ): The planner recovers a pedagogical sequence of concepts (e.g., inputs $\to$ modules $\to$ outputs $\to$ takeaway) based on $K_F$ and the figure structure. The drafter converts this plan into natural language narration segments.
Critic & Reviser ( $A_{crit}, A_{rev}$ ): Four specialized critics check the draft against $K_F$ for faithfulness (no hallucinated modules), coverage (no omitted branches), coherence (logical traversal order), and salience (emphasizing key contributions). A reviser agent merges feedback to produce the final narration.

2. The Perception Team ( $\Phi$ )

This team determines which parts of the figure are valid targets for highlighting, ensuring the system does not hallucinate locations. It operates independently of the narration:

Text & Vision ( $A_{ocr}, A_{vis}$ ): Uses OCR to extract text regions ( $T$ ) and open-vocabulary detection/segmentation (e.g., SAM3) to extract visual regions ( $V$ ) like icons, blocks, and arrows.
Clustering ( $A_{clus}$ ): Groups visually close detections (e.g., binding a label to its corresponding diagram block) into clusters ( $C$ ) to be highlighted as single units.
Builder ( $A_{reg}$ ): Assigns stable IDs and bounding boxes to create a whitelist of valid targets $U = T \cup V \cup C$ . This inventory is derived solely from the figure image.

3. The Grounding Team ( $\mathcal{G}$ )

This team aligns each narration step $s_i$ to regions in the whitelist $U$ through a staged, semantically constrained selection process:

Stage 1 (Narration-to-text): Grounds narration steps to OCR-derived text regions first (e.g., linking "STANZA" in the text to the STANZA module label).
Stage 2 (Candidate visual grounding): Uses spatial proximity and cluster membership to inherit candidate narration steps for visual regions based on nearby grounded text.
Stage 3 (Constrained visual selection): A vision-language model selects the correct narration steps for a highlighted region from the restricted candidate set, preventing broad over-highlighting.
Validation & Rendering: A deterministic validator removes invalid bindings outside $U$ and repairs incomplete selections. The renderer synthesizes the video by overlaying highlights on the original figure, synchronized with TTS audio.

Benchmark: FigTalk

The authors introduce FigTalk, the first benchmark for narrated architectural-figure walkthroughs derived from conference presentations.

Composition: 114 figure-video pairs. FigTalk-Gold (49 videos) contains fully verified human narration and exact grounding traces. FigTalk-Extended (65 videos) contains verified architectural explanations but with weaker or incomplete grounding traces.
Stratification: Figures are categorized into Easy, Medium, and Hard tiers based on structural complexity (component count, DAG depth, sub-figure count), not image size.
Metrics:
- Narration: Evaluated on Order Matching, Internal/External Faithfulness, Concept Coverage, and Takeaway Recall.
- Grounding: Uses Soft-DTW (frame-level similarity via VLM judge, applicable to all systems) and IoU-based DTW (strict region-level precision/recall for systems outputting explicit regions). For the Extended set, a rubric evaluates Component Faithfulness, Concept Coverage, and Excess Highlight Rate.

Key Results

Experiments on FigTalk demonstrate that MINARD significantly outperforms existing baselines, particularly on complex figures.

Narration Quality: Paper-grounded narration (MINARD) substantially improves Concept Coverage and Takeaway Recall compared to figure-only narration. It recovers scientific intent (e.g., method names, design rationales) that is invisible in the figure alone.
Grounding Performance:
- On FigTalk-Gold, MINARD achieves the best scores on both Soft-DTW and IoU-based DTW across all backbones (GPT-5, Claude, Gemini).
- Complexity Scaling: The performance gap widens significantly on Hard figures. While unconstrained VLM grounding and diffusion-based generators (e.g., Veo, CogVideoX) collapse on complex structures (often hallucinating or failing to preserve causal order), MINARD maintains high fidelity.
- Human Preference: In human studies, MINARD is preferred in 74% of narration comparisons and 63% of grounding comparisons over baselines, especially regarding faithfulness and the absence of excess highlighting.
Ablation Studies:
- Retrieval is the primary driver of scientific faithfulness; removing it causes a sharp drop in Concept Coverage and Takeaway Recall.
- Structured Decomposition: The staged grounding process (Perception $\to$ Grounding) is critical. Removing constraints leads to over-highlighting and loss of temporal alignment.

Significance and Claims

The paper claims that MINARD represents a shift from treating figures as opaque images to treating them as structured sequences of explainable components.

Modularity: By separating narration, perception, and grounding, the system makes failures interpretable and allows for targeted improvements.
Fidelity: Unlike re-rendering systems (e.g., Manim-based or diffusion models) that regenerate visuals and often drift from the source, MINARD preserves the exact component-level correspondence of the original figure, which is critical for scientific communication.
Educational Value: The system aligns with multimedia learning theory, providing temporally aligned, segmented explanations that guide attention to salient components, thereby improving comprehension of complex scientific pipelines.

The authors note limitations: the current scope is restricted to architecture-style figures (pipelines, system diagrams) rather than charts or statistical plots, and the pipeline is computationally more expensive than single-pass baselines due to the refinement loops. However, the results suggest that for complex scientific figures, structured, paper-grounded decomposition is essential for generating faithful, human-like explanations.