🤖 AI

SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

This paper introduces SciFigQual-Bench, a novel full-text contextual benchmark for scientific figure quality assessment that evaluates images across five dimensions using expert-annotated data from top computer science conferences, alongside the SFQ-Agent framework which achieves state-of-the-art automated scoring performance.

Original authors: Zihan Deng, Chuanzhi Xu, Huiqi Liang, Haoyang Li, Xiaozhen Zhong, Lequan Yu

Published 2026-07-30

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Zihan Deng, Chuanzhi Xu, Huiqi Liang, Haoyang Li, Xiaozhen Zhong, Lequan Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the clues are scattered across three different rooms. In one room, you have a photograph. In another, a handwritten note describing the photo. In the third, a diary entry where someone mentions the photo while telling a story. To solve the case, you can't just look at the photo; you have to check if the note matches the picture, and if the diary story makes sense with what's actually in the photo. This is exactly the challenge scientists face when reviewing academic papers. For a long time, computers were great at looking at photos and saying, "This is blurry," or "The colors are nice," but they were terrible at reading the story around the photo. They couldn't tell if a graph was lying about its data or if the caption was describing the wrong chart. This matters because in science, a picture isn't just art; it's proof. If the proof doesn't match the story, the whole experiment might be a bust.

Enter SciFigQual-Bench, a new tool designed to teach computers how to be better scientific detectives. The researchers behind this project realized that existing tools were like a judge who only looks at a painting without reading the title or the artist's statement. They built a massive database of over 7,600 real scientific figures from top computer science conferences, but with a twist: every single image is glued to its caption and the specific paragraphs in the paper that talk about it. They then asked human experts to grade these images on five things: how clear they are, how well they are laid out, if the caption matches the image, if the text in the paper matches the image, and if the image is trying to trick the reader.

The paper's main finding is that when you force a computer to look at the image, the caption, and the text separately before combining them, it becomes a much better judge. They created a system called SFQ-Agent that acts like a team of specialists: one looks at the pixels, one reads the text, and a third one compares the notes. When they tested this on 1,200 images, SFQ-Agent made fewer mistakes than any other method, getting the score right within one point of the human experts 93.4% of the time. The paper evaluates standard "all-in-one" models as baselines and finds that these single-pass approaches often get confused, mixing up what they see with what they read. Instead, the paper suggests that breaking the job into steps—checking the visual evidence first, then the text, then fusing them—is the secret to getting it right. The results are measured and specific: the best system had an average error of just 0.418 points on a 1-to-10 scale, proving that for scientific figures, context is king.

Technical Summary: SciFigQual-Bench

Problem Statement

Existing Image Quality Assessment (IQA) paradigms are predominantly designed for natural photographs or AI-generated content, relying on low-level statistics or perceptual metrics (e.g., PSNR, SSIM, NIQE) that evaluate visual clarity in isolation. These methods fail to address the unique requirements of scientific figures, which derive their evaluative value from their integration with the full academic manuscript.

Current scholarly benchmarks for charts and diagrams are limited by several factors:

Isolation: They often evaluate figures as isolated crops, decoupled from captions and citing paragraphs.
Surface-Level Focus: They prioritize visual-surface comparisons or visual question answering (VQA) accuracy rather than holistic quality assessment.
Missing Context: They fail to verify caption alignment, citation relevance, or the risk of visual misleadingness (e.g., axis truncation, missing baselines) that can only be detected by cross-referencing the figure with the text.

Scientific figure quality is inherently tri-modal, depending on the visual content, the figure caption, and the claims made in the citing paragraphs of the manuscript. Existing single-modality or loosely coupled approaches cannot effectively assess these interdependencies.

Methodology

1. SciFigQual-Bench Dataset Construction

The authors propose SciFigQual-Bench, a benchmark designed to evaluate scientific images within their full-manuscript context.

Data Source: The dataset comprises 7,609 curated figures extracted from 1,144 qualified papers across top-tier computer science conferences (ACL, EMNLP, ICML, NeurIPS) published between 2020 and 2025.
Context Binding: Unlike previous datasets, each figure instance is strictly bound to its source PDF. The pipeline extracts the figure image ( $I$ ), its caption ( $c$ ), and the specific citing paragraphs ( $T$ ) from the body text using index-driven resolution (matching figure indices in the PDF).
Annotation: 6,308 instances were independently scored by multiple domain experts on a unified 1–10 scale across five orthogonal dimensions:
1. Visual Clarity (VC): Legibility of text, marks, and encoding.
2. Structure & Layout (SL): Composition, panel organization, and avoidance of "chartjunk."
3. Caption Consistency (CC): Alignment between the caption and visible content.
4. Context Consistency (CTX): Alignment between claims in the citing text and what the figure supports.
5. Misleading Risk (MR): Likelihood of reader misinterpretation (e.g., truncated axes).
Gating Mechanism: To avoid penalizing missing metadata, the benchmark employs an L1 evidence gating strategy. If a caption or citing text is absent, the corresponding dimension (CC or CTX) is marked as null and excluded from the overall score calculation.

2. SFQ-Agent: Staged Cross-Modal Evaluation

To enable automated evaluation that mirrors human expert reasoning, the authors introduce SFQ-Agent, a staged cross-modal judge designed to be auditable and evidence-grounded. It avoids the "black box" nature of monolithic Vision-Language Models (VLMs) by separating evidence collection from fusion.

The agent operates in four stages:

Step 0 (Gating): Determines which dimensions are evaluable based on the presence of $I$ , $c$ , and $T$ .
Step 1 (Vision Evidence): A vision module (using PaddleOCR-VL and classical CV descriptors) extracts visual facts (text regions, layout, axis units) to score VC and SL. Crucially, the language module does not access pixels at this stage.
Step 2 (Language Evidence): An LLM analyzes only the caption and citing text (without pixel input) to extract textual claims and rule violations to support CC and CTX scoring. This prevents the generation of plausible but unfaithful rationales.
Step 3 (Cross-Modal Fusion): A judge fuses the structured evidence from Steps 1 and 2 to score CC, CTX, and MR. It specifically looks for conflicts (e.g., a figure showing an upward trend while the text describes degradation).
Step 4 (Runner): A deterministic post-processor locks VC and SL scores to the vision outputs, applies rule-based caps for MR, and aggregates the final scores.

The paper evaluates three protocols on a fixed test subset (eval1200):

Direct: A single end-to-end VLM prompt.
Sidecar: A single prompt augmented with OCR/CV side features.
SFQ-Agent: The full staged pipeline described above.

Key Results

Experiments were conducted on eval1200 (1,200 stratified instances) using 11 state-of-the-art VLM backends.

Performance: The SFQ-Agent equipped with GPT-5.6-Sol achieved the best overall performance:
- Mean Absolute Error (MAE): 0.418 (lowest among all configurations).
- Within-1 Consistency: 93.4% (highest rate of predictions within ±1 point of human gold labels).
- Spearman Correlation: 0.598.
Protocol Comparison:
- SFQ-Agent consistently outperformed both Direct and Sidecar protocols across all backends.
- The Sidecar protocol (adding OCR features to a single prompt) showed modest improvements over Direct, particularly for weaker encoders, but failed to match the staged fusion of the Agent.
- The results indicate that gains are driven by protocol–evidence alignment (separating visual and textual verification) rather than just model scale.
Failure Analysis:
- Caption Consistency (CC) and Context Consistency (CTX) were identified as the dominant axes of disagreement between humans and models.
- Monolithic judges often conflated visual perception with textual verification, leading to hallucinated rationales. The staged approach successfully mitigated this by enforcing modality separation.
- Qwen-VL-Max showed a significant gap between calibration (W-1) and ranking (Spearman) in Direct mode, which improved with staging but still struggled with caption verification.

Significance and Claims

The paper claims the following contributions and significance:

First Full-Manuscript Benchmark: SciFigQual-Bench is the first benchmark to bind scientific figures to their captions and citing paragraphs from the source PDF, moving beyond isolated visual evaluation to evidence-grounded, full-context assessment.
Auditable Evaluation Framework: By proposing SFQ-Agent, the authors demonstrate that staged, cross-modal evaluation is superior to monolithic VLM prompting for scientific tasks. The design ensures that every score is traceable to specific evidence (visual facts or textual claims), addressing the "faithfulness" problem in LLM-as-judge paradigms.
Diagnostic Capability: The benchmark serves as a stress test for tri-modal scientific literacy. It reveals that current models struggle most with verifying the consistency between figures and the narrative claims in the text (CC and CTX), a gap that existing chart QA or text QA benchmarks have not addressed jointly.
Practical Utility: The framework provides a reproducible testbed for elevating AI-assisted figure inspection from simple visual QA to context-aware scientific reasoning, which is critical for peer review and quality control in scientific publishing.

The authors position this work not as a solution to all scientific image problems, but as a necessary step toward rigorous, context-aware quality assessment that respects the tri-modal nature of scientific communication.