SciClaimEval: Cross-modal Claim Verification in Scientific Papers

1. Problem Statement

Scientific claim verification involves determining whether claims in research papers are supported or refuted by accompanying evidence (figures and tables). While the rise of generative AI has increased the volume of scientific submissions, existing tools struggle to automate the verification of multimodal claims.

Key Limitations of Existing Datasets:

• Synthetic Claims: Many datasets (e.g., SciVer) use claims written by experts or LLMs rather than extracting authentic claims from published papers.
• Artificial Negatives: Refuted claims are often created by simply negating supported claims (e.g., adding "not") or modifying the text, leading to heuristic shortcuts and artifacts rather than genuine reasoning challenges.
• Limited Modalities: Existing resources often focus solely on text or tables, lacking comprehensive cross-modal evidence (combining figures and tables) with diverse structural representations.

2. Methodology

The authors propose SciClaimEval, a new dataset constructed through a rigorous three-step pipeline designed to ensure authenticity and robustness.

A. Data Collection & Preparation

• Sources: Papers were collected from three domains: Natural Language Processing (NLP) and Machine Learning (ML) from arXiv/ACL/NeurIPS, and Medicine from PeerJ.
• Extraction: The team extracted main text, figures (PNG), and tables. Crucially, tables were preserved in multiple formats: LaTeX source, HTML, JSON, and PNG images.

B. Claim-Evidence Pairing

• Using regular expressions, the system identified sentences explicitly referencing specific tables or figures (e.g., "Table 1 shows...").
• Each sentence was paired with its corresponding evidence, including preceding sentences as short context to reduce ambiguity.

C. Novel Evidence Modification Strategy (The Core Innovation)

Unlike prior work that alters the claim to create a negative example, SciClaimEval modifies the evidence itself while keeping the claim authentic.

• Process: Annotators first verify if a claim is "Supported" by the original evidence. If supported, they modify the evidence (figure or table) to make the claim "Unsupported."
• Modification Operations:
  • Tables: Changing cell values, swapping rows/columns, adding/removing rows/columns.
  • Figures: Flipping graphs, swapping legends, exchanging subgraphs, swapping category labels, or adding fake data points.
• Result: This creates "Refuted" samples where the claim remains a genuine statement from a paper, but the evidence has been altered to contradict it, forcing models to reason about the visual/structural data rather than relying on text negation.

D. Annotation & Validation

• Annotators: 11 graduate students and expert researchers in NLP/ML.
• Labels: Supported, Subjectively Supported, Unsupported, or Skip.
• Human Baseline: Evaluated on 80 samples, achieving macro-F1 scores of ~88% (tables) and ~90% (figures).

3. Key Contributions

1. Authentic Claims: The dataset contains 1,664 samples from 180 papers where both supported and refuted claims are extracted directly from real scientific literature, avoiding synthetic generation.
2. Evidence-Centric Negatives: Introduces a novel approach of modifying the evidence (figures/tables) to create negative examples, eliminating the "negation bias" found in previous datasets.
3. Cross-Modal & Multi-Format: Provides diverse evidence types:
   • Figures: PNG images.
   • Tables: Available in PNG, LaTeX, HTML, and JSON, facilitating research on structured vs. unstructured table processing.
4. Comprehensive Benchmark: Evaluates 11 state-of-the-art Multimodal Large Language Models (MLLMs), including open-source (InternVL, Qwen, LLaVA) and proprietary (OpenAI o4-mini) models.

4. Experimental Results

The authors evaluated models using Macro-F1 and a stricter metric called Pair Accuracy (requiring the model to correctly predict both the supported and refuted versions of the same claim pair).

• Overall Performance:
  • Proprietary vs. Open Source: OpenAI's o4-mini outperformed all open-source models across all settings.
  • Table vs. Figure: Models performed significantly better on table-based evidence than figure-based evidence.
    • Table Example: o4-mini achieved 85.4 Macro-F1 on tables vs. 78.1 on figures (validation set).
    • Gap: A substantial performance gap remains between the best models and the human baseline, particularly for figures.
• Context Impact: Surprisingly, models often performed better without additional context (short context or full paper) than with it, suggesting that extra text can introduce noise or that captions already contain sufficient context.
• Pair Accuracy: Scores were consistently lower than Macro-F1, indicating that many models rely on superficial features or dataset biases rather than genuine reasoning. When forced to distinguish between a supported and a subtly modified refuted pair, performance dropped.
• Figure Modifications: Operations like "Graph Swap" and "Category Swap" were particularly challenging for models, while "Graph Flip" was slightly easier.

5. Significance and Future Impact

• Real-World Relevance: By using authentic claims and evidence modification, SciClaimEval better reflects the challenges of verifying scientific literature in the age of AI-assisted writing.
• Benchmarking Standard: It establishes a new, stricter benchmark for multimodal reasoning, highlighting that current MLLMs still struggle with visual reasoning (figures) and fine-grained structural changes in tables.
• Resource for Research: The inclusion of tables in multiple formats (JSON, LaTeX, HTML, PNG) provides a unique resource for developing models that can parse and reason over structured scientific data, not just image recognition.
• Limitations: The dataset currently has more table samples than figure samples, and the JSON conversion of tables had some minor accuracy issues (though mostly correctable).

In conclusion, SciClaimEval bridges the gap between synthetic benchmarks and real-world scientific reasoning, demonstrating that while MLLMs are making progress, significant challenges remain in accurately verifying scientific claims across diverse modalities.