Technical Summary: MSEarth

Problem Statement

Despite the rapid advancement of Multimodal Large Language Models (MLLMs), their application in Earth science remains underexplored, particularly at the graduate level. Existing benchmarks often rely on synthetic data, simple figure-caption pairs, or textbook materials that lack the depth and complexity required for professional geoscientific reasoning. A critical gap exists in evaluating the ability of MLLMs to uncover Earth science phenomena from observational imagery when paired with the high-density reasoning, hypotheses, and evidence found in the main text of academic papers. Current approaches that generate questions solely from figure captions often oversimplify tasks, failing to capture the nuanced, domain-specific reasoning necessary for real-world scientific discovery.

Methodology

The authors introduce MSEarth, a comprehensive multimodal benchmark and dataset curated from high-quality, open-access Earth science publications. The construction process involves a scalable, adaptive framework with two primary innovations:

1. Refined Caption Generation: Unlike traditional benchmarks that use raw captions, MSEarth generates "refined captions" by integrating the original figure caption with relevant contextual information, hypotheses, evidence, and analytical reasoning extracted from the main body of the academic paper. This process transforms brief descriptions into detailed, scientifically meaningful image descriptions (averaging 136.29 tokens compared to 37.56 for raw captions).
2. Multi-Agent Automated Validation with Expert Oversight: To ensure question quality and difficulty alignment with graduate-level tasks, the authors employ a rigorous, multi-stage validation pipeline:
   • Automated Filtering: A multi-agent voting system utilizing diverse MLLMs (e.g., Qwen2.5-VL, InternVL, GPT-4o) evaluates generated questions. Questions are categorized based on model performance with and without refined captions:
     • Easy QA: Answerable without refined context.
     • Specialized QA: Requires domain-specific knowledge found in refined captions.
     • Hard QA: Requires high-performing models to interpret complex image content.
   • Expert Validation: Domain experts (Ph.D. candidates in Earth sciences) manually review and annotate the curated QA pairs to verify accuracy, relevance, and scientific reasoning, filtering out invalid or low-quality entries.

The dataset covers the five major spheres of Earth science (atmosphere, cryosphere, hydrosphere, lithosphere, and biosphere) across 64,560 publications, resulting in 289,891 figures and over 448,000 questions. The benchmark supports three task types: Scientific Figure Captioning, Multiple-Choice Questions (MCQ), and Open-Ended Reasoning.

Key Contributions

• MSEarth Benchmark: A graduate-level multimodal dataset containing 289,891 figures and 448,980 questions (including 2,784 MCQs and 1,411 open-ended questions in the test set), specifically designed to evaluate Earth science phenomenon discovery.
• Refined Caption Methodology: A novel approach to enriching image descriptions with paper-level context, enabling the generation of questions that test deep scientific reasoning rather than simple visual perception.
• Scalable Adaptive Framework: A semi-automated tool for VQA generation and machine-assisted filtering that combines multi-agent evaluation with expert human validation, providing a robust pipeline for domain-specific benchmark construction.
• Training Resources: A diverse training corpus (captioning, open-ended QA, MCQ) derived from the benchmark, used to demonstrate significant performance gains in post-training (instruction tuning and GRPO-based reinforcement learning) for open-source models.

Results

The authors evaluated a wide range of state-of-the-art MLLMs (both proprietary and open-source) on MSEarth:

• Performance Gaps: Current models struggle significantly with scientific reasoning tasks requiring specialized knowledge. While models often exceed 75% accuracy on perception-based questions (direct feature extraction), their performance drops sharply on reasoning tasks (e.g., Gemini-2.5-Pro-Thinking dropped from 77.06% on perception to 56.31% on reasoning).
• Proprietary vs. Open-Source: Proprietary models (e.g., GPT-4o, Gemini-3-Flash) consistently outperform open-source models across all tasks, though fine-tuned open-source models (e.g., Intern-S1) show substantial improvements.
• Impact of Training: Post-training on the MSEarth training set yielded substantial gains across all tasks, with the most significant relative improvements observed in reasoning tasks.
• Human Baseline: Human experts (Ph.D. students) significantly outperformed all MLLMs across all Earth science domains, highlighting the current limitations of AI in complex geoscientific reasoning.
• Evaluation Metrics: The authors introduced LLM-based metrics (OE-Eval and Cap-Eval) which demonstrated higher correlation with human judgment compared to traditional lexical overlap metrics (BLEU, ROUGE).

Significance

The paper positions MSEarth as a critical resource for advancing MLLMs in scientific reasoning, specifically addressing the lack of graduate-level benchmarks in Earth science. By bridging the gap between visual data and the deep contextual reasoning found in scientific literature, MSEarth provides a high-fidelity resource for developing and evaluating models capable of professional-level geoscientific discovery. The authors claim that their work not only identifies the limitations of current models in handling complex, domain-specific problems but also offers a scalable framework and training data to close the reasoning gap, fostering further research and innovation in multimodal scientific applications.