🤖 machine learning

MMClima: A Framework for Multimodal Climate Science Data and Evaluation

The paper introduces MMClima, a large-scale, expert-validated multimodal climate science framework comprising over 104,000 question-answer pairs across text, video, and figures, designed to benchmark and improve AI models' ability to reason across diverse climate data modalities.

Original authors: Muhammad Umer Sheikh, Hassan Abid, Khawar Shehzad, Ufaq Khan, Muhammad Haris Khan

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muhammad Umer Sheikh, Hassan Abid, Khawar Shehzad, Ufaq Khan, Muhammad Haris Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a robot to understand the complex, urgent story of climate change. Right now, most robots (AI models) are like students who have read a few short encyclopedia entries but have never seen a weather map, watched a documentary, or studied a scientific graph. They might know the word "hurricane," but they struggle to look at a chart showing wind speeds and explain exactly what it means.

The paper introduces MMClima, a massive new "textbook and test bank" designed specifically to train and test these robots on climate science. Here is how it works, broken down into simple concepts:

1. The Problem: The "Text-Only" Trap

Existing tests for climate AI are like a quiz that only asks questions about words. They are small, mostly text-based, and don't force the AI to look at pictures, charts, or video transcripts.

The Analogy: Imagine trying to learn to drive a car by only reading the manual, never looking at the road, the speedometer, or the traffic signs. The paper argues that to truly understand climate change, AI needs to "see" the data, not just read about it.

2. The Solution: A Giant, Multi-Format Library

The authors built MMClima, a framework containing over 104,000 expert-verified questions and answers.

The Ingredients: They didn't just scrape random websites. They gathered information from:
- Articles: Like Wikipedia pages (the "textbook").
- Videos: Transcripts from educational YouTube videos (the "lecture").
- Figures: Scientific charts, graphs, and maps from major reports like the IPCC (the "visual aids").
The Five Rooms: The data is organized into five specific "rooms" of climate science:
1. Air quality and what's in the sky.
2. Oceans and coastlines.
3. Ice sheets and glaciers.
4. Extreme weather (storms, heatwaves).
5. Policies and rules governments make.

3. How They Built It: The "Fact-Checking Factory"

You can't just ask a computer to make up 100,000 questions; it might lie or get things wrong. The authors built a "factory" pipeline:

Scraping: They pulled text from reliable sources.
Cutting: They chopped the text into small, manageable pieces (like cutting a long movie into scenes).
Claim Extraction: They used AI to pull out specific, verifiable facts (e.g., "Permafrost thaw releases methane").
Fact-Checking: They cross-referenced these facts with authoritative sources to ensure they were true.
Human Review: Real climate experts looked at the questions to make sure they made sense and weren't confusing. This is the "human-in-the-loop" part, acting like a strict teacher grading the test bank.

4. The Test: Putting the Robots to Work

The authors used this new library to test 28 different AI models (both free/open-source and expensive/closed-source).

The Challenge: The tests weren't just "What is the capital of France?" They were tricky, like:
- Cloze Tests: "When permafrost thaws, organic material becomes available for ______." (The AI must fill in the blank with the exact scientific term).
- Visual Tests: Looking at a graph of ocean heat and answering, "Which line shows the fastest warming?"
The Result:
- Most standard AI models struggled, especially with the visual charts and the "fill-in-the-blank" precision tests.
- The authors took a powerful open-source model (Llama 3.3 70B) and fine-tuned it specifically on their new data.
- The Winner: This custom-trained model (called MMCLIMA-70B-TXT) beat almost every other model, including the most expensive ones from big tech companies. It proved that if you give an AI the right "textbook" (data), it can become a climate expert.

5. The Takeaway

The paper concludes that to make AI useful for climate science, we need more than just general knowledge. We need:

Scale: A huge amount of data (100k+ questions).
Modality: The ability to read text, watch videos, and interpret charts.
Rigor: Expert validation to ensure the answers are scientifically accurate.

The authors are releasing the dataset, the tools to build it, and their trained model to the public, hoping this becomes the new standard for testing how well AI understands our changing planet.

Technical Summary: MMClima: A Framework for Multimodal Climate Science Data and Evaluation

Problem Statement

Climate change research increasingly demands AI systems capable of reasoning across text, dynamic visual content, and scientific figures. However, existing climate Question Answering (QA) benchmarks suffer from three primary limitations:

Scale and Scope: Most datasets are small (hundreds to a few thousand items), restricting their utility for large-scale training or robust evaluation.
Modality: Existing resources are predominantly unimodal (text-only) or rely on textual descriptions of figures rather than requiring direct pixel-level figure reasoning.
Validation: Automatically generated datasets often lack rigorous human validation, leading to noise and unreliable benchmarks, while expert-curated sets are too small for modern foundation models.

Consequently, general-purpose Large Language Models (LLMs) and Vision-Language Models (VLMs) struggle with fine-grained climate tasks such as subpanel localization, sign/unit fidelity, and event attribution, leading to unreliable performance in high-stakes decision-making contexts.

Methodology

The authors introduce MMClima, a large-scale multimodal framework designed to address these gaps through a modular, automated pipeline with human-in-the-loop validation.

1. Data Sources and Domains

The corpus spans five core climate science domains:

Atmospheric Composition & Air Quality
Oceanic and Coastal Dynamics
Cryosphere and Glacial Systems
Climate-Driven Extreme Events
Climate Policy, Governance, and Mitigation Pathways

Data is sourced from:

Textual: Wikipedia articles (structured foundational knowledge) and YouTube video transcripts (explanatory discourse).
Visual: Figures from Intergovernmental Panel on Climate Change (IPCC) Assessment Reports and Our World in Data (OWID), including statistical graphics, schematic diagrams, and geospatial maps.

2. Automated QA Generation Pipeline

The framework employs a multi-stage pipeline to generate over 100,000 QA pairs:

Retrieval & Segmentation: Articles and transcripts are retrieved and segmented using sliding windows (3200 chars for text, 500 chars for transcripts) to preserve semantic coherence.
Claim Extraction: An LLM ($gpt-4.1-nano$) extracts atomic, verifiable factual claims from chunks.
Deduplication: Claims are embedded and filtered using cosine similarity ( $>0.9$ ) to remove redundancy.
Fact Verification: A "FactGen" module cross-checks claims against authoritative sources (e.g., IPCC reports) using Retrieval-Augmented Generation (RAG) to filter unverifiable statements.
QA Synthesis: Validated claims are transformed into diverse formats (Multiple-Choice, Cloze, Free-form) using a different LLM ($Llama-3.3-70B-Instruct-Turbo$) to decouple extraction from synthesis and reduce bias.
Visual QA (VQA): A hybrid approach combines manual expert identification of visual insights with automated template-based generation to create figure-grounded questions.

3. Human-in-the-Loop Validation

To ensure scientific reliability, five domain experts invested over 210 hours in iterative review. They filtered ambiguous stems, unclear units, implausible distractors, and answer leakage. Inter-annotator agreement reached $\kappa = 0.83$ . This process reduced noise while preserving coverage, resulting in a final corpus of 104,902 expert-validated QA pairs.

4. Benchmarking and Evaluation

The framework evaluates 28 LLMs and 8 VLMs (both proprietary and open-source) in a zero-shot setting.

Metrics: Accuracy for MCQ/Yes-No; BERTScore for Free-form; and a composite metric (Exact Match + ROUGE-1 + Cosine Similarity) for Cloze tasks.
Baseline: The authors fine-tuned Llama 3.3 70B on the textual training split using Parameter-Efficient Fine-Tuning (LoRA) to create MMCLIMA-70B-TXT, a domain-adapted baseline.

Key Contributions

Large-Scale Multimodal Dataset: A corpus of 104k+ expert-validated QA pairs spanning text, video transcripts, and scientific figures across five climate domains.
Systematic Multimodal QA: Integration of figure-grounded tasks requiring direct visual reasoning, moving beyond textual descriptions of images.
QA Generation Framework: A modular pipeline combining automated claim extraction, RAG-based verification, and decoupled synthesis to ensure accuracy and reduce bias.
Extensive Benchmarking: The first systematic evaluation of 28 LLMs and 8 VLMs on climate science, covering both closed and open-source families.
Domain-Adapted Baseline: Release of MMCLIMA-70B-TXT, demonstrating the efficacy of fine-tuning on high-quality, domain-specific data.

Results

Textual QA: The fine-tuned MMCLIMA-70B-TXT achieved a new state-of-the-art overall score of 78.24, outperforming frontier closed-source models (e.g., GPT-5-Nano at 69.66) and strong open-source baselines (DeepSeek-Chat at 75.03).
Task Difficulty: Free-form QA was found to be comparatively easy (scores >90 BERTScore), while Cloze QA remained the most discriminative task (scores <50 for all systems), serving as a rigorous stress test for lexical and numerical precision.
Visual QA: Closed-source VLMs (e.g., GPT-5) generally led overall, but open-source models (e.g., Pixtral-Large-2411) showed competitive performance in specific sub-tasks.
Domain Adaptation: Fine-tuning significantly improved performance across architectures (tested on Qwen 2.5 7B and GPT OSS 20B), confirming that gains are not architecture-specific but stem from the quality of the supervision data.

Significance

The paper positions MMClima as a substantive advance in climate science evaluation, addressing the critical need for a benchmark that is simultaneously large-scale, multimodal, and rigorously curated. By unifying text, video, and figures, it provides a foundation for evaluating both language and vision-language models in high-stakes climate applications. The authors emphasize that while the dataset supports more reliable evaluation and development of systems for climate education and analysis, it is not a substitute for expert judgment in safety-critical decisions. The release of the dataset, pipeline, and model weights aims to standardize multimodal evaluation and support the evolution of climate reasoning capabilities in AI.