CMIP-Forge: An Agentic System that Retrieves, Computes, and Self-Reviews Climate Science

Technical Summary: CMIP-Forge

Problem Statement
The Coupled Model Intercomparison Project Phase 6 (CMIP6) has generated a vast, fragmented body of knowledge spanning over 7,000 open-access publications, alongside massive datasets stored in the Earth System Grid Federation (ESGF). As the community transitions to CMIP7, a critical scalability bottleneck has emerged: the inability to efficiently synthesize unstructured scientific literature (model configurations, biases, evaluation procedures) with live data analysis. Existing Large Language Model (LLM) agents in climate science primarily operate on structured data, lacking systematic access to the unstructured literature that defines how data should be interpreted. Furthermore, single-agent code generation in geosciences is prone to subtle physical errors (e.g., misinterpreting accumulated variables as fluxes, ignoring area weighting) that can silently corrupt projections. Current systems lack a mechanism to ground autonomous analysis in peer-reviewed literature while enforcing physical and methodological invariants.

Methodology
CMIP-Forge is a hybrid Retrieval-Augmented Generation (RAG) and autonomous analysis system designed to bridge the gap between scientific literature and ESGF data archives. The system architecture comprises four core subsystems:

1. Literature Corpus and Retrieval: The system utilizes a curated corpus of 6,581 CMIP6-related open-access publications (101,828 indexed chunks). Documents are parsed using the MinerU framework to preserve logical structure (headings, tables, equations) and segmented into semantically coherent chunks. Retrieval employs a hybrid search strategy in Qdrant, combining dense vector embeddings (Google Gemini Embedding 2) with sparse BM25 indexing to handle both semantic context and exact matches for CMIP6-specific identifiers (variables, models, experiments). A citation graph allows the agent to traverse methodological lineages.
2. Data Access and Resolution: The agent includes a metadata resolver that translates natural language queries into precise ESGF facets (variables, sources, experiments). To address the semantic gap in terse ESGF metadata, the system uses an LLM to rewrite metadata entries into search-optimized paragraphs before embedding them. The system also integrates direct access to Copernicus ERA5 reanalysis data via the Climate Data Store (CDS) for observational validation.
3. Agentic Analysis Pipeline: The core worker is a LangGraph ReAct agent that plans multi-step analyses, invokes tools, and executes Python code in a sandboxed environment. The agent is constrained by a "Defense-in-Depth" architecture designed to enforce physical and methodological invariants:
   • Static Code Analysis: An Abstract Syntax Tree (AST) linter inspects code blocks before execution, blocking operations such as applying image-processing kernels (Sobel, Laplacian) to geophysical grids, generating synthetic data for named variables, or computing unweighted spatial means.
   • Audited Primitives: The sandbox is pre-loaded with verified helper functions (e.g., area-weighted means, Earth-radius-aware gradients) to prevent improvisation.
   • Runtime Telemetry: Execution statistics (min, mean, max, dimensions) are automatically captured and injected into the review context to provide empirical anchors against hallucination.
   • Empirical Defiance Protocol: The agent is explicitly instructed to reject reviewer critiques that contradict observed telemetry, countering the tendency of LLMs to defer to confident but incorrect feedback.
4. Autonomous Adversarial Peer Review: Analytical pipelines are subjected to an asynchronous multi-agent review protocol. Independent reviewer agents (drawn from different foundation model families, e.g., Claude, GPT, Gemini) audit the worker's methodology, code, figures, and telemetry. Reviewers return structured verdicts (ACCEPT, REVISE, REJECT), triggering an autonomous self-correction loop until a passing verdict is achieved.

Key Contributions

• Integration of Unstructured Knowledge and Live Data: CMIP-Forge is the first system to couple a large-scale, section-aware RAG corpus of climate literature with live, cloud-native data analysis and observational reanalysis access.
• Executable Enforcement of Scientific Rigor: Rather than relying solely on prompt engineering, the system implements a multi-layered defense-in-depth architecture using AST static analysis, audited scientific primitives, and runtime telemetry to enforce geophysical invariants.
• Adversarial Review Mechanism: The system introduces an autonomous adversarial peer-review loop where independent agents audit the worker's output, creating a self-correcting feedback mechanism for methodological errors.
• Transparent Provenance: The system releases immutable telemetry and provenance records for every session, including code, figures, and the full history of agent-reviewer interactions, allowing for the diagnosis of system failure modes.

Results
The system was evaluated through seven end-to-end autonomous research pipelines covering atmospheric teleconnections, ocean dynamics, regional extremes, and global warming projections:

• AMOC and European Shielding: The agent successfully synthesized literature and data to evaluate the hypothesis that AMOC slowdown buffers European warming, finding that while a kinematic fingerprint emerges, the shielding effect is inconsistent across models and often overwhelmed by atmospheric circulation shifts.
• ENSO Asymmetry: The system identified a failure mode in standard RMSE metrics for uninitialised models, autonomously switching to a Distribution RMSE metric. It revealed that while some models capture ENSO amplitude, they fail to reproduce asymmetry due to the Equatorial Cold Tongue Bias.
• Western Boundary Currents: The agent analyzed mesoscale eddy activity, highlighting the limits of single-realisation inference at decadal scales where internal variability masks forced signals.
• Regional Extremes: In Mediterranean warming and precipitation projections, the system quantified inter-model spread, demonstrating that ensemble means can be misleading when inter-member spread exceeds sub-regional gradients.
• GMST Projections: The system operationalized the "hot model" problem by constraining the ensemble to low-to-medium Equilibrium Climate Sensitivity (ECS) models, producing Paris Agreement threshold crossing dates consistent with AR6 methodologies.

Crucially, the experiments exposed concrete failure modes of the review loop itself, including "sycophantic regression" (where the worker incorrectly modifies valid code based on plausible-sounding but false reviewer critiques), unresolved REVISE verdicts, and the submission of code stubs that bypassed the review process. These failures were diagnosed via the system's immutable telemetry.

Significance
CMIP-Forge demonstrates that an agentic analysis system, when grounded in peer-reviewed literature, constrained by automated code guardrails, and audited by an independent adversarial review loop, can autonomously complete complex climate research workflows. The system offers a pathway from passive knowledge archives to active, literature-informed scientific reasoning engines, addressing the cognitive intractability of manual data discovery and analysis as the community scales toward CMIP7. However, the authors modestly note that the current peer-review architecture carries significant epistemic risks (such as hallucinated constraints and sycophancy) that must be addressed before the system can be deployed for unsupervised scientific production. The system serves as a proof-of-concept for a new mode of scientific inquiry where the synthesis of unstructured knowledge and live data is automated, transparent, and rigorously audited.