🤖 AI

TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data?

This paper introduces TerraBench, a comprehensive benchmark and executable framework designed to evaluate and advance the ability of AI agents to perform grounded, multi-step reasoning across heterogeneous Earth-system data by integrating language models with specialized scientific tools for environmental analysis, simulation, and verification.

Original authors: Dat Tien Nguyen, Thao Nguyen, Fadillah Adamsyah Maani, Huy M. Le, Muhammad Umer Sheikh, Numan Saeed, Muhammad Haris Khan, Salman Khan

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dat Tien Nguyen, Thao Nguyen, Fadillah Adamsyah Maani, Huy M. Le, Muhammad Umer Sheikh, Numan Saeed, Muhammad Haris Khan, Salman Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex mystery about the weather, but you have two very different kinds of detectives on your team:

The "Word Wizard" (Large Language Model): This detective is amazing at reading books, understanding language, and planning a strategy. They can tell you what steps to take to solve the problem. However, they have never seen a weather map, can't read a satellite photo, and if you ask them to calculate a number, they might just guess.
The "Data Cruncher" (Scientific Tools): This detective is a robot with a supercomputer. It can instantly read satellite images, run complex climate simulations, and calculate exact numbers. But it doesn't speak human language, can't understand a question, and needs someone to tell it exactly what to do.

The Problem:
Right now, these two detectives don't work well together. The Word Wizard can't use the Data Cruncher's tools effectively, and the Data Cruncher can't understand the Word Wizard's plans. This makes it very hard to build an AI that can actually help scientists make real decisions about climate change, farming, or disaster response.

The Solution: TerraBench and TerraAgent
The authors of this paper built a new testing ground called TerraBench and a new "manager" called TerraAgent to see if AI can finally learn to make these two detectives work as a team.

Think of TerraAgent as a project manager. When you ask a question like, "How will a hurricane affect the crops in Florida next week?", the manager:

Plans: Breaks the question down into steps.
Calls Tools: Tells the Data Cruncher to look up satellite images, check weather history, and run a simulation.
Checks Work: Looks at the results the robot gives back.
Reports: Writes the final answer in a clear, structured format.

The Test (TerraBench)
To see if this manager is good, the authors created a massive exam called TerraBench. It's not a simple multiple-choice test. It's more like a series of 403 complex "missions" that require the AI to:

Fundamentals: Read a map and a weather chart to answer a basic question.
Simulation: Pretend a disaster happens (like a flood) and use a simulator to predict the damage.
Verification: Check if the AI's answer matches real scientific papers and data.

The test is incredibly strict. It doesn't just care if the AI picked the right tool; it cares if the AI used the right settings on that tool and if the final number is correct within a tiny margin of error.

The Results: A Reality Check
The authors tested the smartest AI models available (like Claude Sonnet 4.6 and Qwen3.5) on this exam. The results were surprising:

Good at Planning, Bad at Math: The top AI models were decent at figuring out which tools to use and in what order (scoring about 59% on the "process" part).
Terrible at Precision: However, when it came to getting the final numbers right, they failed miserably. The best model only got the final answer correct about 23% of the time.
The "Almost" Trap: Most of the time, the AI picked the right tool but used the wrong settings (like looking at the wrong date or the wrong location), leading to answers that were close but scientifically useless.

The Analogy of the "Wrong Recipe"
Imagine you ask a chef (the AI) to bake a cake using a specific recipe (the scientific data).

The chef knows exactly which ingredients to grab (Tool Selection).
But when they measure the flour, they use a cup instead of a gram scale (Wrong Arguments).
The result is a cake that looks like a cake but tastes like sand.

The paper shows that current AI is great at knowing what to do, but it struggles to do it precisely enough to be trusted with real-world science.

Conclusion
The paper concludes that to build a truly useful AI for Earth science, we can't just give it access to tools. We need to teach it how to coordinate those tools with extreme precision, handle complex workflows, and ensure the final numbers are scientifically accurate. TerraBench is the first tool to measure exactly how far we are from that goal.

Technical Summary: TerraBench and TerraAgent

Problem Statement

Current climate and environmental decision-making requires reasoning across heterogeneous data sources, including gridded physical data, satellite imagery, geospatial context, and simulator outputs. A significant gap exists in the capabilities of existing AI systems:

Weather/Climate Foundation Models: Can forecast well but lack the ability to reason interactively in natural language.
Large Language Models (LLMs): Can reason in language but cannot operate directly on high-dimensional Earth-system data.

Existing benchmarks (e.g., OpenEarthAgent, ThinkGeo) typically isolate these capabilities into narrow tasks or focus primarily on tool-trace alignment, failing to evaluate whether an agent can jointly reason over satellite imagery, gridded data, GIS context, simulation outputs, and document-grounded evidence within a unified, auditable workflow. Consequently, real-world scientific workflows in Earth science remain underserved, and numeric-tolerance failures often go undetected.

Methodology

TerraAgent Framework

The authors introduce TerraAgent, a ReAct-style executable framework designed to couple LLM planning with scientific tools. Key architectural features include:

Workflow Orchestration: The model coordinates workflows, invoking domain-specialized tools while the quantitative outputs are generated by data retrieval, geospatial processing, forecasting, simulation, or explicit computation rather than language-model generation.
Tool Registry: The framework integrates 77 sub-tools across categories including reanalysis/environmental data retrieval, forecasts, satellite/EO processing, GIS/OpenStreetMap analysis, deterministic simulation, visualization, and web search.
Simulation Stack: Includes deterministic simulators for diverse impact domains, such as AquaCrop (crop-water), DSSAT (crop systems), CLIMADA (event impact), UTCI (heat stress), EnergyPlus (building demand), and SUMO (traffic disruption).
Output: Returns structured, provenance-bearing outputs (NetCDF, GeoTIFF, CSV, PNG) and an evidence trace.

TerraBench Benchmark

Built upon TerraAgent, TerraBench is a benchmark for grounded Earth-science reasoning comprising 403 extensive agentic tasks.

Structure: Organized into three tracks:
1. Fundamentals: Deterministic execution over gridded data, satellite imagery, and GIS layers.
2. Simulator-Grounded: Intervention, planning, and counterfactual reasoning under explicit assumptions.
3. Document-Grounded Verification: Reproducing or approximating published quantities using public data and tools.
Domains: Covers eight application areas: weather prediction, hurricane/cyclone analysis, infrastructure planning, emergency response, water resource management, agricultural adaptation, air quality, and environmental monitoring.
Reasoning Levels: Tasks are annotated across four levels (0–3), adapting Pearl's causal hierarchy:
- Level 0: Observational grounding (resolving spatiotemporal coordinates, aligning data grids).
- Level 1: Associational reasoning.
- Level 2: Interventional reasoning.
- Level 3: Counterfactual/retrospective reasoning.
Annotation Protocol: A rigorous, semi-automated, human-in-the-loop pipeline. Only 50.9% of proposed samples were retained after human review, and 74.4% of accepted items required multiple execution passes to finalize canonical traces. This prioritizes annotation depth and scientific grounding over raw scale.

Evaluation Methodology

The benchmark employs a fine-grained evaluation protocol that decouples process-level tool use from final-answer correctness:

Process Metrics: Six metrics including Instruction Validity (InstAcc), Tool Call Success Rate, Tool Selection Accuracy (ToolAcc), Argument Correctness (ArgAcc), and Workflow Order Consistency (OrderScore). These are combined into a composite ToolUseScore.
Outcome Metrics:
- Hit@tol: A strict pass/fail metric based on whether the absolute error is within a defined tolerance ( $\lambda = \max(\tau_{abs}, \tau_{rel}|y|, f)$ ).
- NumScore: A tolerance-aware scoring metric that provides partial credit for near misses using an exponential decay function (base 2) for errors exceeding the tolerance band.

Key Results

Model Performance

Empirical evaluation reveals significant limitations in current models, even among frontier systems:

Frontier Models: The top-performing model, Claude Sonnet 4.6, achieved a 59.22 ToolUseScore but only a 22.88 Hit@tol and 28.44 NumScore.
Open-Weight Models: The strongest open-weight model, Qwen3.5-35B, trailed significantly with a 39.95 ToolUseScore and 5.89 Hit@tol.
Process vs. Outcome Gap: A consistent pattern across all models is that process quality (ToolUseScore) is substantially higher than answer quality (NumScore/Hit@tol). For instance, Qwen3.5-9B achieved a non-trivial 31.18 ToolUseScore but only 1.20 Hit@tol.

Failure Analysis

Dominant Failure Modes: The primary bottlenecks are not tool selection errors but argument grounding and numeric misses. Over 84% of numerical answers fell outside acceptable error margins across all models.
Specific Errors: "Wrong Argument Values" and "Numeric Miss Beyond Tolerance" were the most universal failure modes.
Tool Usage: Stronger models utilized a broader subset of tools (reanalysis, GIS, forecast), while weaker models collapsed onto a narrower subset. Simulation and visualization tools were rarely used (e.g., <1.5% of calls), suggesting models optimize for direct textual answers rather than externalizing evidence.
Track Difficulty: Simulator-Grounded tasks were identified as the most difficult overall, while Fundamentals were the easiest.

Significance and Claims

The paper claims that TerraBench is the first benchmark to unify Earth observation imagery, gridded environmental data, GIS reasoning, deterministic simulation, and document-grounded verification under a single executable interface. Its significance lies in:

Unified Evaluation: It provides a unified, auditable workflow to compare Earth-science agent capabilities, addressing the fragmentation of prior benchmarks.
Tolerance-Aware Scoring: By pairing process-level metrics with numeric tolerance scoring, it exposes a systematic gap between tool orchestration proficiency and scientific correctness that prior "tool-trace-only" evaluations missed.
Reproducibility: TerraAgent and TerraBench offer a reproducible testbed with structured output contracts and artifact-backed computation, enabling the development of scientifically grounded LLM agents.
Realistic Constraints: The benchmark demonstrates that reliable Earth-science agents must go beyond simple tool access to coordinate heterogeneous workflows, parameterize tools precisely, and preserve artifact provenance.

The authors conclude that while frontier models outperform open baselines on process metrics, a significant gap remains in producing numerically correct, scientifically grounded outputs, indicating that current agents are not yet ready for robust, autonomous Earth-science workflows.