Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agents

The paper introduces Earth-Agent, a novel agentic framework that unifies RGB and spectral Earth observation data within an MCP-based tool ecosystem to enable complex, multi-step quantitative reasoning, accompanied by the Earth-Bench benchmark for comprehensive evaluation of such capabilities.

The Gist

Imagine you are trying to understand the health of our entire planet. You have a massive library of satellite photos, weather data, and ocean measurements. In the past, trying to make sense of this data was like hiring a very smart but very narrow-minded intern.

The Old Way: The "One-Tool" Intern
Previously, the AI tools used for Earth observation were like interns who only knew how to look at color photos (RGB images).

• If you asked them, "Is this a forest?" they could say yes or no.
• But if you asked, "How much water is in the soil?" or "What is the temperature of the ocean surface?" they would freeze. They couldn't see the invisible data (like heat or chemical signatures) hidden in the raw satellite signals.
• They could only look at one picture at a time. If you wanted to track a hurricane over a month, they couldn't connect the dots between 30 different daily photos.
• Most importantly, they couldn't use tools. They had to guess the answer based on what they memorized during training, rather than going out and calculating it.

The New Way: Earth-Agent
The paper introduces Earth-Agent, which is like upgrading that intern to a super-powered scientific detective.

Here is how it works, using some simple analogies:

1. The "Swiss Army Knife" Toolkit

Instead of just looking at pictures, Earth-Agent has a massive toolbox with 104 different specialized tools.

• The Analogy: Imagine you are a chef. The old AI was a chef who could only chop vegetables. Earth-Agent is a chef who has a blender, a sous-vide machine, a thermometer, a scale, and a spice grinder.
• How it helps: If you ask, "How dry is the land in California?" Earth-Agent doesn't just guess. It grabs a tool to measure vegetation, another to measure heat, combines them to calculate a "drought index," and gives you a precise number. It can handle color photos, heat maps, and raw chemical data all at once.

2. The "Chain of Thought" Detective

Old AI tried to answer complex questions in one giant leap, which often led to mistakes. Earth-Agent uses a step-by-step reasoning process (called ReAct).

• The Analogy: Imagine you are trying to solve a mystery.
  • Old AI: "I think the butler did it because he looks suspicious." (Guessing).
  • Earth-Agent: "Step 1: I need to check the alibi. Step 2: I need to check the weapon. Step 3: I need to compare the timeline."
• How it helps: If you ask, "How has the building volume in Shanghai changed over 40 years?" Earth-Agent doesn't just guess. It:
  1. Finds the 40 years of data files.
  2. Calculates the volume for each year.
  3. Compares the start and end points.
  4. Calculates the percentage change.
     It breaks a huge, scary problem into small, manageable steps.

3. The "Team of Experts" (The Benchmark)

To make sure this new detective is actually good, the researchers built a giant test called Earth-Bench.

• The Analogy: Think of this as a "Driver's License Test" for AI, but instead of driving a car, the AI has to drive a spaceship through a storm.
• The Test: They created 248 difficult questions (like "Count the number of drought spikes in the Yellow River") and gave them to Earth-Agent and other AI models.
• The Result: Earth-Agent didn't just get the right answer; it showed its work. The researchers checked not just the final answer, but every step the AI took to get there. This ensures the AI isn't just lucky; it's actually thinking correctly.

Why Does This Matter?

Before this, if a scientist wanted to study climate change, they had to write complex computer code to process the data, which took days.

• With Earth-Agent: A scientist can just ask a question in plain English: "Show me the trend of building growth in New York from 1990 to 2020."
• The Agent figures out which data to grab, which math to do, and which tools to use, and gives the answer in minutes.

In Summary:
Earth-Agent is the first AI that acts like a real Earth Scientist. It doesn't just "see" the planet; it measures it, calculates it, and reasons about it using a vast library of scientific tools, turning complex satellite data into clear, actionable answers.

Technical Summary

1. Problem Statement

Earth Observation (EO) is critical for understanding the Earth system, yet current AI approaches face significant limitations:

• Modality Constraints: Existing Multimodal Large Language Models (MLLMs) are primarily restricted to RGB imagery, failing to process diverse EO data types like thermal infrared (TIR), synthetic aperture radar (SAR), hyperspectral data, or processed geospatial products (e.g., from Google Earth Engine).
• Reasoning Depth: Current methods rely on single-step or shallow reasoning (e.g., simple classification or captioning) and struggle with complex, multi-step scientific analysis requiring quantitative spatiotemporal reasoning.
• Tool Integration: MLLMs lack the ability to dynamically integrate external scientific tools or expert models, limiting their utility to the static knowledge encoded in their parameters.
• Evaluation Gaps: Existing benchmarks focus on final answers, overlooking the reasoning trajectory (the steps taken to reach a solution), and lack systematic protocols for evaluating tool usage in complex workflows.
• Scalability: Most systems process only one or a few images at a time, making them unsuitable for large-scale EO corpora.

2. Methodology: Earth-Agent Framework

The authors propose Earth-Agent, the first agentic framework designed to unify perceptual (RGB) and spectral (raw data/products) EO data within a structured tool ecosystem.

A. Core Architecture

• ReAct-Based POMDP: The agent operates as a Partially Observable Markov Decision Process (POMDP). It iteratively performs a ReAct (Reasoning + Acting) loop:
  1. Think: The LLM controller analyzes the task goal and interaction history.
  2. Act: It selects and invokes the most suitable tool from a predefined toolkit.
  3. Observe: It receives tool outputs (text, numerical values, images).
  4. Update: The interaction history is updated, and the loop continues until the task is solved.
• MCP Integration: The framework utilizes the Model Context Protocol (MCP) to ensure interoperability and extensibility with external tools.

B. The Toolkit (104 Specialized Tools)

Earth-Agent integrates 104 domain-specific tools organized into five functional kits:

1. Index: Calculates standard EO indices (e.g., NDVI, NDWI, NBR).
2. Inversion: Retrieves geophysical parameters (e.g., Land Surface Temperature, Precipitable Water Vapor, Sea Ice Concentration).
3. Perception: Handles vision tasks like scene classification, object detection (e.g., ships, buildings), and segmentation.
4. Analysis: Performs spatiotemporal reasoning (trend detection, seasonality decomposition, change point analysis).
5. Statistics: Manages large-scale data preprocessing, batch operations, and statistical computations.

C. Dual-Level Evaluation Protocol

To address the lack of systematic evaluation, the authors introduce a protocol assessing:

• End-to-End: Accuracy of the final answer and Efficiency (trajectory length relative to expert solutions).
• Step-by-Step: Quality of the reasoning trajectory, measured by:
  • Tools-Any-Order: Coverage of necessary tools.
  • Tools-In-Order: Correct sequence of tool invocation.
  • Tools-Exact-Match: Prefix-level accuracy of the trajectory.
  • Parameter Accuracy: Correctness of tool arguments.

3. Key Contributions

1. Earth-Agent Framework: A novel agentic system that bridges the gap between LLMs and domain-specific EO tools, supporting multi-modal (RGB, Spectrum, Products) and multi-step quantitative analysis.
2. Earth-Bench: A comprehensive benchmark comprising:
   • 248 expert-curated questions requiring multi-step quantitative reasoning.
   • 13,729 images spanning Spectrum, Products, and RGB modalities.
   • Dual-level annotations: Both final answers and full reasoning trajectories (tool calls, inputs, outputs) are annotated.
   • Two query regimes: Auto-Planning (agent plans steps) and Instruction-Following (agent follows explicit steps).
3. Systematic Evaluation: The introduction of a rigorous evaluation protocol that diagnoses both the correctness of outcomes and the quality of the reasoning process.

4. Experimental Results

The authors evaluated Earth-Agent using 13 LLM backbones (including GPT-5, Gemini-2.5, DeepSeek-V3.1, Kimi-K2, and Qwen3) against general agents and MLLM-based EO methods.

• Performance on Earth-Bench:

  • Earth-Agent significantly outperforms general-purpose agents (e.g., Operator, Manus, Coze) which fail on complex, multi-modal tasks due to a lack of domain tools.
  • GPT-5 achieved the highest final accuracy (65.99% in Auto-Planning), while open-source models like DeepSeek-V3.1 and Kimi-K2 showed superior performance in tool-use accuracy and trajectory alignment.
  • Instruction-Following generally improved tool calling accuracy but did not always guarantee higher final accuracy, highlighting the complexity of autonomous planning.

• Comparison with MLLMs:

  • Earth-Agent surpassed state-of-the-art remote sensing MLLMs (e.g., LHRS-Bot, VHM) across classification, detection, and segmentation benchmarks.
  • Unlike MLLMs that struggle with tasks outside their training distribution, Earth-Agent's modular design allows it to adaptively invoke specialized tools for specific tasks.

• Error Analysis:

  • Common errors included Invalid Parameters and Tool Hallucinations.
  • Open-source models tended to hallucinate more due to reinforcement learning strategies encouraging exploration, whereas closed-source models (GPT-5) made fewer hallucinations but struggled with parameter precision.

• Scalability:

  • Performance peaked when using 3–15 tools per task. Excessive tool chains led to performance degradation, indicating current LLM limitations in handling very long reasoning chains.

5. Significance

• Paradigm Shift: Earth-Agent moves the field from "perception-only" MLLMs to "scientific analysis" agents capable of quantitative, multi-step reasoning.
• Bridging Modalities: It successfully unifies the processing of raw spectral data, processed products, and visual imagery, enabling a holistic view of Earth observation.
• Foundation for Future Research: By releasing Earth-Bench and the Earth-Agent framework, the authors provide a standardized platform for evaluating and advancing agentic AI in geoscience.
• Real-World Applicability: The framework supports complex scientific workflows (e.g., drought monitoring, urban heat island analysis, disaster assessment) that were previously intractable for AI systems.

In conclusion, Earth-Agent establishes a new standard for AI in Earth observation, demonstrating that integrating LLMs with structured, domain-specific tool ecosystems is the key to unlocking complex, quantitative spatiotemporal analysis.