🤖 AI

EO-Agents: A Three-Agent LLM Pipeline for Earth Observation Hypothesis Generation

This paper introduces EO-Agents, a three-agent LLM pipeline grounded in the NASA Earth Observation Knowledge Graph that successfully generates and evaluates scientifically coherent, novel research hypotheses across multiple Earth-science domains by leveraging a heterogeneous graph neural network to identify promising dataset pairings.

Original authors: Mahyar Ghazanfari, Amin Tabrizian, Armin Mehrabian, Peng Wei

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mahyar Ghazanfari, Amin Tabrizian, Armin Mehrabian, Peng Wei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Earth Observation (EO) research as a massive, chaotic library containing over 8,000 different "books" (datasets) about our planet. Each book holds a specific piece of information: one might tell you about soil moisture, another about cloud height, and a third about ice velocity.

For decades, scientists have been trying to figure out which two books make the best story when read together. The problem? There are over a million possible pairings. It's like trying to find the perfect two ingredients for a recipe by tasting every single combination in a giant kitchen. No human can do that alone.

This paper introduces EO-Agents, a new "smart kitchen assistant" designed to help scientists find these perfect ingredient pairings and write the recipe (the hypothesis) for a new study.

Here is how the system works, broken down into simple steps:

1. The Map (The Knowledge Graph)

First, the system doesn't just read the books; it builds a giant, complex map of how they relate to each other. Think of this as a subway map where every station is a dataset.

The Connections: If two datasets were used together in a past scientific paper, there is a direct train line between them.
The Goal: The system wants to find two stations that don't have a train line yet but should have one.

2. The Scout (The AI Ranker)

The system uses a specialized AI (a Graph Neural Network) to act as a scout. It looks at the map and predicts which two disconnected stations are most likely to be good partners.

The Analogy: Imagine a scout looking at a map of cities. Even if City A and City B have never been connected by a road, the scout sees that they both have similar terrain and resources, so it predicts a road should exist there.
The Result: The scout picks the top 200 most promising "unexplored" pairs of datasets.

3. The Three-Agent Team (The LLM Pipeline)

Once the scout finds the top 200 pairs, the system hands them off to a team of three AI "agents" (using large language models like GPT-5.2 and Claude Sonnet 4.6). They work like a production line:

Agent 1: The Filter (The Gatekeeper)
- Job: It looks at the 200 pairs and asks, "Is this combination actually sensible? Is it new?"
- Action: It scores them and picks the best 40. It's like a talent scout who says, "These 40 pairs look like they could actually work together in a real experiment."
Agent 2: The Generator (The Inventor)
- Job: It takes those 40 pairs and writes a full research proposal.
- Action: It doesn't just say "Study these." It writes a specific story: "If we combine Dataset X and Dataset Y, we can test if [Specific Scientific Question] is true. Here is how we would do it, and here is what we expect to find."
- The Output: It creates 40 structured, ready-to-test hypotheses.
Agent 3: The Judge (The Critic)
- Job: It reads the 40 proposals and grades them.
- Action: It rates them on three things:
  1. Importance: Does this matter to science?
  2. Tractability: Can we actually do this study with current tools?
  3. Novelty: Is this a fresh idea?
- The Twist: The judge does this twice for every proposal: once blind (seeing only the text) and once contextual (seeing the actual data descriptions). This helps check if the idea sounds good just because of the words, or because the data actually supports it.

What Did They Find?

The team ran this entire process on 1,475 NASA datasets and ended up with 160 new, structured research hypotheses. These cover diverse fields like glaciology (ice), ecohydrology (water and plants), and atmospheric chemistry.

Key Discoveries:

The System Works: The AI-generated "new" pairings were rated almost as plausible by the judges as real, historical pairings found in actual scientific papers. This means the system isn't just making things up; it's finding scientifically coherent combinations humans haven't tried yet.
The "Judge" Matters: The paper found a funny quirk: who acts as the judge changes the score. If you use AI Model A as the judge, it might give a hypothesis a 4/5. If you use AI Model B, it might give the same hypothesis a 3/5.
- The Lesson: You can't trust a single AI judge to give a perfect absolute score. However, the ranking stays stable. Both judges agree on which ideas are the "best" and which are the "worst," even if they disagree on the exact number.
Real-World Examples: The system came up with ideas like combining ice-penetrating radar data with magnetic field data to study how ice flows over bedrock, or mixing soil moisture data with vegetation indices to see how plants react to drought.

The Bottom Line

This paper presents a tool that helps scientists navigate a massive ocean of data. Instead of drowning in millions of possibilities, the system uses a map to find the best hidden connections, then uses a team of AI agents to write the research proposal and grade it. It's not a robot that does the science for you; it's a robot that hands you a list of the most promising "recipes" so you can go cook the meal.

Technical Summary: EO-Agents

Problem Statement

Earth observation (EO) research is fundamentally combinatorial, relying on the specific pairing of measurement products (e.g., soil moisture with vegetation indices) rather than single datasets. While NASA distributes over 8,000 EO datasets, the combinatorial space of potential pairings ( $\sim 10^6$ ) far exceeds human capacity for survey. Existing Large Language Model (LLM) approaches for scientific hypothesis generation typically rely on unstructured literature retrieval, treating hypotheses as free-form textual claims. However, in data-rich observational domains, a useful hypothesis is inseparable from the specific, executable measurement products required to test it. Current literature-grounded pipelines often fail to directly anchor hypotheses to named datasets, treating dataset identification as a side-effect of text generation rather than a structural constraint.

Methodology

The authors propose EO-Agents, a pipeline that grounds hypothesis generation directly in the NASA Earth Observation Knowledge Graph (EO-KG) rather than unstructured text. The system operates in three stages:

1. Knowledge Graph and GNN Retrieval

Data Source: The pipeline utilizes the EO-KG, a heterogeneous graph with 150,351 nodes (including Datasets, Publications, Instruments, Platforms, etc.) and 436,203 edges.
Supervision: Co-usage supervision is derived from publications citing multiple datasets. Pairs are labeled based on the publication year, creating a temporal split: training (papers $\le$ 2022), validation (2023), and test (2024).
Ranking Model: A heterogeneous Graph Neural Network (GNN) based on GraphSAGE is trained via link prediction to rank candidate dataset pairings.
- Architecture: The model uses type-specific aggregators for different edge types (e.g., co-usage, platform mounting, keyword membership). Dataset nodes are initialized with 768-dimensional SPECTER2 embeddings; other node types use learnable embeddings.
- Scoring: Two scoring heads are evaluated: a dot-product scorer ( $z_{dot}$ ) and an MLP scorer ( $z_{mlp}$ ). The dot-product head performs better on "cold-start" pairs (where one dataset has no training co-usage), while the MLP head excels on "cross-DAAC" pairs (datasets from different archives).
- Output: The GNN ranks pairs by predicted co-usage likelihood, surfacing the top 200 novel pairs (those not observed in training/validation/test sets).

2. Three-Agent LLM Pipeline

The top 200 candidate pairs are processed by a cascade of three LLM agents, each instantiated with either GPT-5.2 or Claude Sonnet 4.6. All agents operate at temperature 0 with strict JSON output to minimize sampling variance.

Agent 1 (Filter): Receives the dataset pair and their abstracts. It scores the pair on Plausibility (1–5) and Novelty (1–5) without seeing any generated hypothesis text. It selects the top 40 pairs using a tiered rule that balances high plausibility with sufficient novelty.
Agent 2 (Generator): Receives the 40 selected pairs and generates a structured research hypothesis for each. The output includes a research question, testable hypothesis, analysis method, expected finding, scientific importance, and domain tag. Crucially, it does not see Agent 1's scores to prevent regression toward filter preferences.
Agent 3 (Judge): Evaluates the generated hypotheses on Importance, Tractability, and Novelty. This agent operates under two conditions:
- Blind: Sees only the hypothesis text.
- Contextual: Sees the hypothesis text plus the underlying dataset metadata.

3. Factorial Evaluation

To assess the reliability of LLM-based evaluation, the authors conduct a $2 \times 2 \times 2$ factorial experiment across the three agent roles (Filter, Generator, Judge) and the two judging conditions (Blind, Contextual). This yields 640 judgments across 160 hypotheses, allowing for a variance decomposition of the scoring process.

Key Results

1. Retrieval Performance

The heterogeneous GNN significantly outperforms baselines (Popularity, Common Neighbors, Adamic–Adar, MF-SVD, and content-only models like SPECTER2 and BGE-base) across all test pools.

Cold-Start: The GNN achieves a Hits@10 of 0.519, outperforming content baselines (0.477) and structural heuristics (0.211).
Cross-DAAC: The GNN achieves a Hits@10 of 0.175 (dot-product) and 0.266 (MLP), demonstrating the ability to link datasets across different archives where content similarity often fails due to generic scientific boilerplate.
Novelty vs. Plausibility: The GNN-predicted novel pairs receive plausibility scores from LLM judges that are nearly indistinguishable from held-out real co-usages from 2024 publications (e.g., 4.38 vs. 4.57 for GPT), indicating the system surfaces scientifically coherent, unexplored combinations.

2. Evaluation Reliability and Variance Decomposition

The factorial experiment reveals critical insights into LLM-as-judge reliability:

Ranking Stability: Hypothesis rankings are highly stable across different agent configurations. 159 of 160 hypotheses change by at most 2 points in rank across different judge conditions.
Absolute Score Variance:
- Importance: Heavily dependent on Judge Identity (25.1% of variance). GPT-5.2 consistently rates importance higher than Claude Sonnet 4.6.
- Tractability: Dominated by the Blind/Contextual Condition (15.8% of variance). Hypotheses appear more executable when judges see only the text; providing dataset metadata lowers tractability scores, suggesting text alone overstates feasibility.
- Novelty: Shows low variance across all factors, with judges struggling to discriminate novelty between pair types.
Inter-Rater Agreement: Agreement is highest for tractability ( $\kappa_q \approx 0.47$ ) and lowest for importance ( $\kappa_q \approx 0.14$ ), confirming that "meaningful advance" is the most subjective metric and most sensitive to model priors.

3. Generated Output

The pipeline produced 160 structured hypotheses spanning five flagship domains: ecohydrology, glaciology, aerosol–cloud interactions, vegetation phenology, and stratospheric chemistry. A representative example links IceBridge airborne magnetometer data with InSAR-derived Antarctic ice velocity to hypothesize that subglacial magnetic anomalies predict basal-friction patterns.

Significance and Claims

The paper claims three primary contributions:

Structured Grounding: It introduces a hypothesis-generation pipeline for observational science that ties every LLM-generated hypothesis to a specific pair of NASA datasets via a knowledge-graph ranker, moving beyond free-form text retrieval.
Scientific Coherence: It demonstrates that model-predicted novel pairings are rated as plausible as real, held-out literature co-usages, providing evidence that the pipeline generates scientifically coherent ideas beyond the model's training distribution.
Evaluation Limitations: It provides empirical evidence that while hypothesis rankings are stable, absolute scores in LLM-as-judge evaluations are highly dependent on the specific judge model and the availability of contextual data. This highlights the risk of relying on single-judge evaluations for absolute scoring in scientific discovery.

The authors emphasize that the system is designed for ideation, not execution. The 160 hypotheses serve as seeds for human-led investigation and require domain-expert verification and actual data analysis before being treated as findings. The work does not claim to replace human scientists but to augment the discovery layer by surfacing actionable, dataset-grounded research directions that would otherwise remain unexplored in the vast combinatorial space of Earth observation data.