🤖 machine learning

Bringing Agentic Search to Earth Observation Data Discovery

This paper presents an agentic search system that leverages large language models and a knowledge graph to significantly improve the discovery of NASA Earth Observation datasets and tools, validated by a new benchmark (NASA-EO-Bench) and a hybrid retrieval pipeline that combines supervised scoring with zero-shot agentic reranking to achieve substantial gains in retrieval performance.

Original authors: Minghan Yu, Youran Sun, Chugang Yi, Yixin Wen, Haizhao Yang

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minghan Yu, Youran Sun, Chugang Yi, Yixin Wen, Haizhao Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine NASA has a massive, dusty library containing thousands of books (datasets) and special tools (like Worldview or Giovanni) for studying Earth. The problem isn't that the books are missing; it's that the library is so huge and disorganized that even expert librarians (geoscientists) struggle to find the exact book they need for a specific research question.

This paper introduces a new "Smart Librarian" system designed to solve this. Here is how it works, broken down into simple concepts:

1. The Problem: Too Many Choices, Too Little Clarity

Finding the right data is like trying to find a specific needle in a haystack, but the haystack is made of other needles, and the labels on them are written in different languages.

The Old Way: You ask a question, and a computer tries to match your words to the book titles. If you use the wrong word (e.g., "flood" instead of "inundation"), it might miss the right book entirely.
The New Way: The authors built a system that uses a "Smart Librarian" (an AI) that can actually think and look things up to find the right data.

2. The New Tool: "NASA-EO-Bench" (The Training Ground)

Before building the Smart Librarian, the team needed a way to test if it was actually good. They created a giant practice test called NASA-EO-Bench.

How they made it: They looked at thousands of real scientific papers published by NASA. They asked an AI to turn the "goal" of each paper into a question a researcher might ask (e.g., "I want to measure rainfall in the Amazon").
The Answer Key: They used the actual list of data sources those scientists cited in their papers as the "correct answer."
The Scale: This isn't a small quiz; it's a massive exam with nearly 50,000 questions and answers, allowing them to train the system properly.

3. The Three-Stage Search Pipeline

The system doesn't just guess; it follows a strict three-step process (like a detective solving a case):

Stage 1: The Quick Check (The Official Tools)
First, the system asks: "Can NASA's existing official tools answer this directly?" If you ask for a simple map visualization, the system sends you straight to the right tool and stops there. No need to search the whole library.
Stage 2: The Hybrid Search (The Detective's First Pass)
If the official tools can't do it, the system searches the library. It uses two methods at once:
1. Keyword Matching (BM25): Like a classic index card catalog. It looks for exact words (e.g., matching "MODIS" to "MODIS").
2. Smart Understanding (Neural Network): Like a librarian who understands the meaning of your question, even if you use different words.
- The Trick: They combine these two. The "Keyword" method is great for exact names, while the "Smart Understanding" method is great for concepts. By blending them, they found the right data 5 times better than using just the "Smart Understanding" method alone.
Stage 3: The "Agentic" Rerank (The Expert Consultant)
This is the paper's biggest innovation. The system takes the top 10 candidates from Stage 2 and asks a Large Language Model (LLM) to pick the best one.
- The "Single-Shot" Librarian: This is a smart AI that reads the query and the 10 book descriptions and picks the best one based on what it already knows.
- The "Agentic" Librarian: This is the super-powered version. Before picking, this AI is allowed to go out and do research. It can:
  - Search the web for current context.
  - Look up scientific papers on arXiv to see what other scientists usually use for this specific problem.
  - Clarify confusing terms.
- The Result: Even without retraining the AI, letting it "do its homework" (use tools) improved the ranking quality by 28%. It proved that an AI that can act and search is better than one that just thinks.

4. Why This Matters

The paper shows that in the age of advanced AI, simply having a smart model isn't enough. You need to give that model tools to use.

The Analogy: Imagine you are trying to find the best restaurant in a new city.
- Old AI: Reads a static list of restaurants and guesses which one is best based on its training.
- Agentic AI: Reads the list, then goes online to check recent reviews, looks up the menu, and asks locals what they recommend before giving you an answer.

Summary of Claims

The Benchmark: They created the largest dataset of its kind (47,000+ pairs) to test Earth science data search.
The Retrieval: Combining keyword search with AI understanding improved finding the right data by over 5 times.
The Agentic Step: Giving the AI the ability to search the web and read papers during the search process made it significantly better at ranking results, even without any extra training.

The paper concludes that for complex scientific data, the future isn't just about smarter AI brains, but about giving those brains the ability to use tools to verify and refine their answers.

Technical Summary: Bringing Agentic Search to Earth Observation Data Discovery

Problem Statement

Earth observation data discovery is characterized not by data scarcity, but by the difficulty of navigating fragmented metadata, diverse tools, and complex access pathways across thousands of datasets hosted by NASA and its Distributed Active Archive Centers (DAACs). While Large Language Models (LLMs) offer a natural interface for expressing retrieval intents, their direct application in geoscience faces two primary challenges:

Domain Knowledge Gap: General LLM pre-training corpora lack specific geoscience observational data and domain knowledge, leading to inaccurate understanding of domain-specific queries.
Context Limitations: Even with Retrieval-Augmented Generation (RAG), the LLM's context window limits the number of candidates it can process. Attention decay in long contexts means only top-ranked candidates influence the final answer, making ranking quality a critical bottleneck. Furthermore, general embedding models often exhibit systematic mean bias that misaligns distances for geoscience terminology.

The paper posits that the latent value of Knowledge Graphs (KGs) can be amplified through agentic search, where an LLM autonomously invokes external tools to ground its reasoning, rather than relying solely on static pre-built indices or internal knowledge.

Methodology

The proposed system is a three-stage agentic search pipeline designed to retrieve and rerank Earth observation datasets:

1. Pipeline Architecture

Stage 1: Router (Official Tools): The system first attempts to resolve queries using NASA's existing ecosystem (Harmony, Science Discovery Engine, WorldView, Giovanni). If a tool fully satisfies the request, the pipeline terminates early with provenance-annotated output.
Stage 2: Hybrid Retrieval: If official tools are insufficient, the system retrieves candidates from the NASA Common Metadata Repository (CMR) corpus using a hybrid approach:
- Lexical Anchoring: Uses BM25 to capture exact scientific identifiers (mission names, instrument codes) which are high-precision signals.
- Task-Adaptive Semantic Scoring: Addresses domain shift and pair-specific bias. The authors propose two methods:
  - NN-SSC (Neural Score Correction): Keeps the backbone encoder frozen and trains a lightweight Multi-Layer Perceptron (MLP) on concatenated query-dataset embeddings to output a relevance score. This allows for non-linear, pair-specific corrections.
  - Encoder Fine-tuning: Directly fine-tunes the sentence transformer on (query, positive, hard-negative) triples.
- Hybrid Fusion: Combines BM25 and neural scores via a convex combination ( $\alpha \cdot \hat{s}_{NN} + (1-\alpha) \cdot \hat{s}_{BM25}$ ). The weight $\alpha$ is derived analytically based on the relative standalone performance of the scorers to avoid test-set tuning.
Stage 3: Agentic Reranking: A top- $K$ $K$ window of candidates is passed to an LLM. Unlike standard reranking, the agentic version equips the LLM with autonomous tool access. Before ranking, the model executes a five-step routine:
1. Search arXiv for relevant papers.
2. Use web search for additional context.
3. Disambiguate candidate descriptions via web search.
4. Reason step-by-step on fit.
5. Output the final ranked list.
  This contrasts with a single-shot LLM rerank, which relies solely on inlined candidate text without tool calls.

2. Benchmark Construction: NASA-EO-Bench

To enable rigorous evaluation, the authors constructed NASA-EO-Bench, derived from the NASA Earth Observation Knowledge Graph (NASA EO-KG).

Source: 10,636 top-cited peer-reviewed publications.
Ground Truth: "Silver labels" are formed by the datasets explicitly cited in the "USES DATASET" edges of the KG.
Scale: 47,654 query-dataset pairs (21,272 task-based queries) split into training (38k pairs) and test (9.5k pairs) sets.
Query Generation: LLMs generate "I want to..." style queries from paper abstracts, simulating expert intent.
Metrics:
- Citation-based: Recall@K, Mean Average Precision (MAP), Mean Reciprocal Rank (MRR).
- Semantic: LLM-as-a-Judge Precision@K (using Qwen3.6-35B-A3B to judge if a dataset measures the required physical variable).

Key Results

Experiments were conducted on the NASA-EO-Bench test set using various baselines and the proposed methods:

Retrieval Suite Performance:
- The unadapted cosine baseline performed poorly (R@10 = 0.0755), demonstrating significant domain shift.
- The NN-SSC + BM25 (Hybrid) approach achieved the best Ground Truth metrics, lifting Recall@10 to 0.4275 and MRR to 0.2517.
- This represents a >5× improvement over the unadapted cosine baseline for both R@10 and MRR.
- The hybrid fusion was critical; removing BM25 caused a drop in MAP and MRR, confirming the necessity of exact lexical signals for geoscience.
Efficiency of NN-SSC:
- NN-SSC (460K trainable parameters) matched or exceeded the performance of a fully fine-tuned encoder (~125M trainable parameters) while being significantly more parameter-efficient.
Agentic vs. Single-Shot Reranking:
- Single-shot LLM reranking (e.g., GPT-5.5) consistently improved MAP and MRR over the retrieval backbone alone, confirming that LLM reasoning complements supervised retrieval.
- Agentic reranking (LLM with tool access) yielded further directional gains over single-shot reranking on a stratified $N=200$ $N = 200$ subset.
  - For Claude Opus 4.7: MRR increased from 0.367 (single-shot) to 0.388 (agentic).
  - For DeepSeek v4 pro: MRR increased from 0.366 to 0.374.
- The agentic approach was 5–10 times more expensive per query due to tool round trips but demonstrated that autonomous tool use provides complementary value to static prompting.

Significance and Claims

The paper claims to operationalize "trustable" and "verifiable" agentic search in the geoscience domain through three main contributions:

NASA-EO-Bench: The first large-scale benchmark (21k queries) for geoscience dataset retrieval that supports supervised training and uses citation-grounded silver labels, moving beyond small-scale manual scoring or coarse LLM-as-a-judge evaluations.
Retrieval Correction Suite: A demonstration that combining lexical (BM25) and semantic (NN-SSC) signals via analytically derived fusion significantly outperforms unadapted baselines, addressing the specific domain shift of geoscience terminology.
Controlled Agentic Comparison: A rigorous, same-model comparison showing that agentic reranking (LLM + autonomous tool use) provides a measurable gain over single-shot reranking. The paper argues that the ability to ground ranking decisions in external context (web/arXiv) is a distinct and valuable capability for dataset discovery, particularly for disambiguating candidates and handling event-specific queries.

The authors maintain a modest stance, acknowledging that citation-based ground truth is a "silver label" that may miss relevant but uncited datasets (popular dataset bias) and that the agentic experiments were limited to a stratified subset without paired-bootstrap confidence intervals. However, the results provide consistent evidence that agentic search, when grounded in external tools, enhances the retrieval pipeline beyond what static LLM prompting can achieve.