Technical Summary: Bidirectional Semantic Complementary Tool Retrieval for Remote Sensing Agents

1. Problem Statement

Large Language Model (LLM)-based agents offer a promising paradigm for automating Remote Sensing (RS) data processing. However, effective agent workflows rely on accessing extensive specialized tool libraries (e.g., QGIS, GDAL, Orfeo ToolBox). Current tool retrieval methods face a critical "semantic asymmetry" bottleneck:

• Query Side: User queries typically express macro-level, abstract intentions (e.g., "analyze crop health") that lack the specific functional semantics required to trigger technical tools.
• Tool Side: Tool documentation provides fine-grained technical specifications (inputs/outputs) but lacks the contextual semantics regarding how tools fit into specific operational workflows.

Existing approaches often address only one side of this gap—either rewriting queries to add functional details or constructing graphs to add context to tools—but fail to bridge the fundamental misalignment between abstract user intent and rigid technical specifications. Furthermore, RS tasks exhibit "strong coupling," where the output of one tool imposes hard constraints on the selection of subsequent tools, a dependency often ignored in standard retrieval.

2. Methodology

The paper proposes Bidirectional Semantic Complementary Tool Retrieval (BSCTR), a framework designed to align macro-level user intentions with fine-grained technical specifications through a three-module architecture:

A. Plan Module (Query-side Functional Enhancement)

To address the lack of functional semantics in user queries, this module leverages the reasoning capabilities of LLMs.

• Mechanism: It employs a planning-based strategy (e.g., Plan-and-Solve prompting) to decompose a coarse-grained user query ($Q_{coarse}$) into a sequence of logical sub-tasks ($S = \{s_1, s_2, ..., s_n\}$).
• Enhancement: Each sub-task is fused with the original macro-intention to create an enhanced query ($q_{enhance}^{(i)}$). This process actively supplements the query with missing functional semantics, transforming abstract goals into explicit technical sub-tasks that better match tool documentation.

B. Graph Module (Tool-side Contextual Injection)

To address the lack of contextual semantics in tool descriptions, this module constructs a dynamic tool dependency graph.

• Graph Construction: The tool library is modeled as a directed graph $G=(N, E)$, where edges represent sequential invocation dependencies based on historical trajectories.
• Context Aggregation: Inspired by Simplified Graph Convolution (SGC), the module performs multi-order linear aggregation of neighborhood features. It explicitly injects contextual information from precursor tools into the current tool node's representation.
• Directionality: The system supports Predecessor (upstream), Successor (downstream), and Symmetric modes. Experimental analysis indicates that the Predecessor mode is most effective for RS, as a tool's identity is primarily defined by its upstream prerequisites (e.g., radiometric correction must precede feature extraction).
• Dynamic Evolution: The graph possesses continual learning capabilities; upon successful task execution, new tool dependencies observed in the trajectory are dynamically added to the graph, updating the contextual representations ($V_t$).

C. Bidirectional Semantic Alignment Module

This module serves as the operational hub, integrating the enhanced query representations ($V_q$) and the context-enriched tool representations ($V_t$).

• Retrieval: It calculates cosine similarity between the enhanced sub-queries and tool nodes to retrieve the top-$k$ relevant tools.
• Execution & Evolution: The agent executes tasks via a ReAct reasoning loop. If a task succeeds, the trajectory is verified, and the tool graph is updated to reflect new causal relationships, ensuring the system adapts to open-set environments.

3. Key Contributions

1. Bidirectional Complementary Framework: The paper introduces a novel paradigm that simultaneously bridges the "semantic asymmetry" by enhancing queries with functional semantics and enriching tools with contextual semantics. This synergistic approach aligns macro-intentions with micro-specifications more effectively than single-perspective methods.
2. Graph-based Contextual Modeling: A dynamic tool dependency graph is designed to encode pragmatic contexts into node representations. By aggregating neighborhood features, the framework captures the "strong coupling" and sequential logic inherent in professional RS workflows, ensuring retrieved tools are logically consistent with preceding steps.
3. Empirical Validation: The method is rigorously validated on the specialized RS dataset GeoPlan-bench and the general-purpose API-Bank. The results demonstrate significant improvements in retrieval accuracy and execution logic over state-of-the-art baselines.

4. Experimental Results

Experiments were conducted on GeoPlan-bench (248 training/996 test trajectories, average chain length 10.67) and API-Bank Level-3 (50 test tasks, 70 APIs).

• Performance on GeoPlan-bench:

  • BSCTR achieved an overall Recall@k of 0.7485 and Similarity (LCS) of 0.3237, significantly outperforming baselines like BM25, Dense retrieval, Graph RAG-Tool Fusion, and Query Rewriting.
  • In the "Complex" difficulty tier, BSCTR maintained a Recall@k of 0.6970, whereas the Query Rewriting baseline dropped to 0.6713.
  • Ablation studies revealed that removing the planning module caused a catastrophic drop in RS performance (Recall@k fell from 0.7485 to 0.1883), highlighting the critical role of functional decomposition in specialized domains where user queries share few keywords with technical tools.

• Performance on API-Bank:

  • BSCTR achieved SOTA performance with a Recall@k of 0.9067 and F1-Score of 0.8804.
  • The method demonstrated robust generalization to general-domain tasks, even in a cold-start setting (no prior training trajectories), proving the versatility of the bidirectional mechanism.

• Parameter Sensitivity:

  • Optimal performance was found with an aggregation order $k=1$ and a smoothing factor $\alpha=0.5$. Higher $k$ values introduced noise from distant tools, while extreme $\alpha$ values led to either insufficient context or feature homogenization.

5. Significance and Claims

The paper claims that the proposed BSCTR framework effectively resolves the "semantic misalignment" that hinders LLM agents in complex, tool-heavy domains like Remote Sensing.

• Domain Specificity: The authors argue that RS tasks require a unique approach due to their "strong coupling" and rigid causal workflows. The bidirectional approach is presented as essential for bridging the gap between non-expert natural language and expert-level technical tools.
• Generalizability: While optimized for RS, the framework's success on API-Bank suggests it is a versatile retrieval paradigm capable of handling semantic gaps in diverse, open-domain tool-use environments.
• Continual Adaptability: The dynamic evolution mechanism allows the system to autonomously learn new tool dependencies from successful executions, addressing the challenge of maintaining relevance in evolving tool libraries.

The authors acknowledge limitations, including reliance on the LLM's initial planning accuracy (risk of logical hallucinations), cold-start challenges in data-sparse sub-fields, and the current lack of multi-modal integration (e.g., sensor metadata). Future work is proposed to integrate visual cues and optimize information aggregation via parametric learning.