Technical Summary: Compass: Navigating Global Marine Lead Data Integration through Expert-Guided LLM Agent

Problem Statement

Marine lead (Pb) and its isotopes are critical tracers for understanding ocean circulation and anthropogenic pollution history. However, high-quality in-situ observations are costly and sparse, with the majority of historical data buried within the unstructured text, tables, and figures of academic papers. This creates a "data silo" problem where critical datasets are inaccessible for global-scale synthesis.

Existing solutions face two primary limitations:

1. Manual Curation: While programs like GEOTRACES ensure high data fidelity, they are labor-intensive and unscalable against the exponential growth of scientific literature.
2. Automated Extraction: General-purpose Large Language Models (LLMs) and domain-specific fine-tuned models often fail in high-stakes scientific contexts. General LLMs lack the necessary domain-specific logical constraints, leading to hallucinations and scientifically invalid outputs (e.g., misinterpreting isotope notations or unit conversions). Conversely, fine-tuned models often degrade instruction-following capabilities required for complex, multi-step workflows.

The core challenge is to achieve both scalability and scientific rigor by bridging the gap between general-purpose language understanding and the strict logical constraints required for marine geochemistry.

Methodology: The Compass Framework

To address these challenges, the authors propose Compass, an LLM agent framework that operationalizes an expert-guided adaptation approach. Instead of relying on black-box fine-tuning or unstructured retrieval-augmented generation (RAG), Compass integrates domain expertise directly into the agent's workflow via a Knowledge Tree.

1. Knowledge Tree Design

Co-designed with marine scientists, the Knowledge Tree ($T$) is a hierarchical structure where nodes represent specific tasks or subtasks. Each node is associated with a four-dimensional knowledge set ($K$):

• Background Knowledge ($BK$): Domain concepts and terminology.
• Logical Constraints ($LC$): Explicit rules and conditional dependencies defining valid execution paths.
• Operational Guidelines ($OG$): Step-by-step procedures and output format specifications.
• Validation Criteria ($VC$): Quality metrics and physical constraint checks for verifying output correctness.

This structure encodes not just what the agent should know, but how it should reason and validate its outputs, ensuring the workflow is interpretable and scientifically trustworthy.

2. Agent Architecture

Compass decomposes the complex data integration task into a three-phase hierarchical pipeline:

• Collection Phase: A paper classification component retrieves candidate papers from repositories (e.g., Semantic Scholar, AceMap) and uses LLMs guided by the Knowledge Tree to identify papers containing target marine Pb data.
• Extraction Phase: A table classification component utilizes PDF parsing tools (MinerU) to extract tabular content. LLMs then analyze tables with contextual information (captions, descriptions) to classify and filter target data tables.
• Aggregation Phase: Three components handle integration:
  • Data Association: Links related elements within papers (e.g., matching Pb measurements with coordinates from footnotes).
  • Cross-source Fusion: Normalizes heterogeneous table headers into a unified schema.
  • Data Standardization: Converts values into consistent units using LLM-generated conversion functions.

3. Validation Mechanism

Compass employs a multi-layered validation strategy informed by the Knowledge Tree's Validation Criteria:

• Automated Physical Constraint Checks: Every extracted record undergoes value range verification, unit conversion cross-validation, and geographical outlier filtering. Failures trigger a rollback mechanism to re-process the specific subtask, preventing error propagation.
• Expert Manual Validation: A stratified random sample is reviewed by marine scientists to assess end-to-end accuracy.

Key Contributions

1. Compass Framework: The development of a Knowledge Tree-enhanced LLM Agent that hierarchically decomposes complex scientific workflows, enabling accurate extraction and integration of fine-grained data (text and tables) from heterogeneous academic sources.
2. Expert-Guided Adaptation: A demonstration that injecting structured expert knowledge via a co-designed Knowledge Tree can effectively bridge the gap between general-purpose LLMs and high-stakes scientific domains without the need for costly fine-tuning.
3. Largest Integrated Marine Pb Database: The deployment of Compass across over 230,000 open-access papers to recover 3,751 previously unincorporated Pb records, establishing the largest integrated marine Pb database to date.

Results and Performance

The authors evaluated Compass against general-purpose LLMs (e.g., GPT-4o, Gemini-2.5-pro) and domain-specific fine-tuned models (e.g., K2, OceanGPT) using a benchmark dataset of 337 tables across 10 categories.

• Benchmark Performance: Compass (using a Qwen2.5-32B backbone) achieved the highest scores across all tasks.
  • Paper Classification: 96.0% accuracy.
  • Table Classification: 94.7% accuracy.
  • End-to-End Extraction: Achieved an F1-score of 0.465, outperforming GPT-4o (0.373) and Gemini-2.5-pro (0.404).
  • Comparison with Fine-tuned Models: Domain-specific models like OceanGPT performed poorly (F1: 0.009), highlighting that fine-tuning alone degrades instruction-following capabilities essential for multi-step integration.
• Ablation Studies: Removing the tree-structured logic reduced the F1-score to 0.402, and removing knowledge nodes reduced it to 0.381. Enabling the automated validation rollback mechanism further improved the F1-score to 0.619.
• Deployment Scale: Compass processed 230,000 papers, identifying 110 relevant papers and extracting 3,751 new records.
• Data Quality:
  • Automated checks achieved 100% pass rates for value range and geographical boundary checks, and 95% for unit conversion.
  • Expert manual validation on a subset of 22 papers (946 data points) confirmed 92% accuracy (95% CI: ±1.7%).
  • The primary sources of error were semantic confusion (e.g., misclassifying rainwater samples as seawater) and data association errors.

Significance and Impact

The authors claim that this work demonstrates the viability of expert-guided agents for bridging the gap between general-purpose LLMs and rigorous scientific domains.

• Data Expansion: The integration adds 16,455 new records to the existing GEOTRACES dataset (an 86% increase), significantly expanding coverage in under-sampled regions such as the East China Sea, the Arabian Sea, and the Southern Ocean.
• Scientific Utility: The enriched dataset supports global Pb inventory estimation, source identification via stable isotope ratios, and the investigation of oceanic responses to the global phase-out of leaded gasoline.
• Efficiency: The pipeline required approximately 52 GPU hours, representing an efficiency improvement of over four orders of magnitude compared to manual curation.
• Open Science: The authors released an interactive visualization platform (JingWei) and the codebase to facilitate open scientific access and reproducibility.

The paper concludes that while limitations exist regarding figure extraction and upstream PDF parsing dependencies, the modular design allows for future enhancements. The work establishes a practical, low-cost alternative to domain-specific model training for scalable data discovery in geosciences.