Technical Summary: HERMES – A Multi-Agent Framework for Structured Knowledge Extraction from Ultra-Long Geoscience Documents

1. Problem Statement

Authoritative scientific knowledge in geoscience remains largely trapped in legacy monographs and historical literature. These documents present significant barriers to computational access due to their unstructured text, complex layouts, and extreme length (often spanning hundreds or thousands of pages). Key information—such as fossil taxonomic names, sampling coordinates, stratigraphic ages, and geochemical compositions—is frequently distributed across narrative descriptions, tables, figure captions, and appendices.

Existing extraction systems often fail to address three core challenges specific to this domain:

1. Document Length and Sparsity: Relevant information is sparsely distributed across ultra-long documents, leading to "lost-in-the-middle" phenomena where models fail to utilize context effectively.
2. Evidence Traceability: Scientific data extraction requires not just the identification of values but the ability to trace them back to precise evidence (specific sentences, table cells, or figure regions) for verification.
3. Domain Complexity: Geoscience literature requires linking entities to attributes across heterogeneous regions, normalizing domain-specific units/nomenclature, and preserving uncertainty markers, which standard entity-relation extraction models often overlook.

2. Methodology: The HERMES Framework

HERMES (Hybrid Entity-Relation Multi-agent Extraction System) is a scalable, multi-agent framework designed to extract structured data from ultra-long scientific documents. It integrates natural language processing, computer vision, and human-in-the-loop validation within a unified, document-level extraction process.

Architecture

The system follows the ReAct (Reasoning + Acting) paradigm, orchestrated by a central Orchestrator Agent (a geoscience-specialized Large Language Model, GeoGPT-R1-Preview, based on Qwen2.5-72B). The Orchestrator coordinates a suite of specialized sub-agents, each responsible for a distinct phase of the workflow:

1. Parser Agent: Digitizes input PDFs (scans or manuscripts), extracting raw text, figures (with captions), and tabular data into a shared context.
2. Entity Recognizer Agent: Identifies relevant entity mentions (e.g., fossil genera, sample IDs) and records their sentence-level sources. It employs rule-based filtering (e.g., taxonomic name patterns) and LLM-based secondary judgment to isolate relevant passages.
3. Annotator Agent: Performs attribute extraction using a Retrieval-Augmented Generation (RAG) approach. For each target attribute, it retrieves the most relevant text chunks to ensure extraction is confined to the immediate context, thereby minimizing hallucinations.
4. Validator Agent: Applies domain-specific deterministic rules (e.g., zoological nomenclature, chronostratigraphic conventions) to check consistency, standardize formats, and handle uncertainty markers.
5. Tracer Agent: Records the source evidence for every extracted entity and attribute, including page-level bounding-box coordinates within the original document.

Operational Workflow

The process begins with a user query defining the entities and attributes of interest. The Orchestrator establishes a shared context that connects all sub-agents, allowing parsed text, intermediate entities, validation results, and evidence locations to be updated and inspected throughout the process. This design supports modularity, parallelization, and error handling. For instance, the system can independently process complex table structures or replace the orchestrator with a different LLM for other disciplines.

The system is designed for human-in-the-loop validation, where expert-adjudicated data can be written back to the shared context to refine the extraction process and improve subsequent validation.

3. Key Contributions

• Unified Extraction Framework: HERMES provides a deployable framework that integrates layout-aware parsing, cross-table reasoning, domain-specific validation, and evidence-level provenance for ultra-long geoscience literature.
• Scalability and Efficiency: By decomposing extraction into coordinated sub-tasks, the system achieves high performance while significantly reducing manual effort compared to fully manual curation.
• Cross-Domain Transferability: The framework was validated across three distinct geoscience domains (Palaeontology, Palaeomagnetism, and Geochemistry) without additional model training, demonstrating adaptability to different document structures and data types.
• Evidence-Traceable Output: Unlike many extraction systems, HERMES explicitly links every data point to its source location in the original document, facilitating auditability and trust.

4. Results and Performance

Case Study 1: Palaeontology (Treatise on Invertebrate Paleontology)

HERMES was applied to the 55-volume Treatise on Invertebrate Paleontology (TIP), a foundational series containing over 50 volumes of text and images.

• Output: A structured database of 32,277 fossil taxonomic entities and 451,878 attributes.
• Efficiency: The system reduced the per-volume turnaround time by approximately sixfold compared to a fully manual baseline (7 days vs. 45 days per volume). The total cumulative labor for the 55 volumes was 53 person-days.
• Accuracy: Evaluation against an expert-adjudicated reference dataset showed:
  • Entity Extraction: F1 score of 0.90 (Precision: 0.90, Recall: 0.89).
  • Attribute Extraction: F1 score of 0.91 (Precision: 0.95, Recall: 0.87).
  • Performance remained stable across different fossil groups despite variations in document length and layout.

Case Study 2: Palaeomagnetism

Applied to 567 palaeomagnetic articles, focusing on quantitative measurements and table-driven structures.

• Accuracy: On a subset of 13 fully annotated articles:
  • Entity Extraction: Precision 0.93, Recall 0.96.
  • Attribute Extraction: Precision 0.81, Recall 0.77.
• Observation: The system successfully handled numerical attributes embedded in complex tables, though linking values to the correct sites/samples remained challenging.

Case Study 3: Geochemistry

Applied to geochemical literature (specifically Neogene to Quaternary intraplate volcanic rocks) involving high-dimensional attribute organization.

• Accuracy:
  • Entity Extraction: F1 score of 0.81 (Precision 0.74, Recall 0.89).
  • Attribute Extraction: F1 score of 0.51 (Precision 0.93, Recall 0.35).
• Analysis: The lower recall in attributes was attributed to the difficulty of locating and binding selected values within wide, high-dimensional tables where units and context were often implicit in headers or footnotes. However, the high precision (0.93) indicates the system prioritizes correct alignment over generating uncertain candidates.

5. Significance and Claims

The paper claims that HERMES provides a practical pathway to transform historical scientific literature into FAIR-oriented (Findable, Accessible, Interoperable, Reusable) structured data. Its significance lies in:

1. Infrastructure for Data-Intensive Science: It offers a sustainable infrastructure for disciplines reliant on historical records, enabling the organization and analysis of existing data at a scale previously unattainable due to cost and labor constraints.
2. Bridging the Gap: It addresses the gap between the abundance of legacy literature and the need for computable, evidence-traceable knowledge resources.
3. Modularity and Adaptability: The multi-agent architecture allows for the independent development of specific components (e.g., table recognition algorithms) and adaptation to other scientific domains by modifying extraction schemas and validation rules rather than retraining foundation models.
4. Real-World Impact: The successful digitization of the Treatise on Invertebrate Paleontology demonstrates that large-scale, series-wide structured database construction is now feasible, making previously scattered paleontological knowledge accessible and analyzable.

The authors acknowledge limitations, particularly regarding error propagation (where missed entities lead to missed attributes), challenges with complex table structures (cross-page tables, merged cells), and the dependency on OCR quality for older scanned documents. They also note that the human verification step remains a bottleneck for throughput, highlighting a trade-off between data quality and scalability in current human-in-the-loop systems.