Technical Summary: StructSense

Problem Statement

The rapid expansion of scientific literature, with millions of new articles published annually, has overwhelmed researchers' capacity to track advances and integrate findings. While Structured Information Extraction (SIE) is critical for curating scientific knowledge, existing approaches face significant hurdles:

• Rule-based and ML/DL systems often suffer from brittleness to linguistic variability, require extensive domain-specific training data, or fail to generalize across diverse document formats without manual re-engineering.
• Large Language Models (LLMs) offer flexible, few-shot capabilities but struggle with domain-specific terminology, polysemous terms (e.g., "cortex" in neuroscience vs. plant biology), and adherence to strict output schemas. They frequently lack integration with curated domain ontologies and often fail to provide transparent provenance for extracted facts.
• Current gaps include inadequate provenance tracking, lack of FAIR (Findable, Accessible, Interoperable, Reusable) compliance, and insufficient mechanisms for handling domain ambiguity without fine-tuning.

Methodology: The StructSense Framework

The authors introduce StructSense, a modular, task-agnostic, open-source, multi-agent framework designed to extract structured information from unstructured scientific text. It integrates ontology-guided symbolic knowledge, agentic self-evaluative refinement, and human-in-the-loop (HIL) validation.

Architecture

The system orchestrates four specialized agents using the Crew.AI framework:

1. Extractor Agent: Performs the initial extraction from unstructured text (e.g., PDFs parsed via GROBID). Unlike systems relying solely on LLMs, it employs a weighted ensemble of domain-specific models (e.g., BioBERT, NCBI-disease models) and general-purpose models (SpaCy) alongside an LLM.
   • Verification: It implements a multi-level verification process to mitigate hallucinations. Extracted entities are checked against source text spans and sentence-level grounding. Entities failing these checks are discarded.
   • Chunking: For long documents, text is segmented, processed in parallel, and then merged with global span verification.
2. Alignment Agent: Maps extracted entities to canonical concepts in ontologies or knowledge graphs (e.g., UBERON, PO, NDFRT). This agent resolves polysemy by linking terms to the correct ontology ID based on context, utilizing a configurable local concept mapping service or external BioPortal.
3. Judge Agent: Acts as a critic, evaluating the alignment agent's output. It assigns a confidence score (0–1) and provides rationale remarks for each assessment, supporting interpretability and self-evaluation.
4. Feedback Agent: Integrates Human-in-the-Loop (HIL) validation, allowing domain experts to correct outputs or provide guidance in natural language before final generation.

Local Concept Mapping Service

A standalone service implemented in Python (FastAPI) handles the retrieval of ontological concepts. It treats mapping as an information retrieval task with three stages:

• Indexing: Constructs rich-text representations of ontology concepts (concatenating labels, definitions, synonyms, parents) and generates dense embeddings (using bge-small-en-v1.5) and sparse indexes (BM25).
• Retrieval: Supports keyword-based (BM25), vector-based, and hybrid retrieval strategies.
• Re-ranking: Reorders candidates using an LLM-based re-ranker, a late-interaction re-ranker (jina-colbert-v2), and a biomedical keyword re-ranker. Configurations allow for single, dual, or ensemble re-ranking.

Provenance and Memory

• Provenance: A first-class design principle. The system tracks the source model for every extraction, the method of concept mapping (tool vs. LLM parametric knowledge), and the rationale for judge scores.
• Memory: Agents share contextual, entity, and long-term memory to maintain continuity across the pipeline.

Key Contributions

1. Task-Agnostic Agentic Framework: A modular system that operates without fine-tuning across diverse extraction tasks (schema extraction, resource extraction, NER) by orchestrating specialized agents.
2. Ontology-Guided Grounding: Integration of curated domain knowledge via ontologies to resolve ambiguity and ensure semantic normalization, addressing the "knowledge gap" in LLMs.
3. Robust Verification and Provenance: Explicit source grounding (span and sentence verification) and systematic provenance tracking (model source, mapping origin, judge rationale) to enhance trust and reproducibility.
4. Human-in-the-Loop Integration: A mechanism for expert validation to correct hallucinations and refine outputs, particularly for high-uncertainty predictions.
5. Local Concept Mapping Service: A high-performance, open-source retrieval and re-ranking service for ontology mapping that reduces dependence on proprietary APIs.

Experimental Results

The framework was evaluated across three case studies using three foundation models (GPT-4o-mini, Gemini 3.1 Flash Lite, Qwen):

• Case Study I (Neuroscience Domain):
  • Schema Extraction: Achieved 91–100% accuracy in converting assessment instruments to machine-readable metadata (ReproSchema). GPT-4o-mini achieved a perfect composite score.
  • Resource Extraction: Achieved 86–93% overall recall in extracting metadata (models, datasets, tools) from scientific papers.
  • Neuroscience NER: Achieved 58–75% label accuracy across 8,882 extracted entities. Performance varied by entity type, with near-perfect accuracy for well-defined categories (Cell Type: 99%, Species: 99%) and lower accuracy for broader categories (Organization: 54%).
• Case Study II (Biomedical NER Benchmarks):
  • Evaluated on NCBI Disease and S800 Species datasets.
  • Achieved ≥90% relaxed recall (97.0% on NCBI Disease with Qwen) and 62.5–85.8% strict recall.
  • The system extracted 1,000–3,600 additional entities beyond gold annotations (e.g., identifying genes in disease-focused papers), demonstrating broader extraction capabilities.
• Case Study III (Local Concept Mapping):
  • Achieved Hits@1 of 62–82% under strict matching and 68–86% under semantic matching.
  • Single LLM re-ranking performed best (81.8% Hits@1 strict), but fully local configurations (dual late-interaction) remained competitive (86.8% Hits@5), suggesting viability without proprietary APIs.

Significance and Claims

The paper claims that StructSense demonstrates generalizability across tasks and domains while maintaining source grounding and provenance transparency. By combining LLM-based extraction with ontology-driven concept mapping, the system transforms unstructured content into semantically normalized, structured outputs that support FAIR principles.

The authors position StructSense as a solution to the limitations of current IE systems:

• It overcomes the brittleness of rule-based systems and the data hunger of ML/DL approaches by operating without fine-tuning.
• It addresses the lack of domain knowledge and schema adherence in LLMs through ontology alignment and agentic refinement.
• It enhances trustworthiness through explicit provenance tracking and human-in-the-loop validation, making outputs more suitable for reliable downstream use in scientific discovery.

The authors acknowledge limitations, including occasional cross-domain misalignments in ontology mapping due to limited local ontology coverage and the reliance on heuristic consensus dictionaries for neuroscience NER evaluation rather than purely manual expert annotations. Future work is proposed to expand ontology coverage, develop expert-annotated benchmarks, and extend the framework to relation extraction.