A Registry-Bound LLM Pipeline for Evidence-Grounded Trait Extraction across Tropical Plants, Aquatic Species, and Exotic Pets

Technical Summary: A Registry-Bound LLM Pipeline for Evidence-Grounded Trait Extraction

Problem Statement
Structured species traits (e.g., light tolerance, toxicity, propagation methods) are essential for downstream applications ranging from ecological niche modeling to regulatory compliance. However, producing these records at scale via manual curation is infeasible for large, cross-domain datasets like the Tropical Species Encyclopedia, which covers ~410,000 cultivated tropical plants, aquatic species, and exotic pets. While Large Language Models (LLMs) offer a path to scale, they introduce critical trust issues: hallucinated values, lack of evidence grounding, registry out-of-vocabulary (OOV) errors, and uncalibrated confidence. Furthermore, the substrate text itself is LLM-generated (Hop 1), creating a two-hop pipeline where the extraction process (Hop 2) must be rigorously auditable to ensure safety-critical data (e.g., toxicity, CITES status) is not published as authoritative fact without verification.

Methodology: The Registry-Bound Framework
The paper proposes a four-mechanism framework to render LLM-derived trait extraction auditable and operable at scale. The pipeline processes the Tropical Species Encyclopedia substrate, where each species record contains seven canonical Chinese-language bio-sections (morphology, distribution, etc.).

1. Versioned Closed-Vocabulary Registry: A strict schema defines 39 trait keys across four domains (universal, plants, aquatic, pets) and five value types (text, enum, multi_enum, int, range). Every admitted value must conform to this typed schema. Four keys are flagged as "red-zone" (safety-critical: human/pet toxicity, physical hazards, CITES status).
2. Per-Row Verbatim Evidence Quotes: The LLM is required to extract a evidence_quote that must appear as an exact substring within the source bio-section text. This prevents the model from fabricating evidence.
3. Model-Assigned Confidence Labels: The model assigns a confidence level (High, Medium, or Low). Low-confidence candidates are routed to null returns and discarded; only High and Medium confidence rows are persisted.
4. Multi-Version Preservation: The system preserves all extraction runs under a unique key of (species_id, trait_key, model_version). This allows for cross-model history and divergence analysis without overwriting previous extractions.

Extraction Pipeline and Filtering
The pipeline utilizes the mimo-v2.5 model via an OpenAI-compatible API. The process involves three distinct outcome paths for candidate cells:

• Admission: Passes both the substring verification (evidence quote exists in source) and registry conformance (key and value are valid) filters.
• Rejection:
  • Substring/Evidence-Unsupported: The quote is not found verbatim in the source text (9.46% of candidates).
  • Registry-OOV/Enum: The key or value violates the closed vocabulary or type bounds (0.09% of candidates).
• Abstention: The model explicitly returns null when it cannot support a claim, rather than guessing (37.47% of candidates). This is treated as a feature, not a failure.

Key Results
Applied to 409,880 publishable species, the pipeline executed 706,220 runs and persisted 5,489,881 trait records (99.985% coverage of the target population).

• Confidence Distribution: 81.57% of records are High confidence; 18.43% are Medium. Red-zone traits show a higher High-confidence rate (87.82%) due to their reliance on explicit lexical signals rather than inference.
• Validation Layers:
  • Layer 1 (Automated, Full Population): Substring verification confirms that 90.12% of evidence-bearing rows contain a verbatim quote from the source. This rises to 93.49% when excluding the cites_appendix_in_bio trait, which draws evidence from structured fields rather than prose.
  • Layer 2 (Manual, n=100): A stratified audit of non-red-zone rows confirmed that 100/100 evidence quotes semantically support the asserted values (Lower bound 96.30%).
  • Layer 3 (Manual, n=50): A face-validity audit of red-zone rows (toxicity) resulted in 50/50 Accept (Lower bound 92.86%).
• Divergence: Among 2,064 species-trait pairs extracted by multiple model versions, 73.5% produced identical values. Divergences were largely cosmetic (e.g., ordering in multi-enum lists) rather than contradictory.

Significance and Claims
The paper explicitly frames its contribution as a methodological framework rather than a claim of per-record correctness.

• Audit-Ready Substrate: The deposit is positioned as an "audit-ready" resource. It does not claim 100% correctness; all 5.5 million rows are currently marked pending for human review. The value lies in the provenance machinery (evidence quotes, confidence labels, versioning) that enables downstream consumers to stratify trust based on their specific use case.
• Scalability of Rigor: The work demonstrates that a registry-bound, evidence-grounded extraction pipeline can operate at the scale of millions of records while maintaining strict adherence to schema and source text, effectively managing the "per-row trust" problem inherent in LLM extraction.
• Transparency: By publishing the rejection counters, abstention rates, and audit samples, the paper provides a transparent accounting of the pipeline's behavior, distinguishing between model failure, protocol features (abstention), and infrastructure issues.

Limitations and Scope
The authors explicitly state what is not validated:

• Hop-1 Fidelity: The quality of the upstream substrate text (generated by a different LLM) is not validated here.
• Per-Record Correctness: No claim is made that any individual trait row is factually correct without human curation.
• Inter-Rater Agreement: Cohen's κ has not been computed; the audit samples are published to enable external blind review.
• Cross-Family Validation: Cross-model agreement is only observed on a small opportunity sample; a controlled cross-family study is deferred.

The paper concludes that the four-mechanism framework successfully enables the production of a massive, structured, and auditable trait dataset, shifting the burden of final verification to downstream consumers equipped with the necessary provenance tools.