Automated Standardization of Legacy Biomedical Metadata Using an Ontology-Constrained LLM Agent

This paper presents an LLM-based system that enhances the automated standardization of legacy biomedical metadata by integrating real-time queries to authoritative terminology services, demonstrating significantly improved prediction accuracy over static prompt-based approaches when evaluated on HuBMAP datasets.

The Gist

The Big Problem: A Library of Confusing Notes

Imagine a massive library where scientists have been storing their research data for years. However, instead of using a standard filing system, everyone wrote their notes on sticky notes in their own handwriting, using their own slang.

• One scientist wrote "Lung Tissue."
• Another wrote "Pulmonary Sample."
• A third wrote "Lungs."

To a human, these all mean the same thing. But to a computer trying to organize this library, they look like three completely different things. This makes it impossible to find all the lung studies at once or combine them to learn something new. This is the problem of bad metadata (the "labels" on the data).

Scientists have tried to fix this by writing rulebooks (called Ontologies and Templates) that say, "You must use the word 'Lung' and nothing else." But these rulebooks are often just long text documents that computers can't easily "read" or enforce automatically.

The Old Solution: The "Guessing" Robot

Researchers tried using AI (Large Language Models) to fix these messy notes. They gave the AI the messy note and said, "Hey, here is a rulebook that says 'Use Lung.' Please fix the note."

The problem? The AI was like a student who had memorized a textbook years ago but didn't have the book in front of them during the test.

• It tried to guess the right word based on what it remembered.
• Sometimes it guessed right.
• Often, it guessed wrong (hallucinated) or used a word that was close but not the exact official term required by the rulebook.
• It couldn't check if a specific word actually existed in the official dictionary at that exact moment.

The New Solution: The "Super-Researcher" Agent (ARMS)

The authors of this paper built a smarter system called ARMS (Agentic Real-Time Metadata Standardization). Instead of just asking the AI to guess, they gave the AI a set of tools and a direct line to the official library.

Think of the AI not as a student taking a test, but as a Super-Researcher with a team of assistants:

1. The Librarian Tool: Before the AI tries to fix a note, it calls a tool to pull up the exact current rulebook (the CEDAR template) to see what the rules actually say.
2. The Dictionary Tool: If the rulebook says "You must use a term from the 'Anatomy' dictionary," the AI doesn't guess. It uses a tool to search the live dictionary (BioPortal) in real-time. It asks, "Does 'Lung' exist in the 'Respiratory System' section of the dictionary right now?"
3. The Logic Tool: If the dictionary returns a list of options, the AI uses its brain to pick the one that fits the context best.

How They Tested It

They tested this on 839 old research records from a huge project called HuBMAP (which maps the human body).

• They had a "Gold Standard" version of these records, where human experts had manually fixed every single label perfectly.
• They let the Old AI (just guessing) try to fix the records.
• They let the New Agent (using tools) try to fix the records.

The Results: A Massive Improvement

The results were like night and day:

• The Old AI (Guessing): Got about 54% of the labels right. It struggled especially with specific scientific terms, often making up words or picking the wrong ones.
• The New Agent (Tool-User): Got about 79% of the labels right.
• The "Magic" Part: For the specific scientific terms that had to come from a dictionary (Ontology-constrained fields), the New Agent jumped from 46% accuracy to 78%. In some specific types of experiments, it got 100% of those terms perfect!

Why This Matters

This paper proves that giving AI access to live tools is better than just giving it a textbook.

• Static vs. Dynamic: You can't rely on an AI's memory because scientific dictionaries change every day. The new system checks the dictionary while it works.
• Scalability: Humans can't fix millions of messy records one by one. This system can do it automatically, making scientific data "FAIR" (Findable, Accessible, Interoperable, and Reusable).

The Bottom Line

Imagine trying to organize a messy garage.

• The Old Way: You ask a friend who thinks they remember where things go. They might put a hammer in the toolbox, or maybe in the garden shed. It's a mess.
• The New Way: You give your friend a smartphone with a map app and a live inventory list. They look up exactly where the hammer should go according to the current rules, and they put it there.

This paper shows that when we give AI agents the right tools to look up facts in real-time, they stop guessing and start getting the job done right. This helps scientists share and reuse their data much more easily.

Technical Summary

1. Problem Statement

Scientific research generates vast amounts of data, but the associated metadata is often incomplete, inconsistent, and non-compliant with community standards (e.g., FAIR principles). This lack of standardization hinders data findability, interoperability, and reuse.

• The Limitation of Current Standards: While reporting guidelines (like MIAME) exist, they are typically textual documents. They specify what fields are needed but lack machine-actionable constraints on how values should be formatted or which specific terms from an ontology must be used.
• The Limitation of Static LLM Approaches: Previous work used Large Language Models (LLMs) guided by static text prompts containing template names and ontology constraints. However, this approach relies on the model's parametric (training) knowledge.
  • Hallucination: LLMs may generate terms that sound correct but are not canonical.
  • Static Knowledge: LLMs cannot access the current state of evolving ontologies or specific sub-trees (branches) defined in a template.
  • Lack of Verification: Without real-time access, the model cannot verify if a generated term actually exists within the required ontology branch.

2. Methodology: Agentic Real-Time Metadata Standardization (ARMS)

The authors propose ARMS, an LLM-based agent system that moves beyond static prompting to tool-use (agentic reasoning). The system dynamically retrieves and enforces constraints at execution time.

Core Architecture

• Agent Framework: Built using Python, LangChain, and LangGraph, wrapping an LLM (GPT-5-mini or GPT-4.1-mini).
• Model Context Protocol (MCP): The agent uses MCP to connect to external services via a standardized server interface.
• Three Key MCP Tools:
  1. get_cedar_template: Queries the CEDAR REST API to retrieve the full machine-readable template specification (field definitions, data types, string patterns, and ontology constraints) for a given assay.
  2. term_search_from_ontology: Queries the BioPortal REST API to find ontology terms matching a search string within a specific ontology.
  3. term_search_from_branch: Queries BioPortal for terms matching a search string within a specific branch (sub-tree) of an ontology, enforcing strict hierarchical constraints.

Workflow

1. Input: The agent receives a legacy metadata record and a CEDAR template identifier.
2. Retrieval: The agent calls get_cedar_template to understand the full schema, including which fields are constrained to specific ontologies or branches.
3. Reasoning & Tool Use:
   • For ontology-constrained fields: The agent extracts the legacy value, identifies the required ontology/branch, and calls the appropriate search tool to retrieve a list of valid, canonical candidate terms.
   • For non-ontology fields: The agent uses the retrieved template specifications (data types, string patterns) to format values correctly (e.g., converting "true" to "Yes" or formatting DOIs).
4. Selection: The agent reasons over the retrieved candidates to select the best match or infers missing values based on context.
5. Output: A fully standardized metadata record.

Optimization Strategies

To handle 839 records efficiently, the authors implemented:

• Asynchronous Tool Calls: Reducing idle time by issuing multiple API requests concurrently.
• Response Caching: Storing API responses for duplicate search queries to reduce latency and API load.
• Parallel Execution: Processing records in parallel to accelerate the experimental timeline.

3. Experimental Setup

• Dataset: 839 legacy metadata records from the Human BioMolecular Atlas Program (HuBMAP), covering 12 distinct assay types (Sequencing, Imaging, Mass Spectrometry).
• Gold Standard: Expert-curated versions of these records, manually corrected by domain experts against CEDAR standards. This allows for exact-match assessment.
• Baseline: A "prompt-only" LLM approach where the model receives the legacy record, target field names, and ontology names as static text, without tool access or full template retrieval.
• Metrics: Prediction accuracy (proportion of fields exactly matching the gold standard), stratified by:
  1. Ontology-constrained fields.
  2. Non-ontology-constrained fields.
  3. All fields combined.

4. Key Results

The ARMS agent significantly outperformed the prompt-only baseline across all categories and assay types.

• Overall Accuracy:
  • Baseline: 54%
  • ARMS: 79%
  • Improvement: 46% relative increase.
• Ontology-Constrained Fields (The Critical Gain):
  • Baseline: 46% accuracy (ranging 28–63% per assay). The model struggled to guess correct terms without live lookup.
  • ARMS: 78% accuracy (ranging 55–100% per assay).
  • Notable: ARMS achieved 100% accuracy on ontology-constrained fields for the MIBI and Lightsheet assay types.
  • Relative Improvement: ~70% increase.
• Non-Ontology-Constrained Fields:
  • Baseline: 59% accuracy.
  • ARMS: 79% accuracy.
  • Relative Improvement: ~35% increase. This demonstrates that providing the full template specification (data types, patterns) improves formatting even without ontology lookups.
• Model Robustness: The performance gains were consistent across both GPT-5-mini and GPT-4.1-mini, indicating the method is model-agnostic regarding the tool-use architecture.

5. Key Contributions

1. Agentic Metadata Standardization: Demonstrated that treating metadata constraints as executable specifications (via tool use) rather than static text prompts significantly improves standardization accuracy.
2. Real-Time Ontology Grounding: Proved that querying authoritative terminology services (BioPortal) at inference time is essential for satisfying complex, branch-specific ontology constraints that LLMs cannot reliably recall from training data.
3. Comprehensive Evaluation: Provided a rigorous evaluation using a unique, expert-curated gold standard from HuBMAP, allowing for exact-match validation rather than proxy metrics.
4. Scalable Framework: Developed a practical, optimized pipeline (caching, async, parallelization) that makes automated standardization feasible for large-scale legacy datasets.

6. Significance and Implications

• Solving the "FAIR" Gap: This approach directly addresses the barrier to making legacy data FAIR (Findable, Accessible, Interoperable, Reusable) by automating the conversion of messy, free-text metadata into machine-actionable, standardized formats.
• Beyond "Prompt Engineering": The paper argues that for high-stakes scientific tasks requiring precise adherence to evolving standards, tool-use agents are superior to prompt engineering alone. The model must be able to verify and retrieve rather than just generate.
• Scalability: The method offers a scalable alternative to manual curation, which is too slow and expensive for the millions of legacy records in public repositories.
• Future Directions: The authors note that while highly effective, the system still relies on the quality of external APIs (e.g., BioPortal search robustness) and may miss values not present in the legacy record but derivable from external protocol documents. Future work could integrate document retrieval tools to bridge this gap.

In conclusion, ARMS represents a paradigm shift in metadata management, moving from static, knowledge-based generation to dynamic, constraint-enforced retrieval, thereby enabling the reliable automation of biomedical data standardization.