MetaHarmonizer: robust biomedical metadata harmonization and a contamination control for inflated LLM performance on public benchmarks

Technical Summary of MetaHarmonizer

Problem Statement
Public biomedical repositories possess significant potential for data reuse, yet inconsistent metadata routinely obstructs integration across studies. While recent Large Language Model (LLM)-based approaches attempt to address harmonization at scale, they suffer from critical limitations: non-determinism, the generation of hallucinated ontology terms, and a reliance on proprietary APIs or labeled fine-tuning data for high accuracy. Furthermore, the paper identifies a fundamental flaw in current evaluation protocols: apparent LLM performance on widely used public benchmarks is likely inflated due to data contamination. Under a contamination-controlled evaluation, the perceived advantage of LLM-only methods on the GDC schema-mapping benchmark is inverted, revealing that many models rely on direct memorization rather than transferable capability.

Methodology
To address these challenges, the authors present MetaHarmonizer, an automated metadata harmonization system designed to be robust by construction. The system comprises two primary modules:

1. SchemaMapper: Aligns attribute names across different schemas.
2. OntologyMapper: Standardizes values to controlled vocabularies.

Both modules utilize a multi-stage cascade architecture. This design escalates to more resource-intensive methods only when earlier, lighter stages fail to produce a match. Crucially, all candidate outputs are grounded in pre-defined controlled vocabularies to preclude hallucinations. LLMs are employed strictly as bounded preprocessing components (e.g., generating alias dictionaries) rather than as inference-time dependencies.

The system is fully local, deterministic, and computationally efficient. Inference times are measured in seconds for schema mapping and under one minute for ontology mapping of up to ~7,000 terms against a pre-indexed corpus of 33,230 terms. Additionally, the system generates calibrated confidence scores (AUC 0.73–0.94) to distinguish correct from incorrect predictions, facilitating principled human-in-the-loop triage.

Key Results

• Schema Mapping (GDC Benchmark):
  • SchemaMapper, utilizing a deployment-optimized LLM-generated alias dictionary, achieved 71.6% Top-1 accuracy.
  • It demonstrated higher Recall@Ground Truth (Recall@GT) than Magneto bipartite variants, recovering significantly more ground-truth mappings.
  • With the best-performing alias dictionary, it reached the highest Top-1, Top-5, and Recall@GT scores and matched the performance of the best Magneto reranker (a fine-tuned LLM-reranker) on Mean Reciprocal Rank (MRR).
  • It outperformed LLM-only baselines when evaluated under contamination-controlled conditions.
• Ontology Mapping (EFO Benchmarks):
  • OntologyMapper achieved 77.9% – 95.5% Top-1 accuracy across four EFO benchmarks.
  • It outperformed the text2term baseline by up to 16.4 percentage points and direct LLM inference (against a smaller corpus) by 19.2 percentage points. The authors note that memorization is not a viable shortcut for this specific task, highlighting the robustness of their approach.
• Contamination Control:
  • The study's evaluation protocol revealed that under contamination-controlled settings, three out of five tested LLMs recovered 80–100% of GDC identifiers from zero-schema context, suggesting their prior high scores were driven by memorization rather than generalization.

Significance and Claims
The paper claims that MetaHarmonizer provides a scalable, domain-agnostic foundation for improving the FAIRness (Findable, Accessible, Interoperable, Reusable) of biomedical data and enabling cross-study integration. By eliminating hallucinations and reducing reliance on non-deterministic LLM inference, the system offers a reliable alternative for metadata harmonization.

Furthermore, the paper posits that its contamination-controlled evaluation methodology is applicable to any LLM-augmented bioinformatics benchmark built on public datasets, offering a necessary correction to inflated performance claims in the field. The system is released as a Python package, emphasizing its utility as a practical tool for immediate deployment.