Towards understanding the disease landscape of clinical trials in Germany: Ontology and embedding-based pipelines versus Large Language Models for ICD-10 Harmonization

Technical Summary: Ontology and Embedding-Based Pipelines versus Large Language Models for ICD-10 Harmonization in German Clinical Trials

Problem Statement
Clinical trials in Germany are registered across multiple heterogeneous systems, including the German Clinical Trials Register (DRKS), ClinicalTrials.gov, the EU Clinical Trials Register (EUCTR), and the Clinical Trials Information System (CTIS). These registries utilize disparate classification terminologies—such as ICD-10-GM, MeSH, MedDRA, and free-text descriptions—to record health conditions. This lack of standardization impedes cross-registry analysis, limits interoperability with global taxonomies like the Global Burden of Disease (GBD), and obscures the true landscape of disease research priorities. While manual coding is possible, it is labor-intensive and prone to inter-coder variability. The study addresses the need for an automated, reproducible method to harmonize these diverse condition descriptions into a unified WHO ICD-10 classification.

Methodology
The authors developed and evaluated two distinct approaches for mapping condition mentions to WHO ICD-10 codes: a deterministic hybrid pipeline and a Large Language Model (LLM) constrained by the same decision logic used by human experts.

1. Data Sources and Preprocessing:

   • The study analyzed 23,061 German-affiliated interventional trials registered between 2013 and 2025.
   • Condition fields were extracted, normalized (including German-to-English translation and synonym consolidation), and split into individual mentions.
   • A hierarchical linkage method merged records across registries using identifiers and title matching.

2. The Hybrid Pipeline (Four-Stage Approach):

   • Stage 1 (Extraction & Normalization): Parses condition fields, distinguishing actual disease terms from narrative study descriptions.
   • Stage 2 (Mention Classification): Classifies mentions as ICD-mappable (Tier 1: A–Q, U; Tier 2: R, S, T, O, Z) or non-mappable (Tier 3: healthy volunteers, pharmacokinetic studies). This uses a supervised elastic-net logistic regression classifier trained on silver-standard labels, with a conservative override for explicit disease keywords.
   • Stage 3 (Deterministic Ontology Mapping): Generates candidate codes via UMLS pathways:
     • DRKS: Direct mapping from ICD-10-GM.
     • ClinicalTrials.gov: MeSH terms linked to ICD-10 via UMLS.
     • CTIS/EUCTR: MedDRA terms linked via UMLS.
     • Fallback: Jaro-Winkler string matching if ontology links fail.
   • Stage 4 (Embedding-Supported Ranking): Uses SapBERT embeddings to retrieve top-10 semantically similar ICD-10 candidates. A hybrid confidence score combines ontology evidence, embedding similarity (SapBERT), lexical similarity (Jaro-Winkler), and specific bonuses/penalties (e.g., for exact matches, vague codes, or therapeutic area alignment). A cross-encoder reranker (Method 2) was also tested to refine top candidates.

3. LLM Evaluation:

   • GPT-4o was evaluated in parallel against a stratified sample of 500 mentions (including high-confidence, review-band, reject-band, and Tier-3 cases).
   • Crucially, the LLM was not asked to generate codes freely. It was constrained to follow the exact same decision cascade as human reviewers: attempting direct assignment, using trial context if necessary, mapping to GBD categories if ICD-10 was impossible, and assigning "NO_ICD" only as a last resort.

4. Evaluation Framework:

   • A gold standard was created by two medical reviewers independently coding the 500-sample subset (inter-rater agreement $\kappa \ge 0.70$).
   • Performance was measured using accuracy, precision, recall, F1-score, and Cohen's $\kappa$ at the three-character, block, and chapter levels.
   • Error analysis distinguished between within-family errors (same disease domain) and cross-chapter errors.

Key Contributions

• Pipeline Development: The creation of a registry-aware, four-stage pipeline that integrates deterministic ontology mapping (UMLS) with embedding-based semantic retrieval (SapBERT) for harmonizing multi-registry trial data.
• Comparative Benchmarking: A rigorous head-to-head comparison of a complex deterministic pipeline against a constrained LLM (GPT-4o) using a unified expert reference standard.
• Error Characterization: Detailed analysis revealing that a significant portion of pipeline errors (approx. 25%) stem from candidate generation failures (the correct code was never retrieved), rather than ranking errors.
• Recovery Analysis: Demonstration of the LLM's ability to assign clinically plausible codes to mentions the pipeline rejected (82.4% of reject-band mentions) or excluded (62.8% of Tier-3 mentions).

Results

• Pipeline Performance: The baseline hybrid pipeline achieved 49.0% accuracy and a Cohen's $\kappa$ of 0.487 at the three-character ICD-10 level against the expert standard. Performance improved at broader levels (58.7% accuracy, $\kappa$ = 0.561 at the chapter level). The cross-encoder reranker (Method 2) provided only marginal improvements.
• Candidate Recall: The correct code was present in the retrieved candidate set in only 73.7% of cases, indicating a ceiling on performance for ranking-based methods if the correct candidate is missing.
• LLM Performance: The constrained LLM significantly outperformed the pipeline, achieving 96.7% accuracy and a Cohen's $\kappa$ of 0.966 at the three-character level.
• Adjudication: The reported 96.7% accuracy reflects adjudicated results where expert reviewers resolved initial disagreements between the LLM and the reference standard. Before adjudication, the LLM's raw accuracy against the reference was 62.7% (Cohen's $\kappa$ = 0.625); after expert review of these disagreements, accuracy rose to 96.7%, suggesting that many initial discrepancies were due to ambiguous phrasing or context interpretation rather than model failure.
• Error Types: Most pipeline errors were within the same disease family (64.3%), while cross-chapter misclassifications accounted for 30.4%.

Significance and Claims
The paper claims that automated harmonization of clinical trial condition data across heterogeneous registries is feasible and supports the use of a common ICD-10 framework for cross-registry analyses. The authors conclude that while deterministic ontology and embedding pipelines achieve moderate agreement with expert coding, LLMs constrained to a structured decision framework achieve near-perfect agreement (when adjudicated against expert standards).

The study posits that the primary value of modern LLMs in this domain lies not in generating codes from scratch, but in clinical reasoning, disambiguation, and candidate evaluation within a constrained set of plausible diagnoses provided by external knowledge sources. The findings suggest that LLMs can effectively support the analysis of the distribution of health conditions in German clinical trials and serve as a promising tool for classifying other non-standardized trial characteristics, provided they operate within a grounded framework that mitigates hallucination and ensures alignment with established coding logic.