Development of a European Union Time-Indexed Reference Dataset for Assessing the Performance of Signal Detection Methods in Pharmacovigilance using a Large Language Model

1. Problem Statement

Pharmacovigilance (PV) relies on signal detection methods to identify potential adverse drug reactions (ADRs). However, the development and validation of these methods are hindered by the lack of reliable, large-scale reference datasets. Existing datasets suffer from three critical limitations:

• Lack of Temporal Indexing: Most datasets do not record when an adverse event (AE) was officially recognized by regulatory authorities. This prevents researchers from restricting analyses to the "pre-confirmation" period, a crucial step for evaluating a method's ability to detect new signals rather than confirming known associations.
• Scope and Currency: Existing resources (e.g., SIDER 4.1) are often outdated, static, or limited in scope (e.g., specific to Japan or pediatric populations).
• Regulatory Gap: While the European Union (EU) maintains a comprehensive archive of historical Summary of Product Characteristics (SmPCs) for Centrally Authorized Products (CAPs), no systematic effort had been made to extract and compile this data into a time-indexed reference dataset.

2. Methodology

The study developed a fully automated, reproducible pipeline to construct a time-indexed reference dataset for the EU.

• Data Source: The authors retrieved current and historical SmPCs for 1,513 Centrally Authorized Products (CAPs) from the European Commission's Union Register of Medicinal Products (data lock: December 15, 2025).
• Extraction Workflow:
  1. Web Scraping: Automated retrieval of product metadata (INN, ATC codes, procedure dates) and direct links to SmPC PDFs.
  2. Text Extraction: PDFs were parsed to extract Section 4.8 ("Undesirable effects") using pdfplumber and other libraries to handle various formats (tables, scanned images).
  3. LLM Processing: The unstructured text of Section 4.8 was processed using DeepSeek V3 (a Mixture-of-Experts LLM with 671B parameters). The model was configured with a temperature of 0.0 for deterministic output to extract AE terms.
  4. Terminology Mapping: Extracted terms were mapped to MedDRA v28 (Preferred Terms, HLT, HLGT, SOC) and ATC classifications.
     • Strategy: A two-pass approach was used. First, exact string matching against the MedDRA dictionary. Second, for unmatched terms, the LLM predicted the most likely System Organ Class (SOC) to narrow the search space, followed by a "SOC-Filtered Match" to identify the best PT.
  5. Time Indexing: Each drug-AE pair was assigned a "Date Added" based on the specific SmPC version in which the AE first appeared, serving as the regulatory confirmation timestamp.
• Validation: Two researchers manually validated a subset of the data against source SmPCs. The extraction accuracy was measured by categorizing errors as correct, incorrect, missing, duplicate, or triplicate.
• Software: The entire workflow was executed in Python (3.2) and R, with code and data made publicly available on GitHub.

3. Key Contributions

• First EU Time-Indexed Dataset: This is the first reference dataset for the EU that captures the precise timing of AE inclusion in regulatory labels, enabling "pre-inclusion" performance analysis.
• Scale and Scope: The dataset covers 1,479 active CAPs (excluding withdrawn products) and 110,823 drug-AE associations spanning 30 years (1995–2025).
• LLM-Driven Extraction: Demonstrates the efficacy of using advanced LLMs (DeepSeek V3) to extract safety information from unstructured regulatory text, achieving high accuracy where traditional rule-based methods struggle with semi-structured formats.
• EU Analogue to PVLens: Creates a European counterpart to the US-based PVLens initiative, facilitating cross-regulatory comparisons.

4. Key Results

• Dataset Composition:
  • Total Associations: 110,823 drug-AE pairs in the processed active database.
  • Temporal Distribution: 74.5% of AEs were identified during pre-marketing (clinical trials), while 25.5% were post-marketing discoveries.
  • Update Frequency: Safety label updates peaked around 2012 (coinciding with the establishment of the PRAC). The median time to the first post-approval SmPC update is approximately 2 years.
• Therapeutic and Organ Class Distribution:
  • Dominant Class: Antineoplastic and immunomodulating agents account for 45.3% of associations.
  • Top SOCs: Gastrointestinal disorders (10.9%), Skin/Subcutaneous tissue (9.0%), and Nervous system disorders (9.3%).
  • Drug Type Differences: Small molecules showed higher rates of Gastrointestinal and Psychiatric disorders, while biologics/targeted therapies showed higher rates of Infections and Skin disorders.
• Specificity: 21.7% of unique Preferred Terms (PTs) were product-specific, while the majority were shared across multiple products.
• Validation Metrics:
  • Extraction Accuracy: 95.1% (Missing events were the most common error).
  • Mapping Success: 95.7% of terms were successfully mapped to MedDRA (73.5% exact match, 22.2% SOC-filtered match).
  • Unresolved Terms: Only 0.5% remained unmatched after automated processing.

5. Significance and Limitations

Significance:

• Benchmarking: The dataset allows for rigorous, longitudinal benchmarking of signal detection algorithms by strictly separating data into "pre-confirmation" and "post-confirmation" periods.
• Methodological Advancement: It validates the use of LLMs for extracting safety data from regulatory documents, offering a scalable alternative to manual curation.
• Transparency: The inclusion of regulatory metadata (procedure types, decision dates) and full traceability to source PDFs enhances the dataset's utility for regulatory science.

Limitations:

• Scope: Covers only Centrally Authorized Products (CAPs), excluding nationally authorized medicines, which limits generalizability to the entire EU market.
• Temporal Lag: The "Date Added" reflects the regulatory approval of the label update, not the exact date the signal was first detected internally by the manufacturer or agency (typically a lag of several months).
• LLM Limitations: While accurate, the LLM approach is sensitive to linguistic variations. The mapping strategy for multi-axial MedDRA terms (where a PT belongs to multiple SOCs) relies on file ordering in the ASCII distribution rather than an explicit primary-flag lookup, which may introduce minor classification ambiguities.

Conclusion:
This study successfully bridges a critical gap in pharmacovigilance by providing a high-quality, time-indexed reference dataset for the EU. It enables more accurate assessment of signal detection methods and supports the development of robust, early-warning safety systems.