Technical Summary: AMEND++: Benchmarking Eligibility Criteria Amendments in Clinical Trials

Problem Definition

Clinical trial protocols frequently undergo post-approval modifications, known as amendments, which introduce significant delays, increased costs, and administrative burdens. Among the various components of a protocol, Eligibility Criteria (EC) are the most commonly amended section. Changes to inclusion and exclusion criteria can alter the study population mid-trial, affecting study integrity and interpretability.

Despite the operational importance of anticipating these changes, the computational literature lacks large-scale resources and models to predict whether the EC text of an initial protocol will be modified later. Existing studies have primarily been descriptive or statistical, characterizing the frequency and causes of amendments but failing to frame amendment forecasting as a predictive modeling problem or release machine learning-ready datasets.

Methodology

The authors propose a comprehensive pipeline to address this gap, centered on the AMEND++ benchmark suite and a novel pretraining strategy.

1. Data Collection and Processing (AMEND++)

The benchmark is constructed from interventional drug trials curated from ClinicalTrials.gov. The authors programmatically scraped the "Record History" tab to reconstruct longitudinal EC version histories, which are not available via standard APIs. The suite comprises two datasets:

• AMEND: A large-scale collection (161,970 trials) capturing raw EC version histories and amendment labels derived directly from public records.
• AMEND_LLM: A refined subset (64,641 trials) where labels are "denoised" to isolate substantive changes from minor administrative edits (e.g., formatting, typos).

2. Label Construction and Denoising

To generate high-quality labels for training without prohibitive manual costs, the authors employ a LLM-based denoising pipeline for the training and validation splits, while reserving human annotation for the test split (gold standard).

• Decomposition: Instead of a monolithic judgment, the LLM performs three distinct checks: identifying added criteria, removed criteria, and modified criteria (e.g., changes in numeric thresholds or logical operators).
• Input: The LLM processes the textual differences ($\Delta E$) between the initial and final EC versions.
• Aggregation: A logical OR of the three checks determines the final binary amendment label.
• Validation: This approach achieved a 99.34% match rate with human annotations, significantly outperforming raw ClinicalTrials.gov labels and baseline Chain-of-Thought approaches.

3. Task Formulation

The task is defined as a supervised binary classification problem. Given the initial eligibility criteria text ($E^{(0)}$) and trial-level metadata ($M$) available at initiation, the model predicts whether the protocol will undergo future EC amendments ($\hat{y} \in \{0, 1\}$).

4. Change-Aware Masked Language Modeling (CAMLM)

To improve representation learning, the authors propose CAMLM, a revision-aware pretraining strategy.

• Motivation: Standard Masked Language Modeling (MLM) treats all tokens equally, whereas EC amendments are typically sparse and localized.
• Mechanism: CAMLM leverages historical EC versions to identify "unstable spans"—tokens that were deleted, replaced, or modified in later versions.
• Masking Policy: During pretraining, tokens within these unstable spans are masked with a high probability ($p_{span}$), while stable tokens are masked with a lower background probability ($p_{low}$).
• Architecture: CAMLM does not alter the model architecture (e.g., it works with BERT, BioBERT) but modifies the masking policy to encourage the encoder to learn representations sensitive to content that is historically prone to revision.

Key Contributions

1. New NLP Task: The formulation of eligibility criteria amendment prediction as a novel task to forecast future protocol modifications based on initial drafts.
2. AMEND++ Benchmark Suite: The release of two large-scale datasets:
   • AMEND: Raw, large-scale EC version histories.
   • AMEND_LLM: A high-fidelity subset with LLM-denoised labels that decompose amendments into added, removed, and modified components.
3. CAMLM Pretraining Strategy: A method that utilizes historical edits to learn amendment-sensitive representations, improving downstream prediction without architectural changes.

Experimental Results

Experiments were conducted on both AMEND and AMEND_LLM datasets using various baselines (BioBERT, BERT, Longformer) and classifiers (Logistic Regression, Random Forest, Fine-tuned BioBERT).

• Performance Gains: CAMLM consistently improved prediction performance across all model classes and metrics.
  • On fine-tuned BioBERT, CAMLM improved AUROC by ~1.4–1.5%, AUPRC by ~2.1%, and Accuracy by 4.0–4.4% relative to baselines.
  • Statistical significance (DeLong's test) confirmed these improvements ($p < 0.05$) for Logistic Regression and Fine-tuned models on both datasets.
• Ablation Studies: Comparisons showed that while standard MLM and span-aware MLM provided modest gains, CAMLM (modeling long-horizon protocol evolution) consistently outperformed all other pretraining strategies, demonstrating that modeling EC evolution yields richer representations than static text alone.
• Label Quality: Models trained on LLM-denoised labels (AMEND_LLM) outperformed those trained on raw CTGov labels when evaluated against human-annotated test data, validating the denoising pipeline.

Significance and Claims

The paper claims that the information contained in initial eligibility criteria text holds predictive signals for future amendment risks. By introducing AMEND++ and CAMLM, the authors provide a scalable framework for:

• Anticipating Protocol Amendments: Enabling sponsors to identify ECs at elevated risk of modification early in the design process.
• Improving Trial Design: Helping sponsors improve protocol quality, reduce avoidable revisions, and streamline trial execution.
• Cost Reduction: Potentially mitigating the substantial costs and delays associated with post-approval modifications.

The authors note that while their work focuses exclusively on Eligibility Criteria as a starting point due to their frequency and complexity, the framework could be extended to other protocol sections (e.g., outcomes, study design) in future work. They also acknowledge the assumption that "no amendment" labels in public registries are correct, though false negatives are expected to be rare.