Technical Summary: CaresAI at CT-DEB'26

Problem Statement

Medication errors, specifically dosing errors within clinical trials (CT), pose significant risks to patient safety, trial integrity, and financial viability. While preventable, these errors are difficult to identify early due to the reliance on large volumes of unstructured textual data in trial protocols and reports. Manual verification of correct dosing schedules across study arms is time-consuming and prone to oversight. The challenge addressed in this study is the automated detection of clinical trials that meet a predefined elevated risk criterion for dosing errors, utilizing Natural Language Processing (NLP) to extract and analyze methodological details from unstructured text.

Methodology

Dataset and Problem Formulation

The study utilized data from the CT-DEB'26 challenge, derived from ClinicalTrials.gov. The dataset comprised 29,478 training and 6,316 validation records, each representing a single clinical trial. The target variable was a binary label indicating whether the lower bound of the 95% Wilson confidence interval for the dosing error rate exceeded 0.01%. The dataset included a mix of structured categorical/numerical variables and unstructured text fields (e.g., briefSummary, detailedDescription, armDescriptions, interventionDescriptions).

Feature Engineering

• Text Encoding: Unstructured text fields were encoded using four domain-specific pre-trained transformer models: BioBERT, ClinicalBERT, PubMedBERT, and MedCPT. A maximum sequence length of 256 tokens was applied, with sentence-level embeddings generated via mean pooling.
• Structured Features: Categorical variables (e.g., interventionModel, allocation, oversightHasDmc, primaryPurpose, phase) were one-hot encoded. Continuous numerical variables (e.g., enrollmentCount, numArms) were excluded from the final pipeline due to weak statistical association or redundancy.
• Exclusions: The protocolPdfText field was excluded due to extreme data sparsity in the training set (81.3% missing) and excessive document length, which would necessitate aggressive truncation and introduce distributional bias.

Experimental Framework

The research followed a controlled, iterative framework with two primary phases:

1. Phase 1 (Encoding & Features): Evaluated different transformer encoders and the incremental impact of categorical features using a logistic regression baseline.
2. Phase 2 (Model Architecture): Tested various classical machine learning models (Logistic Regression, SVC, XGBoost, LightGBM, etc.) and neural network architectures (MLP, Residual Networks, Text-Tower) on the selected features.
   • Class Imbalance: Given the dataset's imbalance (~95.4% negative class), experiments varied class weighting strategies (fixed ratios and automatic proportional weighting).
   • Optimization: Neural networks utilized early stopping, learning rate scheduling, and loss function comparisons (Binary Cross-Entropy vs. Focal Loss).

Evaluation

Performance was primarily assessed using ROC-AUC on a held-out validation set. Secondary metrics included precision, recall, and balanced accuracy to address class imbalance. A final robustness check was performed using stratified 5-fold cross-validation on the combined dataset.

Key Results

Text Encoding Performance

• BioBERT emerged as the superior encoder, achieving a ROC-AUC of 0.794 with logistic regression. This represented a 3.95% improvement over the ClinicalBERT baseline (0.764).
• PubMedBERT (0.775) and MedCPT (0.789) performed moderately, while ClinicalBERT (0.764) underperformed, likely due to its pre-training on ICU notes rather than structured protocol language.
• Combination Strategy: Concatenating embeddings from multiple encoders did not improve performance (ROC-AUC dropped to 0.764), suggesting that domain alignment is more critical than representational stacking for this task.

Model Architecture Performance

• Classical Models: Gradient boosting models (LightGBM and XGBoost) and Support Vector Classifiers (SVC) achieved the strongest results. LightGBM reached a validation ROC-AUC of 0.831, and the calibrated SVC achieved 0.831. Logistic Regression achieved 0.812.
• Neural Networks: A Residual Neural Network (with skip connections) achieved the highest performance among neural architectures with a ROC-AUC of 0.831. Simple MLPs (0.829) and wider networks (0.814) performed slightly lower. Text-tower architectures did not yield significant gains over standard concatenation.
• Class Weighting: While varying class weights had minimal impact on Logistic Regression (ROC-AUC remained ~0.795), it significantly improved Neural Network performance, increasing ROC-AUC from 0.806 (no weighting) to 0.838 (automatic weighting).

Feature Importance

Statistical analysis (Chi-squared and Cramér's V) identified interventionModel, allocation, and oversightHasDmc as the most significant predictors. Clinically, complex designs (crossover/factorial) and the absence of Data Monitoring Committees correlated with higher error risks.

Significance and Claims

The paper claims that the integration of domain-specific transformer embeddings (specifically BioBERT) with structured metadata enables the effective discrimination of clinical trials meeting elevated dosing error risk criteria.

Key contributions include:

1. Demonstrating Domain Alignment: Showing that models pre-trained on biomedical literature (BioBERT) outperform those trained on general clinical notes (ClinicalBERT) for protocol analysis.
2. Validating Hybrid Approaches: Proving that combining text embeddings with categorical features yields superior results compared to text-only or structured-only approaches.
3. Practical Utility: The proposed framework supports safety monitoring and informed regulatory decision-making by automating the identification of high-risk trial protocols.

The authors maintain a modest stance, noting that while the models achieved strong discrimination (ROC-AUC ~0.82–0.85), the study did not exhaustively tune hyperparameters for neural networks, and the exclusion of PDF text limits the scope to specific textual fields. The work advances the application of NLP in clinical trial integrity but acknowledges the ongoing challenge of data scarcity and the need for further large-scale experimentation.