CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Prediction

1. Problem Statement

The paper addresses the critical challenge of predicting clinical trial outcomes before results are publicly available. This is a high-stakes problem for pharmaceutical companies, clinicians, and patients, as many trials fail, leading to significant resource waste and delayed medical advancements.

The core difficulty lies in data contamination. Traditional benchmarks often fail because:

• Trial results appear in scattered sources (news, conference abstracts, financial disclosures) long before official registry updates.
• Large Language Models (LLMs) trained on web-scale corpora may have "memorized" these partial results, leading to inflated performance that reflects recall rather than genuine predictive ability.
• Static benchmarks become obsolete as models are retrained on newer data, making fair evaluation impossible.

2. Methodology: The CT Open Platform

The authors introduce CT Open, a dynamic, live platform designed to evaluate AI systems on clinical trial prediction while ensuring strict data decontamination.

A. Platform Architecture & Workflow

• Dynamic Challenges: The platform hosts four challenges annually (Winter, Spring, Summer, Fall). Each challenge covers a 3-month window.
• Submission Protocol: Participants submit predictions before the challenge window begins.
• Evaluation Logic: After the window closes, the system identifies trials that:
  1. Had no public results prior to the challenge start date.
  2. Acquired usable public results during the challenge window.
• Benchmark Generation: Only questions derived from this "clean" subset of trials are used to generate the time-stamped leaderboard and benchmark.

B. The Novel Decontamination Pipeline

The paper's most significant technical contribution is a fully automated, robust decontamination pipeline designed to detect the earliest mention of trial results across the entire internet.

• Multi-Modal Search: It combines:
  • LLM-powered Web Search: Using models like GPT-5.2 and Gemini 3 to perform iterative, grounded searches.
  • Customized Web Scraping: Utilizing Brave Search API, Python Requests, and ZenRows to fetch content, with DeepSeek OCR for PDFs.
  • Database Search: Querying PubMed, PMC, BioRxiv, and MedRxiv via NCT IDs.
• Verification Steps:
  1. Date Extraction: A specialized pipeline extracts publication dates from HTML, JSON-LD, and metadata to verify if a document falls before the cutoff date.
  2. Two-Round LLM Verification:
     • Round 1: Confirms the document refers to the exact clinical trial (triangulating NCT IDs, sponsors, design, and eligibility, not just relying on IDs).
     • Round 2: Confirms the document reports actual results (efficacy/safety) rather than just trial initiation or related studies.
• Validation: The pipeline was validated against the proprietary Embase database and manual expert review (50 trials, 200+ pages/trial), achieving >98% accuracy in identifying uncontaminated trials.

C. Question Formulation

CT Open generates questions at the outcome measure level (fine-grained) rather than just trial-level. The questions fall into three classes:

1. Superiority: Binary (Did treatment significantly outperform control?).
2. Comparative Effect: Three-way (Better, Worse, or No Difference?).
3. Endpoint: Binary (Did the trial meet the pre-specified endpoint?).

3. Key Contributions

1. CT Open Platform: The first live, open-access platform specifically for clinical trial outcome prediction that updates quarterly with uncontaminated data.
2. Robust Decontamination Pipeline: A novel, automated system using iterative LLM search and multi-source verification to eliminate "leakage" from obscure sources (blogs, financial reports, conference posters).
3. Time-Stamped Benchmarks: Release of Winter 2025 and Summer 2025 benchmarks with strict cutoff dates, ensuring models cannot rely on pre-training data for these specific trials.
4. Comprehensive Baselines: Evaluation of diverse methodologies including Prompt-based LLMs, Retrieval-Augmented Generation (RAG), Agentic Systems, and traditional Machine Learning (ML).

4. Experimental Results

The authors evaluated models (o3-mini, GPT-5, Claude Opus 4.5) against traditional baselines (Random Forest, Neural Networks) on the Winter and Summer 2025 benchmarks.

• Traditional ML vs. LLMs: Surprisingly, simple Machine Learning (Random Forest) and Neural Network baselines often outperformed or matched state-of-the-art LLMs. This suggests LLMs struggle to effectively utilize structured tabular data (e.g., enrollment numbers, phase, country) compared to models explicitly trained on such correlations.
• Retrieval-Augmented Generation (RAG):
  • RAG provided modest and inconsistent gains.
  • While it offered useful evidence from similar completed trials, models often failed to reason about subtle differences between the retrieved trials and the target trial, sometimes leading to degraded performance.
• Agentic Systems:
  • Agentic workflows (iterative search) were computationally expensive (>$1000 per trial) and yielded mixed results.
  • In some cases, agents discovered unique, high-value sources (e.g., patient blogs, specific industry sites like WithPower) that improved predictions, but in others, they underperformed baselines.
• Contamination Impact:
  • Claude Opus 4.5 (knowledge cutoff Aug 2025) significantly outperformed other models on Winter 2025 (cutoff Jan 2025), likely due to data leakage in its training set.
  • However, on Summer 2025 (cutoff Aug 2025), Opus 4.5's performance dropped relative to others, demonstrating that dynamic, uncontaminated benchmarks are essential to distinguish genuine prediction from memorization.

5. Significance and Future Directions

• Advancing AI Research: CT Open highlights a critical gap in current LLM capabilities: while they excel at reasoning and knowledge retrieval, they struggle with predictive tasks involving structured data and real-world uncertainty.
• Scientific Integrity: By ensuring no prior knowledge of results, the platform allows researchers to test if AI can truly forecast scientific outcomes, a prerequisite for "AI Scientists" that can design and predict experiments.
• Clinical Impact: Accurate prediction could help pharmaceutical companies terminate failing trials early, saving billions of dollars and accelerating the delivery of effective treatments.
• Future Work: The paper suggests future directions include improving LLMs' ability to learn correlational structures in tabular data and developing more efficient agentic systems that can synthesize evidence without excessive computational cost.

In summary, CT Open establishes a new standard for evaluating AI in high-stakes scientific prediction by solving the pervasive problem of data contamination through a rigorous, automated decontamination pipeline, revealing that current LLMs still lag behind specialized ML methods in this specific domain.