TrialScout links published results to trial registrations using a large language model

The paper introduces TrialScout, a large language model-based tool that efficiently and reliably links registered clinical trials to their published results, achieving high accuracy compared to human coders and successfully identifying result publications for 63.6% of a large sample of trials.

The Gist

Imagine the world of medical research as a massive, chaotic library. In this library, scientists write "recipes" for new medicines (called clinical trials) and file them away in a central catalog called ClinicalTrials.gov.

The problem? Just because a recipe is filed doesn't mean the chef actually cooked the dish and wrote down the results. Sometimes, the results are published in a different, dusty section of the library (like a scientific journal), but the catalog card never gets updated with a link to that new book. Other times, the results are never written down at all.

This creates a huge headache for doctors, patients, and researchers who need to know: "Did this experiment work, and where can I find the proof?" Usually, finding these answers requires a human detective to spend hours, days, or even weeks searching through thousands of books, hoping to match a recipe to its final dish.

Enter "TrialScout": The AI Librarian

The authors of this paper built a new tool called TrialScout. Think of it not as a simple search engine, but as a super-smart, tireless AI Librarian equipped with a "brain" (a Large Language Model) that can read and understand context.

Here is how TrialScout works, using a simple analogy:

1. The Recipe Card (The Trial): TrialScout looks at the original trial registration (the recipe card). It knows what the experiment was supposed to test.
2. The Fingerprint Search: Instead of just looking for a specific code number (like an NCT-ID) that might be missing, TrialScout casts a wide net. It scans millions of scientific articles (the library shelves) looking for anything that might be related to that recipe.
3. The "Taste Test" (The AI Judgment): This is the magic part. TrialScout doesn't just look for keywords; it reads the abstract (the summary) of the potential article and asks its AI brain: "Does this article actually contain the results of this specific trial, or is it just a plan for the trial, a review of other trials, or a completely different study?"
   • Analogy: If you asked a human to find a specific cake recipe, they might grab every book with "cake" in the title. TrialScout is like a baker who opens the book, smells the pages, and says, "No, this is a book about baking bread, not this specific chocolate cake."

Did the AI Librarian Get It Right?

The researchers tested TrialScout against a team of human detectives who had already spent years manually finding these results.

• The Score: TrialScout was incredibly accurate. It found the right results 92.5% of the time when they existed (Sensitivity) and correctly said "no results found" 81.2% of the time when they didn't (Specificity).
• The Twist: When the AI and the humans disagreed, the researchers looked closer. They found that humans were actually wrong more often than the AI! In many cases, the AI found a result the human missed, or the human thought a result was missing when it was actually there. The AI was essentially the better detective in these specific scenarios.

What Did They Find When They Used TrialScout?

The researchers used TrialScout to scan a random sample of 9,600 completed trials. Here is what the AI Librarian discovered:

• The Good News: TrialScout found published results for 63.6% of the trials. This is higher than previous estimates, suggesting that maybe more research is being published than we thought, or that previous methods just weren't looking hard enough.
• The Bad News: About 26% of the trials had no results found at all (neither in a journal nor on the registry).
• The Patterns:
  • Big Trials: Trials with more participants were much more likely to have their results published. (Think of it like a blockbuster movie getting a review; a small indie film might get ignored).
  • Industry vs. Academia: Trials funded by pharmaceutical companies were slightly less likely to have published results compared to those funded by universities or the government.
  • Gender Gap: Trials that only included men were less likely to have published results than those including everyone.
  • The "Terminated" Problem: Trials that were stopped early (Terminated) were much less likely to have results published than those that finished.

Why Does This Matter?

In the past, checking if a trial was reported was a slow, expensive, and boring job that only a few experts could do. TrialScout automates this.

• Speed: It can scan thousands of trials in the time it takes a human to scan a handful.
• Transparency: It helps ensure that patients and doctors aren't making decisions based on incomplete information. If a drug trial is hidden, we might think a drug is safe when it's actually dangerous, or vice versa.
• The Future: TrialScout isn't perfect, but it's a powerful first step. It acts like a "first pass" filter, doing the heavy lifting so human experts can focus their energy on the tricky cases where the AI isn't sure.

In short: TrialScout is a high-tech, tireless librarian that helps us finally find the missing "receipts" for medical experiments, ensuring that the truth about what works and what doesn't is finally brought to light.

Technical Summary

1. Problem Statement

Clinical trials are essential for medical advancement, but their value is compromised if results are not reported. Despite ethical obligations and legal guidelines (e.g., Declaration of Helsinki, FDAAA 801), a significant portion of completed trials lack published results.

• The Gap: Summary results are often missing from trial registries (like ClinicalTrials.gov), and publications are frequently not explicitly linked to their corresponding trial registrations (often due to missing NCT IDs in abstracts).
• Current Limitations: Existing automated tools (e.g., TrialsTracker, Smalheiser-Holt model) rely heavily on explicit NCT-ID matches or metadata similarity. These methods suffer from:
  • Low Sensitivity: They miss publications that do not explicitly cite the trial ID.
  • Scalability Issues: Some models (like Smalheiser-Holt) require pairwise comparisons that are computationally expensive and slow as databases grow.
  • Manual Burden: Accurate identification of results currently requires labor-intensive manual searches by researchers.

2. Methodology: TrialScout

The authors developed TrialScout, an automated tool designed to link clinical trial registrations with result publications using a Large Language Model (LLM).

• Architecture & Workflow:
  1. Data Retrieval: Retrieves metadata from trial registries (ClinicalTrials.gov, EUCTR, DRKS).
  2. Candidate Generation: Uses a multi-strategy search to find candidate publications in PubMed, Google Scholar, and via Citation Matcher APIs. Strategies include:
     • Direct links provided in the registration.
     • Predefined PubMed queries based on trial metadata.
     • Dynamic queries generated by the LLM.
  3. LLM Classification: The core innovation. TrialScout prompts an LLM (specifically ChatGPT-5.1, version gpt-5.1-2025-11-13) with the trial registration details and the abstracts of candidate publications.
  4. Decision Logic: The LLM determines if a publication reports the results of the specific trial, distinguishing them from protocols, systematic reviews, or unrelated studies.
• Technical Specifications:
  • Language: Node.js (v22.7.0).
  • Model Parameters: Temperature set to 1.0; reasoning effort set to "medium".
  • Cost/Efficiency: Average cost of ~$0.043 per trial; processed 5,774 trials in ~1 day on consumer hardware (M2 MacBook Air).
  • Constraint: Can only find results in PubMed-indexed publications that have an available abstract.

3. Validation and Evaluation

The authors validated TrialScout against a "silver standard" of human-curated data derived from two previous rigorous studies (IntoValue dataset and Nilsonne et al. dataset), totaling 5,774 trials.

• Performance Metrics (vs. Human Coders):
  • Sensitivity: 92.5%
  • Specificity: 81.2%
  • F-score: 92.7%
  • Positive Predictive Value (PPV): 92.8%
• Discrepancy Analysis: A manual review of 200 cases where TrialScout and humans disagreed revealed that human error was the dominant factor.
  • False Positives: 79% of cases where TrialScout found a publication but humans did not were actually true positives (humans missed the publication).
  • False Negatives: 44% of cases where TrialScout missed a publication were actually true negatives (no publication existed).
  • Conclusion: TrialScout's actual performance is likely higher than the calculated metrics suggest, as the human reference standard contained significant errors.

4. Key Results: Large-Scale Application

TrialScout was applied to a random sample of 9,600 completed or terminated interventional trials from ClinicalTrials.gov (completed before Oct 2022).

• Overall Reporting Rates:
  • Published Results: Found in 63.6% (6,110) of trials.
  • Reported Results (Published + Registry Summary): Found in 72.9% (6,998) of trials.
  • Note: This published rate (63.6%) is notably higher than the 53% estimated in recent meta-analyses, likely due to TrialScout's superior ability to find unlinked publications.
• Subgroup Analysis:
  • Enrollment: Strong positive correlation between participant numbers and result reporting.
  • Phase: Early-phase trials (Phase 1) had significantly lower reporting rates (60.4%) compared to later phases (79.6%).
  • Funder: Industry-funded trials had lower reporting rates (70.5%) compared to non-industry (73.6%) and NIH-funded (81.0%) trials.
  • Sex: Trials with male-only subjects had significantly lower reporting rates (68.6%) compared to those with all participants (73.3%).
  • Status: Terminated trials had much lower reporting rates (56.9%) compared to completed trials (74.8%).

5. Significance and Contributions

• Technological Advancement: TrialScout demonstrates that LLMs can effectively solve the "linkage problem" in metascience by interpreting semantic content rather than relying solely on metadata tags (NCT IDs). It achieves higher sensitivity than previous models while being computationally faster.
• Scalability: The tool enables the analysis of massive datasets (thousands of trials) that were previously infeasible to audit manually.
• Transparency & Monitoring: It provides a robust mechanism for monitoring trial reporting practices, identifying gaps in transparency (e.g., industry bias, early-phase non-reporting), and assisting systematic reviewers.
• Practical Utility: With a high Positive Predictive Value, TrialScout is recommended as a "first-pass" screening tool. Researchers can use it to automatically link trials to results, reserving manual review only for the cases where the tool fails to find a match.

6. Limitations

• No True Gold Standard: Validation relied on human-curated datasets which were found to contain errors (humans missed valid publications).
• Scope: Currently limited to PubMed-indexed publications with abstracts; it cannot detect results in non-PubMed sources or full-text papers without abstracts.
• Geographic Bias: The large-scale sample was drawn exclusively from ClinicalTrials.gov, which may overrepresent trials from the Americas.

In conclusion, TrialScout represents a significant step forward in research transparency, leveraging AI to automate the detection of clinical trial results with high accuracy and efficiency, thereby facilitating better evidence synthesis and accountability.