More Signal vs. More Noise - Comparing Full Text and Abstract as Inputs for Large Language Model-based Classification of Oncology Trial Eligibility Criteria

This study demonstrates that providing full-text articles rather than just abstracts significantly improves the accuracy of GPT-5 in classifying oncology trial eligibility criteria, as the additional signal from the complete text outweighs the potential negative impact of increased noise.

The Gist

Imagine you are a detective trying to solve a mystery: "Who is allowed to join this exclusive medical club?" (In this case, the "club" is a clinical trial for cancer patients).

To solve the mystery, you have two different sources of information to read:

1. The "Elevator Pitch" (The Abstract): A short, 200-word summary at the beginning of a research paper. It's like a movie trailer—exciting, condensed, and hits the high notes, but it leaves out the gritty details.
2. The "Whole Movie" (The Full Text): The entire research paper, which can be 10,000+ words. It has the trailer, but also the behind-the-scenes footage, the script, the boring interviews, and the tiny details hidden in the background.

The Big Question:
Does reading the whole movie help you solve the mystery better, or does all that extra "noise" (irrelevant words) confuse you? Or, does the "Elevator Pitch" give you just enough info to get it right without the headache?

The Experiment

The researchers in this paper used a super-smart AI detective (called GPT-5) to play this game. They took 200 real cancer studies and asked the AI two questions for each one:

• "Can patients with localized cancer (cancer stuck in one spot) join?"
• "Can patients with metastatic cancer (cancer that has spread) join?"

They ran the test twice for every single study:

• Round 1: The AI only read the short "Elevator Pitch" (Abstract).
• Round 2: The AI read the entire "Whole Movie" (Full Text).

The Results: More Signal, Less Noise?

Here is what happened, using a simple analogy:

The "Metastatic" Case (The Easy Clue):
Finding out if a study included patients with spread cancer was like finding a neon sign in a dark room. The AI found this clue almost perfectly (99% accuracy) whether it read the short summary or the whole book. The "neon sign" was so bright in the abstract that the extra noise of the full text didn't matter.

The "Localized" Case (The Hidden Clue):
Finding out if a study included patients with localized cancer was like looking for a specific needle in a haystack.

• With the Abstract: The AI got it right 86% of the time. Sometimes, the "Elevator Pitch" forgot to mention that local patients were allowed, or it was too vague. The AI missed the needle.
• With the Full Text: The AI got it right 92% of the time. Even though the full text was full of distractions (like long lists of chemicals, patient histories, and statistical tables), the AI was smart enough to ignore the noise and find the specific sentence in the fine print that said, "Yes, local patients can join."

The Verdict

Reading the whole book won.

The study concluded that for this specific task, the extra "signal" (the hidden details found only in the full text) was worth the extra "noise" (the thousands of irrelevant words). The AI was strong enough to filter out the junk and find the gold.

Why This Matters

Think of it like searching for a specific ingredient in a recipe.

• If you only read the summary on the back of the box, it might just say "Delicious soup!" and miss the fact that it requires "fresh truffles."
• If you read the full recipe, you see the truffles, but you also have to wade through instructions on how to chop onions and boil water.

This study proves that modern AI is smart enough to chop the onions and boil the water while finding the truffles. It suggests that if we want to build better tools to help doctors find the right clinical trials for their patients, we shouldn't just rely on the short summaries. We need to let the AI read the whole story to get the full picture.

In short: Don't just read the movie trailer; let the AI watch the whole movie. It turns out, the AI is smart enough to ignore the boring parts and find the plot twists that matter.

Technical Summary

1. Problem Statement

The rapid growth of biomedical literature creates a bottleneck for clinicians and researchers trying to synthesize evidence, particularly regarding clinical trial eligibility. While Natural Language Processing (NLP) and Large Language Models (LLMs) offer automation potential, a critical trade-off exists regarding input data:

• Abstracts: Provide condensed, high-density information but often omit critical eligibility details (e.g., specific disease stages like "locally advanced" vs. "metastatic").
• Full Texts: Contain the complete "signal" necessary for accurate classification but introduce significant "noise" (thousands of irrelevant words) that may degrade LLM reasoning capabilities.

The study addresses the open question: Does the additional signal in full-text manuscripts outweigh the potential performance degradation caused by noise when using state-of-the-art LLMs for classifying oncology trial eligibility?

2. Methodology

Data Collection & Preprocessing:

• Dataset: A random sample of 200 randomized controlled oncology trials published between 2005 and 2023 in six high-impact journals (BMJ, JAMA, JAMA Oncology, The Lancet, Lancet Oncology, NEJM).
• Exclusion: Trials from the Journal of Clinical Oncology were excluded due to licensing restrictions on full-text PDFs.
• Processing: Full-text PDFs were downloaded and converted to plain text using the PyMuPDF library.

Ground Truth Annotation:

• Labels: Trials were labeled as "LOCAL" (localized/locally advanced), "METASTATIC," "Both," or "Neither."
• Process:
  1. Initial annotation by one author (P.W.) based on title/abstract.
  2. Independent re-annotation by a second author (J.W.) based on full text.
  3. Discrepancies were resolved by a third author (D.R.Z.).
  4. Inter-annotator agreement: 95.5% (191/200). Disagreements were largely resolved in favor of the full-text annotation, confirming that abstracts often lacked necessary details.

LLM Configuration:

• Model: OpenAI's GPT-5 (snapshot gpt-5-2025-08-07).
• Parameters: Temperature set to 1, reasoning level set to "high."
• Approach: Zero-shot classification with structured output (Python list).
• Input Strategy: Each trial was processed twice:
  1. Abstract-only input.
  2. Full-text input.
• Task: Determine eligibility for patients with localized disease, metastatic disease, both, or neither.

Evaluation Metrics:

• Accuracy, Precision, Recall, and F1 Score were calculated for both input types.
• Statistical significance of differences was assessed using the McNemar test.
• 95% Confidence Intervals (CI) were computed using normal approximation.

3. Key Contributions

• Empirical Evidence on Input Modalities: This is one of the first studies to systematically compare abstract vs. full-text inputs for LLM-based clinical trial classification, moving beyond the historical reliance on abstracts due to cost/complexity.
• Noise Robustness Demonstration: It demonstrates that modern LLMs (specifically GPT-5) possess sufficient reasoning capacity to filter out the "noise" of full manuscripts and extract specific eligibility criteria without performance degradation.
• Identification of Information Gaps: The study highlights that abstracts frequently omit "locally advanced" or "unresectable" disease details, leading to under-classification of localized disease eligibility when relying solely on summaries.

4. Results

Performance on Localized Disease:

• Abstract Input: Accuracy = 86% (F1 = 0.90).
• Full Text Input: Accuracy = 92% (F1 = 0.92).
• Significance: The improvement was statistically significant (p = 0.027).
• Analysis: The full text significantly improved Recall (from 0.83 to 0.92), meaning the model missed fewer eligible trials when given the full text.

Performance on Metastatic Disease:

• Abstract Input: Accuracy = 99% (F1 = 1.00).
• Full Text Input: Accuracy = 98% (F1 = 0.99).
• Significance: No statistically significant difference (p = 0.5). Performance was near-perfect for both inputs, likely because metastatic status is explicitly stated in most abstracts.

Combined Label Accuracy (Overall):

• Abstract: 86% accuracy.
• Full Text: 92% accuracy (p = 0.027).
• Error Reduction: Misclassifications where trials enrolling both localized and metastatic patients were labeled only as "Metastatic" dropped from 13.5% (abstract) to 6% (full text).

Key Observation: The primary benefit of full-text input was capturing the "localized" label in trials where the abstract emphasized the metastatic population, a common reporting bias in oncology literature.

5. Significance and Conclusion

Scientific Impact:
The study concludes that for the specific task of extracting detailed eligibility criteria (particularly disease stage) in oncology, the benefit of the additional signal in full texts outweighs the potential negative impact of noise. The common practice of relying solely on abstracts for NLP tasks may lead to an underestimation of an LLM's true potential and result in incomplete evidence synthesis.

Practical Implications:

• Evidence Synthesis: Automated literature review systems should prioritize full-text ingestion for tasks requiring granular eligibility data, despite higher computational costs.
• Model Robustness: State-of-the-art LLMs are robust enough to handle long-context inputs without losing reasoning fidelity, challenging the assumption that "noise" inevitably degrades performance.

Limitations & Future Work:

• Generalizability: Limited to structured abstracts from high-impact journals.
• Data Leakage: The model's training cutoff post-dates some trials, though this likely makes the reported performance gap a conservative estimate.
• Multimodality: The study used plain text extraction, ignoring figures and tables. Future work should explore Vision-Language Models (VLMs) to process PDFs directly.
• Task Specificity: The benefit of full text is likely task-dependent; for primary endpoints (usually in abstracts), full text may offer diminishing returns compared to eligibility criteria.

In summary, the paper argues that "More Signal" (Full Text) > "More Noise" (Full Text) for complex extraction tasks in oncology, provided the LLM is sufficiently advanced.