Does the sensitivity- and precision-maximizing RCT filter find all 'included' records retrieved by the sensitivity-maximizing filter on Ovid MEDLINE? An investigation using 14 Cochrane reviews

This study demonstrates that the sensitivity- and precision-maximizing RCT filter on Ovid MEDLINE significantly reduces the screening burden compared to the sensitivity-maximizing filter while retrieving nearly all eligible records, with any missed studies being recoverable through citation searching.

The Gist

Imagine you are a detective trying to solve a massive mystery. Your job is to find every single clue (a specific type of evidence called a "Randomized Controlled Trial") hidden inside a library that contains millions of books.

The problem? The library is huge, and most of the books are irrelevant noise. If you try to read every single book, you'll never finish your case. So, you need a special search filter—a magical sieve—to sift through the books and only let the relevant ones through.

For years, detectives have used two main sieves:

1. The "Catch-All" Sieve (SM): This sieve has huge holes. It catches almost every single clue you need, but it also lets in a lot of trash (irrelevant books). You have to read a lot of trash to find the good stuff.
2. The "Smart" Sieve (SaPM): This sieve has slightly smaller, smarter holes. It tries to catch the clues you need and keep the trash out. It's supposed to be more efficient, but there was a worry: "Does it miss any important clues that the Catch-All sieve would have found?"

This paper is a report card comparing these two sieves to see which one is better for modern detectives.

The Experiment: A Test Drive

The researchers took 14 real-life detective cases (Cochrane reviews) that had already been solved using the "Catch-All" sieve. They then ran those same cases through the "Smart" sieve to see what happened.

Here is what they found, using some simple analogies:

1. The "Missed Clues" (Sensitivity)

• The Catch-All Sieve: Found 98.2% of the clues it was supposed to find.
• The Smart Sieve: Found 95.9% of the clues.
• The Takeaway: The Smart sieve missed a tiny bit more (about 2.3% more). It's like if you were looking for 100 gold coins, the Catch-All sieve found 98, and the Smart sieve found 96. It missed two coins.

But wait! The researchers checked if those two missing coins could be found another way. They used a technique called "Citation Searching" (basically asking the people who wrote the missing books, "Who did you talk to?" and "Who talked to you?").

• Result: They found almost all the missing clues just by asking the neighbors! So, even if the Smart sieve misses a clue, you can usually find it by doing a little bit of extra digging.

2. The "Trash" Problem (Precision)

This is where the Smart sieve shines.

• The Catch-All Sieve: For every 1 good clue it found, it brought you 189 books to read. That's like finding one diamond in a pile of 189 rocks. You have to dig through a mountain of rocks!
• The Smart Sieve: For every 1 good clue it found, it only brought you 68 books to read. That's like finding one diamond in a pile of 68 rocks.

The Analogy: Imagine you are looking for a specific needle in a haystack.

• The Catch-All sieve hands you a haystack the size of a house. You have to dig through the whole thing.
• The Smart sieve hands you a haystack the size of a small shed. You still have to dig, but it's much faster and less tiring.

3. The "Unique Finds"

Sometimes, the Smart sieve finds things the Catch-All sieve doesn't.

• The Catch-All sieve found 12 unique clues that the Smart sieve missed.
• The Smart sieve found 3 unique clues that the Catch-All sieve missed.
• Why? The Smart sieve is better at spotting clues where the word "trial" is in the title (like a book cover), while the Catch-All sieve is better at spotting clues where "drug therapy" is mentioned in a specific technical way.

The Verdict: Which Sieve Should You Use?

The paper concludes that the Smart Sieve (SaPM) is usually the better choice, but with a few caveats:

1. Save Time: Using the Smart sieve cuts your work down by about 28% on average. If you have a deadline or a small team, this is a huge win. It saves you from reading thousands of useless books.
2. The Risk is Low: You might miss a tiny fraction of clues, but you can fix that easily by doing a quick "neighbor check" (citation searching).
3. Know Your Topic: If your case is specifically about drugs, the Catch-All sieve might be slightly better because the Smart sieve sometimes misses specific drug-related tags. But for most other topics, the Smart sieve is the winner.

The Bottom Line

Think of the Catch-All Sieve as a safety net that catches everything but is heavy and hard to carry. Think of the Smart Sieve as a lightweight, high-tech net that catches almost everything but is much easier to carry.

If you are a detective with a heavy workload, the Smart Sieve is the way to go. You save a massive amount of time, and if you miss a tiny clue, you can easily find it by asking the right questions. The paper tells us: Don't be afraid to use the lighter, smarter net.

Technical Summary

1. Problem Statement

Systematic reviewers face a growing volume of literature in databases like Ovid MEDLINE, necessitating the use of study filters to limit results to Randomized Controlled Trials (RCTs). The Cochrane Handbook provides two primary filters for this purpose:

• Sensitivity-Maximizing (SM): Designed to retrieve the maximum number of relevant studies, often at the cost of retrieving many irrelevant records (low precision).
• Sensitivity-and-Precision-Maximizing (SaPM): Designed to balance high sensitivity with higher precision to reduce the screening burden.

While the SM filter is the standard recommendation, the SaPM filter is often underutilized due to a lack of updated comparative data. Specifically, there was no recent evidence comparing the 2023 updated versions of these filters. Key questions remained:

• Does the SaPM filter miss eligible records that the SM filter would catch?
• What is the trade-off in terms of screening burden (precision, Number-Needed-to-Read, Overall Number-Needed-to-Screen)?
• Can citation searching mitigate the risk of missing records when using the SaPM filter?
• How do recent changes, such as machine-indexing of MeSH terms (introduced April 2022), impact filter performance?

2. Methodology

The study employed a retrospective analysis using 14 recently published or updated Cochrane reviews (published/updated after April 2022) that had utilized the SM filter.

Data Collection:

• Datasets: The final list of "included" records (studies meeting eligibility criteria after full-text screening) from 14 Cochrane reviews were extracted.
• Sample Size: A total of 394 unique "included" records were identified as available on Ovid MEDLINE across the 14 datasets.
• Filter Application: The 2023 updated SM and SaPM filters were applied to these specific datasets to determine recall (sensitivity) and unique yield.

Analysis Steps:

1. Recall & Unique Yield: Calculated the percentage of "included" records retrieved by each filter and identified records unique to one filter.
2. Missed Record Analysis: Investigated why specific records were missed by analyzing metadata (e.g., presence of abstracts, specific subject headings, free-text terms in titles/abstracts).
3. Citation Searching: Performed forward and backward citation searching (using Web of Science and CitationChaser) on the missed records to determine if the risk of missing eligible studies could be mitigated.
4. Reproduction & Efficiency Metrics: For 9 of the 14 reviews (excluding those with <1,000 initial records or specialized registers only), the original search strategies were reproduced. The SM and SaPM filters were applied to calculate:
   • Precision: Proportion of relevant records among total retrieved.
   • Number-Needed-to-Read (NNR): Inverse of precision (records to read to find one "include").
   • Overall Number-Needed-to-Screen (ONNS): Total records requiring screening.
   • Percentage Reduction: The reduction in screening volume compared to unfiltered searches.

3. Key Contributions

• Updated Comparative Analysis: Provides the first direct comparison of the 2023 revised SM and SaPM filters, accounting for recent database indexing changes (e.g., new subject headings like Equivalence Trial and Pragmatic Clinical Trial).
• Granular Failure Analysis: Detailed the specific metadata reasons why records were missed (e.g., lack of abstracts, specific floating subheadings like drug therapy, or title/abstract term variations).
• Mitigation Strategy Validation: Quantified the effectiveness of citation searching as a safety net for the SaPM filter.
• Efficiency Metrics: Provided concrete data on the reduction in screening burden (ONNS, NNR) achievable by switching to the SaPM filter.

4. Key Results

Recall and Sensitivity:

• SM Filter: Retrieved 98.2% (387/394) of the "included" records.
• SaPM Filter: Retrieved 95.9% (378/394) of the "included" records.
• Difference: The SaPM filter missed 9 additional records compared to the SM filter (a 2.3% difference).
• Unique Yield: The SM filter found 12 unique records not found by SaPM; the SaPM filter found 3 unique records not found by SM.

Reasons for Missed Records:

• SM Misses: Often due to records lacking abstracts or relying on specific subject headings like Clinical Trials as Topic.
• SaPM Misses: Primarily due to the absence of specific free-text terms in the abstract (e.g., "trial," "groups") or the floating subheading drug therapy. Notably, Dataset 13 (a drug-focused review) showed a significant drop in SaPM recall (75.5%), highlighting a topic-specific weakness.

Efficiency and Screening Burden:

• Precision: SaPM mean precision was 4.4% vs. SM at 1.5%.
• Number-Needed-to-Read (NNR): SaPM required reading 68 records to find one "include," compared to 189 for the SM filter.
• Overall Number-Needed-to-Screen (ONNS):
  • SM reduced the screening burden by 54.8%.
  • SaPM reduced the screening burden by 83.0%.
  • Net Gain: Using SaPM over SM resulted in an additional 28.2% reduction in the number of records to screen.

Citation Searching Mitigation:

• Citation searching effectively recovered missed records.
• CitationChaser was superior to Web of Science.
• Using bidirectional citation searching (forward and backward) on the missed records allowed for the retrieval of 100% of records missed by the SM filter and 93.7% (15/16) of records missed by the SaPM filter.

5. Significance and Implications

• Strategic Filter Selection: The study challenges the default use of the SM filter. It demonstrates that the SaPM filter can drastically reduce the screening workload (an extra ~28% reduction) with a minimal risk of missing eligible records (approx. 2.3%).
• Topic-Specific Adaptation: The results suggest that for reviews focused on drug interventions, the SM filter (or an adapted SaPM filter including the drug therapy floating subheading) remains superior due to the SaPM's lower sensitivity in this specific domain.
• Mitigation via Citation Searching: The risk of missing records with the SaPM filter is low and can be effectively mitigated by performing citation searching on the few records the filter misses. This supports a workflow where the SaPM filter is used for the primary database search, followed by citation searching to capture the "tail" of missed studies.
• Machine Indexing Context: The study notes that the introduction of machine-indexing in MEDLINE (April 2022) may affect filter performance, particularly regarding the accuracy of subject headings, warranting future research.
• Recommendation: Information specialists should consider the SaPM filter for most reviews to improve efficiency, particularly for large-scale or rapid reviews, provided that citation searching is employed to capture the small percentage of missed records. Decisions should be tailored to the specific topic (e.g., avoiding SaPM for drug-heavy reviews without adaptation).