Limiting to English language records: A comparison of five methods on Ovid MEDLINE and Embase versus removal during screening

This study compares five English language limit methods on Ovid MEDLINE and Embase against manual screening removal, finding that while automated limits perform similarly to screening with minor errors due to metadata issues, the risk of excluding eligible English records can be mitigated through citation searching.

The Gist

Imagine you are a detective trying to solve a massive mystery: "What is the best treatment for a specific health problem?"

To solve this, you need to read every single report ever written about the topic. But here's the catch: you only speak English. The world, however, is full of reports written in French, Spanish, Chinese, and dozens of other languages.

You have two choices:

1. The "Filter" Approach: Before you even start reading, you tell your assistant, "Only bring me reports written in English. Throw everything else in the trash immediately."
2. The "Sieve" Approach: You let all the reports (English and non-English) come to your desk. You read the English ones first. When you see a foreign language report, you pause, check if it's actually in English (maybe the title is in Spanish, but the article is in English?), and only then do you decide to throw it away.

This paper is a scientific experiment to see which of these two approaches is safer and smarter. The researchers wanted to know: Does using the "Filter" (limiting your search to English) accidentally throw away important English reports because of a mistake in the filing system?

The Setup: The Two Giant Libraries

The researchers looked at two of the world's biggest medical libraries: MEDLINE and Embase. Think of these as massive warehouses where millions of medical studies are stored on shelves.

They took seven real-life detective cases (systematic reviews) that had already been solved. They knew exactly which reports were the "winners" (the English ones that helped solve the case) and which were the "losers" (the non-English ones that were thrown out).

Then, they ran a simulation. They asked: "If we had used the 'Filter' approach on these libraries, would we have found the same winners? Or would we have accidentally thrown away some English reports?"

The Five "Filters" Tested

The researchers didn't just try one way to filter; they tried five different types of filters to see which was the best:

1. The Strict Filter: "Only English."
2. The "Maybe" Filter: "English OR 'No Language Listed'." (Sometimes the library doesn't know the language, so this filter keeps those mystery boxes just in case).
3. The "Undefined" Filter: "English OR 'No Language' OR 'Undetermined'." (This is for those weird, old records where the language is a mystery).
4. The "Not-Not" Filter: A clever trick that says, "Keep everything except the ones that are definitely not English."
5. The "Super" Filter: A variation of the "Not-Not" filter that tries to be extra careful about records with mixed languages.

The Big Reveal: The "Labeling Error" Problem

Here is the twist in the story. The researchers found that all five filters worked almost exactly the same way. They were like five different brands of the same security gate; they all let 99.8% of the English reports through.

However, they did make a few mistakes.

Imagine a library where a librarian is in a rush. They pick up a report that is written in English and Spanish (a bilingual report). Instead of labeling it "English/Spanish," they accidentally slap a label on it that says "Spanish Only."

If you use the "Strict Filter" (Method 1), your computer sees the "Spanish Only" label and throws the report in the trash. But the report was actually in English! You just lost a clue because of a bad label.

In this study:

• The "Strict Filter" accidentally threw away 5 English reports because the library labels were wrong.
• The "Sieve" approach (checking during screening) caught these mistakes. The human reviewers saw the "Spanish Only" label, looked at the report, realized, "Hey, this is in English too!" and saved it.

The "False Alarm" Problem

The researchers also looked at the reports that were supposed to be thrown away (the non-English ones).

• Some of these were actually in English, but the library labeled them as "French" or "Polish."
• When the researchers used the "Filter," these English reports got through the gate because the filter didn't know they were English.
• When the researchers used the "Sieve" (screening), they saw the "French" label, looked at the report, and realized, "Oh, this is actually English," and kept it.

The Analogy: The Airport Security Check

Think of the Search Strategy (The Filter) as the automated X-ray machine at an airport.

• If the machine is set to "Only allow people with English passports," it might accidentally stop a person who speaks English but has a passport with a smudge that looks like a different language. They get stuck in the security line.
• If you rely on the Screening (The Human Check), you have a human officer at the gate. They see the smudged passport, ask the traveler, "Do you speak English?" The traveler says "Yes," and they get through.

The Takeaway: What Should You Do?

The paper concludes with some practical advice for anyone doing research:

1. The "Filter" is mostly safe: If you are short on time or money, using a filter to limit your search to English is usually fine. You will only miss about 0.2% of English reports (that's 2 out of 1,000).
2. But be careful of "Bilingual" reports: The biggest risk is with reports written in two languages (like English and French). The library labels often get confused here.
3. The "Sieve" is the safety net: If you can, don't just rely on the computer filter. Let the non-English reports come through, and have a human check them. If you see a report labeled "German" but the title is in English, don't throw it away!
4. Double-check the "Mystery" boxes: If a report has no language listed, or says "Undetermined," don't ignore it. It might be an English report that the library forgot to label.

The Final Verdict

The researchers say: "It's okay to use the English filter to save time, but don't be 100% sure it's perfect."

If you are doing a very important medical review where missing even one clue could be dangerous, you should use the "Sieve" method (check everything) or do a "back-up search" (look at the references of the papers you found) to make sure you didn't miss any English reports that got lost because of a bad library label.

In short: The computer filter is a great helper, but it's not perfect. Sometimes, a human eye is the only thing that can spot a mistake in the filing system.

Technical Summary

1. Problem Statement

In evidence syntheses (systematic reviews), excluding non-English language (NEL) records is a common practice to manage resources, but it introduces "language bias," potentially skewing findings by missing relevant evidence. Review teams typically apply language restrictions in two ways:

1. Within search strategies: Using database limits to filter out NEL records before screening.
2. During screening: Retrieving all records and manually excluding NEL records during the title/abstract or full-text screening phase.

The literature suggests that applying limits during screening is more sensitive because it avoids removing records with missing or incorrect language metadata. However, the specific performance of different technical methods for applying English Language (EL) limits on Ovid MEDLINE and Embase has not been rigorously compared against the "screening" approach. Furthermore, the accuracy of language metadata in these databases remains a concern, as errors could lead to the inadvertent exclusion of eligible English-language records.

2. Methodology

The study employed a retrospective analysis using data from seven existing evidence syntheses conducted at the Centre for Reviews and Dissemination (University of York).

• Data Sources: The authors utilized the final "includes" (records meeting eligibility criteria at full-text screening) and "NEL excludes" (records excluded solely due to language) from these seven reviews.
  • Total records tested: 1,509 from MEDLINE and 1,584 from Embase.
• Investigation of Metadata: The authors contacted NLM (for MEDLINE) and Elsevier (for Embase) to understand how language metadata is generated, updated, and applied. They found that metadata is applied separately by each provider and is not automatically corrected by Ovid unless reported.
• Five Limiting Methods Tested: The authors developed and tested five distinct search strategies to limit results to English on Ovid MEDLINE and Embase:
  1. Method 1: Standard inbuilt limit to "English language."
  2. Method 2: Limit to "English language" OR "no language specified."
  3. Method 3: Limit to "English," "no language specified," OR "undefined languages" (e.g., undetermined, multilingual, polyglot).
  4. Method 4: The "not (NEL not EL)" technique using the language field code (lg.). This removes records tagged with NEL metadata that do not also have EL metadata.
  5. Method 5: A variation of Method 4 that also excludes records tagged with "undefined" languages alongside NELs.
• Analysis: The authors applied these five methods to the datasets to calculate:
  • The percentage of "includes" (eligible English records) that were incorrectly removed (false negatives).
  • The percentage of "NEL excludes" (records marked as non-English) that were incorrectly retrieved (false positives, i.e., actually English).
  • The accuracy of the underlying language metadata for these specific records.

3. Key Contributions

• Technical Comparison of Search Strategies: The study provides a direct empirical comparison of five specific search syntaxes and limit options available on Ovid platforms, moving beyond theoretical advice to practical performance data.
• Metadata Accuracy Audit: It highlights specific failure modes in language metadata, particularly regarding bilingual/multilingual journals and records where only the abstract is in English.
• Validation of "Screening" vs. "Search" Limits: It quantifies the risk difference between applying limits in the search strategy versus during the screening process.
• Guidance on Undefined Languages: The paper clarifies the handling of ambiguous metadata tags (e.g., undetermined, multilingual) and their impact on search sensitivity.

4. Results

Performance of the Five Methods:

• Identical Performance: On the datasets tested, all five methods performed identically.
• Retrieval of Eligible Records ("Includes"):
  • MEDLINE: 99.8% (1,020/1,022) of eligible English records were retrieved. 0.2% (2 records) were incorrectly removed.
  • Embase: 99.7% (1,137/1,140) of eligible English records were retrieved. 0.3% (3 records) were incorrectly removed.
  • Cause of Error: All 5 incorrectly removed records had partially correct metadata. They were published in bilingual/multilingual journals but were tagged only with the non-English language, missing the English tag.

Retrieval of Non-English Records ("NEL Excludes"):

• MEDLINE: 98.2% of NEL excludes were correctly removed. 1.8% (9 records) were retrieved (incorrectly identified as English by the limits).
  • Of these 9, 8 were actually English (correctly retrieved by the limit, but excluded by reviewers during screening), and 1 was a Polish journal with English abstracts only (incorrectly retrieved due to metadata error).
• Embase: 94.6% of NEL excludes were correctly removed. 5.4% (24 records) were retrieved.
  • Of these 24, 9 were actually English. The remaining 15 were retrieved due to incorrect or partially correct metadata (e.g., tagged as NEL but containing English text, or vice versa).
  • Pattern: Metadata errors were most common in bilingual journals where the article was only in NEL, or NEL journals with English abstracts.

Key Findings on Metadata:

• Language metadata is often inaccurate for articles in bilingual/multilingual journals.
• The same article can have different language tags on MEDLINE vs. Embase (e.g., "Scottish Gaelic" vs. "Irish Gaelic").
• Records with "no language specified" or "undefined" languages are significant and should be included in sensitive searches.

5. Significance and Implications

• Risk Assessment: While the risk of missing eligible English records via search limits is low (~0.2–0.3%), it is non-zero and stems from metadata errors rather than search syntax flaws.
• Strategic Recommendation:
  • For Resource-Constrained Reviews: Applying EL limits in the search strategy is a pragmatic shortcut, provided the team understands the small risk of missing records.
  • For High-Stakes Reviews: The authors recommend removing NEL records during screening rather than in the search strategy. This approach captures records with missing/incorrect metadata and allows for translation if resources permit.
• Mitigation Strategies:
  • Citation Searching: Forward and backward citation searching is essential to mitigate the risk of missing eligible records due to language restrictions or metadata errors.
  • Metadata Awareness: Information specialists should include "undefined" languages (e.g., undetermined, multilingual) in their search limits to ensure sensitivity.
  • Screening Protocols: Review teams should check records excluded on language grounds for availability in multiple languages, as DOIs or page numbers may differ between language versions.
• Conclusion: The study concludes that while EL limits are generally effective, they are not infallible due to metadata quality issues. The decision to restrict by language should be transparent, justified, and accompanied by mitigation strategies like citation searching to prevent language bias.