Removing animal and nonhuman records in Ovid Embase: A comparison of 11 filters

This study evaluates the performance of 11 different filters for removing animal and nonhuman records in Ovid Embase by analyzing 3,000 records, revealing significant variations in sensitivity and specificity among the methods and highlighting the need for search teams to carefully select filters based on their specific research context.

The Gist

Imagine you are a chef trying to make a delicious soup that only contains human ingredients. You have a massive, chaotic pantry (the Ovid Embase database) filled with millions of jars. Some jars contain human recipes, some contain dog recipes, some contain cat recipes, and some contain a weird mix of both.

Your goal is to find the human recipes without accidentally throwing away the ones that mention humans but also have a dog in the title (like a study about how dogs help humans).

To do this, you have 11 different "magic sieves" (search filters). Each sieve is designed to shake out the animal jars and keep the human ones. But here's the problem: nobody knew which sieve actually worked the best. Some might be too coarse (letting dog jars through), and some might be too fine (throwing away human jars that had a tiny speck of dog in the description).

This paper is like a taste-test competition where the authors tried all 11 sieves to see which one saved the most human recipes while still getting rid of the animal ones.

The Setup: The Three Piles of Jars

To test the sieves, the authors didn't just look at random jars. They created three specific piles of 1,000 jars each:

1. The "Dog Only" Pile: Jars that are definitely about animals.
2. The "Human Only" Pile: Jars that are definitely about people.
3. The "Mixed Bag" Pile: Jars that are about both animals and humans (like a study on how a dog's presence affects a human's heart rate).

They then ran each of the 11 sieves through these piles to see what happened.

The Results: The "Goldilocks" Sieves

The authors measured two things:

• Sensitivity (The "Don't Miss" Score): How many good human jars did the sieve keep? If you throw away a human jar by mistake, your sensitivity is low.
• Specificity (The "Don't Let In" Score): How many bad animal jars did the sieve catch? If you let a dog jar slip through, your specificity is low.

Here is what they found:

1. The "Super Catcher" (Method 11)
This sieve was the most sensitive. It kept 90.6% of the human jars. It was very careful not to throw anything away. However, it was a bit sloppy; it let about 28% of the animal jars slip through.

• Analogy: Imagine a security guard who is so afraid of missing a VIP guest that they let almost everyone into the building, even the people who shouldn't be there. You get all the VIPs, but you have to deal with a lot of extra people.

2. The "Strict Bouncer" (Method 3)
This sieve was the most specific. It caught 91.7% of the animal jars. It was very good at keeping the building clean of dogs. However, it was too strict; it accidentally threw away 25% of the human jars.

• Analogy: Imagine a bouncer who checks IDs so strictly that they accidentally kick out a few VIPs because their ID looked slightly different. The club is very clean of intruders, but you lost some good guests.

3. The "Middle Ground" (Methods 1-9)
Most of the other sieves were in the middle. They kept about 78% of the human jars and caught about 90% of the animal jars. They were "okay" but not the best at either extreme.

The Big Surprise: What Got Thrown Away?

The authors looked closely at the human jars that the sieves accidentally threw away. They found a pattern:

• The most common human jars to get tossed were the ones that actually used human participants or data.
• Why? Because the database (the pantry) sometimes forgets to put a "Human" label on a jar, even if it has humans in it. The sieve looks for that label, doesn't find it, and throws the jar away.

It's like if a library catalog forgot to tag a book as "Fiction," and your robot librarian threw it in the trash because it didn't see the tag, even though it was a great story.

The "Why" Behind the Sieves

Why did Method 11 work so well at keeping humans, but Method 3 work so well at catching animals?

• Method 11 was designed to be gentle. It only looked for specific "Animal Experiment" labels. If a jar didn't have that label, it stayed. This is great for keeping humans, but bad for catching animals that didn't get labeled correctly.
• Method 3 looked at the Title and Abstract (the text on the jar) in addition to the labels. It searched for words like "animal study" or "excluded animals." This made it very good at catching animals, but it also accidentally caught human jars that mentioned animals in passing (e.g., "We excluded animal studies from our review").

The Takeaway for You

If you are a researcher (the chef) trying to find human studies:

1. Don't just pick a sieve blindly. You need to know your ingredients.
2. If you are terrified of missing a single human study (high sensitivity), use Method 11. You will have to do a bit more work later to manually check the "animal" jars that slipped through, but you won't lose any human gems.
3. If you are terrified of wasting time reading animal studies (high specificity), use Method 3. But be warned: you might accidentally throw away some human studies that mention animals.
4. Talk to your team. Before you start cooking, ask: "Do we care about studies where animals helped humans?" (Like therapy dogs). If yes, you can't use a sieve that throws those out.

The Bottom Line

There is no perfect sieve. Every tool has a trade-off.

• Method 11 is the safest bet for not missing human studies.
• Method 3 is the best at cleaning out the animal noise.

The authors conclude that information specialists (the librarians) and researchers (the chefs) need to sit down and have a chat about what they are willing to lose in exchange for what they want to find. It's all about balancing the risk of missing a good recipe against the risk of cooking with the wrong ingredients.

Technical Summary

1. Problem Statement

Evidence syntheses (e.g., systematic reviews) focusing on human populations often require the exclusion of records related to animals or nonhumans to reduce noise and focus on eligible studies. While Ovid Embase contains an inbuilt limit for the Emtree subject heading human/, relying solely on this risks excluding relevant records that lack specific indexing. Consequently, information specialists routinely employ various pre-tested search filters to remove animal/nonhuman records.

However, a critical gap exists:

• Lack of Performance Data: Many filters currently in use are non-validated, lack performance metrics (sensitivity/specificity), or have unknown authorship.
• Design Variability: Filters differ significantly in their construction (e.g., use of Emtree subject headings vs. free-text in titles/abstracts, specific Emtree terms used).
• Indexing Complexities: The definition of "animal" vs. "human" records is complex (e.g., animal-assisted therapies, zoonotic diseases), and indexing practices (manual vs. automatic) vary by publication type (e.g., conference abstracts vs. full-text articles).

The study aims to evaluate and compare the performance of 11 commonly used filters in Ovid Embase to determine which offers the best balance of retaining relevant human records while removing irrelevant animal records.

2. Methodology

The study employed a quantitative, comparative design using a "gold standard" dataset created by the authors.

• Dataset Construction:

  • Three search strategies were developed in Ovid Embase to retrieve records likely to fall into three categories: (1) Animals/Nonhumans, (2) Humans, and (3) Both Animals/Nonhumans and Humans.
  • These strategies utilized filters for Systematic Reviews and Randomized Controlled Trials (RCTs) to ensure relevance to evidence synthesis.
  • Sample Size: A total of 3,000 records were exported (500 removed + 500 retained per category).
  • Screening: Two reviewers independently screened titles and abstracts to categorize records as "should be removed" (irrelevant to humans) or "should be retained" (relevant to humans). Disagreements were resolved via discussion.
  • Final Test Set: Due to accession number retrieval issues, the final analyzed dataset consisted of 2,689 records (1,102 "should be retained" and 1,587 "should be removed").

• Filters Tested:

  • 11 Methods: The study tested 11 distinct search strategies (Methods 1–11). These included filters from the Cochrane Common Mental Disorders Group, York Health Economics Consortium (YHEC), Canada's Drug Agency, and the Embase Guide.
  • Design Variations: The filters varied in their use of:
    • Emtree subject headings (e.g., exp animals/, animal experiment/, human/).
    • Free-text fields (Title, Abstract).
    • Logical operators (NOT, AND).
    • Specific exclusions (e.g., excluding plants/fungi).

• Performance Metrics:

  • Sensitivity: The ability to correctly retain records that "should be retained."
  • Specificity: The ability to correctly remove records that "should be removed."
  • Overall Accuracy: The proportion of total correct decisions (retained + removed).
  • Error Analysis: Records incorrectly removed (false negatives) were categorized into seven content types (e.g., "uses human participants," "review of human condition") to identify patterns in exclusion errors.

3. Key Contributions

• First Comparative Analysis: This is the first study to empirically compare the performance of 11 different animal-exclusion filters specifically within Ovid Embase.
• Granular Error Analysis: Beyond simple metrics, the study categorizes why relevant records are excluded, linking errors to specific content types (e.g., studies using human data but indexed with animal terms).
• Indexing Insights: The study highlights the impact of Elsevier's indexing policies (manual vs. automatic) on filter performance, noting that conference abstracts and pre-prints (which often undergo automatic indexing) are more prone to incorrect exclusion.
• Practical Guidance: Provides evidence-based recommendations for information specialists on filter selection based on the trade-off between sensitivity (not missing relevant studies) and specificity (removing noise).

4. Results

Performance Metrics (All Datasets Combined):

• Highest Sensitivity: Method 11 was the most sensitive, correctly retaining 90.6% of records that should be kept. Method 10 followed closely at 88.3%.
• Highest Specificity: Method 3 had the highest specificity, correctly removing 71.5% of records that should be excluded.
• Lowest Specificity: Method 11 had the lowest specificity (41.2%), meaning it retained many animal records that should have been removed.
• Overall Accuracy: Method 3 achieved the highest overall accuracy (73%), while Method 11 had the lowest (61.4%) due to its low specificity.

Key Findings on Design:

• Subject Headings vs. Free Text: Filters relying heavily on free-text fields (Title/Abstract) in addition to subject headings (e.g., Method 3) tended to have higher specificity but lower sensitivity. This is likely because free-text terms can be metaphorical or denote exclusion (e.g., "animal studies excluded"), leading to false positives.
• The "Human" vs. "Animal" Terms: Methods 10 and 11 were the only filters that did not explicitly remove records indexed with nonhuman/ or broad animal terms, relying instead on the presence of animal experiment/. This design choice contributed to their superior sensitivity.
• Indexing Impact: Records with automated indexing (conference abstracts, pre-prints, non-English records) were disproportionately affected by incorrect exclusions.

Error Analysis (Incorrectly Removed Records):

• The most frequently excluded category of relevant records was "uses human participants or data" (44.9% of all errors).
• The second most excluded was "review or overview of a condition or intervention affecting humans" (23.6%).
• Methods 10 and 11 deviated from the average error pattern, excluding fewer records in the "review" category compared to other filters.

5. Significance and Conclusion

Significance:
The study demonstrates that there is no single "perfect" filter; the choice depends on the review team's priorities.

• If the priority is maximizing recall (ensuring no relevant human study is missed), Method 11 is the superior choice, despite retaining more animal records.
• If the priority is maximizing precision (removing as many animal records as possible), Method 3 is more effective, though it risks missing relevant human studies.

Recommendations:

1. Contextual Discussion: Information specialists and review teams must discuss the specific scope of the review. For example, reviews on animal-assisted therapies or zoonotic diseases should avoid strict animal filters.
2. Preferred Filter: The authors recommend Method 11 for general use due to its high sensitivity, acknowledging that the cost of retaining some animal records is lower than the risk of missing critical human data.
3. Caution with Publication Types: Filters should be used with caution for conference abstracts, pre-prints, and non-English literature, as these often lack the manual indexing required for accurate Emtree assignment.
4. Future Research: Further validation is needed for specific study types (e.g., economic evaluations, qualitative studies) and for different languages.

In summary, this paper provides the first empirical evidence base for selecting animal-exclusion filters in Embase, moving the field from reliance on unvalidated, ad-hoc strategies to data-driven decision-making.