Ranked (In)direct Citation Searching in Systematic Reviews: A methodological case study

This methodological case study introduces a framework and open-source tool for Ranked (In)direct Citation Searching (RICS) to systematically compare it with Unranked Direct Citation Searching (UDCS) in systematic reviews, demonstrating that RICS offers comparable or improved precision while identifying additional relevant literature.

The Gist

Imagine you are a detective trying to solve a complex case (a Systematic Review). Your first step is to search through massive libraries of books and articles (databases) using specific keywords to find clues (studies) that might help solve the case. You find a list of "seed" clues that look promising.

But, you know that searching by keywords alone might miss some hidden gems. So, you decide to use a second strategy: Citation Searching. This is like asking, "Who wrote this book?" (looking backward) and "Who has read and referenced this book?" (looking forward).

This paper introduces a new, smarter way to do this detective work, comparing two methods: the old standard way and a new, ranked way.

The Two Detective Methods

1. The Old Way: "Unranked Direct Citation Searching" (UDCS)
Think of this as asking your seed clues: "Who wrote you, and who read you?"

• You get a list of everyone who directly cited your seed clues.
• It's a straightforward, "blunt" list. You get everyone who has a direct connection, but you don't know which ones are the most important.
• The Problem: It can be a bit messy. You might get a lot of irrelevant books just because they happened to be on the same shelf.

2. The New Way: "Ranked (In)direct Citation Searching" (RICS)
This is the paper's main innovation. Imagine asking your seed clues: "Who wrote you? Who read you? AND who else did you both read? Who else did you both write about?"

• Direct Links: Just like the old way (cited/citing).
• Indirect Links (The Secret Sauce): It also finds "co-citations" (books that were read by the same people as your seed) and "co-citing" (books that read the same sources as your seed).
• The Ranking: Because this method finds thousands of potential connections (not just the direct ones), the authors created a tool called Co*Citation Network. This tool acts like a smart filter. It scores every single book based on how many different ways it connects to your seed clues.
• The Cut-off: It then says, "Okay, we only need to look at the top 100 most connected books to keep our workload manageable." This ensures the new method doesn't overwhelm the detective with too much paperwork.

The Experiment: Putting Them to the Test

The researchers built a free, open-source tool (the Co*Citation Network) to automate this process. They wanted to see if the "Smart Filter" (RICS) was better than the "Blunt List" (UDCS).

They tested this in two ways:

1. Looking Backward (Retrospective): They took three past detective cases they had already solved and ran both methods on them to see what they would have found.
2. Looking Forward (Prospective Case Study): They ran a brand new investigation on "young-onset dementia" and used both methods simultaneously to see which one found better clues.

What They Found

• More Overlap with the "Good Stuff": In the past cases, the "Smart Filter" (RICS) found a list of books that looked much more similar to the high-quality books the detectives had already found in the main library search. This suggests RICS is better at finding relevant material that fits the specific case.
• The "Need to Read" Score: In the new dementia case, they had to read through the titles and abstracts to find the winners.
  • With the old method (UDCS), they had to read about 57 articles to find 1 winner.
  • With the new method (RICS), they had to read about 48 articles to find 1 winner.
  • Translation: The new method was slightly more efficient; you wasted less time reading irrelevant papers.
• The Surprise: Interestingly, the old method (UDCS) found one final winner that the new method (RICS) missed. Why? Because that specific winner wasn't "connected" enough to the seed clues to make the top 100 list in the new method. If they had lowered the cut-off to include it, they would have had to read over 10,000 articles, which is impossible.

The Bottom Line

The paper claims that this new Ranked (In)direct Citation Searching (RICS) method is a promising tool. It seems to find clues that are more relevant to the specific case and requires slightly less reading to find the "winners."

However, the authors are careful to say: "We don't know for sure yet if this is the perfect method."

• In their test, the old method found one unique winner that the new method missed.
• They haven't proven that RICS is always better.

The Goal: The main point of this paper isn't to declare a winner today. It's to build the tool (Co*Citation Network) and the workflow so that detective teams all over the world can use them together. By sharing their data, they hope to run a massive, global comparison to finally answer the question: "Is the Smart Filter better than the Blunt List?"

In short: They built a new, smarter magnifying glass for finding research clues. Early tests look good, but they need more detectives to use it before they can say it's the new standard.

Technical Summary

Technical Summary: Ranked (In)direct Citation Searching in Systematic Reviews

Problem Statement
Systematic Review (SR) authors frequently employ citation searching (CS) as a supplementary technique to primary database searches to ensure comprehensive evidence retrieval. The current standard workflow, Unranked Direct CS (UDCS), identifies literature directly citing or cited by "seed references" (eligible records found in the primary search). While manageable in volume, UDCS may miss relevant literature. Researchers have proposed Ranked (In)direct CS (RICS), which expands the search to include co-cited and co-citing literature (indirect citations) and applies a relevance ranking and cut-off procedure. However, RICS workflows remain non-standardized, tedious to execute manually, and their comparative effectiveness and efficiency against UDCS in a supplementary search context are unclear. Existing tools for automated indirect CS are scarce or unsupported, and no large-scale comparative studies have evaluated RICS as a supplementary method alongside UDCS.

Methodology
The authors developed a methodological framework and a software tool, Co*Citation Network, to facilitate the parallel execution and comparison of UDCS and RICS.

• Tool Development: The Co*Citation Network is an open-source application built on the OpenAlex citation index. The frontend integrates with the existing Local Citation Network for UDCS. To handle the high volume of indirect citations (co-cited/co-citing), the backend utilizes local processing of OpenAlex snapshots rather than direct API calls. It implements a relevance ranking algorithm based on Belter's method, summing citation relationships (cited, citing, co-cited, co-citing) between a candidate paper and the seed references.
• Rank Cut-off Strategy: To ensure RICS remains feasible for SR screening loads, the system automatically determines a relevance rank cut-off. This cut-off is set to return a number of RICS results that is approximately equal to, but slightly greater than, the volume of UDCS results. This empirical approach aims to balance the retrieval of relevant indirect citations with the practical constraints of screening.
• Study Design: The study employed a mixed approach:
  1. Retrospective Analysis: Parallel UDCS and RICS searches were conducted on three previously completed SRs (Neuhaus et al., Baroutsou et al., Reid et al.) using their included records as seed references. Results were deduplicated against the original primary database searches.
  2. Prospective Case Study: A new SR on non-pharmacological interventions for young-onset dementia was conducted. Seed references (11 included records + 6 related protocols/reviews) were used to generate parallel UDCS and RICS results. These were subjected to independent title/abstract and full-text screening by two reviewers.

Key Contributions

• Co*Citation Network Application: The introduction of the first open-source tool capable of automating parallel UDCS and RICS searches based on the OpenAlex index. It provides a flexible, empirically determined relevance rank cut-off.
• Standardized Workflow: The establishment of a replicable workflow for comparing supplementary UDCS and RICS, including deduplication against primary search results and eligibility screening.
• Research Infrastructure: The creation of a framework for an international research consortium ("RICS crowd study") to gather data from multiple SR teams to statistically evaluate the effectiveness of RICS versus UDCS.

Results

• Retrospective Analysis: In all three retrospective SRs, RICS results showed a higher overlap with the primary database search results compared to UDCS (e.g., 40.9% vs. 22.6% in the Neuhaus study). Consequently, after deduplication, the number of unique records requiring screening was lower for RICS in two of the three cases.
• Prospective Case Study:
  • Volume: UDCS yielded 917 results; RICS (with an automatic rank cut-off of 10) yielded 1,151 results.
  • Overlap: RICS had a higher overlap with the primary search (12.9%) than UDCS (7.5%).
  • Screening Efficiency: At the title/abstract stage, RICS demonstrated a slightly higher precision (2.1%) and a lower Number Needed to Read (NNR) of 48, compared to UDCS precision of 1.8% and NNR of 57.
  • Inclusion: Two additional eligible articles were identified through supplementary searching. One was found by both UDCS and RICS; the other was found exclusively by UDCS. Notably, the UDCS-exclusive article had a RICS rank of 6, falling below the automated cut-off of 10. Lowering the cut-off to include this article would have generated over 10,000 results, rendering the workflow unfeasible.

Significance and Claims
The paper claims to provide the first framework for the prospective international comparison of supplementary UDCS and RICS. The results suggest that RICS results may be more relevant (i.e., more similar to seed references and primary search results) than UDCS results, potentially offering higher precision. However, the authors explicitly state that the current case study lacks the statistical power to definitively determine whether RICS is superior to UDCS. The study identifies a potential trade-off: while RICS may offer higher precision, a strict rank cut-off (necessary for feasibility) might miss relevant literature that UDCS captures. The authors conclude that the proposed workflow and the open-source tool enable the formation of a research consortium to generate conclusive evidence on whether RICS should be adopted as a standard supplementary search technique in systematic reviews.