Development of a System of Indicators for Assessing the Quality of Search Strategies in Systematic Reviews: A Methodological Document Review

In the world of scientific research, a systematic review is a powerful tool. Imagine a team of scientists who want to know the truth about a specific medical treatment or a social issue. Instead of conducting a single new experiment, they gather every single study that has ever been done on that topic. They read them all, check their quality, and combine their findings to create one clear, reliable answer. This process is the gold standard for decision-making in fields like medicine and public health because it cuts through the noise of individual studies to reveal the bigger picture. However, the entire process rests on a single, fragile foundation: the search strategy. This is the specific set of instructions the researchers use to find those studies. If the search is too narrow, they miss important evidence. If it is too messy or poorly written, they might find irrelevant junk or fail to find the right papers at all. When the search fails, the final conclusion of the review is flawed, potentially leading doctors and policymakers to make decisions based on incomplete or biased information.

Despite how critical this step is, the quality of these search strategies has often been overlooked. Researchers have long known that they need to be thorough, but there was no standard way to measure whether a search was actually done well. A new study by Alireza Jahani and Keyvan Salehi from the University of Tehran aims to fix this gap. The researchers set out to build a practical system of indicators—a checklist of specific, measurable signs—that anyone can use to judge the quality of a search strategy in a systematic review. Their goal was to move beyond vague advice and provide a concrete framework that researchers, reviewers, and journal editors could use to ensure these studies are built on solid ground.

To create this system, the authors conducted a two-phase investigation. First, they performed a deep dive into existing knowledge. They gathered and read thirty-five different documents, including books, international guidelines, and previous studies that discussed how to write and evaluate search strategies. From this vast collection of text, they extracted 128 initial ideas about what makes a search good. They then carefully organized these ideas, merging those that meant the same thing and removing duplicates. Through this process of refinement, they distilled the 128 ideas down to a final set of 39 specific indicators, which they grouped into seven main categories. These categories cover the entire life cycle of a search, from the initial planning to the final report.

The first category, which the authors call "Selection Comprehensiveness," focuses on where the search takes place. The study found that a high-quality search must look in many different places, including major specialized databases and multidisciplinary collections. The researchers explicitly noted that relying on a single general search engine is not enough. They found that using only one source, or depending too heavily on a general web search, increases the risk of missing important studies and introduces bias. A good search strategy must be broad enough to catch relevant evidence from diverse scientific sources while avoiding the temptation to use a single, easy tool as the only source of truth.

The second category deals with the "Search Strategy Structure Design." This is about the logic behind the search. The study emphasizes that a strong search begins with a clear, well-defined question, often organized around specific elements like the population being studied and the outcomes being measured. The researchers found that effective strategies group key concepts together and find every possible way to say those concepts, including using both technical medical terms and everyday language. They also highlighted that this process should be iterative, meaning the researchers should test their search, see what comes back, and then refine their questions and terms based on what they find. This cycle of testing and improving prevents the search from being too narrow or missing studies that use different words to describe the same idea.

The third category, "Syntactic Formulation," addresses the technical rules of the search. Just as a sentence must follow grammar rules to make sense, a computer search must follow specific syntax rules to work correctly. The study points out that every database has its own unique set of rules for how to combine words, use symbols for "and" or "or," and handle special characters. The researchers found that errors in these technical details are common and can cause a search to miss relevant documents or return thousands of irrelevant ones. To avoid this, high-quality searches use the correct symbols and often employ tools to translate the search from one database to another, ensuring that the logic remains consistent across different platforms.

Beyond the main search, the fourth category, "Supplementary Searching," looks for evidence that might be hiding in the shadows. The study identifies that many important findings, particularly those that did not show a dramatic effect, are never published in standard journals. To find these, a high-quality review must go further than just searching databases. It should involve checking the reference lists of articles that were already found, looking through conference papers, searching for unpublished clinical trials, and even contacting experts in the field. The authors argue that skipping these extra steps leaves the review vulnerable to "publication bias," where the final result only reflects the studies that were lucky enough to get published, rather than the full reality of the evidence.

The fifth category is "Validation." This step is about checking the work before it is finished. The study found that the best practice involves having an information specialist, a professional trained in finding information, review the search strategy. This expert can spot errors that the original researchers might miss. Additionally, the search should be tested against a known list of important articles to see if it successfully finds them. If the search fails to find these known key studies, it is not sensitive enough and needs to be adjusted. This validation process acts as a safety net, ensuring that the search is capable of retrieving the critical evidence needed for the review.

The sixth category, "Reporting Transparency," focuses on how the search is documented. The study stresses that for a review to be trustworthy, the authors must write down exactly what they did. This includes listing every database used, the exact date the search was performed, and the full list of search terms. They must also explain why they included or excluded certain studies and how they removed duplicate results. Without this level of detail, other scientists cannot repeat the search to verify the results. The researchers found that following established reporting checklists helps ensure that no critical step is left out and that the process is clear to everyone reading the study.

Finally, the seventh category, "Process Updating," ensures the review stays current until it is published. Because new studies are published every day, a search that was perfect a year ago might be outdated by the time the review is released. The study recommends that the search be run again shortly before publication to catch any new evidence. It also suggests that the process of selecting studies should be done independently by at least two researchers to reduce individual errors, and that the team should use software to manage the large number of articles they find.

After developing this system of 39 indicators, the researchers tested it. They applied their new checklist to twenty systematic reviews that had been published in scientific journals. They evaluated each review to see if it met the indicators they had identified. The results showed that while some reviews were strong, many had significant gaps, particularly in the technical areas of syntax and the crucial step of having a specialist validate the search. The authors concluded that their new system provides a practical and necessary tool for improving the quality of systematic reviews. By using these indicators, researchers can reduce bias, make their findings more reproducible, and ensure that the evidence they gather truly reflects the world as it is. The study suggests that journals should require authors to use this system as a standard part of the review process, ensuring that the foundation of these critical scientific summaries is as strong as the conclusions they draw.