Chained Prompting for Better Systematic Review Search Strategies

Technical Summary: Chained Prompting for Better Systematic Review Search Strategies

Problem Statement
Systematic reviews (SRs) are the cornerstone of evidence-based research, yet their validity relies heavily on the rigor of the underlying literature search (LS). Traditional manual search strategies, while methodologically sound, are resource-intensive, time-consuming, and susceptible to subjectivity. Conversely, existing automated and heuristic approaches often struggle with recall (the ability to retrieve all relevant studies) unless heavily supplemented by expert input. Current Large Language Model (LLM) applications in this domain frequently rely on single-prompt approaches that yield broad, imprecise results, or ensemble methods that require generating and aggregating multiple queries to achieve acceptable performance. There is a need for an automated framework that replicates the structured, multi-step logic of manual search design while leveraging LLM capabilities to ensure high recall and reproducibility without extensive human intervention.

Methodology
The authors propose a chained prompt engineering pipeline built upon the GPT-4o-mini model. Unlike single-prompt methods, this framework decomposes the search strategy construction into four sequential, interdependent stages, mirroring the procedural structure of manual librarian-led searches:

1. PICO Extraction: The model decomposes the research objective (or title/abstract) into structured PICO elements (Population, Intervention, Comparison, Outcome). The authors note that their specific prompt design for this step yields more accurate and structured PICO elements than baseline methods.
2. Concept Identification: The extracted PICO components are expanded into precise, domain-specific conceptual categories, clarifying the informational scope.
3. Keyword Generation: For each identified concept, the model generates a comprehensive set of keywords, including synonyms, spelling variants, and related terminologies to maximize retrieval space.
4. Query Construction: The accumulated concepts and keywords are integrated into a final Boolean search strategy. The model applies logical connectors (AND, OR) and proximity syntax consistent with systematic review guidelines to form the executable query.

The pipeline was evaluated using a subset of the LEADSInstruct dataset, comprising 81 high-quality PubMed-indexed systematic reviews published between 2012 and 2016. The evaluation focused primarily on recall, defined as the proportion of target articles (those included in the original review) successfully retrieved by the generated strategy.

Key Results
The evaluation demonstrated that the chained prompting approach significantly outperforms existing methods:

• Baseline Comparison: A direct prompting approach using GPT-4o achieved an average recall of only 0.10. The existing LEADS model achieved 0.24.
• Chained Pipeline Performance:
  • When the pipeline started with PICO elements provided by the LEADS authors, the average recall was 0.62 (rising to 0.80 when excluding strategies with recall <0.2).
  • When using the full pipeline (including the authors' own prompt for reformulating structured objectives from titles/abstracts before PICO extraction), the average recall reached 0.87 across all 81 reviews.
  • When excluding strategies with recall <0.2, the full pipeline achieved an average recall of 0.90.
• Comparative Gains: The full pipeline represents a 362% relative improvement over the LEADS model (0.24 to 0.87) and an 8% gain over the LEADS+ensemble method (0.82 to 0.90) under comparable filtering conditions.
• Distribution: The full pipeline resulted in 62 out of 81 strategies achieving perfect recall (1.0), with only 3 strategies resulting in zero recall.

Error Analysis
An analysis of strategies with low recall (<0.5) identified three primary failure modes:

1. Terminology Mismatch: Discrepancies between standard medical terminology and the specific phrasing used in articles (e.g., "workers" vs. "adults").
2. Dataset Noise: Inclusion of non-systematic reviews in the evaluation set.
3. Objective Formulation Errors: Insufficient scope capture when deriving objectives solely from titles and abstracts.
   Crucially, the authors found that manually replacing poorly formulated objectives with those derived from systematic review protocols restored recall to 1.0 in these cases, highlighting that the pipeline's performance is highly dependent on the precision of the input objective.

Significance and Claims
The paper claims that this work demonstrates the capacity of LLM-based pipelines to generate transparent, reproducible, and high-performing search strategies. By adhering to a chained prompting structure, the system achieves superior recall compared to both direct prompting and ensemble-based methods, without requiring the generation of multiple queries or extensive human intervention.

The authors emphasize that the quality of the search strategy is fundamentally tied to the quality of the PICO extraction and objective specification. Their results suggest that improved prompt engineering for these initial steps creates a stronger foundation for downstream keyword expansion and query synthesis. The framework is presented as a scalable instrument for supporting evidence synthesis, potentially reducing the reliance on manual expertise while maintaining the methodological rigor required for high-quality systematic reviews. Future work is noted to focus on extending the approach to other bibliographic databases and refining query reformulation strategies.