SLRMentor: An LLM-Based Tool Supporting Learning of SLR in Software Engineering

Technical Summary: SLRMentor

Problem Statement
Conducting Systematic Literature Reviews (SLRs) is a foundational activity in Evidence-Based Software Engineering (EBSE), yet it presents significant challenges for novice researchers. The planning phase—specifically the formulation of research questions, construction of search strings, and definition of inclusion/exclusion criteria—is identified as critical to review quality but is often time-consuming and error-prone due to terminology variations and complex database syntax. While existing AI-based tools have been explored to improve efficiency in study selection and keyword suggestion, they often lack support for the methodological understanding and learning of the SLR process itself. Current educational approaches struggle to make the underlying methodological reasoning explicit, often leading students to focus on producing artifacts rather than engaging with the decision-making logic required for rigorous reviews.

Methodology
This study employs the Design Science research method to develop and evaluate a software artifact named SLRMentor. The tool is a conversational assistant built upon Large Language Models (LLMs) (specifically OpenAI and Google Gemini) designed to support the planning activities of SLRs.

• Architecture & Mechanism: SLRMentor utilizes Retrieval-Augmented Generation (RAG) to ground its responses in curated SLR guidelines and methodological documentation (e.g., [10, 12, 14]). This design choice ensures that generated outputs are consistent with established practices and improves transparency, preventing the model from relying solely on uncontextualized generation.
• Functional Components: The tool is structured around three learning-oriented conversational modules:
  1. Mentor Chat: Provides general guidance on SLR stages, methodological principles, and bias sources to support conceptual understanding.
  2. Search String Chat: Assists in translating research goals into structured search strings, explaining keyword selection, synonyms, and Boolean operators.
  3. Criteria Chat: Supports the derivation of inclusion and exclusion criteria from study goals, explaining how these criteria operationalize scope and relevance.
• Validation: A pilot validation was conducted in a graduate software engineering course involving eight students (four doctoral, four master's), all of whom were novice reviewers. Four participants voluntarily engaged with the tool after completing their manual course assignments. The evaluation was exploratory and formative, utilizing a structured questionnaire with Likert-scale items and open-ended questions to assess clarity, consistency with known practices, and perceived support for learning and reflection.

Key Contributions
The primary contribution of this work is the SLRMentor artifact itself, a conversational interface designed not to automate review tasks, but to act as a scaffold for learning methodological reasoning. Key technical features include:

• Guideline-Grounded Explanations: Unlike generic chatbots, SLRMentor explicitly ties its reasoning to established SLR guidelines via RAG, ensuring educational content aligns with EBSE standards.
• Iterative Refinement Support: The system allows users to revise goals and artifacts across multiple interactions, mirroring the iterative nature of protocol development in practice.
• Focus on Reasoning over Automation: The tool is engineered to provide "why" and "how" explanations for planning decisions (e.g., why a specific term is included in a search string) rather than simply outputting a final, unexamined artifact.

Results
The pilot validation yielded qualitative and quantitative insights from four participants:

• Perception as a Learning Tool: Participants consistently viewed SLRMentor as a "learning-oriented support" rather than a tool for producing definitive artifacts. It was described as a "beginner-friendly scaffold" that helped structure complex planning tasks.
• Clarity and Alignment: The Mentor Chat received high marks for clarity and alignment with established practices. Participants noted it helped "parsing through complicated systematic literature review requirements."
• Search String Construction: The Search String Chat was praised for being educational rather than prescriptive. Participants felt it effectively explained the reasoning behind term and operator selection, though they acknowledged the need for further student refinement.
• Criteria Definition: Results for the Criteria Chat were more mixed. While some participants found it useful for illustrating common categories and deriving initial criteria, others noted variability in the comparability between tool-generated criteria and their manually defined counterparts.
• Active Judgment Required: A recurring theme was that the tool supports learning but does not replace the need for active methodological judgment. Participants emphasized that the tool aids in understanding and reflection but requires the researcher to make final decisions.

Significance and Claims
The paper claims that SLRMentor addresses a gap in current educational tools by embedding conversational, guideline-grounded assistance directly into SLR planning activities. Its significance lies in:

1. Lowering Barriers: It reduces initial barriers for novice researchers by providing on-demand explanations and structured guidance, making the SLR process more accessible.
2. Explicit Reasoning: It makes methodological reasoning explicit, allowing learners to compare tool-generated guidance with their own decisions, thereby fostering critical reflection.
3. Complementary Role: The authors modestly position the tool as a complement to, not a substitute for, established teaching approaches and human supervision. It is designed to reinforce methodological concepts and structure practice while maintaining the necessity of researcher responsibility.

The authors conclude that while the pilot suggests the tool is effective for learning and reflection, larger-scale empirical studies are required to assess long-term learning outcomes and transferability across diverse student populations. Future work aims to expand the retrieval corpus and extend support to later SLR stages, such as study selection and data synthesis.