AISysRev -- LLM-based Tool for Title-abstract Screening

1. Problem Statement

Conducting Systematic Reviews (SRs) in fields like Software Engineering (SE) is a labor-intensive process, often taking an average of 67 weeks. A significant bottleneck occurs during the title-abstract screening phase, where researchers must manually filter thousands of papers (e.g., ~25,000) against specific inclusion/exclusion criteria.

• Limitations of Current Solutions: Existing open-source tools (e.g., AIReview, ASReview) often lack support for multiple Large Language Model (LLM) providers, have poor documentation, or are no longer maintained. Commercial tools (e.g., Elicit, Rayyan) are closed-source, limiting transparency regarding underlying models, parameters, and prompting techniques.
• LLM Challenges: While LLMs show promise, they suffer from prompt sensitivity, randomness, and false negatives. There is a need for a tool that supports "human-in-the-loop" workflows, allowing humans to verify LLM suggestions rather than relying on fully automated, potentially error-prone outputs.

2. Methodology and Tool Architecture

The authors developed AISysRev, an open-source, containerized web application designed to support rapid, scalable, and transparent title-abstract screening.

System Architecture

• Backend: Built with Python, leveraging asyncio for high-concurrency parallel processing of LLM calls. It uses Pydantic AI to enforce structured output (JSON schemas) and automatically correct malformed responses.
• Frontend: A React.js web interface providing a Graphical User Interface (GUI) for non-technical users.
• Infrastructure:
  • Database: PostgreSQL for persistent data storage.
  • Caching: Redis for managing real-time I/O and scheduling API calls.
  • Storage: MinIO (S3-compatible) for object storage.
  • Deployment: Docker-based, allowing independent scaling of components.
• LLM Integration: Supports a wide range of providers via OpenRouter (unifying access to OpenAI, Google, Anthropic, Meta, Mistral, etc.), direct OpenAI SDK connections, and locally hosted models.

Key Functional Features

1. Input/Setup: Users upload CSV files containing paper metadata (titles, abstracts) and define inclusion/exclusion criteria.
2. Prompting Strategies:
   • Zero-Shot (ZS): LLMs evaluate papers based solely on criteria.
   • Few-Shot (FS): Users provide examples of included/excluded papers to guide the LLM, improving accuracy.
3. Dual-View Screening Interface:
   • Individual Screening: Displays LLM recommendations (binary, ordinal, and probability scores) alongside the paper for human verification.
   • Prioritized Screening: Orders papers by the probability of inclusion (averaged across multiple LLMs). This supports "Rapid Reviews" where humans screen only the top-ranked evidence, potentially stopping after a set number of consecutive exclusions.
4. Parallelization: The system parallelizes LLM calls, achieving speeds of 100–300 papers per minute, depending on the model and host.
5. Output: Results are exported to CSV, including metadata, decisions from multiple LLMs, per-criteria breakdowns, and confidence scores, facilitating further statistical analysis.

3. Key Contributions

• Open-Source Infrastructure: AISysRev fills a gap by providing a transparent, flexible, and actively maintained tool that supports multiple LLM providers and local deployment, unlike closed commercial alternatives.
• Human-in-the-Loop Design: The tool is explicitly designed to augment, not replace, human judgment. It highlights uncertainty through probability scores and offers interfaces for manual review of "boundary" cases.
• Scalability and Speed: By utilizing asynchronous programming and parallel LLM calls, the tool significantly accelerates the screening process compared to manual methods.
• Methodological Transparency: The tool exposes model parameters (temperature, top_p), prompting techniques (ZS/FS), and allows for the comparison of different models on the same dataset.

4. Results and Evaluation

The authors conducted a qualitative trial using a subset of 137 papers from a systematic review on "time pressure in software engineering." Three LLMs were tested: Google Gemini 2.5 Flash, Mistral Small 3.2-23B, and OpenAI GPT-4.1 Mini.

Findings:
The study categorized papers into four distinct groups based on LLM performance and human judgment:

1. Easy Includes (n=6): Papers explicitly studying time pressure in SE. LLMs classified these with very high accuracy.
2. Boundary Includes (n=7): Papers in an SE context with ambiguous evidence of time pressure (e.g., mentioned implicitly). LLMs frequently made errors here, often excluding relevant papers that required full-text review.
3. Boundary Excludes (n=17): Papers related to SE but where time pressure was only a background motivation, not the empirical focus. LLMs frequently misclassified these as relevant, failing to distinguish between context and contribution.
4. Clear Excludes (n=107): Papers outside the SE domain (e.g., medicine, aviation). Both humans and LLMs accurately excluded these.

Conclusion from Trial:
While LLMs excel at filtering "Easy" cases (both includes and excludes), they struggle with Boundary cases. These cases highlight the necessity of human intervention to verify ambiguous results. The tool effectively reduces the volume of papers requiring human attention but does not eliminate the need for human review.

5. Significance

• Practical Impact: AISysRev provides a practical solution for researchers and practitioners to manage the overwhelming volume of literature in systematic reviews, potentially reducing the time and effort required for screening.
• Research Foundation: The tool serves as a platform for studying the efficacy of different LLMs, prompting techniques, and human-in-the-loop strategies in evidence synthesis.
• Future Directions: The authors plan to expand the tool to support full-text screening, data extraction, and search strategy generation. They also intend to benchmark AISysRev against commercial tools and conduct user studies to quantify the reduction in human workload.

In summary, AISysRev represents a significant step toward democratizing and accelerating systematic reviews by combining the speed of modern LLMs with the necessary rigor of human oversight, all within an open, transparent, and scalable technical framework.