Skip to content
BranchesTags

Repository files navigation

AISysRev - LLM-based Tool for Title-abstract Screening

GitHub Actions Workflow Status GitHub commit activity GitHub License GitHub Repo stars GitHub contributors

Claude ChatGPT Google Gemini MistralAI Ollama

Important

AISysRev is a Minimum Viable Product (MVP) with core functionality working. Some features are missing and there might be bugs. You can also checkout command line alternative AISysRevCmdLine

Table of contents

Introduction

The AISysRev web-application offers research-backed and AI-based support for Systematic Literature Reviews. Currently, only one step is supported: title–abstract screening. Although the application runs in a web browser, all data is stored locally on your machine. LLMs are accessed through OpenRouter, OpenAI or through a local provider (OpenAI SDK). Data for screening can be imported from two sourced as a CSV from: 1) Scopus and from 2) Github. For Github data see our tool Github Query Tool that queries and collects github repository data.

Scopus data screening works like normal title-abstract screening for academic papers. For GitHub we apply the same logic. Title from GitHub project becomes the title and abstract is the project description + README.md contents. GitHub screening can be useful if one is interested in finding software tools that satisfy inclusion and exclusion rules. Our paper on tool discovery with LLMs.

Features

The application allows you to:

  • Import a CSV file with paper titles and abstracts. You can also use our Demo CSV file
  • Specify include/exclude criteria for paper screening
  • Evaluate papers against the criteria using multiple LLMs
  • Receive LLM evaluations as binary decisions (include/exclude), ordinal ratings (1-7), or inclusion probabilities (0–1)
  • Perform manual evaluation of titles and abstracts alongside LLM evaluations
  • Export evaluation results to CSV for further analysis in Microsoft Excel, Google Sheets, R, Python, etc. All of the above steps are also done for Github repositories.

The application is based on our research papers on this topic. Please consider citing if you use the application 1–3.

Showcase


Figure 1: Main view shows LLM screening tasks.


Figure 2: Manual evaluation view, with LLM evaluations (binary, ordinal, probability) alongside manual review.


Figure 3: Manual evaluation list view, with papers sorted by inclusion probability according to all executed LLMs.

Getting started

Data

The tool has been developed and tested with CSV data exported from Scopus and [Github] with data exported by our custom Github Query Tool. Support for Web of Science can be achieved by editing the columns headers to match the ones from Scopus. The minimum required fields are: Document title, DOI, Abstract, Authors, and Source title.

image

LLMs Access

The application is integrated with OpenRouter, which supports multiple LLMs ranging from very affordable to top-tier models like OpenAI’s ChatGPT, Google’s Gemini, Anthropic’s Claude, Meta's LLama, and Mistral. To use the models, you need to provide an OpenRouter key. You can set spending limits for each key directly on the OpenRouter website.

{585DBE92-5A2F-412E-BEF1-A727015EE872}

LLM screening speed

LLM calls are parallelized, and you should achieve a screening speed exceeding 100 papers per minute when using OpenRouter. The screening speed depends on the model used.

PDF Screening

AISysRev supports full-text PDF screening in addition to title-abstract screening.

Uploading PDFs

PDFs can be attached to papers in two ways:

  • Manual - on a paper's card, click Upload full text and select a PDF file to upload.
  • Bulk upload with Zotero/EndNote XML
    1. Click Download papers missing full text to export a RIS file containing all papers in the project that don't have a PDF attached.
    2. Import the RIS file into Zotero and use Find Full Text to retrieve full-text PDFs.
    3. Export the Zotero collection with the automatically retrieved PDFs in EndNote XML format with Export notes and Export files checked.
    4. In AISysRev, click Import full text (Zotero Export Folder) and select the exported folder. PDFs are attached to papers based on DOI.

Screening modes

When creating a screening task, choose which screening mode to use:

  • Abstract - the paper's title and abstract are given to the LLM.
  • PDF - excerpts from the paper's full-text PDF are given to the LLM.
  • Automatic - uses PDF screening mode for papers with a PDF attached and Abstract for papers without a PDF attached.

See pdf_screening.md for details on how PDF screening mode works.

System and software requirements

Tip

See https://docs.docker.com/desktop/ for Docker installation instructions. Docker Desktop includes Docker Compose, Docker Buildx, Docker Engine and the Docker CLI.

Note

If Docker Desktop did not include the Buildx plugin, see: https://github.com/docker/buildx

Verifying Docker setup and environment

  1. Run docker info to verify you have Docker installed
    • Docker 26.0.0 has been tested as working. For MacOS computers with Colima, Docker version 28.5.1 confirmed to be working.
  2. Run docker buildx version to verify you have Docker Buildx installed. For MacOS computers, Buildx plugin version 0.29.1 confirmed to be working.
  3. Run docker compose version to verify you have Compose installed. For MacOS computers, Compose plugin version 2.40.3 confirmed to be working.
    • Version 2.33.1 has been tested as working, newer versions should also work.
    • Note: Older versions of Compose use docker-compose as the compose command. We don't provide support for legacy Compose versions.

Running the AISysRev application

First, clone the repository to your local computer.

git clone https://github.com/EvoTestOps/AISysRev.git

move to correct directory

cd AISysRev

MacOS, Linux and Windows (WSL)

Start the application in production mode:

make start-prod

If you want to develop the app, run:

make start-dev

Note

The startup of the app may a while due to the download of corresponding Docker images & services, application dependencies and building of the application.

After startup, open the application:

If you ran start-prod, navigate to https://localhost:3000 (the Caddy server's root CA is by default untrusted. You can bypass the browser warning).

If you used make start-dev, navigate to http://localhost:3001

Windows (non-WSL)

If you do not have Windows Subsystem for Linux (WSL), start the application with

./start-prod.bat

Technology

The AISysRev tool is built on bleeding front-end and back-end technology, with industry-demonstrated stability. We follow the best known coding conventions and toolkit.

Front-end

TypeScript, React, Tailwind CSS, Vite, Wouter, Zod, Redux

Back-end

Python, FastAPI, PostgreSQL, SQLAlchemy, Alembic

System design

See Architecture.md

Development requirements

Running in development mode

MacOS, Linux and Windows (WSL)

make start-dev

Windows (non-WSL)

./start-dev.bat

Getting started with development

Open up the client: https://localhost:3001

Note

/api is internally proxied to the backend container, e.g. http://localhost:3001/api/v1/health will be proxied to http://localhost:8080/api/v1/health.

API: https://localhost:3001/api/v1

API docs: https://localhost:3001/docs

Adminer GUI: http://localhost:8081/?pgsql=postgres&username=your_username&db=your_database_dev&ns= password: your_password

Mock data

Mock data is located in data/mock -folder.

Tests

Client

Run in client/:

  • npm test for unit and component tests
  • npm run test:e2e for e2e tests

Server

Run in repository root:

  • make backend-test (./backend-test.bat for Windows non-WSL) for backend tests
  • make backend-test-html (./backend-test-html.bat for Windows non-WSL) for backend tests and HTML coverage report

Makefile Commands

The project includes a Makefile for common development and database operations:

Development

Command Description
make start-dev Start dev containers with live reloading and build on startup (default setup)
make start-test Start test containers and rebuild images (isolated test environment)
make start-prod Start production container and rebuild images

Note: Run all commands from the project root.
Containers are isolated by environment using the Docker Compose -p flag.

Database Migrations (Alembic)

Command Description
make m-create m="Message" Create a new migration with an autogenerated diff (replace Message)
make m-up Apply all pending migrations (upgrade to latest)
make m-hist Show the full migration history with details
make m-current Display the current migration version in the database

Supported LLMs

Currently, we support models provided via Openrouter, OpenAI or via a local provider (OpenAI SDK). The list of supported LLMs is continuously evolving and changing, which is why we don't provide a comprehensive list.

BibTeX Citation

Please use the following BibTeX citation to cite our work:

Conference proceedings

Coming soon.

Preprint

@misc{huotalaAISysRevLLMbasedTool2025,
	title = {{AISysRev} -- {LLM}-based {Tool} for {Title}-abstract {Screening}},
	url = {http://arxiv.org/abs/2510.06708},
	doi = {10.48550/arXiv.2510.06708},
	publisher = {arXiv},
	author = {Huotala, Aleksi and Kuutila, Miikka and Turtio, Olli-Pekka and Mäntylä, Mika},
	month = oct,
	year = {2025},
	note = {arXiv:2510.06708 [cs]},
	keywords = {Computer Science - Artificial Intelligence, Computer Science - Software Engineering}
}

License

MIT

References

[1] Huotala, A., Kuutila, M., Ralph, P., & Mäntylä, M. (2024). The promise and challenges of using llms to accelerate the screening process of systematic reviews. Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 262–271. https://doi.org/10.1145/3661167.3661172

[2] Huotala A, Kuutila M, Mäntylä M. SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews. In Proceedings of the The 19th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) 2025 Oct 218 (pp. 1-12) https://arxiv.org/abs/2507.19027

About

LLM-based Tool for Title-abstract Screening

Topics

llmsystematic-reviewwebapp

Resources

Readme
MIT license
Activity
Custom properties

Stars

21 stars

Watchers

0 watching

Forks

3 forks
Report repository

Used by

Contributors

Languages