ABSTRACT

Objective:

To describe the main functions of the “Systematic Review Support” web-based system for removing duplicate articles and aiding eligibility analysis during the process of conducting systematic review studies.

Methods:

The system was developed based on the incremental build model using the Agile methodology. The software is proprietary source code and was published on a proprietary platform. The architecture of the production environment allows the infrastructure used to increase or decrease according to demand. The system functions are presented with insertion of screenshots of the interfaces of the version for personal computers during the simulation of a systematic review.

Results:

After importing the files containing the abstracts retrieved from the Pubmed, Embase, and Web of Science databases, the system identifies and removes duplicates for later reading and analysis of title and abstract, a stage which can be performed by one or more reviewers independently. After unblinding of reviewers, the decisions on the eligibility of the studies are compared automatically to help the researchers reach a consensus on any disagreements. Results can be filtered and a PDF produced containing the eligible studies.

Conclusion:

Version 1.0 of the system is available on the web (sysrev.azurewebsites.net) to assist researchers in the initial stages of systematic reviews.

Keywords:
Databases, bibliographic; Cloud computing; Systematic review; Software; Information technology

INTRODUCTION

The number of health-related scientific articles published in databases is constantly increasing. Reading and critically evaluating this production represents a challenge for researchers, so that systematic review studies are essential tools for scientific updating¹1 Egger M, Smith GD, Altman DG. Systematic reviews in health care: meta-analysis in context. 2nd ed. London: BMJ Publishing Group; 2001..

Unlike traditional literature reviews, systematic and scoped reviews require protocols with the criteria to be followed to guarantee the reliability of the results and the reproducibility of the process²2 Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol 2018; 18(1): 143. https://doi.org/10.1186/s12874-018-0611-x
https://doi.org/10.1186/s12874-018-0611-... .

The search for studies in different databases generates a considerable number of identical articles, which are called duplicates. Their elimination is usually labor intensive and requires time and supporting computational tools³3 Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Marshall C, et al. Technical supplement to chapter 4: searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston MS, Li T, Page MJ, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions Version 6.2. [Internet]. 2021 [cited on May 3, 2022]. Available at: https://training.cochrane.org/handbook/current/chapter-04-technical-supplement-searching-and-selecting-studies#section-4-3/
https://training.cochrane.org/handbook/c... . A multidisciplinary team with enough members is also necessary, especially in the eligibility phase, due to the demand for time. At least two reviewers are required for initial study selection, quality analysis, and data extraction from scientific publications to ensure comprehensiveness of decisions and avoid bias¹1 Egger M, Smith GD, Altman DG. Systematic reviews in health care: meta-analysis in context. 2nd ed. London: BMJ Publishing Group; 2001.,⁴4 Muka T, Glisic M, Milic J, Verhoog S, Bohlius J, Bramer W, et al. A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research. Eur J Epidemiol 2020; 35(1): 49-60. https://doi.org/10.1007/s10654-019-00576-5
https://doi.org/10.1007/s10654-019-00576... .

The use of automation or semi-automated processes for the selection of articles in systematic reviews is a reason for study and debate at the International Collaboration for Automation of Systematic Reviews (ICASR), favoring the construction and evaluation of computational tools that contribute to the quality of studies of review of evidence, as they reduce selection bias, time, and workload⁵5 O’Connor AM, Glasziou P, Taylor M, Thomas J, Spijker R, Wolfe MS. A focus on cross-purpose tools, automated recognition of study design in multiple disciplines, and evaluation of automation tools: a summary of significant discussions at the fourth meeting of the International Collaboration for Automation of Systematic Reviews (ICASR). Syst Rev 2020; 9(1): 100. https://doi.org/10.1186/s13643-020-01351-4
https://doi.org/10.1186/s13643-020-01351... .

On The Systematic Review Toolbox platform, free and paid software can be found that help in one or more stages of the process of different types of reviews, such as the identification of duplicate articles and the selection of studies. Among them are the Brazilian computational tools Parsifal and StArt⁶6 Marshall C, Sutton A, O’Keefe H, Johnson E, eds. The systematic review toolbox [Internet]. 2022 [cited on Jul. 28, 2022]. Available at: http://www.systematicreviewtools.com/
http://www.systematicreviewtools.com/... .

Despite the existence of different software, in practice, it is observed that students use non-specific programs for this purpose, such as text editors, spreadsheets and reference managers, which require manual work to complete the necessary tasks. Thus, the need for methodological and technological support that contributes to the rapid dissemination of knowledge for application in the health area is identified⁷7 Miranda ES, Figueiró AC, Potvin L. Are public health researchers in Brazil ready and supported to do knowledge translation? Cad Saude Publica 2020; 36(4): e00003120. https://doi.org/10.1590/0102-311X00003120
https://doi.org/10.1590/0102-311X0000312... .

Thus, the objective of this study was to describe the main functionalities of the “Support for Systematic Review” system in the identification and exclusion of duplicate articles and in assisting in the analysis of eligibility during the conduction of systematic literature reviews.

METHODS

This methodological research describes the main functionalities of the computer system called “Support for Systematic Review” (Apoio à Revisão Sistemática) (version 1.0). This web system was created to assist researchers in the initial phases of a systematic review project:

1. Organization of abstracts, with identification and exclusion of duplicate studies; and

2. Eligibility analysis, which includes the definition of eligibility criteria, selection of studies by reading titles and abstracts, according to established criteria, and comparing decisions on the eligibility of studies to resolve disagreements.

An abstract is the initial part of a scientific article, usually presented in a paragraph that highlights the main elements of the research (background, objectives, methods, results, and conclusion). It is preceded by its reference (authorship, title, journal, date of publication) and indexed in the databases to facilitate search⁸8 Universidade de São Paulo. Faculdade de Saúde Pública. Guia de apresentação de teses. 2ᵃ ed. atual. São Paulo: Faculdade de Saúde Pública da USP; 2017 [Internet]. [cited on May 03, 2022]. Available at: http://www.biblioteca.fsp.usp.br/guia
http://www.biblioteca.fsp.usp.br/guia... . In the developed system, the term “summary” was considered to include its reference.

For the construction of the system, the following items were considered:

• Target audience: professionals and researchers, undergraduate and graduate students from programs in the health area, whose objective is to carry out a literature review.

• Objective: to provide an efficient alternative to manual work, containing specific functions to optimize researchers’ time in the initial stages of systematic review studies and ensure the production of reliable results.

• Infrastructure: access to the system via computer, tablet or smartphone, considering the different types of hardware, screen sizes, and operating systems available in the market. We opted for development on a web platform to access the system on a device with an updated browser connected to the internet.

• Language: Portuguese.

• Ensuring the security of the data stored in the system.

• Priority to ease of use and the construction of a friendly and responsive interface (adaptable to different screen sizes).

An incremental process model⁹9 Pressman RS, Maxim BR. Engenharia de software: uma abordagem profissional. 8ᵃ ed. Porto Alegre: AMGH; 2016. was used, which makes it possible to add new functionalities to each version of the system. Thus, the functionalities referring to the other phases of a systematic review may be implemented in later versions. The Agile¹⁰10 Beck K, Beedle M, van Bennekum A, Cockburn A, Cunningham W, Fowler M, et al. Manifesto para desenvolvimento ágil de software [Internet]. [cited on May 03, 2022]. Available at: https://agilemanifesto.org/iso/ptbr/manifesto.html
https://agilemanifesto.org/iso/ptbr/mani... methodology was adopted to deliver a version with new features, after each development and testing period. This methodology has the following principles: to satisfy the user through the continuous and early delivery of value-added software; deliver working software in a few weeks, with the shortest time preferred; work together (users and developers) on the entire project; talk face-to-face with the development team to convey information more efficiently and effectively; keep continuous attention on technical excellence and good design to increase agility; seek simplicity as an essential element.

Kanban¹¹11 Kniberg H, Skarin M. Kanban and Scrum making the most of both [Internet]. 2010 [cited on May 3, 2022]. Available at: https://www.infoq.com/minibooks/kanban-scrum-minibook/
https://www.infoq.com/minibooks/kanban-s... was adopted to monitor the activities under development; thus, the functionalities and requirements defined for the delivery were transformed into cards classified according to To Do, Doing, and Done status. The cards were arranged on a panel according to the version to be delivered. The system was developed with closed source code and published on a proprietary platform (Azure DevOps, by Microsoft)¹²12 Microsoft. Azure devops documentation [Internet]. 2015 [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/azure/devops/
https://docs.microsoft.com/en-us/azure/d... to control activities, display cards, store source code versions and publish versions on the production server.

Functional requirements were established based on theoretical references¹1 Egger M, Smith GD, Altman DG. Systematic reviews in health care: meta-analysis in context. 2nd ed. London: BMJ Publishing Group; 2001.,¹³13 Centre for Reviews and Dissemination. Systematic reviews. CRD's guidance for undertaking reviews in health care. York: University of York; 2009 [Internet]. [cited on May 3, 2022]. Available at: https://www.york.ac.uk/crd/guidance/
https://www.york.ac.uk/crd/guidance/... ,¹⁴14 Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. 2021 [cited on May 03, 2022]. Available at: http://training.cochrane.org/handbook/
http://training.cochrane.org/handbook/... and on the experience acquired in the elaboration of review studies. Technical requirements were defined in partnership with a systems analyst. System development activities were organized by an information technology (IT) professional, voluntarily and independently, according to the functionalities described in Chart 1.

For the development of the system, a set of three software components was defined for its constitution:

1. User interface (UI), responsible for displaying information and entering data by users;

2. Application programming interface (API), responsible for communicating data with the user interface and for executing the processing; and

3. Database (DB), responsible for storing and retrieving data from each revision.

The concepts and technologies used in each component are detailed below.

User interface

For the construction of the UI, the concept of Single Page Application¹⁵15 Santamaria JMA. The single page interface manifesto [Internet]. 2015 [cited on May 3, 2022]. Available at: http://itsnat.sourceforge.net/php/spim/spi_manifesto_en.php/
http://itsnat.sourceforge.net/php/spim/s... was used, in which there is a single HTML page that alternates its visible elements according to user interaction and data update. This approach allows for a more dynamic user experience as only the elements where data has been updated are redrawn. Another advantage of this concept is that part of the processing, referring to the rules for presenting information and entering data by the user, is performed on the device used, allowing more users to access the system without this processing being added to the API server.

The programming language used for the development of the UI was TypeScript¹⁶16 TypeScript, TypeScript documentation [Internet]. [cited on May 3, 2022]. Available at: https://www.typescriptlang.org/docs/
https://www.typescriptlang.org/docs/... , using the Angular framework, version 9¹⁷17 Introduction to the angular docs [Internet]. [cited on May 3, 2022]. Available at: https://angular.io/docs/
https://angular.io/docs/... . For the page layout, the admin model was used, which has: a side menu, fixed header and the rest of the page reserved for displaying information. The tool used to build the page layout was the Bootstrap framework¹⁸18 Introduction [Internet]. [cited on May 3, 2022]. Available at: https://getbootstrap.com/docs/4.1/getting-started/introduction/
https://getbootstrap.com/docs/4.1/gettin... .

To adapt the UI to different screen sizes, we used the responsive development of pages through Media Queries¹⁹19 Marcotte E. Responsive web design [Internet]. 2010 [cited on May 3, 2022]. Available at: https://alistapart.com/article/responsive-web-design/
https://alistapart.com/article/responsiv... defined in the Cascading Style Sheet (CSS)²⁰20 W3C. Cascading style sheets. What is CSS [Internet]. [cited on Aug. 30, 2021]. Available at: https://www.w3.org/Style/CSS/
https://www.w3.org/Style/CSS/... .

Application programming interface

The API was developed using the C#²¹21 Microsoft. C# documentation [Internet]. [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/dotnet/csharp/
https://docs.microsoft.com/en-us/dotnet/... programming language, with the .Net Core framework, version 3.0²²22 Microsoft. What is .NET? Introduction and overview [Internet]. [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/dotnet/core/introduction/
https://docs.microsoft.com/en-us/dotnet/... . The use of this framework allows hosting the API on Windows or Linux servers, allowing greater flexibility in defining the infrastructure.

For communication with the UI, the Asp. Net Core Web API framework, version 3.0²³23 Microsoft. Visão geral do ASP.NET core [Internet]. [cited on May 3, 2022]. Available at: https://docs.microsoft.com/pt-br/aspnet/core/introduction-to-aspnet-core?view=aspnetcore-3.0/
https://docs.microsoft.com/pt-br/aspnet/... , was used, which allows communication of the Representational State Transfer (REST) type²⁴24 Fielding RT. Representational State Transfer (REST). In: Fielding RT. Architectural styles and the design of network-based software architectures [dissertation] Irvine: University of California; 2000. [cited on May 3, 2022]. Available at: https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm
https://www.ics.uci.edu/~fielding/pubs/d... , in which most of the information is not stored in the server's memory. In this way, a greater number of users can use the system without increasing memory consumption to keep their information.

The UI access authorization to use the API is done through Json Web Token (JWT)²⁵25 JWT. Introduction to JSON web tokens [Internet]. [cited on May 3, 2022]. Available at: https://jwt.io/introduction/
https://jwt.io/introduction/... , in which the user's access information is transmitted through an encrypted token, scheduled to expire after a certain time interval.

Communication with the database was developed using the concept of Object Relational Mapping (ORM)²⁶26 Hibernate. What is object/relational mapping? [Internet]. [cited on May 3, 2022]. Available at: https://hibernate.org/orm/what-is-an-orm/
https://hibernate.org/orm/what-is-an-orm... , whereby queries and data updates are generated by the ORM framework, reducing development time and allowing the database management system of data to be changed without having to develop new queries and specific updates for a new management system.

Database

The database chosen for storing revision and user data was Microsoft SQL Server²⁷27 Microsoft. SQL server technical documentation [Internet]. [cited on May 03, 2022]. Available at: https://docs.microsoft.com/en-us/sql/sql-server/
https://docs.microsoft.com/en-us/sql/sql... , as its performance supports large volumes of data. The data model chosen was a hybrid of the relational model, in which data is separated into related tables, with a non-relational model, in which data is stored in JavaScript Object Notation (JSON) objects²⁸28 Introducing JSON [Internet]. [cited on Apr. 4, 2021]. Available at: https://www.json.org/json-en.html
https://www.json.org/json-en.html... . Tables were created to store data from users, reviews, their abstracts and groups, authors, eligibility criteria, comments, and reviewers.

Production environment

The production environment chosen for publishing the software components necessary for the system to work was Azure²⁹29 Microsoft. Azure documentation [Internet]. [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/azure/
https://docs.microsoft.com/en-us/azure/... , which uses cloud services architecture. This architecture allows the infrastructure used to increase or decrease according to the system's demand, paying only for what was used in memory, processing, and storage.

We created a web application service for the user interface, a web application service for the API, and a Microsoft SQL Server relational database service for storing the data.

Publishing of UI and API components is done by auto publishing, configured in Azure DevOps¹²12 Microsoft. Azure devops documentation [Internet]. 2015 [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/azure/devops/
https://docs.microsoft.com/en-us/azure/d... , allowing the storage of published versions and reverting to a previous version, in case the publication has problems.

The database is updated using Data Definition Language (DDL) Strucutred Query Language (SQL) scripts, generated with a database structure comparison tool provided by the Visual Studio Community³⁰30 Microsoft. Visual studio documentation [Internet]. [cited on May 3, 2022]. Available at: https://docs.microsoft.com/en-us/visualstudio/windows/?view=vs-2019/
https://docs.microsoft.com/en-us/visuals... . The comparison is made between the local database used for development and the database service in production. Scripts are also created to return to the previous version, if necessary.

For security in the transfer of information between client and server, the HTTPS protocol was used, through Secure Socket Layer (SSL), which encrypts all information, preventing unauthorized persons from viewing it over the network.

Tests performed

The developer performed specific tests for each requirement created. The bugs identified were fixed prior to deployment to the production environment.

Description of the computer system

After the system was implemented, its functionalities were described according to the menu, simulating its use in conducting a systematic review study on cystic fibrosis³¹31 Egan ME, Greene DM, Voynow JA. Fibrose cística. In: Kliegman RM, Stanton BF, St Geme III JW, Schor NF, Behrman RE. Nelson: Tratado de pediatria. 20ᵃ ed. Rio de Janeiro: Elsevier; 2018. p. 2098-112.,³²32 Hauschild DB, Rosa AF, Ventura JC, Barbosa E, Moreira EAM, Ludwig Neto N, et al. Association of nutritional status with lung function and morbidity in children and adolescents with cystic fibrosis: a 36-month cohort study. Rev Paul Pediatr 2018; 36(1): 31-8. http://dx.doi.org/10.1590/1984-0462/;2018;36;1;00006
http://dx.doi.org/10.1590/1984-0462/;201... . The search phrases (Chart 2) were inserted into the databases to obtain the files to be imported into the system. Screen images were inserted to visualize the functionalities of version 1.0, for access through computers (desktop and notebook).

Ethical aspects

The work followed the terms of Resolution of the National Health Council (Conselho Nacional de Saúde – CNS) 466/12 and was approved by the Research Ethics Committee of the School of Public Health of Universidade de São Paulo (CAAE 36397420.8.0000.5421).

RESULTS

The “Support for Systematic Review” system is available at the electronic address (sysrev.azurewebsites.net). Explanatory videos can be accessed on a platform (www.apoioarevisaositematica.com.br) for self-learning and better use of its main features: the identification and exclusion of duplicates, assistance in the selection of eligible studies by reading titles and abstracts and assistance in resolving disputes.

Identification and exclusion of duplicates

First, one needs to register. In the “Reviews” (Revisões) menu, researchers are allowed to create, change or delete a systematic review. The system allows the inclusion of one or more bibliographic review projects as a main reviewer and as a guest reviewer. In the “Groups” (Grupos) menu, the researcher can create, change or delete one or more groups to import the abstracts as needed. In the “Summary” (Resumos) menu, the files that contain the records found in the electronic databases can be imported into the groups created by the researcher. After import, the status of each summary is set by the system as “Imported” (Importado).

After clicking on the “Duplicates” (Duplicados) button, researchers must select a priority database, which will serve as a reference for the system to assign the “Duplicate” status to existing abstracts in the other groups. By clicking on the “Exclude duplicates” (Excluir duplicados) button, the studies to be included in the eligibility phase are obtained.

By using the system, a review was created with the name “Anthropometric assessment in cystic fibrosis”. Next, three groups were created with the names of the origin databases, namely PubMed, Embase, and Web of Science. Each file was imported into the system and saved in the group according to its origin. In all, 834 records were imported, resulting in 644 studies after excluding duplicates (Figure 1).

Assistance in the selection of eligible studies

In the eligibility phase, in the “Criteria” (Critérios) menu, researchers are allowed to include, change, and exclude the eligibility criteria predefined in the research protocol.

In the “Reviewers” (Revisores) menu, researchers are allowed to include or remove reviewers for reading and analyzing titles and abstracts. Each guest reviewer is notified of their inclusion in a review.

In the “Abstracts” (Resumos) menu, after clicking on “View” (Visualizar), review participants can access the summaries and independently decide on the eligibility criteria by selecting from the “Complies” (Atende) list an answer for each criterion (yes, no or not clear). Next, they must select a result for each abstract (eligible, non-eligible or unclear), and it is possible to create comments.

In the application of the system, it is possible to notice that the eligibility criteria are presented on the same page of the summary to facilitate and guide analysis. The selection of responses in the “Complies” (Atende) and “Result” (Resultado) lists is not mandatory, but a status should be included in the “Result” (Resultado) list to favor the comparison of responses in the verification of the existence of divergences (Figure 2).

Assistance in resolving disputes

At an appropriate time, the primary reviewer may break the blinding to release results on the eligibility of abstracts to participants. In the “Abstracts” (Resumos) menu, by clicking on “Show comparison” (Exibir comparação), the system automatically compares the answers, and the differences pointed out can be resolved by the participants. At this stage, discussions can take place in person or remotely, provided by the sharing of data by the system. Reviewers can make the necessary changes directly into the system and download a PDF file with eligible abstracts. In the “Dashboard” menu, the main reviewer can check, through follow-up charts, the progress and completion of the abstracts reading by the invited reviewers and identify the existence of divergences (Figure 3).

DISCUSSION

This study presented the “Support to Systematic Review” system, which excludes duplicate articles, assists in the selection of eligible studies and in the resolution of divergences, organizing teamwork.

These functionalities aim at the rational use of time in the initial phase of a systematic review. Time savings are found in research on the influence of the use of automation tools in review studies. In it, among 189 participants, 79% reported using them in various tasks of the eligibility stage. As participants were divided into systematic reviews, guides and health technology evaluators, it was observed that 80, 88, and 73%, respectively, reported time savings when using automated tools³³33 Scott AM, Forbes C, Clark J, Carter M, Glasziou P, Munn Z. Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: a survey. J Clin Epidemiol 2021; 138: 80-94. https://doi.org/10.1016/j.jclinepi.2021.06.030
https://doi.org/10.1016/j.jclinepi.2021.... .

The use of computational tools is strongly encouraged in review studies¹⁴14 Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. 2021 [cited on May 03, 2022]. Available at: http://training.cochrane.org/handbook/
http://training.cochrane.org/handbook/... . Considering that a typical systematic review can take at least a year to be produced³⁴34 Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 2017; 7(2): e012545. https://doi.org/10.1136/bmjopen-2016-012545
https://doi.org/10.1136/bmjopen-2016-012... , the use of the system is important in these phases, which are usually very time consuming when performed manually.

A software frequently used in systematic reviews is EndNote, a reference manager that allows the identification of duplicates and the automatic insertion of citations in text editors³⁵35 Thomson Reuters. EndNote X9. Quick reference guide [Internet]. [cited on Jul. 29, 2022]. Available at: https://support.clarivate.com/Endnote/servlet/fileField?entityId=ka14N000000EcsXQAS&field=CA_Attachment_1__Body__s
https://support.clarivate.com/Endnote/se... , but which does not allow the inclusion of eligibility criteria and the comparison of decisions about eligibility of studies. Another software, Rayyan, has functionalities similar to those existing in “Support for Systematic Review”, with the difference that it is presented in English and requires the researcher to make a decision on the existence of duplicates based on similarity analysis, in addition to allowing the inclusion of keywords and terms that are highlighted in the abstract to facilitate their analysis³⁶36 Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan-a web and mobile app for systematic reviews. Syst Rev 2016; 5(1): 210. https://doi.org/10.1186/s13643-016-0384-4
https://doi.org/10.1186/s13643-016-0384-... .

A common problem in the identification of duplicate studies is the errors that occur due to variations in the records of studies in the databases, mainly in the name of authors and journals. With this, computational tools can point out a study as a duplicate when in fact it is not (false-positive); or they may point out as not duplicated a study that actually is (false-negative)³⁷37 Rathbone J, Carter M, Hoffmann T, Glasziou P. Better duplicate detection for systematic reviewers: evaluation of systematic review assistant-deduplication module. Syst Rev 2015; 4(1): 6. https://doi.org/10.1186/2046-4053-4-6
https://doi.org/10.1186/2046-4053-4-6... . A false-positive study, when excluded, can introduce a selection bias if it is not retrieved later by the researcher. The evaluation of the performance of the “Support for Systematic Review” in the identification of duplicates showed a sensitivity of 98.4%, specificity of 99.8%, and accuracy of 99.6%. The sensitivity result was higher than that of EndNote (sensitivity: 59.4%, specificity and accuracy: 100%) and that of Rayyan (sensitivity: 96.4%, specificity: 100%, accuracy: 99.3%) (data under publication).

In the eligibility phase, one of the advantages of “Support for Systematic Review” in the selection of studies is the ease of use, the reproducibility of the process and the participation of several reviewers in an organized and independent way, with the possibility of breaking the blinding in time defined by the main reviewer. The system is practical for storing eligibility criteria and sharing data between reviewers. On the same screen, an abstract and a form with the eligibility criteria for marking responses are presented, facilitating the abstract selection process.

The system is geared toward the production of typical systematic reviews, but is extended to researchers who opt for bibliographic surveys without the formalities of a systematic review. A simulation of a systematic review was carried out using a set of studies on cystic fibrosis, however different topics may be used in other studies, provided that the surveys are carried out in the bibliographic databases enabled in this system. Images of the interfaces of the computer version were inserted, but users can also access the system through smartphones and tablets, as it is responsive to different screen sizes.

We can identify as weaknesses the availability of version 1.0 only in Portuguese, the proprietary code, and the reduced number of databases. However, the Portuguese language version favors its use in other countries that have this language as their mother tongue. The number of databases included can be increased, prioritizing those used in the health area⁴4 Muka T, Glisic M, Milic J, Verhoog S, Bohlius J, Bramer W, et al. A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research. Eur J Epidemiol 2020; 35(1): 49-60. https://doi.org/10.1007/s10654-019-00576-5
https://doi.org/10.1007/s10654-019-00576... ,³⁸38 Gupta S, Rajiah P, Middlebrooks EH, Baruah D, Carter BW, Burton KR, et al. Systematic review of the literature: best practices. Acad Radiol 2018; 25(11): 1481-90. https://doi.org/10.1016/j.acra.2018.04.025
https://doi.org/10.1016/j.acra.2018.04.0... . The system has the potential for improvements, and may include quality assessment instruments, data extraction by more than one evaluator and comparison of these results.

As potentialities, there is the use of the Agile methodology, which provides the possibility of constantly adding new resources and improvements, such as the implementation of the Latin American and Caribbean Literature on Health Sciences (Lilacs), the inclusion of checkboxes for deleting false negatives, a recycle bin for retrieving false positives and exporting text files for use in reference managers. As it is a web tool, all updates can be accessed immediately by users.

As a vision of the future, it is intended to expand its functionality according to the steps of a systematic review and improve the usability and accuracy of the duplicate identification process, as well as move toward the (semi) automation of the selection of studies.

The production of scientific knowledge can be reinforced in the presence of computational tools that reduce manual work and allow better synthesis and use of information. Thus, it is expected that the system will be used in Portuguese-speaking countries for the continuous production of knowledge syntheses for application in health actions.

The “Support for Systematic Review” system is available on the web for the exclusion of duplicates and assistance in the selection of eligible studies and resolution of divergences in systematic literature reviews, ensuring greater speed, reliability, and reproducibility for the completion of these steps.

REFERENCES

Publication Dates

History