ABSTRACT

OBJECTIVE:

to evaluate the Brazilian Laboratorial Environment Management System (GAL) as a data source for the Sentinel Surveillance of Influenza-Like Illness (SSILI).

METHODS:

this is an evaluation study of GAL, through the analysis of its simplicity, flexibility, data quality, acceptability, representativeness, timeliness, stability and usefulness for SSILI, based on the guide for system evaluation in public health of the Centers for Disease Control and Prevention (Atlanta/GA, USA).

RESULTS:

in 2011 and 2012, a total of 13,765 exams for respiratory viruses were registered. GAL presented simple structure, flexibility to changes, good data quality, acceptability and timeliness in the access to test results, being representative and stable in 23 Brazilian states.

CONCLUSION:

the system is useful in meeting the goals of SSILI; however, there are some recommendations for adjustments and for encouraging the adherence by the states that do not use the system yet.

Keywords:
Sentinel Surveillance; Health Evaluation; Information Systems; Influenza Human

Introduction

The disease caused by the Influenza virus is considered one of great social impact, due to the high rates of mortality and economic losses, especially during pandemics. To fight the possible influenza outbreaks or epidemics, the Ministry of Health of Brazil implemented, in 2000, a sentinel surveillance system, as a part of the World Health Organization (WHO) Global Influenza Surveillance Network. The system aims to identify the circulation of respiratory viruses (Influenza A and B, Parainfluenza 1, 2 and 3, Adenovirus and Respiratory syncytial virus), isolate viral specimens, identify new subtypes early and disseminate epidemiological information related to influenza.¹1. World Health Organization. WHO global technical consultation: global standards and tools for influenza surveillance. Geneva: World Health Organization; 2011 [Cited 2013 Jan 30]. Available from: Available from: http://www.who.int/influenza/resources/documents/technical_consultation/en/index.html .
http://www.who.int/influenza/resources/d... ^,22. Mistério da Saúde (BR). Secretaria de Vigilância em Saúde. Departamento de Vigilância Epidemiológica. Guia de vigilância epidemiológica [Internet]. 7 ed. Brasília: Ministério da Saúde; 2010 [citado 2013 mar 03]. (Série A. Normas e manuais técnicos). Disponível em: Disponível em: http://www.epi.uff.br/wp-content/uploads/2013/10/Guia-de-Vigil%C3%A2ncia-Epidemiol%C3%B3gica-%E2%80%93-7%C2%AAedi%C3%A7%C3%A3o-2010.pdf
http://www.epi.uff.br/wp-content/uploads...

Laboratory data on respiratory viruses is important in order to recommend control actions, and must be organized in an integrated way, through information systems (IS), defined as computer technologies (softwares) able to store and recover data in a structured way. ³3. Guimarães EMP, Évora YDM. Sistema de informação: instrumento para tomada de decisão no exercício da gerência. Ci Inf. 2004 jan-abr;33(1):72-80.^,44. Richieri FL. A nova geração de ferramentas de TI. Gazeta Mercantil. 22 Fev. 2001. These systems must be robust, flexible, safe and contain trustworthy data, generated by systematic processes and organized in databases (BD) that facilitate the conversion of data into information, so they can assist users and subsidize the decisions of health surveillance taken by workers and managers.³3. Guimarães EMP, Évora YDM. Sistema de informação: instrumento para tomada de decisão no exercício da gerência. Ci Inf. 2004 jan-abr;33(1):72-80.^,44. Richieri FL. A nova geração de ferramentas de TI. Gazeta Mercantil. 22 Fev. 2001.

In 2008, aiming to perfect laboratory information, the Ministry of Health (MS), through the General Coordination of Public Health Laboratories (CGLAB) and IT Department of the Brazilian National Health System SUS (DATASUS), elaborated the Laboratory Environment Management System (GAL), with its own communication patterns, distributed, robust and flexible architecture, multiplatforms and in a free software, it can also be integrated to the other health surveillance and information systems used in Brazil.⁵5. Mistério da Saúde (BR). Secretaria de Vigilância em Saúde. Departamento de Vigilância Epidemiológica. Coordenação Geral dos Laboratórios de Saúde Pública. Gerenciador de Ambiente Laboratorial. Datasus: Modúlos do sistema [Internet]. Brasília: Ministério da Saúde; 2011 [citado 2012 dez 20]. Disponível em: Disponível em: http://gal.datasus.gov.br/GAL/default.php?area=02
http://gal.datasus.gov.br/GAL/default.ph... ^,66. Jesus R, Guimarães RP, Bergamo R, Santos LCF, Matta ASD, Paula Júnior FJ. Sistema Gerenciador de Ambiente Laboratorial: relato de experiência de uma ferramenta transformadora para a gestão laboratorial e vigilância em saúde. Epidemiol Serv Saude. 2013 jul-set;22(3):525-9. GAL contains seven modules. The data related to respiratory viruses are contained in the Medical Biology module.

GAL's objective is to computerize Brazil's entire public health laboratory network, including the registry of human, animal and environmental origin samples; to supply quality control of diagnosis results; and to provide data for epidemiological and environmental surveillance at a city, district, state and national level.⁶6. Jesus R, Guimarães RP, Bergamo R, Santos LCF, Matta ASD, Paula Júnior FJ. Sistema Gerenciador de Ambiente Laboratorial: relato de experiência de uma ferramenta transformadora para a gestão laboratorial e vigilância em saúde. Epidemiol Serv Saude. 2013 jul-set;22(3):525-9. Furthermore, the system is a database for Brazil's Sentinel Surveillance of Influenza-Like Illness (SSILI).

Considering this is an information system established in 2008 and that so far it has not gone through any evaluations, it is important to proceed to this evaluation, so that its results and performance may be confirmed; If those do not meet the expectations, corrections and improvements must be suggested in order to turn it into a more efficient and effective system.

Therefore, the objective of the present study is to evaluate GAL as a database for SSILI.

Methods

An evaluation study was performed, based on the criteria established in the Guidelines for Evaluating Surveillance Systems provided by the Centers for Disease Control and Prevention (Atlanta/GA, USA). The following attributes were evaluated: qualitative (simplicity, flexibility, data quality and acceptability); quantitative (representativeness, timeliness, stability); and the system's usefulness.⁷7. German RR, Lee LM, Horan JM, Milstein RL, Pertowski CA, Waller MN; Guidelines Working Group Centers for Disease Control and Prevention. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR Recomm Rep. 2001 Jul;50(RR-13):1-35; quiz CE1-E7. The evaluation of the attributes was performed by the main author (trainee at the Training Program in Epidemiology Applied to the Services of the Brazilian National Health System - EpiSUS). The criteria were discussed with the Information Technology (IT) professionals and a chemist from CGLAB/MS, a nurse from the Ministry of Health's technical group of influenza surveillance and a veterinary, who was the trainee's tutor. Each item evaluated arbitrarily was discussed with all the members of this group.

As data source, GAL's database, referring to years 2011 and 2012, was used, along with technical documents and consultation to professionals from CGLAB/MS, from the Surveillance Unit for Airborne and Immuno-preventable Diseases (UVRI/MS), from DATASUS/MS and from the Central Laboratory of Parana State (Lacen-PR).

Simplicity was evaluated based on three criteria connected to SSILI: (i) the flow since the collection of data, the registering and sending of information and the input of exam results in GAL; (ii) access to exam reports; and (iii) export of the database at a national level. To that purpose, a description of the way the GAL system works inside SSILI was done. Each of the processes was classified as either simple or complex, it was taken into consideration the levels of difficulty and the time lapse for execution, given that there was not a specific parameter for each item. In case the attribute as a whole presented two or more criteria evaluated as simple, its final classification would be simple. In case the attribute as a whole presented two or more items evaluated as complex, its final classification would be complex.

Flexibility was evaluated by the percentage of updates to the system, during the years 2010 and 2011, done in order to fit SSILI. The system was classified as flexible if the number of updates was superior to five in both years, or inflexible, if it was inferior to five in at least one of the years evaluated.

Data quality was evaluated by its completeness and validity of the registered data. Completeness, defined by the percentage of records filled for each variable, was evaluated for essential variables that could assist in the tracking of the samples. Sinan8 parameters were applied for the classification of completeness: poor (<70%), regular (70%-89%) and excellent (≥90%). Data completeness average (DCA) was calculated with the sum of the percentages, divided by the total of variables analyzed. Mandatory variables were not part of the calculation, as they were a 100% filled.

In order to evaluate data validity, inconsistencies in the variables "Date of birth incompatible with age"; "Date of birth posterior to request date”; “Date of first symptoms and Date of collection", and "Date of receipt posterior to Processing date" were observed. The parameter was arbitrated according to the percentual of inconsistencies in the records: satisfactory (<5%) and unsatisfactory (≥5%). After the classification of each variable, the data inconsistency average (DIA) was calculated through the sum of the percentages divided by the total of variables analyzed. The system's data quality was classified as good when the DCA was regular or excellent and the DIA was satisfactory; poor when the DCA was poor and the DIA was satisfactory, as well as when the DCA was regular or excellent and the DIA was unsatisfactory; and very poor when the DCA was bad and the DIA was unsatisfactory.

Acceptability was evaluated indirectly through the proportion of the Brazilian population in the states where the system was implemented. Therefore, the proportion of the Brazilian population living in the states where GAL was implemented during the time of the study was calculated, according to the 2010 Census from the Brazilian Institute of Geography and Statistics (IBGE). The parameters were arbitrated along with the SSILI, defined as the following: bad coverage, <50% of the country's population; good, between 50% and 75% of the country's population; and great, >75%.

Representativeness was evaluated by simply comparing the distribution of respiratory viruses by age group, between GAL and the systems described in the epidemiological bulletin on influenza, which uses the database from the Influenza Epidemiological Surveillance System (Sivep-Gripe), and the distribution of respiratory viruses by epidemiological week, comparing seasonality to the data already described in literature. The system was considered representative in case the data presented visibly similar characteristics to those described in literature, and non-representative if any similarity was absent.

Timeliness was evaluated by the reduction of the average time for the release of exam results. It was considered excellent when the result was available in ≤7 days, good when between 8 and 15 days, regular between 16 and 30 days, and poor when over 30 days.

Stability was evaluated indirectly by the following aspects: (i) number of technicians available for the system's development and maintenance at a national level (positive when there was two or more technicians and negative when there was only one); and (ii) loss of data (positive when there was no data loss and negative when there was data loss during the period of evaluation). The data was obtained from accounts given by professionals and reports from DATASUS. The system was classified as stable when the evaluation of both items was positive; or unstable when at least one of those items had a negative evaluation.

Utility was evaluated by GAL's ability to fulfil all of SSILIS's objectives regarding: (i) the flow of processed samples; (ii) the integration between epidemiological surveillance and the laboratory; and (iii) the identification of the circulating viruses. It was classified as useful when the evaluations of two or more of these items were positive, or not useful when two or more items had a negative evaluation.

For data analysis, descriptive statistical analysis was performed, such as frequency estimates, timely analysis, medians and interquartile ranges. To that purpose, Epi Info TM version 7.1.0 software, and Microsoft Excel® 2010 were used.

As for the ethical aspects of the research, since it dealt with secondary, unidentified data, the study was exempted from evaluation by the Ethics Research Committee as established by Resolution of the National Health Council (CNS) No. 510, dated April 7th 2016.

Results

Description of the interaction between GAL and SSILI

The samples of respiratory viruses are collected in sentinel units previously registered by the municipalities. The samples are done by convenience. It is recommended the collection of five weekly samples of nasopharynx discharge from patients classified as having influenza-like illness.

When samples arrive at the laboratory, they are usually handed to the Sample Management Sector (SGA) or to screening, being verified storage conditions and filled all the register files from GAL and Sivep-Gripe. From then on, the system signalizes the approval of the sample and it is then sent to the Specific Analysis Laboratory (LEA), at the Central Laboratory of Public Health.

The samples must be handed to LEA at the day of their arrival, and must be verified again by lab professionals at LEA. Despite this new verification, the system does not have an entry for the date of arrival at the laboratory.

Usually, after the samples are processed, the results are filled in a spreadsheet and taken to the computer room to be registered at GAL and Sivep-Gripe systems; often, Lacen's team sends the results in a spreadsheet to be typed by the surveillance team at Sivep-Gripe, or the team retrieves the laboratory results directly at GAL.

The system does not allow communication between labs, that is to say, the GAL in one Lacen has no possibility of communication with another Lacen's GAL or with reference laboratories, which prevents sample records from being sent directly through the system. Therefore, the sample must be registered again in the system of the laboratory where it will be sent, where it will receive a new identifier. This entire process clearly shows the difficulty in tracking the samples sent to other laboratories.

GAL data is consolidated in the states' databases, by place of register of the samples, which are under the responsibility of the Lacen that processed and released exam results. Therefore, municipalities and states' health surveillance teams must request a password to the system's local manager, in order to access it through the internet, retrieve exam reports and database exports or reports. The data is made available in real-time, which means that when the data is inserted and released in the system, it is accessible to all registered professionals, according to their level of work.

Evaluation of Qualitative Attributes

In the systems' data flow described and represented in Figure 1, it can be seen that GAL is simple, as it allows integration to the epidemiological surveillance, since the collection of the sample until the release of the results by the assigned technician. Furthermore, it is inserted in a surveillance sentinel which is simple and costs less than an universal surveillance.

In order to retrieve exam reports, the professional must be registered, have basic computer knowledge and a computer with internet access. For that reason, the system was also considered simple regarding the access to reports.

Despite that, in order to generate a database at a national level, it was necessary to access the system of each of the 23 states where it was used and generate quarterly databases (total of eight in the period researched). That resulted in 184 processes in the period of study, which turns it into a complex system for this activity. Therefore, considering the three aspects evaluated in the system's description, GAL can be considered, when interconnected to SSILI, as a simple system.

As for flexibility in the years 2011 and 2012, GAL went through 15 and 16 updates in its versions, respectively. Considering the updates done specifically for the identification and monitoring of respiratory viruses, the system went through 4 updates in 2011 and 16 in 2012. Therefore, GAL has 20 updates during that period related to adjustments for exam results and the identification of respiratory viruses, and was, thus, classified as flexible.

As for the evaluation of the completeness criterion, a database with 13,562 records of respiratory viruses exams, processed and released through indirect immunofluorescence (IIF) methodology was analyzed . Out of the existing 64 variables in the database, 37 were identified as important for the monitoring and tracking of the samples, out of which 17 (46%) were mandatory variables.

The 21 variables, listed in Table 1, were considered essential and classified depending on their percentage of completeness, with 10 presenting 99% and 100% (excellent), 10 with percentages between 73% and 82.2% (regular) and 1 with 70% (poor). The DCA was 87% (Table 1), resulting in regular data completeness.

When analyzing the inconsistencies in the system's variables, the existence of date of receipt posterior to the processing date of the sample in 10% (unsatisfactory percentage) and date of birth differing from age in 8% (unsatisfactory) of the exams was verified. Date of birth posterior to the date of first symptoms in 0.4% (satisfactory), the date of request in 0.3% (satisfactory) and the date of collection in 0.3% (satisfactory) of the records was also observed. Therefore, the DIA in this evaluation was of 3.8% (satisfactory) (Table 1)

By observing the averages of completeness and inconsistency, the system was classified as having good data quality.

Acceptability evaluation identified that the system is implemented in 85% of the states, which corresponds to a coverage of 66.4% of the country's population, and the system was classified as having good acceptability.

Evaluation of the quantitative attributes

Representativeness was evaluated based on 2,591 (19%) positive exams for respiratory viruses, classified as released in GAL's database. It was observed that the distribution of respiratory viruses by age group was predominant in the group of 0 to 4 years, followed by the group of 25 to 59 years. These were the same age groups found as predominant in the Ministry of Health's epidemiological bulletin on influenza, as seen in Figure 2, highlighting similar characteristics between the GAL's and Sivep-Gripe's databases. The distribution of respiratory viruses by epidemiological week was also observed and it was verified that, in the last two years analyzed, the highest circulation of the viruses happened between March and September, as presented in Figure 3.

With the evaluation of these aspects, the system was classified as representative for SSILI regarding age groups and the epidemiological week.

The system showed the ability to calculate the many time intervals between the processes, since the collection done by surveillance, until the release of the result by the technician assigned to the analysis, as demonstrated in Figure 4. The average time to gain access to exam results, coming from SSILI, was 11 days, with interquartile variation (IQR 6 - 29 days).

According to professionals, before GAL’s implementation, access to exam results happened only after they were released by the technician, being then necessary to print and send them to the requester.The reason for that, in some cases, resulted from the necessity to send the results to the surveillance team through the laboratory, or even for the surveillance team themselves to be responsible for picking up the exams. After the system was implemented, the average time to gain access to exam results decreased approximately 50%. According to professionals, the average time used to be around 30 days, showing the system's good timeliness when compared to the time spent before it was created. The system was evaluated as showing good timeliness regarding access to exam results, taking in consideration the parameter established by the technical department, as presented in Figure 4.

According to technicians in charge of the system, during the period studied, there was no loss of data. As for the professionals responsible for GAL's development and maintenance at a national level, the project had a project leader, an analyst and two developers at DATASUS, an IT professional at CGLAB and an advisory Technical Group (TG), with five professionals from Lacen, with no connection to the Ministry of Health, amounting to 10 professionals in 2011. In the following year, the team was expanded to 13 professionals, with one junior analyst and a developer being added to DATASUS team, an IT professional to CGLAB and the Technical Group was maintained. Although the team was composed by only one public servant and five state employees, the system is classified as having good stability, since, during the period evaluated, it did not showed any loss of data, nor absence of a qualified technical team for its maintenance and development.

Evaluation of system's usefulness

Regarding the objectives established by the surveillance team, GAL was able to identify the circulation of respiratory viruses through IIF; characterize the seasonality; monitor the samples sent to reference labs that use GAL; and produce and disseminate information through the establishment of databases that allow the creation of reports, informative bulletins and scientific publications. Furthermore, though it is not the objective of this evaluation, it should be noted that the system proved to be able to monitor samples sent to be processed through molecular biology, identifying the subtype of the positive samples for this test.

Based on all the characteristics presented in this evaluation, GAL proved to be useful for influenza surveillance, especially as a tool for sentinel surveillance of influenza-like illness. Even so, it was not being used at a national level due to the inexistence of a national module, which consequently prevents access in a timely manner for the periodic monitoring of influenza surveillance.

Discussion

Overall, GAL was evaluated as simple, flexible to changes, with good data quality and acceptability. It was considered representative when compared to other systems, with good timeliness of exam results and good stability, when evaluated indirectly by professionals involved in its maintenance and development. Furthermore, the system showed to be able to fulfill satisfactorily the objectives established by surveillance, and was, therefore, considered useful.

The use of IS in the health surveillance system in Brazil has been increasing, which makes the evaluation of this component in particular more common. That does not, however, disregard the need to evaluate the remaining components. The evaluation of the information system of public health laboratories is not a simple task.

Since it is a tool developed for laboratories, which operates in a number of public health surveillance systems that need laboratory analysis, GAL's integration should be evaluated in each of the surveillance systems. Nevertheless, the present study chose to evaluate only SSILI. This choice was due to the importance of laboratory results for the surveillance performance as the basis for monitoring the circulation of respiratory viruses.

For national level technicians, the system plans to provide the national module, which allows access to exams released by Lacen, without nominal data. Surveillance technicians will be able to access data, perform analysis and generate information on the diseases of their areas of expertise, in the same way technicians of the states do.

The aforementioned updates show the system's flexibility, which is able to be altered specifically depending on the disease, and even to undergo the adaptations proposed during this evaluation.

The system's acceptability, although good, can be improved with the possibility of GAL's implementation if other states adhere to it. Even though the implementation is not mandatory, in December of 2012, most states had already adhered to the system. The insertion of data from the states of Bahia, São Paulo, Santa Catarina and the Federal District, as soon as possible, would allow the system to encompass all the laboratory results of the country’s federated unities.

It was also observed that, regarding the distribution of respiratory viruses by epidemiological week, as seen in the two years analyzed, there was a higher circulation of the viruses in the period between March and September, as presented in Figure 3. The same behaviour of seasonal distribution by epidemiological week was observed in other studies.¹¹11. Oliveira LF, Carvalho TS. Epidemiologia dos vírus respiratórios no município de Porto Velho-RO. Saber científico. 2008 jan-jun;2(1):69-80.^,1212. Thomazelli LM, Vieira S, Leal AL, Sousa TS, Oliveira DB, Golono MA, et al. Surveillance of eight respiratory viruses in clinical samples of pediatric patients in southeast Brazil. J Pediatr (Rio J). 2007 Sep-Oct;83(5):422-8.

The procedures until the exam results were released took around 30 days to take place, or even longer in municipalities farther from the central laboratory main building (personal communication). Despite the time to access exam results being reduced, it is still higher than a week in a large portion of the samples, which is not ideal for SSILI, which advocates a limit of seven days.

As for limitations, there is the absence of a BD considered gold-standard to be used as reference in analysis, and the use of subjective and indirect criteria to evaluate certain attributes. Furthermore, it should be mentioned that this evaluation did not include the attributes sensibility and positive predictive value.

In spite of having presented a good result in this evaluation, GAL needs a number of adjustments aiming to improve its performance in SSILI, amongst which it is recommended the following ones: (i) to perform other evaluations with different diseases in order to identify specific problems; (ii) to provide an option to select all variables, when generating the database; (iii) to put the national module into practice; (iv) to widen the BD's export period from 3 to 12 months, aiming to facilitate the analysis of collective data; (v) to insert the field "Date of Arrival" of the sample in the specific laboratory for analysis, in order to calculate the sample processing timeliness when arriving at the laboratory; (vi) to block the possibility of inserting the "Date of Birth" posterior to the "Date of Request", "Date of First Symptoms", and "Date of Collection"; and "Date of Receipt" posterior to "Processing Date"; (vii) to capacitate professionals regarding data insertion, showing the importance of filling all the fields correctly, so as to improve data quality; (viii) to encourage the adherence by the states that do not use the system yet, or at least encourage data interface with other systems, in order to reduce the workload of the lab professionals regarding duplicated insertion of the results. With that, the system's analysis capacity could be expanded and the time for processing, releasing and analyzing the results could be reduced, with the purpose to improve the system's acceptability, stability and representativeness and to make it more useful to health surveillance.

References

Publication Dates

History