An evaluation of the Brazilian Mortality Information System

Morais, Rinaldo Macedo de; Costa, André Lucirton

doi:10.1590/0103-11042017s09

ABSTRACT

This paper assesses quality indicators of the Mortality Information System (SIM) with professionals and managers. Each indicator was analyzed with use of Stochastic Oscillator for qualitative evaluation of Likert scales. The evaluation indicated that the system has features that meet users, is effective to the management, provides adequate performance, enables auditing and tracking of accesses and operations, has mechanisms that ensure disaster recovery situations in data, and has a robust interface. Interoperability indicators were poorly evaluated, confirming reports on the lack of integration between the health information systems of the Unified Health System, fragmentation, and duplication of information.

KEYWORDS:
Health management; Mortality registries; Health evaluation

Introduction

The gathering of mortality events in Brazil has improved significantly in recent decades; however, surveys on coverage and completeness of vital events have classified the country in intermediate ranges (MAHAPATRA 2007MAHAPATRA, P. et al. Civil registration systems and vital statistics: successes and missed opportunities. The Lancet, London, v. 370, n. 9599, p. 1653-1663, nov. 2007., MATHERS ., 2005MATHERS, C. D. et al. Counting the dead and what they died from: an assessment of the global status of cause of death data. Bull World Health Organ, Genebra, v. 83, n. 3, p. 171-177, mar. 2005.), with a significant volume of registries with undefined causes (LIMA; QUEIROZ, 2011LIMA, E. E. C.; QUEIROZ, B. L. A evolução do sub-registro de mortes e causas de óbitos mal definidas em Minas Gerais: diferenciais regionais. Rev. bras. estud. popul., Rio de Janeiro, v. 28, n. 2, p. 303-320, dez. 2011., FRANÇA 2013FRANÇA, E. B. et al. Strengthening vital statistics in Brazil: investigation of ill-defined causes of death and implications on mortality statistics. The Lancet, London, v. 381, supl. 2, p. 51-51, jun. 2013.) and problems in death notification and information flow (SIVIERO 2013SIVIERO, P. et al. Indicador de subnotificação de óbitos no Sistema de Informação de Mortalidade no Brasil obtido de pacientes que morreram por doença renal crônica terminal: mensuração baseada nas Autorizações de Procedimentos de Alta Complexidade de 2000 a 2004. Cad. saúde colet., v. 21, n. 1, p. 92-95, mar. 2013., FIGUEIROA 2013FIGUEIROA, B. Q. et al. Análise da cobertura do Sistema de Informações sobre Mortalidade em Olinda, Pernambuco, Brasil. Cad. Saúde Pública, Rio de Janeiro, v. 29, n. 3, p. 475-484, mar. 2013.).

The Mortality Information System (SIM) supports the collection, storage and management process of death registries, in Brazil, which is mandatory in all cities. Mortality data are periodically sent to the State Secretaries of Health and then to the Ministry of Health. The system also has a module available on the web, which accesses the national database for investigation registrations and consultations, such as infant and neonatal deaths and deaths of pregnant and fertile women. The historical data stored by the system produce indicators that subsidize managers through managerial consultations through situational dashboard health consultation apps in consolidated databases.

Studies in literature show deficiencies in the quality of public healthcare information systems in Brazil, such as the lack of integration, fragmentation and duplicity of information (THAINES , 2009THAINES, G. H. L. S. et al. Produção, fluxo e análise de dados do sistema de informação em saúde: um caso exemplar. Texto & Contexto Enfermagem, Florianópolis, v. 18, n. 3, p. 466-474, set. 2009., DAMÉ , 2011DAMÉ, P. K. V. et al. Sistema de Vigilância Alimentar e Nutricional (SISVAN) em crianças do Rio Grande do Sul, Brasil: cobertura, estado nutricional e confiabilidade dos dados. Cad. Saúde Pública, Rio de Janeiro, v. 27, n. 11, p. 2155-2165, nov. 2011.), poor coverage of some systems and uncertainties regarding the reliability of the data they maintain (DAMÉ 2011DAMÉ, P. K. V. et al. Sistema de Vigilância Alimentar e Nutricional (SISVAN) em crianças do Rio Grande do Sul, Brasil: cobertura, estado nutricional e confiabilidade dos dados. Cad. Saúde Pública, Rio de Janeiro, v. 27, n. 11, p. 2155-2165, nov. 2011., BARBUSCIA; RODRIGUES JÚNIOR, 2011BARBUSCIA, D. M.; RODRIGUES JÚNIOR, A. L. Completude da informação nas declarações de nascido vivo e nas declarações de óbito, neonatal precoce e fetal, da região de Ribeirão Preto, São Paulo, Brasil, 2000-2007. Cad. Saúde Pública, Rio de Janeiro, v. 27, n. 6, p. 1192-1200, jun. 2011., FARIAS , 2011FARIAS, L. M. M. et al. Os limites e possibilidades do Sistema de Informação da Esquistossomose (SISPCE) para a vigilância e ações de controle. Cad. Saúde Pública, Rio de Janeiro, v. 27, n. 10, p. 2055-2062, out. 2011., MOTA, 2009MOTA, F. R. L. Registro de informação no sistema de informação em saúde: um estudo das bases SINASC, SIAB e SIM no estado de Alagoas. 2009. Tese (Doutorado em Ciência da Informação) - Escola da Ciência da Informação, Universidade Federal de Minas Gerais, Belo Horizonte, 2009.) and deficiencies in support to the manager in decision-making and planning processes (MOTA, 2009MOTA, F. R. L. Registro de informação no sistema de informação em saúde: um estudo das bases SINASC, SIAB e SIM no estado de Alagoas. 2009. Tese (Doutorado em Ciência da Informação) - Escola da Ciência da Informação, Universidade Federal de Minas Gerais, Belo Horizonte, 2009., VIDOR; FISHER; BORDIN, 2011VIDOR, A. C.; FISHER, P. D.; BORDIN, R. Utilização dos sistemas de informação em saúde em municípios gaúchos de pequeno porte. Revista de Saúde Pública, São Paulo, v. 45, n. 1, p. 24-30, 2011.). Healthcare information in Brazil has multiple sources, poor quality of data and availability in formats that hinder their appropriation by managers and by social control (MORAES, 2010MORAES, I. H. S. Sala de situação em saúde: contribuição à ampliação da capacidade gestora do Estado? In: ORGANIZAÇÃO PAN-AMERICANA DA SAÚDE. Sala de situação em saúde: compartilhando as experiências do Brasil. Brasília, DF: Organização Pan-Americana da Saúde, Ministério da Saúde, 2010.) and monitoring of data quality in healthcare information systems that serve the Unified Healthcare System (SUS) do not follow a regular evaluation plan (LIMA 2009LIMA, C. R. A. et al. Revisão das dimensões de qualidade dos dados e métodos aplicados na avaliação dos sistemas de informação em saúde. Cad. Saúde Pública, Rio de Janeiro, v. 25, n. 10, p. 2095-2109, out. 2009.).

This paper introduces an evaluation process for quality attributes for SIM with two user profiles: healthcare professionals who use the system to feed death registrations in cities in the Northeast of the state of São Paulo, and healthcare managers who analyze system information for management and decision-making in the same region.

Methods

Characterized as a cross-sectional study, this research consisted of a quality evaluation of the SIM that included healthcare professionals from a macro-region of the state of São Paulo and health managers who work at the local, regional and state levels. The option to evaluate the SIM occurred because it is a high capillarity system, of regular and obligatory use in all Brazilian cities and that it has demanded many studies in public healthcare literature.

The evaluation with healthcare professionals occurred through interviews in cities from the 13th Regional Healthcare Network of the State of São Paulo (RRAS-13), which includes the healthcare regions of Araraquara, Barretos, Franca and Ribeirão Preto. City selection criteria adopted a grid with five population ranges, adapted from the Brazilian Institute of Geography and Statistics (IBGE) municipal social indicator grid. Among the 90 cities in the macro region, 18 were selected, observing the proportional population distribution for the region by range, resulting in two cities with up to five thousand inhabitants (Cássia dos Coqueiros and Santa Cruz da Esperança), nine between 5 and 25 thousand (Dumont, Santo Antônio da Alegria, Serra Azul, Luiz Antônio, São Simão, Pradópolis, Miguelópolis, Cajuru and Santa Rosa de Viterbo), four between 25 and 50 thousand (São Joaquim da Barra, Ituverava, Barrinha and Santa Rita do Passa Quatro), two between 50 and 200 thousand (Taquaritinga and Barretos) and one above 200 thousand inhabitants (Ribeirão Preto). The cities in each range were selected out of convenience, including those in which professionals agreed to participate in surveys and who had been using SIM for at least three months.

The interviews with healthcare managers included five professionals who worked in the cities of Ribeirão Preto, Barretos, Araraquara, São Joaquim da Barra and Taquaritinga; four who worked in the Epidemiological Surveillance Groups (GVEs) in the Regional Healthcare Departments of Ribeirão Preto, Araraquara, Franca and Barretos; and a SIM manager from the São Paulo State Secretary of Health Center for Strategic Information on Healthcare Surveillance Office. Selected professionals have worked with SIM for at least six months.

The model proposed by Morais and Costa (2014)MORAIS, R. M.; COSTA, A. L. Um modelo para avaliação dos sistemas de informação do SUS de abrangência nacional: o processo de seleção e estruturação de indicadores. Revista de Adm. Pública, Rio de Janeiro, v. 48, n. 3, p. 377-385, jun. 2014., designed as an instrument for evaluating the quality of SUS information systems on a national scope, was used as a theoretical reference for the evaluation process. This model includes three quality dimensions: software product quality, quality in use and service quality. For each dimension, the model specifies a set of quality indicators for health information systems. The structure and definition of the model's quality attributes are described in chart 1.

Thumbnail

Chart 1
Structuring and definition of quality attributes of the evaluation model

Two questionnaires oriented towards each user profile were prepared for collecting data, with evaluation questions for 45 quality indicators stipulated in the model specified by Morais and Costa. There were 28 questions for healthcare professionals who work in cities and 17 questions for managers, using four-point Likert scales.

The presentation of results used the absolute frequency distributions and percentages observed for the response categories for each indicator evaluated. The evaluation of results reused a procedure for data analysis with Likert scales, formulated by Sanches, Meireles and De Sordi (2011)SANCHES, C.; MEIRELES, M.; DE SORDI, J. O. Análise qualitativa por meio da Lógica Paraconsistente: método de interpretação e síntese de informação obtida por escalas Likert. In: ENCONTRO DE PESQUISA EM ADMINISTRAÇÃO E CONTABILIDADE DA ANPAD (ENEPQ), 3., 2011, João Pessoa. Anais... João Pessoa, 2011. Disponível em: <http://www.anpad.org.br/diversos/trabalhos/EnEPQ/enepq_2011/ENEPQ221.pdf>. Acesso em: 24 abr. 2014.
http://www.anpad.org.br/diversos/trabalh... , with a version adapted from stochastic oscillator proposed by the authors, for evaluating the degree of agreement (DA) for each indicator, on a scale between zero and one, with the inclusion of weight factors for the categories, as in the expression:

GC=100−100×0.011+2×ΣCC+ΣCP+0.25×ΣDP+0.000012×ΣDC+ΣDP+0.25×ΣCP+0.00001

In the expression, the sums for CA, PA, CD and PD refer to the totals obtained from compiling the answers for each category in the evaluation questionnaires (Completely Agree/ Partially Agree/ Completely Disagree/ Partially Disagree). The factor 0.00001 was added to the expression to avoid errors in division by zero. As shown in table 1, the value calculated for the degree of agreement (DA) was mapped for agreement or disagreement very strong, substantial, moderate, low or negligible, adapted from the authors' proposal for the adopted scale.

Thumbnail

Table 1
Qualitative assessment of the degree of agreement (DA)

An indicator was considered well evaluated for very strong, substantial or moderate agreement (DA value greater than 0.7) and poorly evaluated for very strong, substantial or moderate disagreement (DA lower than 0.3). The result was considered inconclusive for the other options (low or negligible agreements or disagreements, with DA values between 0.3 and 0.7).

This study was included in a project submitted and approved by the Research Ethics Committee at the Hospital das Clínicas of the College of Medicine of Ribeirão Preto, University of São Paulo (CAAE/Conep:21337513.0.0000.5440) and there is no conflict of interest.

Results and Discussion

The evaluation with health professionals

Interviews with SIM users at the local level occurred between November 2013 and February 2014 in the epidemiological surveillance departments of the secretaries of health in the selected cities. The application of the questionnaire was conducted personally by the researcher, with the presentation of each proposition associated with the indicator: the respondent indicated the options of answers, with opening for observations and comments.

Table 2 presents the frequency distributions for the answers to the questions that refer to the Software Product Quality (indicators 1 to 21), Quality in Use (indicators 22 to 25), and Quality of Services (indicators 26 to 28), with the respective evaluation, interpretation and result metrics for the evaluated indicators.

Thumbnail

Table 2
Distributions of frequency, metrics and evaluation of indicators – healthcare professionals

FUNCTIONAL SUPPORTABILITY

The indicators for Functional completeness and Functional correctness, the first two in table 2, were very positively evaluated, without disagreement. Observe that for the Functional Correctness, 47.1% of users say SIM has some degree of incorrect information, especially related to causes of death. According to the respondents, this degree of incorrectness is primarily associated with registering the causes of mortality based on death certificates, which affect the quality of information maintained by the system.

With regard to the quality of data maintained by the system, associated with functional correctness, records of deaths with undefined, erroneous or incomplete causes are frequently reported in the literature (FRANÇA , 2013FRANÇA, E. B. et al. Strengthening vital statistics in Brazil: investigation of ill-defined causes of death and implications on mortality statistics. The Lancet, London, v. 381, supl. 2, p. 51-51, jun. 2013., BARBUSCIA; RODRIGUES JÚNIOR, 2011BARBUSCIA, D. M.; RODRIGUES JÚNIOR, A. L. Completude da informação nas declarações de nascido vivo e nas declarações de óbito, neonatal precoce e fetal, da região de Ribeirão Preto, São Paulo, Brasil, 2000-2007. Cad. Saúde Pública, Rio de Janeiro, v. 27, n. 6, p. 1192-1200, jun. 2011., COSTA; FRIAS, 2011COSTA, J. M. B. S.; FRIAS, P. G. Avaliação da completitude das variáveis da declaração de óbitos de menores de um ano residentes em Pernambuco, 1997-2005. Ciênci saúde coletiva, Rio de Janeiro, v. 16, supl. 1, p. 1267-1274, 2011.).

Advances in the improvement of data quality in SIM include greater commitment by the medical professional (MELLO-JORGE; LAURENT; GOTLIEB, 2007MELLO-JORGE, M. H. P.; LAURENTI, R.; GOTLIEB, S. L. D. Análise da qualidade das estatísticas vitais brasileiras: a experiência de implantação do SIM e do SINASC. Ciência saúde coletiva, Rio de Janeiro, v. 12, n. 3, p. 643-654, jun. 2007., CASCÃO; COSTA; KALE, 2012CASCÃO, A. M.; COSTA, A. J. L.; KALE, P. L. Qualidade da informação sobre mortalidade numa coorte de diabéticos - Estado do Rio de Janeiro, 2000 a 2003. Rev. bras. epidemiol., São Paulo, v. 15, n. 1, p. 134-142, mar. 2012.) and by the professional who work at encrypting and feeding the system (COSTA; FRIAS, 2011COSTA, J. M. B. S.; FRIAS, P. G. Avaliação da completitude das variáveis da declaração de óbitos de menores de um ano residentes em Pernambuco, 1997-2005. Ciênci saúde coletiva, Rio de Janeiro, v. 16, supl. 1, p. 1267-1274, 2011., BRAZ , 2013BRAZ, R. M. et al. Avaliação da completude da variável raça/cor nos sistemas nacionais de informação em saúde para aferição da equidade étnico-racial em indicadores usados pelo Índice de Desempenho do Sistema Único de Saúde. Saúde debate, Rio de Janeiro, v. 37, n. 99, p. 554-562, 2013.) and a better integration with other public agencies that have interface with healthcare information systems and services (MELO; VALONGUEIRO, 2015MELO, G. B.; VALONGUEIRO, S. Incompletude dos registros de óbitos por causas externas no Sistema de Informações sobre Mortalidade em Pernambuco, Brasil, 2000-2002 e 2008-2010. Epidemiol. Serv. Saúde, Brasília, DF, v. 24, n. 4, p. 651-660, dez. 2015.). One technical improvement that can reduce incorrect inputs in the system is the inclusion of rules for validating motives of death related to gender, age and context of the death.

For premature deaths, Costa and Frias (2011)COSTA, J. M. B. S.; FRIAS, P. G. Avaliação da completitude das variáveis da declaração de óbitos de menores de um ano residentes em Pernambuco, 1997-2005. Ciênci saúde coletiva, Rio de Janeiro, v. 16, supl. 1, p. 1267-1274, 2011. also report a strategy that has proven efficient in improving how to complete the DCs: the action by Infant Death Surveillance and the Infant Death Prevention Committees, organized by public authorities and civil society, which promote the systematic investigation and discussion of these cases of death within the ambit of cities.

PERFORMANCE EFFICIENCY

The Performance Efficiency characteristic was evaluated satisfactorily by the users, for the local module as well as for the SIM web module, for the three behavior indicators in relation to when they were analyzed, for on-line, lot and authentication operations (indicators 3, 4 and 5 in table 2). The performance problems reported by users mainly relate to deficiencies in the internal network structure and access to the Internet in some cities, but they do not compromise system operability.

USABILITY

For Usability indicators (questions 6 to 16), responses suggest that:

the system has user accessible information; however, you cannot infer whether the system is easy to learn, without long training, with the support of available documentation. These results can be partially justified because, while some users were trained and had the support of professionals in learning system use, such as in encrypting cases of death, others learned without any resources, with deficient or no support,
the system has good operability, with easy and standardized operation, access and navigation, and its functions offer adequate operations feedback to the user, for making corrections easily and helping the user to prevent errors, validating input values and blocking invalid operations and
although users stated that the system has uniform and standardized interfaces, interface aesthetics were poorly evaluated in user flexibility for adjusting field layouts, as well as accessibility, because there were no features for users with special needs or the elderly.

RELIABILITY

In evaluating Reliability attributes (questions 17 to 21), users diverge about the frequency of maintenance for error correction, and the stability of new versions, and they gave a negative evaluation to the prevalence of errors during system operation. Some factors that may be associated with the poor evaluation of these indicators, based on reports, include training and time of system use, available infrastructure for system operation, data quality and availability of user support.

Two other reliability indicators were well evaluated: results show an appropriate degree of availability for the system for the local and the web module for accessing the national database, and the system has reliable backup and restoration resources that guarantee support in cases of failure.

QUALITY IN USE

The results of SIM effectiveness in supporting user activities (questions 22 to 25) also suggest that the system helps reduce the professional's work time and is efficient in information storage and recovery.

QUALITY OF SERVICES

The indicators of Quality of Services (questions 26 to 28), to evaluate the infrastructure to support the user in training and support to solve problems, presented the following results: (a) the majority of users declared themselves satisfied with the training and qualification of the users of the system, although part of this majority evaluates that it is not adequate and (b) the results for the indicators associated with the infrastructure for user service suggest that the technical support team for the system is effective and qualified to help professionals with difficulties in using the system, although the on-line/help-desk service structure is not ideal: users reported they were rarely readily served, the professional was not always available for providing service and there were often delays in feedback for solving pending issues.

EVALUATION OF HEALTHCARE MANAGERS

Interviews with healthcare managers who use SIM occurred between January and March 2014. The interviews took place in the coordinations of epidemiological surveillance departments of the municipal health departments (municipal managers), epidemiological surveillance groups of regional health departments (regional managers) and the Directorate of the Strategic Information Center on Health Surveillance of the State Secretariat of Health (state manager), after a previous scheduling.

The tabulation of quantitative and qualitative results for the questionnaires is shown in table 3, which includes the frequency distributions for the answers to the questions that refer to the Software Product Quality (indicators 1 to 10), Quality in Use (indicators 11 to 16) and Service Quality (indicator 17), with the respective evaluation, interpretation and result metrics for the evaluated indicators.

Thumbnail

Table 3
Distributions of frequency, metrics and evaluation of indicators – healthcare managers

FUNCTIONAL SUPPORTABILITY

Indicators for functional completeness (questions 1 to 3) were positively evaluated, with a predominance of answers of agreement. The respondents commented some observations:

There is a demand for decision-making reports that show consolidated data. Many reports are difficult to extract and do not contain summarized data, which requires manager rework in processing data on electronic spreadsheets;
With regard to legal standards and standardized tables, inconsistencies were reported associated with CID10, Brazilian Classification of Occupations (CBO) and street addresses;
Reports and consultations to the system do not include medical chart data and require manual investigation in other systems.

Indicator 4, for the Functional Adequacy was inconclusive: although 50% of those interviewed completely agree SIM integrates the areas/departments encompassed by the system, 30% indicated a low degree of integration such as between teams that work integrated with grievance and notification teams, that uses Sinan system.

PERFORMANCE EFFICIENCY

SIM users evaluated the Performance Efficiency characteristic satisfactorily, with 100% in agreement, as suggested by the results for indicator 5. Performance problems pointed out by users basically refer to deficiencies in network structure and access to the Internet, but they do not compromise system operation.

COMPATIBILIDADE

Compatibility indicators - Coexistence and Interoperability - were poorly evaluated by respondents, with 87.5 and 80% in disagreement, respectively, according to results for indicators 6 and 7. The users reported that:

- There is no integration with applications that register clinical data for the dead person (as a patient);
- There is not integration with the Information System for Notifiable Diseases (Sinan);
- There are deficiencies in integration with the Live Births System (Sinasc);
- There are demands for more effective integration between the SIM local and web modules;
- Crossing data for managerial reports is obtained manually from several sources/ systems.

The respondents also observed the integration between applications, through data-linkages, is impracticable due to restrictions in access to information laws, which limit the identification of public records on population data up to the district level. Despite this limitation, the development of some specific linkages could help manage access to integrated information from multiple sources.

The indicators for Compatibility attributes were poorly evaluated, which confirms reports in literature about the lack of integration between applications, fragmentation and duplicity of information (THAINES , 2009THAINES, G. H. L. S. et al. Produção, fluxo e análise de dados do sistema de informação em saúde: um caso exemplar. Texto & Contexto Enfermagem, Florianópolis, v. 18, n. 3, p. 466-474, set. 2009., DAMÉ , 2011DAMÉ, P. K. V. et al. Sistema de Vigilância Alimentar e Nutricional (SISVAN) em crianças do Rio Grande do Sul, Brasil: cobertura, estado nutricional e confiabilidade dos dados. Cad. Saúde Pública, Rio de Janeiro, v. 27, n. 11, p. 2155-2165, nov. 2011.). Several actions by the Ministry of Health seek to promote regulations on Interoperability standards, such as Ordinance 2.073/2011, the institutionalization of CIINFO (Healthcare Information and Information Technology Committee), management and improvement of PNIIS (National Healthcare Information and Information Technology Policy) and standardization of technologies (Ministry of Health Ordinances 2.466/2009, 2.072/2011 and 188/2012), but effective integration between applications is still a challenge to be solved.

USABILITY

Two Usability indicators (questions 8 and 9) were evaluated: (a) indicator 8, which checks system flexibility, was poorly evaluated by respondents, interviewed said the SIM does not have a modifiable interface, but rather a fixed standard based on forms approved by the system management committee; therefore, there is no flexibility for customization; (b) indicator 9, associated with interface aesthetics, checks to what degree the system presents clear forms, reports, screens and graphics, with understandable and unambiguous terms. It was evaluated positively, with 90% agreement.

SAFETY

The accountability and traceability of access to operations indicator (question 10) was evaluated positively: those interviewed said the system maintains logs of events executed by users at the local and the web module levels. Access controls are recorded with user data registered by managers who attend to safety requirements for the system.

QUALITY IN USE

For Quality in Use, effectiveness indicator - which refers to the degree the system permits users to access related information during use was positively evaluated, as well as the indicators for satisfaction, utility and coverage of context.

QUALITY OF SERVICES

A single indicator was evaluated for Quality of services (question 17): if the processing of modifications required for the system is adequate in relation to time, modification results and opening to user participation in progressive maintenance processes. In the results, 44.4% agree with the affirmation, whereas 55.6% disagree. The negative evaluations for this indicator were justified by respondents with arguments that the periods in which the required modifications are handled are very long; the modifications implemented do not always meet user needs and the decision on priorities does not involve users at the local and state levels.

Conclusions

Although the literature reports problems associated with coverage, incomplete data, integration and support for management, for SIM (FRANÇA , 2013FRANÇA, E. B. et al. Strengthening vital statistics in Brazil: investigation of ill-defined causes of death and implications on mortality statistics. The Lancet, London, v. 381, supl. 2, p. 51-51, jun. 2013., SIVIERO , 2013SIVIERO, P. et al. Indicador de subnotificação de óbitos no Sistema de Informação de Mortalidade no Brasil obtido de pacientes que morreram por doença renal crônica terminal: mensuração baseada nas Autorizações de Procedimentos de Alta Complexidade de 2000 a 2004. Cad. saúde colet., v. 21, n. 1, p. 92-95, mar. 2013., BARBUSCIA; RODRIGUES JÚNIOR, 2011BARBUSCIA, D. M.; RODRIGUES JÚNIOR, A. L. Completude da informação nas declarações de nascido vivo e nas declarações de óbito, neonatal precoce e fetal, da região de Ribeirão Preto, São Paulo, Brasil, 2000-2007. Cad. Saúde Pública, Rio de Janeiro, v. 27, n. 6, p. 1192-1200, jun. 2011.; MOTA, 2009MOTA, F. R. L. Registro de informação no sistema de informação em saúde: um estudo das bases SINASC, SIAB e SIM no estado de Alagoas. 2009. Tese (Doutorado em Ciência da Informação) - Escola da Ciência da Informação, Universidade Federal de Minas Gerais, Belo Horizonte, 2009.; BRAZ 2013BRAZ, R. M. et al. Avaliação da completude da variável raça/cor nos sistemas nacionais de informação em saúde para aferição da equidade étnico-racial em indicadores usados pelo Índice de Desempenho do Sistema Único de Saúde. Saúde debate, Rio de Janeiro, v. 37, n. 99, p. 554-562, 2013.), in absolute numbers, SIM was well evaluated by both user profiles in this process. For the 45 attributes analyzed for system quality, there were 31 positive evaluations and six negative evaluations, while eight indicators presented divergent or inconclusive results.

The evaluation indicated that the SIM has features that meet user needs; it has adequate performance; and, despite deficiencies in adaptations and interface accessibility, usability was well evaluated. With regard to Reliability, the evaluation indicated the system is available and has mechanisms that guarantee data recovery in fault situations; however, according to respondents, the system has a prevalence of errors and instability in new versions, which compromises normal operations. Safety was also well evaluated. The system enables auditing and tracking of accesses and operations performed by its users.

Interoperability indicators were poorly evaluated, confirming reports in the literature about lack of integration, fragmentation and duplicity of information on information systems for SUS. The quality of data maintained by the system was also one of the problems pointed out by users, due to records of deaths with undefined, misinformed or incomplete causes.

In relation to Quality in Use, the results of the analyzed indicators suggest the SIM has been effective in supporting management, with positive ratings above 70%.

For the Quality of Services dimension, the indicators revealed deficiencies in user support and service. The training and empowerment of professionals who operate the SIM in cities was one of the demands identified in this study: the training of professionals was seen as heterogeneous and investments in qualification could bring gains in the quality of data maintained by the system.

User support infrastructure also lacks the organization of better-structured remote service that offers feedback that is more effective to the user. Results also indicate there is no adequate management of modifications in the system, in relation to time, content of alterations and opening for user participation.

In short, it was possible to evaluate the SIM with its main users, identify the attributes the system has that are adequate and those that need improvement. The positive evaluation for SIM does not extend to others applications. It must be remembered that SIM has considerable stability and that it was the first public healthcare information system implemented in Brazil, back in 1976.

In a recent study, Santos and Teixeira (2016)SANTOS, J. S.; TEIXEIRA, C. F. Política de saúde no Brasil: produção científica 1988-2014. Saúde debate, Rio de Janeiro, v. 40, n. 108, p. 219-230, jan./mar. 2016. carried out a systematic review of studies in the literature related to health policies within the scope of the Ministry of Health between 1998 and 2014. Among the findings of the research, only 2 articles on information systems were found in a universe of 377 studies (0.5%), which shows the lack of literature in this area and a potential for future research.

The process described in this paper may help other evaluation projects for healthcare information systems. It may be totally or partially replicated for other applications and be useful as another reference for studies that involve evaluation processes of the technical quality of software in healthcare or other areas, with adaptations.

Financial support: non-existent

References

BARBUSCIA, D. M.; RODRIGUES JÚNIOR, A. L. Completude da informação nas declarações de nascido vivo e nas declarações de óbito, neonatal precoce e fetal, da região de Ribeirão Preto, São Paulo, Brasil, 2000-2007. Cad. Saúde Pública, Rio de Janeiro, v. 27, n. 6, p. 1192-1200, jun. 2011.
BRAZ, R. M. et al Avaliação da completude da variável raça/cor nos sistemas nacionais de informação em saúde para aferição da equidade étnico-racial em indicadores usados pelo Índice de Desempenho do Sistema Único de Saúde. Saúde debate, Rio de Janeiro, v. 37, n. 99, p. 554-562, 2013.
CASCÃO, A. M.; COSTA, A. J. L.; KALE, P. L. Qualidade da informação sobre mortalidade numa coorte de diabéticos - Estado do Rio de Janeiro, 2000 a 2003. Rev. bras. epidemiol., São Paulo, v. 15, n. 1, p. 134-142, mar. 2012.
COSTA, J. M. B. S.; FRIAS, P. G. Avaliação da completitude das variáveis da declaração de óbitos de menores de um ano residentes em Pernambuco, 1997-2005. Ciênci saúde coletiva, Rio de Janeiro, v. 16, supl. 1, p. 1267-1274, 2011.
DAMÉ, P. K. V. et al Sistema de Vigilância Alimentar e Nutricional (SISVAN) em crianças do Rio Grande do Sul, Brasil: cobertura, estado nutricional e confiabilidade dos dados. Cad. Saúde Pública, Rio de Janeiro, v. 27, n. 11, p. 2155-2165, nov. 2011.
FARIAS, L. M. M. et al Os limites e possibilidades do Sistema de Informação da Esquistossomose (SISPCE) para a vigilância e ações de controle. Cad. Saúde Pública, Rio de Janeiro, v. 27, n. 10, p. 2055-2062, out. 2011.
FIGUEIROA, B. Q. et al Análise da cobertura do Sistema de Informações sobre Mortalidade em Olinda, Pernambuco, Brasil. Cad. Saúde Pública, Rio de Janeiro, v. 29, n. 3, p. 475-484, mar. 2013.
FRANÇA, E. B. et al Strengthening vital statistics in Brazil: investigation of ill-defined causes of death and implications on mortality statistics. The Lancet, London, v. 381, supl. 2, p. 51-51, jun. 2013.
LIMA, C. R. A. et al Revisão das dimensões de qualidade dos dados e métodos aplicados na avaliação dos sistemas de informação em saúde. Cad. Saúde Pública, Rio de Janeiro, v. 25, n. 10, p. 2095-2109, out. 2009.
LIMA, E. E. C.; QUEIROZ, B. L. A evolução do sub-registro de mortes e causas de óbitos mal definidas em Minas Gerais: diferenciais regionais. Rev. bras. estud. popul, Rio de Janeiro, v. 28, n. 2, p. 303-320, dez. 2011.
MAHAPATRA, P. et al Civil registration systems and vital statistics: successes and missed opportunities. The Lancet, London, v. 370, n. 9599, p. 1653-1663, nov. 2007.
MATHERS, C. D. et al Counting the dead and what they died from: an assessment of the global status of cause of death data. Bull World Health Organ, Genebra, v. 83, n. 3, p. 171-177, mar. 2005.
MELLO-JORGE, M. H. P.; LAURENTI, R.; GOTLIEB, S. L. D. Análise da qualidade das estatísticas vitais brasileiras: a experiência de implantação do SIM e do SINASC. Ciência saúde coletiva, Rio de Janeiro, v. 12, n. 3, p. 643-654, jun. 2007.
MELO, G. B.; VALONGUEIRO, S. Incompletude dos registros de óbitos por causas externas no Sistema de Informações sobre Mortalidade em Pernambuco, Brasil, 2000-2002 e 2008-2010. Epidemiol. Serv. Saúde, Brasília, DF, v. 24, n. 4, p. 651-660, dez. 2015.
MORAES, I. H. S. Sala de situação em saúde: contribuição à ampliação da capacidade gestora do Estado? In: ORGANIZAÇÃO PAN-AMERICANA DA SAÚDE. Sala de situação em saúde: compartilhando as experiências do Brasil. Brasília, DF: Organização Pan-Americana da Saúde, Ministério da Saúde, 2010.
MORAIS, R. M.; COSTA, A. L. Um modelo para avaliação dos sistemas de informação do SUS de abrangência nacional: o processo de seleção e estruturação de indicadores. Revista de Adm. Pública, Rio de Janeiro, v. 48, n. 3, p. 377-385, jun. 2014.
MOTA, F. R. L. Registro de informação no sistema de informação em saúde: um estudo das bases SINASC, SIAB e SIM no estado de Alagoas. 2009. Tese (Doutorado em Ciência da Informação) - Escola da Ciência da Informação, Universidade Federal de Minas Gerais, Belo Horizonte, 2009.
SANCHES, C.; MEIRELES, M.; DE SORDI, J. O. Análise qualitativa por meio da Lógica Paraconsistente: método de interpretação e síntese de informação obtida por escalas Likert. In: ENCONTRO DE PESQUISA EM ADMINISTRAÇÃO E CONTABILIDADE DA ANPAD (ENEPQ), 3., 2011, João Pessoa. Anais... João Pessoa, 2011. Disponível em: <http://www.anpad.org.br/diversos/trabalhos/EnEPQ/enepq_2011/ENEPQ221.pdf>. Acesso em: 24 abr. 2014.
» http://www.anpad.org.br/diversos/trabalhos/EnEPQ/enepq_2011/ENEPQ221.pdf
SANTOS, J. S.; TEIXEIRA, C. F. Política de saúde no Brasil: produção científica 1988-2014. Saúde debate, Rio de Janeiro, v. 40, n. 108, p. 219-230, jan./mar. 2016.
SIVIERO, P. et al Indicador de subnotificação de óbitos no Sistema de Informação de Mortalidade no Brasil obtido de pacientes que morreram por doença renal crônica terminal: mensuração baseada nas Autorizações de Procedimentos de Alta Complexidade de 2000 a 2004. Cad. saúde colet., v. 21, n. 1, p. 92-95, mar. 2013.
THAINES, G. H. L. S. et al Produção, fluxo e análise de dados do sistema de informação em saúde: um caso exemplar. Texto & Contexto Enfermagem, Florianópolis, v. 18, n. 3, p. 466-474, set. 2009.
VIDOR, A. C.; FISHER, P. D.; BORDIN, R. Utilização dos sistemas de informação em saúde em municípios gaúchos de pequeno porte. Revista de Saúde Pública, São Paulo, v. 45, n. 1, p. 24-30, 2011.

Publication Dates

Publication in this collection
Mar 2017

History

Received
Apr 2016
Accepted
Oct 2016

Este é um artigo publicado em acesso aberto (Open Access) sob a licença Creative Commons Attribution, que permite uso, distribuição e reprodução em qualquer meio, sem restrições desde que o trabalho original seja corretamente citado.

[1] Financial support: non-existent

Dimension	Characteristics	Sub-characteristics
Product quality	Functional supportability: capacity of the software product to provide functions to meet explicit and implicit needs for which it has been conceived.	Functional completeness: capacity of the software product to provide an appropriate set of functions for specified tasks and objectives of the user.
		Functional correctness: capacity of the software product to provide, with the necessary degree of precision, correct or as agreed results or effects.
		Functional adequacy: capacity of the software product to facilitate the performance of user tasks and objectives.
	Performance efficiency: capacity of the software product to maintain a proper level of performance when used in specified conditions.	Behavior in relation to time: capacity of the software product to provide fitting response and processing times, when the software performs its functions under established conditions.
		Use of resources: capacity of the software product to use fitting types and quantities of resources, when performing its functions under established conditions.
		Capacity: Maximum limits of system parameters (items that can be stored, number of competing users, bandwidth, transaction speed, size of database etc.) that meet requirements.
	Compatibility: capacity of the software product to enable an exchange of information with other applications and/or share the same hardware or software environment.	Coexistence: capacity of the software product to coexist with other independent software products in a common environment and sharing common resources.
		Interoperability: capacity of the software product to interact with one or more specified systems through an exchange of information and the use of the information that is exchanged.
Product quality	Usability: capacity of the software product to be understood, learned, operated and attractive to the user, effectively and efficiently, when used under specified conditions.	Intelligibility: capacity of the software product to enable the user to understand whether the software is fitting and how it can be used for specific-use tasks and conditions. Depends on software documentation.
		Aprehendability: capacity of the software product to enable the user to learn how to use it. Depends on software documentation.
		Operability: capacity of the software product to enable the user to operate and control it easily.
		User error protection: capacity of the software product to protect the user from errors.
		Interface aesthetics with the user: capacity of the software product to be attractive to the user by offering an interface with pleasant interaction.
		Accessibility: capacity of the software product to be used by a broad spectrum of people, including those with special needs and age-associated limitations.
	Reliability: capacity of the software product to execute its functions continuously.	Maturity: capacity of the software product to avoid failures resulting from defects in the software, maintaining normal operations.
		Availability: capacity of the software product to be operational and accessible when its use is required.
		Failure tolerance: capacity of the software product to operate at a specified level of performance in cases of defects in software or hardware.
		Recoverability: capacity of the software product to reestablish its specified level of performance and recover the data directly affected in the case of a failure.
	Safety: capacity of the software product to protect information and data – unauthorized people or systems cannot read them or modify them and access to authorized people and systems is denied.	Confidentiality: capacity of the software product to guarantee the data will be accessible only to people authorized to have access to them.
		Integrity: capacity of the software product to avoid unauthorized access for access to or modification of programs or data.
		Non-questioning: capacity of the software product to guarantee that the occurrence of actions or events can be proved, thus avoiding future questioning.
		Responsibility: capacity of the software product to audit the traceability of access to operations.
		Authentication: capacity of the system to validate the identity of a user.
	Maintainability: capacity of the software product to be modified. Modifications can include corrections, improvements or adaptation of the software due to changes in the environment and in functional requirements or specifications.	Modularity: capacity of the system to have discrete components so a modification in one component has minimal impact on other components.
		Reusability: capacity of software components to be used in other software or in the construction of other components/systems.
		Analyzability: capacity of the software product to allow the diagnosis of deficiencies or causes of failures, or the identification of parts to be modified.
		Modifiability: capacity of the software product to allow implementation of a specified modification.
		Testability: capacity of the software product to allow the validation of the software when it has been modified.
Product quality	Portability: capacity of the software product to be transferred from one environment to another.	Adaptability: capacity of the software product to be adapted to different specified environments, without the need for other actions or means besides those provided for this purpose by the software in question.
		Capacity to be installed: capacity of the software product to be installed in a specified environment.
		Capacity to replace: capacity of the software product to be used to replace another specified software product, with the same purpose and in the same environment.
Quality of use	Effectiveness: capacity of the software product to allow users to achieve specified goals with accuracy and completeness in a specified use context.
	Efficiency: capacity of the software product to enable users to employ an appropriate quantity of resources in relation to effectiveness obtained, in a specified use context.
	Satisfaction: capacity of the software product to satisfy users in a specified use context.	Utility: degree to which the user perceives the software helps in executing his activities.
		Credibility: degree of confidence the user has in the correct behavior of the system.
		Pleasantness: degree of user contentment in using the software.
		Comfort: degree of physical comfort in using the software.
	Absence of risk: capacity of the software product to present acceptable levels of risk of damage to people, business, property or the environment.	Mitigation of economic risk: degree in which the software reduces potential risk of a financial, operations, property or reputation nature in its context of use.
		Mitigation of risks to safety and health: degree in which the software reduces potential risk in its context of use.
		Mitigation of environmental risks: degree in which the software reduces potential risk to the environment in its context of use.
	Coverage of context: capacity of the software product to be used in its context of use and beyond those initially specified.	Completeness of context: degree in which the software is used in all contexts of use (for example: using a low resolution monitor, with a low access to network rate, by an inexperienced user or without access to the network).
		Flexibility: degree in which the software is used beyond its projected context of use (if software is not designed for flexibility, it may not be safe in unplanned contexts).
Quality of services: The degree to which the software product provides infrastructure for user support, in capacitation and support for the solution of problems regarding the system.

DA value	Qualitative interpretation
between 0.9 and 1.0	very strong agreement
between 0.8 and 0.9	substantial agreement
between 0.7 and 0.8	moderate agreement
between 0.6 and 0.7	low agreement
between 0.5 and 0.6	negligible agreement
between 0.4 and 0.5	negligible disagreement
between 0.3 and 0.4	low disagreement
between 0.2 and 0.3	moderate disagreement
between 0.1 and 0.2	substantial disagreement
between 0.0 and 0.1	very strong disagreement

Indicator	Agree				Desagree				Metric (DA)	Qualitative interpretation	Evaluation of Indicator
	completely		partially		partially		completely
	N	%	N	%	N	%	N	%
1. The system has functions that provide and treat the information I need/ use.	13	72,2%	5	27,8%	0	0,0%	0	0,0%	0,96	very strong agreement	Well evaluated
2. The system presents correct and precise information.	10	55,6%	8	44,4%	0	0,0%	0	0,0%	0,93	very strong agreement	Well evaluated
3. Response time for an on-line operation in the system is satisfactory.	9	50,0%	8	44,4%	1	5,6%	0	0,0%	0,89	substantial agreement	Well evaluated
4. Response time for a "job" request in the system is satisfactory.	12	66,7%	6	33,3%	0	0,0%	0	0,0%	0,95	very strong agreement	Well evaluated
5. Access and authentication in the system occur in satisfactory time.	13	72,2%	5	27,8%	0	0,0%	0	0,0%	0,96	very strong agreement	Well evaluated
6. The system provides users documentation/ online assistance for its functions.	11	68,8%	2	12,5%	3	18,7%	0	0,0%	0,88	substantial agreement	Well evaluated
7. The learning of the system does not require long training.	3	16,7%	7	38,9%	3	16,7%	5	27,8%	0,48	negligible disagreement	Inconclusive
8. The system's functions are easy to operate and intuitive.	9	50,0%	5	27,8%	4	22,2%	0	0,0%	0,82	substantial agreement	Well evaluated
9. The system's functions have fast and standardized access/ navigation.	12	66,7%	6	33,3%	0	0,0%	0	0,0%	0,95	very strong agreement	Well evaluated
10. The system's functions produce appropriate feedback, with clear messages.	13	72,1%	5	27,9%	0	0,0%	0	0,0%	0,96	very strong agreement	Well evaluated
11. It is simple, easy and safe to correct an error in the system's functions: operations made by the user are reversible.	9	52,9%	6	35,3%	2	11,8%	0	0,0%	0,88	substantial agreement	Well evaluated
12. The system's functions check whether data input values are valid.	14	77,8%	4	22,2%	0	0,0%	0	0,0%	0,97	very strong agreement	Well evaluated
13. The system avoids the execution of incorrect operations.	9	52,9%	5	29,4%	1	5,9%	2	11,8%	0,79	moderate agreement	Well evaluated
14. The user can adjust the layout of fields in the system function's interfaces to suit his or her work.	0	0,0%	0	0,0%	0	0,0%	15	100,0%	0,00	very strong disagreement	Poorly evaluated
15. The system's functions have uniform and standardized interfaces.	15	88,2%	2	11,8%	0	0,0%	0	0,0%	0,98	very strong agreement	Well evaluated
16. The system includes features for users with special needs or the elderly.	0	0,0%	1	10,0%	2	20,0%	7	70,0%	0,08	very strong disagreement	Poorly evaluated
17. The system undergoes much maintenance to correct errors.	3	18,8%	5	31,2%	5	31,2%	3	18,8%	0,50	negligible agreement	Inconclusive
18. Corrections, improvements or updates in the system cause instability in the system or demand excessive effort or time.	4	25,0%	2	12,5%	4	25,0%	6	37,5%	0,40	negligible disagreement	Inconclusive
19. Errors occur during use of the system.	0	0,0%	5	29,4%	6	35,3%	6	35,3%	0,25	moderate disagreement	Poorly evaluated
20. The system is available for the user, when requested.	15	83,3%	3	16,7%	0	0,0%	0	0,0%	0,98	very strong agreement	Well evaluated
21. The system presents a low level of data loss and efficient mechanisms for restoration.	12	75,0%	2	12,5%	2	12,5%	0	0,0%	0,91	very strong agreement	Well evaluated
22. The system offers adequate support/ features for executing my tasks/ activities.	15	88,2%	2	11,8%	0	0,0%	0	0,0%	0,98	very strong agreement	Well evaluated
23. The system helps reduce my work time.	6	37,5%	7	43,8%	2	12,5%	1	6,2%	0,77	moderate agreement	Well evaluated
24. The use of the system facilitates information storage and recovery.	14	77,8%	4	22,2%	0	0,0%	0	0,0%	0,97	very strong agreement	Well evaluated
25. I do not lose time in solving technical problems associated with the system.	5	35,7%	5	35,7%	2	14,3%	2	14,3%	0,68	low agreement	Inconclusive
26. Training and qualification are offered for using the system.	3	16,7%	7	38,9%	4	22,2%	4	22,2%	0,50	negligible agreement	Inconclusive
27. There is a sized and qualified team for effective support in system use in difficult situations.	7	38,8%	9	50,0%	1	5,6%	1	5,6%	0,82	substantial agreement	Well evaluated
28. System support service includes an on-line (hot line) and/or help-desk service structure.	3	17,6%	8	47,1%	5	29,4%	1	5,9%	0,63	low agreement	Inconclusive

Indicator	Agree				Desagree				Metric (DA)	Qualitative interpretation	Evaluation of Indicator
	completely		partially		partially		completely
	N	%	N	%	N	%	N	%
1 - The system provides the functions I need to support decision-making.	5	50,0%	5	50,0%	0	0,0%	0	0,0%	0,92	very strong agreement	Well evaluated
2 - The system provides functions that obey legal information standards (CID10, DRG, data transmission etc.)	5	50,0%	5	50,0%	0	0,0%	0	0,0%	0,92	very strong agreement	Well evaluated
3 - The system's functions provide clinical/ healthcare documentation in a correct and complete manner.	5	55,6%	3	33,3%	1	11,1%	0	0,0%	0,88	substantial agreement	Well evaluated
4 - System features integrate different areas/ departments.	5	50,0%	2	20,0%	1	10,0%	2	20,0%	0,69	low agreement	Inconclusive
5 - The system generates clinical/ healthcare documentation with a satisfactory response time.	5	55,6%	4	44,4%	0	0,0%	0	0,0%	0,93	very strong agreement	Well evaluated
6 - The system provides its data to other systems that need to access it.	0	0,0%	1	12,5%	2	25,0%	5	62,5%	0,11	substantial disagreement	Poorly evaluated
7 - The system integrates with other systems through the exchange of information and the use of the information that is exchanged.	0	0,0%	2	20,0%	2	20,0%	6	60,0%	0,15	substantial disagreement	Poorly evaluated
8 - For system functions that require or demand customization by the user, the system permits adaptations to meet local/ specific needs.	0	0,0%	0	0,0%	3	30,0%	7	70,0%	0,04	very strong disagreement	Poorly evaluated
9. The system’s functions present clear interfaces (screens/forms/ data input/ reports/graphs), with understandable terms and without ambiguities.	5	50,0%	4	40,0%	1	10,0%	0	0,0%	0,88	substantial agreement	Well evaluated
10. The system’s functions register access and operations information that can be audited/ tracked in the future.	7	77,8%	1	11,1%	1	11,1%	0	0,0%	0,92	very strong agreement	Well evaluated
11. I can obtain the information I need when using the system.	7	77,8%	2	22,2%	0	0,0%	0	0,0%	0,97	very strong agreement	Well evaluated
12. The system effectively contributes to the strategic objectives of the management of professionals (medical, nursing, administrative teams etc.).	3	37,5%	4	50,0%	1	12,5%	0	0,0%	0,84	substantial agreement	Well evaluated
13. The system has been contributing to improving the quality of the offer of healthcare services.	6	66,7%	2	22,2%	0	0,0%	1	11,1%	0,85	substantial agreement	Well evaluated
14. The system has been contributing to improving the management of healthcare services.	4	44,4%	5	55,6%	0	0,0%	0	0,0%	0,91	very strong agreement	Well evaluated
15. Different groups of users are satisfied with the system.	2	20,0%	8	80,0%	0	0,0%	0	0,0%	0,86	substantial agreement	Well evaluated
16. The system operates according to its design and specifications.	6	60,0%	2	20,0%	2	20,0%	0	0,0%	0,85	substantial agreement	Well evaluated
17. The processing of change requests in the system is adequate (time, modifications, opening to user participation).	0	0,0%	4	44,4%	3	33,3%	2	22,2%	0,37	low disagreement	Inconclusive