Abstract

This study compared the reliability of a data group registered in the secondary databases of the National Registry of Health Facilities. A survey was conducted in 2,777 with hospitals to achieve this objective. Visited hospitals provided information on equipment, geographic location, operating status and number of beds. Regarding matching data between visited hospitals and the National Registry, it can be noted that the operating status was updated in 89% of cases, the number of beds in 44%, 82% had the correct amount of equipment and 63% had accurate geographic coordinates. These findings point to a good reliability of information from the National Registry of Health Facilities, regarding the compared categories, excepting for data on the number of registered beds and for some equipment. As a further development of this work, we stress the need to discuss strategies and incentives to improve the reliability of data that still have inconsistencies, in order to improve the instruments used to formulate public policies.

Key words
Information systems; Data sources; Data analysis; Hospitals with less than 100-bed capacity

Introduction

Health information systems (SIS) are defined as a set of interrelated components that collect, process, store and distribute information to support the decision-making process and to assist the organization of the health system¹1. Araújo Lima CR, Schramm JMA, Coeli CM, Silva MEM. Revisão das dimensões de qualidade dos dados e métodos aplicados na avaliação dos sistemas de informação em saúde. Cad Saude Publica 2009; 25(10):2095-2109.. Thus, it is expected that they contribute to support actions geared to improving the supply of health care²2. Safran C, Perreault LE. Management of Information in Integrated delivery networks. In: Shortliffe EH, Perreault LE editors. Medical Informatics computer applications in health care and biomedicine. 2nd ed. Berlim: Springer; 2001. p. 359-396.. Several SIS are available in Brazil, most of which are publicly accessible and administered by the Ministry of Health, through the Department of Informatics of the Unified Health System (DATASUS), whose data has guided the conduction of studies that address the analysis of epidemiological, health and service provision structure and infrastructure parameters²2. Safran C, Perreault LE. Management of Information in Integrated delivery networks. In: Shortliffe EH, Perreault LE editors. Medical Informatics computer applications in health care and biomedicine. 2nd ed. Berlim: Springer; 2001. p. 359-396..

Araújo Lima et al.¹1. Araújo Lima CR, Schramm JMA, Coeli CM, Silva MEM. Revisão das dimensões de qualidade dos dados e métodos aplicados na avaliação dos sistemas de informação em saúde. Cad Saude Publica 2009; 25(10):2095-2109. point out that managers and society appropriation of data conveyed in these SIS must consider the strengths and limitations of these systems. They also emphasize that the knowledge about the quality of data organized in these databases can only arise from regular and systematic evaluations of the data made available. The lack of assurance that data from these systems portray the reality compromises the processes of policy formulation and production of knowledge at their core³3. Rede Interagencial de Informações para a Saúde. Indicadores básicos de saúde no Brasil: conceitos e aplicações. Brasília: Organização Pan-Americana da Saúde; 2002..

The systematic review of literature by Araújo Lima et al.¹1. Araújo Lima CR, Schramm JMA, Coeli CM, Silva MEM. Revisão das dimensões de qualidade dos dados e métodos aplicados na avaliação dos sistemas de informação em saúde. Cad Saude Publica 2009; 25(10):2095-2109. addressed the SIS on the understanding of the concept of quality applied to them, highlighting the following realms: reliability, completeness, coverage, validity, timeliness, non-duplicity, consistency, accessibility and methodological clarity. Results pointed out that the SIS were being used for academic and public management purposes, but not all of them had been evaluated regarding the quality of their data. In addition, there was a geographical imbalance in relation to the federation units whose data were evaluated, with only a few being considered.

Evidence such as these has raised the importance of expanding SIS quality analyses that play a relevant role in the Unified Health System (SUS), such as the National Registry of Health Establishments (CNES), a database that contains data on all the Brazilian health institutions. An establishment is included in the CNES by filling in specific forms with data on physical area, human resources, equipment and outpatient and hospital services in operation, regardless of whether or not they provide care to SUS users. Once registered, the Ministry of Health generates a numerical code for each establishment. The managers responsible for each institution may request changes or even their exclusion from the CNES database.

CNES data are important for health planning, control and evaluation and should reflect the real situation of the health system⁴4. Amorim AS, Pinto Junior VL, Shimizu HE. O desafio da gestão de equipamentos médico-hospitalares no Sistema Único de Saúde. Saúde debate 2015; 39(105):350-362.. However, the scarce recent studies which addressed elements of the CNES revealed inconsistencies in the database with the potential of negatively affecting eventual analyses developed with it.

Matos and Pompeu⁵5. Matos CA, Pompeu JC. Onde estão os contratos? Análise da relação entre os prestadores privados de serviços de saúde e o SUS. Cien Saude Colet 2003; 8(2):621-628., when analyzing the contractual situation of the private healthcare network linked to the SUS, based on CNES data, identified inconsistencies in the number of registered contracts, considering that contracts signed between 1950 and 1980 and not renewed were still there. Discrepancies in the CNES registry were also sources of limitations for the National Health Service Evaluation Program (PNASS) from 2004 to 2006, such as, for example, registrations of High Complexity Centers in Oncology that did not even have chemotherapy, radiotherapy or oncologic surgery activities⁶6. Brasil. Ministério da Saúde (MS). Programa Nacional de Avaliação de Serviços de Saúde-PNASS: resultado do processo avaliativo 2004-2006. Brasília: MS; 2007.. Data available in the system were from the last update of the registry made by the establishment and were outdated⁶6. Brasil. Ministério da Saúde (MS). Programa Nacional de Avaliação de Serviços de Saúde-PNASS: resultado do processo avaliativo 2004-2006. Brasília: MS; 2007..

Santos et al.⁷7. Santos FAZ, Gouveia, GC, Martelli PJ, Vasconcelos EMR. Acupuntura no sistema único de saúde e a inserção de profissionais não-médicos. Rev. Bras. Fisioter São Carlos 2009; 13(4)330-334. and Costa et al.⁸8. Costa LR, Costa JLR, Oishi J, Driusso P. Distribuição de fisioterapeutas entre estabelecimentos públicos e privados nos diferentes níveis de complexidade de atenção à saúde. Rev. Bras. Fisioter São Carlos 2012; 16(5):422-430. carried out studies using data on the distribution of professionals, registered in the CNES by establishment, and were able to verify the limitations regarding data fragility for a portion of the analyzed establishments. Medeiros and Calvo⁹9. Medeiros GAR, Calvo MCM. Serviços Públicos de Média Complexidade Ambulatorial em Fisioterapia Vinculados ao Sistema Único de Saúde em Santa Catarina. Revista de Saúde Pública de Santa Catarina 2014; 7(2):7-16., when describing the distribution of public physiotherapy services registered in the outpatient medium complexity in Santa Catarina, observed that the CNES was outdated and not completed, mainly in relation to the service's telephone number, number of professionals and type of equipment.

The publication of Ordinance N° 134 of April 4, 2011¹⁰10. Brasil. Portaria n° 134, de 04 de abril de 2011. Dispõe sobre responsabilidade dos gestores municipais, estaduais e do Distrito Federal/DF, bem como dos gerentes de todos os estabelecimentos de saúde na correta inserção, manutenção e atualização sistemática dos cadastros no SCNES dos profissionais de saúde em exercício nos seus respectivos serviços de saúde, públicos e privados. Diário Oficial da União, 2011; 4 abr. was an attempt to minimize some of the deficiencies reported in the CNES, since it stipulated new registration rules in order to minimize irregularities, among them all public links of professionals. Ordinance N° 118, dated February 18, 2014¹¹11. Brasil. Portaria n° 118, de 18 de fevereiro de 2014. Dispõe sobre a desativação automaticamente no Cadastro Nacional de Estabelecimentos de Saúde (SCNES) os Estabelecimentos de Saúde que estejam há mais de 6 (seis) meses sem atualização cadastral. Diário Oficial da União, 2014; 18 fev. provides for the automatic CNES deactivation of establishments that do not update their registry every six months, seeking to ensure that data are closer to reality. However, despite these efforts, there is a need to broaden actions to assess the reliability of data provided by CNES.

The comparison of the data registered in the CNES, with those observed in loco can contribute to the identification of criteria with low reliability and direct improvements. We did not observe in the national literature studies that used this comparative strategy in the evaluation of data quality, especially regarding the realm of reliability. Thus, this study aims to compare what was registered in the CNES with what can be observed in small Brazilian hospitals. The choice of this typology of health facilities drew from a census study¹²12. Barbosa AC, Rocha TAH, Silva NC, Thumé E, Facchini LA, Vasconcelos PA, Rocha V, Rocha JVM, De Almeida DG. Análise de desempenho de instituições hospitalares de pequeno porte brasileiras: diagnóstico, avaliação e espacialização. Belo Horizonte: Imprensa Universitária da Universidade Federal de Minas Gerais; 2015., which ended up establishing comparison criteria in relation to CNES for the whole set of health facilities in the country. In a health system, the existence of small hospitals is justified mainly for the implementation of low complexity actions, but not primary health care actions and actions related to urgent/emergency and mother and child care¹³13. Posnett J. The hospital of the future Is bigger better? Concentration in the provision of secondary care. BMJ 1999; 319(7216):1063-1065. services, especially in remote and hard-to-access locations.

The data set obtained in this study allowed discussing the reliability of the CNES data when comparing: operating status / participation of the establishment, number of beds, availability of equipment and geographic location. The hypothesis is that there is a discrepancy between data recorded in the CNES and those observed in loco.

Methods

This is a cross-sectional descriptive study carried out with the approval of the Research Ethics Committee of the Federal University of Pelotas (CEP/UFPEL).

All small hospitals with up to 50 bed-capacity were selected for the study, considering the records of September 2013 at CNES. According to this criterion, 3,524 hospitals distributed in the national territory were eligible for the survey. Table 1 shows the quantitative distribution of the selected establishments.

Most small hospitals are municipalized and obtaining permits to carry out the study had to be addressed on a case-by-case basis. Each facility received an invitation letter signed simultaneously by the research coordinator and by the Directorate of Hospital Care and Urgency of the Ministry of Health, requesting cooperation regarding participation in the study. In addition, there was a negotiation with the National Council of Health Secretaries (CONASS) to obtain the cooperation of the municipalities and hospitals to be visited. Since participation in the research was voluntary, only those who consented were visited. It is important to note that even the establishments that refused to participate were visited for the collection of geographical coordinates and completion of the information regarding the reason for the refusal.

Collection of primary and secondary data

Primary data were collected between February and September 2014 and secondary data obtained at CNES were related to September 2014.

The primary research goal was broader than just the characterization of existing equipment in small hospitals. Thus, more equipment was evaluated by the study in each hospital than those existing in the CNES. Data on bed capacity, geographic coordinates, quantity of facilities, work process compliance, availability of diagnostic support, financial and human resources aspects, among others, were collected locally. Thus, the selection of variables to be compared in this paper considered all the equipment that were registered in the CNES, for the hospitals visited, in both primary and secondary databases. Data extracted from the CNES, listed in Chart 1, were only related to the number of each equipment under conditions of use in each hospital.

The study's collection instrument was designed in electronic format and based on extensible markup language (XML) technology. The XML questionnaire was applied with the help of tablets and the open data kit collect (ODK collect) application. They were completed with a self-reported response and were preferably provided by the clinical director, the head of nursing or the general director of the establishment.

The electronic application was based on the benefits of the possible automated checking of data immediately after collection. All data collected that violated specific validation rules were checked. There were two verification levels: the first one, embedded in the ODK collect, only allowed the submission of complete questionnaires and with data that showed the sum of values compatible between the different sections of the questionnaire: for example, there was a field for entering total expenditure of the establishment and then, right after that, several fields for the breakdown of these expenses. The collection tool only allowed the submission of the questionnaire if the total and the sum of the detailed expenses were the same. The second level used five validation indicators: i) verification of the establishment's geographical coordinates; ii) proportion of missing values; iii) number of beds; iv) proportion of financial data completion; and v) proportion of human resources data completion.

For each of the indicators, criteria were established that, when infringed, entailed carrying out additional checks. Geographic coordinates were verified if they belonged to the municipality of the registered establishment. Mean values were defined for the share of missing, completion of financial data and human resources. These mean values were dynamic and recalculated each time a hospital's data entered the validation system, thus we aimed at defining cut-off levels reflecting the data provision profile of the hospitals that were visited. The only indicator with a fixed comparison criterion was the number of beds, and any questionnaire that reported a hospital with more than 50-bed capacity was submitted for verification. Regarding other indicators, whenever a questionnaire evidenced, for at least one of the indicators monitored, a discrepancy of three standard deviations from the mean criteria defined by data already collected, this questionnaire was automatically submitted for verification.

The entire procedure for calculating indicators, defining reference means, comparing for the selection of questionnaires to check and monitor the resolution of verifications was done automat-ically, in a system specially designed for such task. This system was web-based and provided follow-up charts of these multiple aspects to control study data validation actions.

Once listed for verification, questionnaires were entered in the definitive database only after their data veracity was ascertained. Logistical facilitators were responsible for conducting telephone checking of non-standard data, addressing respondents from the hospitals visited, as well as the interviewers responsible for the collection. If necessary, the interviewer could be asked to return to a previously visited hospital to collect data that were missing or required further details. Once the verification was made, a report was issued, detailing the justification for the apparently inconsistent information, and possible corrections were made in the electronic file containing the data by exclusive programmers. Subsequently, the revised questionnaire was incorporated into the definitive database. These procedures were designed to ensure that large differences were, in fact, a reflection of the hospitals’ situation and not stemming from collection errors.

Data analysis

The descriptive analyses performed observed aspects of data reliability, since they examined the agreement between the measurements obtained by the different sources considered. Therefore, we chose to show differences found by means of proportions.

First, the operating status of the establishment was analyzed to ascertain the existence of outdated registries in the CNES. Secondly, the situation regarding the number of beds was verified, since hospitals having less than 50 beds found in the CNES reported having more than 50 beds during the visit.

Regarding the equipment, quantities listed in the CNES were subtracted from the values found in the face-to-face survey. If no differences were found in the hospital's status regarding a specific equipment, the establishment was defined as “updated CNES”. After comparing all the equipment considered, of all hospitals, a table was drawn up with those that evidenced updated data, by equipment and administrative region.

Finally, the geographic coordinates were analyzed. CNES coordinates were obtained through the registry mirror of each hospital in the DATASUS databases, and those of the survey were collected through the GPS of the tablets used. The formula for calculating distance in large circles was used to examine the distance between both.

1. Δσ = Spherical inner angle

2. Δϕ = Latitude survey – Latitude CNES

3. ϕ_s = Latitude survey

4. ϕ_f = Latitude CNES

5. Δλ = Longitude survey – Longitude CNES

This formula is used to identify the shortest path between two points on the surface of a sphere. Thus, Earth was considered as a perfect sphere, with a radius corresponding to 6,371 km. Latitudes and longitudes collected by GPS were deemed correct and their respective distances, with those registered with the CNES, examined considering km as the reference measurement unit.

Results

According to Table 1, of the universe of 3,524 hospitals, 2,777 consented to participate in the study, of which 2,455 (88%) provided care to SUS and 322 (12%) were exclusively private.

In the country, only 1% (N = 35) of the hospitals were not found or the registered property did not exist, especially in the Federal District, Paraíba, Rio Grande do Norte, Espírito Santo and Rondônia. The number of hospitals not located or closed corresponded to a reduced percentage compared to the total number of hospitals considered.

Despite all articulation to make the study feasible, some establishments refused to participate. Private and military hospitals, for example, had a higher refusal pattern than the others. States such as São Paulo, Alagoas, Rio de Janeiro and the Federal District also fit in this situation, due to the smaller involvement of collegiate bodies that aided in the dissemination of the research.

Table 2 shows the percentage of beds identified through primary data in relation to those recorded in the CNES database. Thus, 44% of the hospitals visited showed between 91% and 110% of the registered beds of the CNES. This category includes those situations of small discrepancy between the two sources used. A small percentage of hospitals (4.5%) owned less than 50% of the CNES beds. In the Southeast region, more than 23% of the hospitals obtained values ranging from 111% to 150% of CNES beds, and in the Northeast region, more than 12% of the establishments were included in the category of more than 150% of CNES beds. Hospitals with values above 111% indicate a greater discrepancy, affecting the reliability of data on the hospitalization capacity of these establishments. Considering the country as a whole, approximately 31% of hospitals fell into categories above 111% and over 150%.

Table 3 shows equipment-related data. Considering the 39 pieces of equipment evaluated, the average number of hospitals with updated data for Brazil was 82%. The proportions of equipment analyzed revealed that the expensive one such as PET/CT, magnetic resonance imaging, hemodialysis, mammography, tomography and others showed a higher updating level than those of lower cost, such as defibrillators, AMBU, ECG monitors and electrocardiographs.

As for the geographic coordinates, Table 4, it is observed that 63% of them showed a difference less than or equal to one kilometer. Of the total, almost 10% of hospitals are more than five kilometers from what is registered in the CNES. In states with a lower population density, this percentage of imprecision increases as in the case of Acre, Amazonas, Mato Grosso, Maranhão, Pará and Roraima.

Discussion

The importance of information systems for the effective management of health services is undeniable. The possibility of obtaining accurate data about a large number of services, with the potential to characterize them in terms of infrastructure, accreditation, localization and characteristics of human resources, equips health managers with data supporting evidence-based decisions.

The initial hypothesis of this study was that data registered in the CNES evidenced lags in relation to those existing in health facilities, which can be partially observed. Different levels of divergence between CNES data and those observed at hospital visits were verified, depending on the type of information analyzed.

In fact, the most dynamic aspects in the health facilities evaluated showed a higher discrepancy pattern. It was possible to observe several closed hospitals that still appeared as assets in CNES databases. The approach to issues of this nature requires a change in the steps and procedures necessary for a given health apparatus to cease to be considered active.

The recent instructions provided by DATASUS, guiding the steps necessary to exclude an establishment from CNES, demonstrates an incipient stance to address this situation¹⁴14. Brasil. Ministério da Saúde (MS). CNES: Cadastro Nacional de Estabelecimentos de Saúde [homepage na Internet]. Brasília. [acessado 2015 ago 18]. Disponível em: http://cnes.datasus.gov.br/Cadastramento_Solicitar_Exclusao.asp
http://cnes.datasus.gov.br/Cadastramento... . Nevertheless, other initiatives are necessary such as the possible establishment of partnerships with other public entities such as commercial boards and federal revenue, so that it is only possible to consider that activities inherent to the provision of health services by an institution are terminated through a certificate of cancellation of records from information systems.

Discrepancies regarding the number of beds interfere in several areas of the formulation of public health policies. Oftentimes, these data are used as a basis for the framework of public funding lines, making health facilities, up to a certain size, eligible to claim resources from public policies¹²12. Barbosa AC, Rocha TAH, Silva NC, Thumé E, Facchini LA, Vasconcelos PA, Rocha V, Rocha JVM, De Almeida DG. Análise de desempenho de instituições hospitalares de pequeno porte brasileiras: diagnóstico, avaliação e espacialização. Belo Horizonte: Imprensa Universitária da Universidade Federal de Minas Gerais; 2015.. Errors or size inconsistencies contribute to the increasing difficulty in differentiating institutions that claim resources, promoting financing imbalances. In addition, the impossibility of estimating precisely the size of a given hospital triggers a cascade effect that negatively affects regulatory activities, since it makes it difficult to know the availability of beds in different health care networks¹²12. Barbosa AC, Rocha TAH, Silva NC, Thumé E, Facchini LA, Vasconcelos PA, Rocha V, Rocha JVM, De Almeida DG. Análise de desempenho de instituições hospitalares de pequeno porte brasileiras: diagnóstico, avaliação e espacialização. Belo Horizonte: Imprensa Universitária da Universidade Federal de Minas Gerais; 2015..

The requirement to update data according to a defined time periodicity alone is a strategy that does not ensure data quality. It makes room for timely changes, not ensuring that data reported are current, making the measure useless for handling the situation. Defining protocols or different forms of control to register criteria that evidence rapid changes, such as the existence of beds or low cost equipment can contribute to the improved reliability of the data available in this SIS.

Some hypotheses were elaborated seeking an approximation to understand these findings. The control of items of lower value, which can be purchased without the mobilization of a large volume of funds, seems to be tied to the delay in adjusting the infrastructure criteria of the CNES to the hospitals visited.

Another topic that draws attention to the reliability of equipment data is the high level of outdated information of items dedicated to support urgent and emergency care, such as: X-ray of 500 mA or less, respirator/ventilator, electrocardiograph, incubators, AMBU and ECG monitors. Thus, findings of this study raise caution regarding the use of some CNES data to analyze the service provision infrastructure.

Finally, the accuracy of the location of health establishments is fundamental so that studies are made that can mark services spatialization criteria. The structuring of care networks, coverage analysis and access demand reliable data on the existence of high-cost equipment, for example. Thus, the geographical distribution of health facilities is an important information for the organization of service supply flows.

Considering that the range of influence of a hospital often exceeds the geographical limits of the municipality of installation, the inaccuracies shown in the geographical coordinates registered in the CNES have little influence on the organization of care networks. Notwithstanding, findings of this study raise issues on whether differences greater than 2 km could not generate deleterious impact when they refer to primary care.

The importance of CNES is to gather all the health facilities in the country. More than the broad scope of coverage, this is about the relevance of high-quality data delivery. Inaccuracies in this SIS can distort information needed for managerial decision-making and undermine policy-making. The imprecise provision of geographical coordinates by the CNES may hinder the induction of studies that seek to evaluate the appropriateness of spatial allocation of health services, for example. This debate is traversed by the conduct of actions that encourage the regular and accurate feeding of data, in order to curb inadequacies.

Previous studies that focused data quality in the Brazilian SIS addressed tangentially the CNES without the possibility of comparing primary and secondary data¹1. Araújo Lima CR, Schramm JMA, Coeli CM, Silva MEM. Revisão das dimensões de qualidade dos dados e métodos aplicados na avaliação dos sistemas de informação em saúde. Cad Saude Publica 2009; 25(10):2095-2109.^,66. Brasil. Ministério da Saúde (MS). Programa Nacional de Avaliação de Serviços de Saúde-PNASS: resultado do processo avaliativo 2004-2006. Brasília: MS; 2007.^–99. Medeiros GAR, Calvo MCM. Serviços Públicos de Média Complexidade Ambulatorial em Fisioterapia Vinculados ao Sistema Único de Saúde em Santa Catarina. Revista de Saúde Pública de Santa Catarina 2014; 7(2):7-16.. There were no reports of broader studies that considered the country as a whole¹1. Araújo Lima CR, Schramm JMA, Coeli CM, Silva MEM. Revisão das dimensões de qualidade dos dados e métodos aplicados na avaliação dos sistemas de informação em saúde. Cad Saude Publica 2009; 25(10):2095-2109.. Some studies⁶6. Brasil. Ministério da Saúde (MS). Programa Nacional de Avaliação de Serviços de Saúde-PNASS: resultado do processo avaliativo 2004-2006. Brasília: MS; 2007.^,77. Santos FAZ, Gouveia, GC, Martelli PJ, Vasconcelos EMR. Acupuntura no sistema único de saúde e a inserção de profissionais não-médicos. Rev. Bras. Fisioter São Carlos 2009; 13(4)330-334. have highlighted, in some establishments and in some federal units, the outdated data, the lack of entries informing new or deactivated establishments, as well as the mismatch between the services registered and those actually in place. The findings of this study partially confirmed this evidence and raised some new questions such as the relation between the cost of the equipment and time elapsed to update the CNES or what leads some regions and states to show a higher level of imprecision regarding geographical coordinates. In spite of study limitation when not addressing issues inherent to this level of care, the findings discussed here point to the importance of carefully examining the geographical location data of health facilities, by level of care analyzed.

The examination of the PHC facilities’ location accuracy appears as a potential future work, since high discrepancies in their real location may modify access criteria, sociodemographic characteristics of the assisted population, attendance by public transportation, among other criteria that would allow better characterization of the environment of health facilities. Recommendations include the possibility of using financial incentive measures, stipulating conditions for participation in public policies aimed at financing actions, as well as the development of information quality indicators using multiple sources of data and which can evidence data validity are examples of strategies that can foster the improved reliability of CNES-linked data.

It is worth highlighting the limitations of this study, which are the impossibility of stratifying apparatuses compared in terms of their availability or not for the SUS, as well as the lack of comparisons of workforce availability aspects. With regard to the latter, the high number of missing resulting from hospitals’ lack of availability of human resources information ended up compromising the comparative possibilities of this aspect.

Due to findings, and taking into account the time required to perform primary data collection, it is possible that, for the hospitals evaluated at the beginning of the survey, there was some change in the quantity of equipment registered at the CNES, which could affect the results of these establishments. Despite challenges and limitations explained above, findings of this study seek to clarify some aspects of CNES’ reliability and may contribute to the examination of certain aspects of health information systems in the country with greater security and validity.

References

Publication Dates

History