Abstract

Introduction: The Patient Safety Self-Assessment Practices is a regulatory action to enhance quality of care. However, validation of its information requires attention. This study aimed to analyze reliability of the Patient Safety Self-Assessment Practices. Methods: Pilot study analyzing the reliability of 21 simple indicators and 1 composite of self-assessment to provide a national sample in future studies. Hospitals with intensive unit care beds participated in the study, and data from self-assessment (SA) and revised self-assessment (RSA) by the health surveillance (HS) were compared with on-site inspection (OSI). Analyses included Kappa and intraclass correlation coefficients. Results: Concordance was satisfactory (Kappa ≥ 0.4) in 12 indicators of SA and 18 indicators of RSA compared with OSI. The least reliable indicators were related to infection prevention protocols. Reliability of the adherence level composite indicator improved with HS revision (SA = 0.89 and RSA = 0.94), despite the low concordance of the high compliance classification. Conclusion: RSA was essential to improve reliability of SA. In addition, some indicators and assessment tools of the HS need revision.

Keywords:
Health surveillance; Patient safety; Health regulation and supervision.

Introduction

Millions of people around the world suffer damages during the health care process. This problem in developing and developed countries leads to high morbidity and mortality and represents 23 million life-years lost annually (JHA , 2013JHA, A. K. et al. The global burden of unsafe medical care: analytic modelling of observational studies. BMJ Qual Saf., v. 22, p. 809-815, 2013. https://doi:10.1136/bmjqs-2012-001748.
https://doi.org/10.1136/bmjqs-2012-00174... ). Thus, investments for improving health care systems and safety of people who have access to health care services are needed to change this scenario and minimize risks (WHO, 2018WORLD HEALTH ORGANIZATION; OECD; INTERNATIONAL BANK FOR RECONSTRUCTION AND DEVELOPMENT . Delivering quality health services: a global imperative for universal health coverage. Geneva: WHO, 2018. Disponível em: Disponível em: https://apps.who.int/iris/handle/10665/272465 . Acesso em: 13 mai. 2019.
https://apps.who.int/iris/handle/10665/2... ).

According to international recommendations, Brazil established the National Patient Safety Program in 2013 to “contribute to health care qualification in all health facilities of the national territory” (BRASIL, 2013aBRASIL. Ministério da Saúde. Portaria nº 529, de 01 de abril de 2013. Institui o Programa Nacional de Segurança do Paciente. Brasília/DF, 2013a. Disponível em: Disponível em: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2013/prt0529_01_04_2013.html . Acesso em: 03 dez. 2018.
https://bvsms.saude.gov.br/bvs/saudelegi... ). To exert its role as a health service regulator, the National Health Surveillance Agency (ANVISA) published health standards in 2013 (BRASIL, 2013bBRASIL. Agência Nacional de Vigilância Sanitária. Resolução da Diretoria Colegiada nº 36, de 25 de julho de 2013. Institui ações para a segurança do paciente em serviços de saúde. Diário Oficial da União, 2013b. Disponível em: Disponível em: https://bvsms.saude.gov.br/bvs/saudelegis/anvisa/2013/rdc0036_25_07_2013.html . Acesso em: 03 dez. 2018.
https://bvsms.saude.gov.br/bvs/saudelegi... ) and the Management of Patient Safety in Health Services Integrated Plan in 2015 (BRASIL, 2015BRASIL. Agência Nacional de Vigilância Sanitária. Plano Integrado para a Gestão Sanitária da Segurança do Paciente em Serviços de Saúde. Monitoramento e Investigação de Eventos Adversos e Avaliação de Práticas de Segurança do Paciente. Brasília-DF: Anvisa, 2015. ). The plan aims to incorporate National Health Surveillance System (NHSS) actions to induce health facilities to adopt patient safety practices and monitor and investigate adverse events.

The Patient Safety Self-Assessment Practices was created to monitor the adoption of safety practices, guide health surveillance operations at the three levels of actions of the NHSS, and encourage improvement actions in services with intensive care unit (ICU) beds (BRAZIL, 2015BRASIL. Agência Nacional de Vigilância Sanitária. Plano Integrado para a Gestão Sanitária da Segurança do Paciente em Serviços de Saúde. Monitoramento e Investigação de Eventos Adversos e Avaliação de Práticas de Segurança do Paciente. Brasília-DF: Anvisa, 2015. ). Self-assessments in health services are a potentially useful regulatory strategy to improve healthcare quality (HEALY; BRAITHWAITE, 2006HEALY, J.; BRAITHWAITE, J. Designing safer healthcare through responsive regulation. Med J Aust. N. 184, p. s56, 2006. https://doi:10.5694/j.1326-5377.2006.tb00364.x.
https://doi.org/10.5694/j.1326-5377.2006... ). This initiative to minimize risks may enhance performance of the NHSS and change its normative and authoritarian role into a health protector of the population (SANTANA et al., 2020SANTANA, H. T. et al. The Protagonism of the Brazilian Health Regulatory System in the Evolution of Patient Safety in the Country: History, Dilemmas, and Current Challenges. Journal of Patient Safety. 2020. Dec;16(4):e260-e266. doi: 10.1097/PTS.0000000000000671. PMID: 32084092. Disponível em: Disponível em: https://pubmed.ncbi.nlm.nih.gov/32084092/ . Acesso em: 08 jan. 2021.
https://pubmed.ncbi.nlm.nih.gov/32084092... ). In addition, the self-assessment is part of a responsive regulation approach since regulators also implement non-normative initiatives to produce regulatory impacts and improve quality of the regulation process. It is a meta-regulation action in which the regulator induces self-regulation based on assessment for the management (KOLIEB, 2015KOLIEB, J. When to Punish, When to Persuade and When to Reward: Strengthening Responsive Regulation with the Regulatory Diamond. Monash University Law Review, v. 41, n. 1, 2015. Disponível em: Disponível em: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2698498 > Acesso em: 13 mai. 2019.
https://papers.ssrn.com/sol3/papers.cfm?... ).

Self-assessment (SA) data are revised by the national, state, and municipal health surveillance (HS), which verifies information before publishing results. However, the society, regulated sector, health professionals, and HS may question the reliability of information reported by health services (BRASIL, 2015BRASIL. Agência Nacional de Vigilância Sanitária. Plano Integrado para a Gestão Sanitária da Segurança do Paciente em Serviços de Saúde. Monitoramento e Investigação de Eventos Adversos e Avaliação de Práticas de Segurança do Paciente. Brasília-DF: Anvisa, 2015. ). Reliability is an essential attribute for quality indicators (BARCLAY; DIXON-WOODS; LYRATZOPOULOS, 2019BARCLAY, M.; DIXON-WOODS, M.; LYRATZOPOULOS, G. The problem with composite Indicators. BMJ Qual Saf., v. 28, p. 338-344, 2019.https://doi:10.1136/bmjqs-2018-007798.
https://doi.org/10.1136/bmjqs-2018-00779... ; MAINZ, 2003MAINZ, J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care, v. 15, n. 6, p. 523-30, 2003. https://doi:10.1093/intqhc/mzg081.
https://doi.org/10.1093/intqhc/mzg081... ), similar to SA, which contains 21 simple and 1 composite indicator. The composite indicator is of main interest because the NHSS discloses services classified as highly compliant with patient safety practices.

Although the Patient Safety Self-Assessment Practices is a promising regulatory practice applied annually, its reliability has not yet been analyzed. Knowledge regarding reliability strengthens trust between the regulated sector and regulatory body and consolidates this strategy for improving patient safety.

This pilot study aims to analyze reliability of the Patient Safety Self-Assessment Practices as measure for monitoring risks in health services. In this sense, this research may provide a national sample for future studies.⁵5This article is the product of a Master's thesis in Public Health at the Federal University of Rio Grande do Norte.

Methods

Background

This study results from an academic master's thesis conducted during 2019 in collaboration with the Sub Coordination of Sanitary Surveillance (SUVISA/RN) and the Graduate Program in Collective Health of the Federal University of Rio Grande do Norte).

The Rio Grande do Norte state (RN) had 29 services with adults, pediatric, or neonatal ICU beds, and 27 of these (97%) responded to the self-assessment in 2019. The RN and other states reached the goal regarding participation in the assessment in 2019 (80%) and stayed above the national average (67%). Among services that responded, five were classified as highly compliant with patient safety practices because they met at least 67% of indicators.

Study design

This validation study analyzed the reliability (MEDRONHO, 2009MEDRONHO, R. A. et al. Epidemiologia. 2ª ed. Rio de Janeiro: Atheneu; 2009. ) of information sent by hospitals that participated in the Patient Safety Self-Assessment Practices and compared with data obtained in the on-site inspection (OSI) conducted by the HS.

Population and sample

The study population included public, private, or philanthropic hospitals with ICU beds in the RN state that completed the Patient Safety Self-Assessment Practices Form in 2019.

We used a non-proportional stratified random sampling considering the high compliance (yes or no) stratification variable. The list of participants in 2019 provided by SUVISA/RN and strict for research purposes was used as data source. Ten hospitals (n) were selected from a total of 27 (N) services that responded to the self-assessment. This sample corresponds to 37% of the population, five with high adherence to safety practices and five that did not achieve high adherence. The number of cases was defined using feasibility analysis and considering the research as pilot study. Exclusion criteria were applied to services in which information was incomplete or impossible to collect.

Study variables

The following variables regarding adherence level to safety practices, according to results of the self-assessment, revision (RSA) by HS, and OSI, were included: compliance (yes or no) with the 21 criteria or simple indicators; compliance with the adherence level composite indicator (percentage of compliant indicators = number of compliant indicators / 21 x 100); and high compliance (yes or no), based on compliant indicators ≥ 67% (BRASIL, 2019aBRASIL. Agência Nacional de Vigilância Sanitária. Instrutivo para a análise do formulário de autoavaliação das práticas de segurança do paciente. Brasília-DF: Anvisa , 2019a. ).

The following variables characterized hospitals: hospital ownership (public or private); type of administration (direct public, indirect public, for-profit, or non-profit private hospitals); quality management (yes or no); risk management (yes or no); presence of accreditation seal (yes or no); certification as teaching hospital (MEC/MS) (yes or no); inclusion in the Sentinel Network (yes or no); number of health professionals in the institution; municipality; size according to number of beds (small, up to 99 beds; medium, between 100 and 199 beds; or large, over 200 beds); number of ICU beds (adult, pediatric, or neonatal); and surgical procedures (yes or no).

Procedures and tools for data collection

SUVISA/RN provided SA and RSA data. Self-assessment responses were available in the FormSUS, completed in August 2019. RSA was consolidated in October 2019, when the “worksheet for the analysis of the Patient Safety Self-Assessment Practices Form - 2019” was completed (BRASIL, 2019b). The form was created by ANVISA and sent to the state HS. In this revision process, the state HS verifies documents sent by services and classifies the service.

For data collection in the OSI, a structured questionnaire was prepared based on the 21 indicators defined by ANVISA⁶6Available at: https://doi.org/10.6084/m9.figshare.12456614.v1. We also followed a script of activities (available online) performed before, during, and after the visit⁷7Available at: https://doi.org/10.6084/m9.figshare.12456662. Before the inspection, the SUVISA/RN team contacted the service to schedule the visit and separated evidence documents (protocols, charts, and other documents). During the visit, Patient Safety Centers (PSC) and the Hospital Infection Control Committees were visited, and documents were revised (including medical records corresponding to indicators). The inspection was completed after a visit to the ICU, drawn in case of more than one ICU. Data were consolidated in the data collection instrument after inspection.

Structure indicators, such as number of lavatories and presence of alcohol-based hand solutions, were evaluated through direct observation of the ICU service. We also verified whether protocols or summaries were available and accessible to healthcare professionals. The researcher and municipal HS applied the instrument; all were blinded to compliance of indicators according to SA form, except for indicators from medical records. The form for registering OSI data was self-explanatory and created on the Google Forms platform to consolidate the database.

Information regarding variables was collected on the National Registry of Health Establishments website (BRASIL, 2019c) between August and November 2019 and during data collection at the OSI.

Data analysis

Reliability analysis was performed after a descriptive analysis of frequencies and measurements of central tendency and dispersion of indicators and variables that characterized hospitals. Compliance with SA and RSA indicators was compared with OSI, represented as gold standard.

Reliability of the 21 simple indicators and the composite indicator categorized in high compliance was calculated based on general agreement indexes (GAI) and Kappa coefficients, which presented a cutoff point of 95% and 0.40, respectively, according to Landis and Koch (1977LANDIS, J. R., KOCH, G. G. The measurement of observer agreement for categorical data. Biometrics, v. 33, n. 1, p. 159-174, 1977. https://doi:10.2307/2529310
https://doi.org/10.2307/2529310... ). For the composite indicator measured as percentage of adherence, reliability was calculated using intraclass correlation coefficient (ICC), which had a cutoff point indicating good correlation values ​​above 0.75. In addition, concordance was analyzed using Bland-Altman plots. Differences in compliance of indicators were also evaluated. Significance level was set at 5% (α), in which null hypotheses were rejected when p-value was less than 0.05. Data were analyzed using the Statistical Package for the Social Sciences (IBM SPSS Statistics 22).

Ethical aspects

The study was approved by the research ethics committee (ID 3.360.654) and concluded according to recommendations.

Results

Sample characterization

Hospitals from four municipalities, representing the four largest health regions (Grande Natal, Mossoró, Caicó, and Pau dos Ferros) out eight in the state, composed the study. The number of public and private health services was the same. Four private hospitals were for-profit, and three public hospitals had direct administration. Seven of ten services were medium-sized, and the average number of health professionals in each establishment was 713. Most services claimed quality and risk management (70%), and no hospital had an accreditation seal. Table 1 details the characteristics of included hospitals.

Compliance of indicators according to different data sources

Results regarding compliance of indicators in the SA were overestimated compared with RSA and OSI. Regarding simple indicators, variation was high among levels of compliance, whereas one of the indicators showed compliance in the three data sources. Adherence to safety practices in SA was 83.5%, 54.8% in the RSA, and 35.8% in the OSI. Although SA revealed seven services with high compliance, RSA considered five, and OSI considered only one. Table 2 shows the percentage of compliance for each indicator according to different data sources (SA, RSA, and OSI).

Reliability estimates according to different data sources

Based on GAI cutoff point, when SA was compared with OSI, only one indicator was considered directly reliable (> 95%). In this comparison, eight indicators showed relatively high GAI (> 80%). When comparing RSA with OSI, six indicators reached the maximum agreement value (100%), whereas ten indicators showed GAI > 80%. Regarding the high compliance indicator, agreement was 40% when SA was compared with OSI and 60% when RSA was compared with OSI. Table 3 presents reliability values for simple and composite indicators.

Regarding Kappa coefficient, 12 indicators from SA and 18 from RSA showed a satisfactory agreement (> 0.40) compared with OSI. Kappa coefficient was not calculated for indicators 14 and 15 in both comparisons due to lack of variability. Kappa coefficient of the composite indicator for classification of high compliance in SA was 0.10 compared with OSI, whereas RSA compared with the OSI was 0.20, indicating low level of agreement.

The ICC for percentage of adherence corroborated with agreement found in the classification of high compliance. Results showed excellent ICC values (> 0.75), indicating a strong correlation in the two comparisons (SA vs. OSI = 0.899 and RSA vs. OSI = 0.941).

Information from Table 3 is presented in Figure 1 (Bland-Altman plots) to better understand data. Bland-Altman evaluates the mean difference between two variables (MARTÍNEZ-GONZÁLEZ; SÁNCHEZ-VILLEGAS; FAJARDO, 2006MARTÍNEZ-GONZÁLEZ, M. A.; SÁNCHEZ-VILLEGAS, À., FAJARDO, F. J. F. Bioestadística amigable. 2. ed. Ediciones: Diaz de Santos, 2006.; HIRAKATA; CAMEY, 2009HIRAKATA, V. N.; CAMEY, S. A. Análise de concordância entre métodos de Bland-Altman. Clinical & Biomedical Research, v. 29, n. 3, p.261-268, 2009. ). Figure 1 shows data dispersion in both plots (Figure 1). The 95% confidence interval was short, with few cases, and no outliers were observed.

While mean difference in the first plot was approximately 34.3% (p=0.000), the second graph shows a value of 15.2% (p=0.000). This result indicates that HS revision reduces differences compared with OSI. In both situations, Mann-Whitney U test was used for independent samples (CONTADOR; SENNE, 2015CONTADOR, J. L.; SENNE, E. L. F. Testes não paramétricos para pequenas amostras de variáveis não categorizadas: um estudo. Gest. Prod., São Carlos, 2015. http://dx.doi.org/10.1590/0104-530X357-15
https://doi.org/10.1590/0104-530X357-15... ).

Differences in compliance between data

Differences between variations of data sources were observed between SA and OSI (from 0 to 60) and between RSA and OSI (from 0 to 50). Regarding p-values (in which low values indicate greater evidence against the null hypothesis), the indicator 20 (SA vs. OSI) and indicators 2, 19, and 20 (RSA vs. OSI) were significantly different (p < 0.05).

Figure 2 illustrates differences in compliance according to criteria between data sources. Differences in percentage of high compliance (% HC) were also found. The thirteenth indicator presented the greatest difference in both comparisons, while the composite indicator of adherence to patient safety practices (% HC) resulted in a 10% difference in both comparisons (SA vs. OSI and RSA vs. OSI). In the first figure, the indicator 13 and other indicators (6, 8, and 14) were different. The second figure showed no differences in indicators 1, 2, 9, 10, 19, and 20.

Discussion

Contribution of the study

Regulation is a recommended strategy to improve quality of care in health systems (WHO, 2018WORLD HEALTH ORGANIZATION; OECD; INTERNATIONAL BANK FOR RECONSTRUCTION AND DEVELOPMENT . Delivering quality health services: a global imperative for universal health coverage. Geneva: WHO, 2018. Disponível em: Disponível em: https://apps.who.int/iris/handle/10665/272465 . Acesso em: 13 mai. 2019.
https://apps.who.int/iris/handle/10665/2... ); however, its methods and results are still discussed internationally (FLODGREN; GONCALVES-BRADLEY; POMEY, 2016FLODGREN, G.; GONCALVES-BRADLEY, D. C.; POMEY, M. P. External inspection of compliance with standards for improved healthcare outcomes. Cochrane Database Syst Rev, v. 12, CD008992, 2016. DOI: 10.1002/14651858.CD008992.pub3.
https://doi.org/10.1002/14651858.CD00899... ; SMITHSON; ROBERTS; WALSHE, 2018SMITHSON, R.; ROBERTS, J.; WALSHE, K. Impact of the Care Quality Commission on provider performance: room for improvement? London: King's Fund, 2018. ). This research contributes to validating an innovative method implemented by HS of health services in Brazil, which uses self-assessment, revision of data by regulators, control actions, and publication of services with high adherence to standards (BRASIL, 2015BRASIL. Agência Nacional de Vigilância Sanitária. Plano Integrado para a Gestão Sanitária da Segurança do Paciente em Serviços de Saúde. Monitoramento e Investigação de Eventos Adversos e Avaliação de Práticas de Segurança do Paciente. Brasília-DF: Anvisa, 2015. ).

This pilot study is the first in Brazil to assess reliability of the Patient Safety Self-Assessment Practices. The study identified signs of reliability in most indicators, issues in other indicators, and essential information for revising measurement instruments. Although these are early findings, they are important because they guide this large-scale national policy and indicate the need to periodically revise the reliability of assessments in upcoming years. The risk monitoring model through assessment of patient safety practices can be useful for protecting the health of the population and promoting regulatory efficiency, mainly if methods are improved and conducted with precaution (PECI, 2011PECI, A. Avaliação do impacto regulatório e sua difusão no contexto brasileiro. Rev. adm. empres., v. 51, n. 4, p. 336-348, 2011. Disponível em: Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0034-75902011000400003&lng=en&nrm=iso . Acesso em: 15 ago. 2020. https://doi.org/10.1590/S0034-75902011000400003.
http://www.scielo.br/scielo.php?script=s... ), considering financing barriers of the HS in the Unified Health System (SUS, Sistema Único de Saúde) (CELUPPI et al., 2019CELUPPI, I. C. et al. 30 anos de SUS: relação público-privada e os impasses para o direito universal à saúde. Saúde em Debate, v. 43, p. 302-313, 2019.; BATTESINI; ANDRADE; SETA, 2017BATTESINI, M.; ANDRADE, C. L. T de; SETA, M. H. de. Financiamento federal da Vigilância Sanitária no Brasil de 2005 a 2012: análise da distribuição dos recursos. Ciênc. Saúde Coletiva, v. 22, n. 10, p. 3295-3306, 2017. Disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S1413-81232017021003295&lng=pt. https://doi.org/10.1590/1413-812320172210.10852017. Acesso em: 15 ago. 2020.
http://www.scielo.br/scielo.php?script=s... ).

Reliability of self-assessment

A consistent interpretation of all analyzes of this study that will probably be confirmed in the future is related to the indispensable need for HS (national, state, and municipal) to revise information sent by the health service. Efforts are needed to improve performance of NHSS when monitoring this activity (SANTANA; COSTA; NOGUEIRA, 2020SANTANA, H. T. et al. The Protagonism of the Brazilian Health Regulatory System in the Evolution of Patient Safety in the Country: History, Dilemmas, and Current Challenges. Journal of Patient Safety. 2020. Dec;16(4):e260-e266. doi: 10.1097/PTS.0000000000000671. PMID: 32084092. Disponível em: Disponível em: https://pubmed.ncbi.nlm.nih.gov/32084092/ . Acesso em: 08 jan. 2021.
https://pubmed.ncbi.nlm.nih.gov/32084092... ).

In general, results of all indicators were close to reality (OSI) when undergoing HS revision. Many indicators were similar to reality (Tables 2 and 3; Figure 2).

The improved reliability of indicators after revision by HS may have been caused by 1) error due to communication problems in requirements of health services; 2) error due to divergences in criteria used by the hospital and by other assessments considered in this study; or 3) intentional submission of erroneous information by the health service. Regarding the first possibility, for example, guidelines on the “patient safety plan” indicator requested a series of requirements that seemed unrealistic for services initiating PSC activities; centers sent plans even without these requirements. This issue was reconsidered in the new assessment conducted in 2020, and criteria are now better suited to different contexts. Requirements for validating HS revision are based on a local analysis and a set of realistic objectives for each situation (BRAZIL, 2020BRASIL. Agência Nacional de Vigilância Sanitária. Instrutivo para a análise do formulário de avaliação das práticas de segurança do paciente - 2020. Brasília-DF: Anvisa , 2020a.).

The second possible explanation for reliability issues and improvement after HS revision relates to data collection guidelines for this study (SA, RSA, and OSI). The first guidance was sent to hospitals (BRASIL, 2019dBRASIL. Agência Nacional de Vigilância Sanitária. Orientações para preenchimento da autoavaliação das práticas de segurança do paciente - 2019. Brasília-DF: Anvisa , 2019d. ), the second was the instruction and revision worksheet for HS (BRASIL, 2019aBRASIL. Agência Nacional de Vigilância Sanitária. Instrutivo para a análise do formulário de autoavaliação das práticas de segurança do paciente. Brasília-DF: Anvisa , 2019a. bBRASIL. Agência Nacional de Vigilância Sanitária. Planilha para a análise do formulário de Autoavaliação das Práticas de Segurança do Paciente - 2019. Brasília-DF: Anvisa , 2019b.), and the third was the instruction for on-site data collection. Differences in the orientation of these three sources and systematization for consolidation of responses may have led to reliability issues (MATOS, 2014MATOS, D. A. S. Confiabilidade e concordância entre juízes: aplicações na área educacional. Est. Aval. Educ., v. 25, n. 59, p. 298-324, 2014.). Also during data collection, representatives of PSC required clarity about writing of protocols and proof of implementation. The importance of harmonizing information of these official documents was discussed in the revision of the new 2020 assessment (BRAZIL, 2020BRASIL. Agência Nacional de Vigilância Sanitária. Instrutivo para a análise do formulário de avaliação das práticas de segurança do paciente - 2020. Brasília-DF: Anvisa , 2020a.) to achieve a single direction (BLACK, 2002BLACK, J. Critical Reflections on Regulation. Australian Journal of Legal Philosophy. Canberra, Austrália, 2002. Disponível em: Disponível em: https://www.researchgate.net/publication/265568938_Critical_Reflection_on_Regulation . Acesso em: 15 ago. 2020.
https://www.researchgate.net/publication... ).

Differences in evaluation criteria also occurred compared to OSI, which reduced reliability. In the indicators related to implementation of protocols, which presented more reliability issues, OSI was more demanding and presented two additional criteria: easy access to the protocol during inspection and proof of training of professionals. This decision was taken because requirements were essential to ensure implementation of the self-assessment protocol, according to the assessment indicator. Thus, protocols sent and remotely approved were considered non-compliant with OSI. Since protocols were not available, professionals were unaware of their existence, or no evidence of training was found (KRAUZER et al., 2018KRAUZER, I. M. et al. A construção de protocolos assistenciais no trabalho em enfermagem. Rev. Min. Enf., v. 22, e-1087, 2018. http://www.dx.doi.org/10.5935/1415-2762.20180017
https://doi.org/10.5935/1415-2762.201800... ).

The third probable reason for reliability issues was related to services that may have intentionally sent a limited or unreal form to reach a high adherence indicator (BARCLAY; DIXON-WOODS; LYRATZOPOULOS, 2019BARCLAY, M.; DIXON-WOODS, M.; LYRATZOPOULOS, G. The problem with composite Indicators. BMJ Qual Saf., v. 28, p. 338-344, 2019.https://doi:10.1136/bmjqs-2018-007798.
https://doi.org/10.1136/bmjqs-2018-00779... ). Although the evaluation of patient safety practices encourages a non-punitive culture (REIS; PAIVA; SOUSA, 2018REIS, C. T.; PAIVA, S. G.; SOUSA, P. The patient safety culture: a systematic review by characteristics of hospital survey on patient safety culture dimensions. International Journal for Quality in Health Care, v. 30, n. 9, p. 660-677, 2018.), falsifying data to a regulatory body (especially if repeatedly and evident) may lead to undesirable consequences and prompt regulators to climb to the top of the responsive regulatory pyramid (HEALY; BRAITHWAITE, 2006HEALY, J.; BRAITHWAITE, J. Designing safer healthcare through responsive regulation. Med J Aust. N. 184, p. s56, 2006. https://doi:10.5694/j.1326-5377.2006.tb00364.x.
https://doi.org/10.5694/j.1326-5377.2006... ; KOLIEB, 2015KOLIEB, J. When to Punish, When to Persuade and When to Reward: Strengthening Responsive Regulation with the Regulatory Diamond. Monash University Law Review, v. 41, n. 1, 2015. Disponível em: Disponível em: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2698498 > Acesso em: 13 mai. 2019.
https://papers.ssrn.com/sol3/papers.cfm?... ).

After revision by the HS, only the ventilator-associated pneumonia prevention indicator showed an unsatisfactory Kappa value. Kappa coefficient was not calculated for surgical site infection prevention and microbial resistance prevention, and they also showed no compliance after OSI. This finding raises doubts about these protocols - whether they are realistic for the services or clearer protocol models are needed to facilitate adaptation and adherence of the regulated sector since this is a relevant change mechanism (HEALY; BRAITHWAITE, 2006HEALY, J.; BRAITHWAITE, J. Designing safer healthcare through responsive regulation. Med J Aust. N. 184, p. s56, 2006. https://doi:10.5694/j.1326-5377.2006.tb00364.x.
https://doi.org/10.5694/j.1326-5377.2006... ; FLODGREN; GONCALVES-BRADLEY; POMEY, 2016FLODGREN, G.; GONCALVES-BRADLEY, D. C.; POMEY, M. P. External inspection of compliance with standards for improved healthcare outcomes. Cochrane Database Syst Rev, v. 12, CD008992, 2016. DOI: 10.1002/14651858.CD008992.pub3.
https://doi.org/10.1002/14651858.CD00899... ).

In addition, the sum of reliability issues of simple indicators is reflected in the composite indicator that classifies services of high adherence, which is a predicted/predictable problem (BARCLAY; DIXON-WOODS; LYRATZOPOULOS, 2019BARCLAY, M.; DIXON-WOODS, M.; LYRATZOPOULOS, G. The problem with composite Indicators. BMJ Qual Saf., v. 28, p. 338-344, 2019.https://doi:10.1136/bmjqs-2018-007798.
https://doi.org/10.1136/bmjqs-2018-00779... ; SATURNO-HERNANDEZ, 2004SATURNO-HERNANDEZ, P. J. La invasión de los indicadores compuestos. Riesgos y beneficios para la gestión de la calidad. Rev Cal Assist., v. 19, n. 6, p. 407-15, 2004. ). Even after revision by the HS, a reduction of high adherent services was observed (from five cases to one case), and Kappa was 0.20. If results of this study are representative of the country, it would suggest low reliability of the high adherence list published in the annual Patient Safety Self-Assessment Practices report of ANVISA (BRASIL, 2020bBRASIL. Agência Nacional de Vigilância Sanitária. Relatório de Autoavaliação Nacional das Práticas de Segurança do Paciente em Serviços de Saúde - 2019. Brasília/DF: 2020b. Disponível em: Disponível em: http://antigo.anvisa.gov.br/documents/33852/271855/Relat%C3%B3rio+de+Autoavalia%C3%A7%C3%A3o+Nacional+das+Pr%C3%A1ticas+de+Seguran%C3%A7a+do+Paciente+em+Servi%C3%A7os+de+Sa%C3%BAde+%E2%80%93+2019/faa6381c-b3c3-4210-8ddf-4e93927c64dd . Acesso em: 25 fev 2020.
http://antigo.anvisa.gov.br/documents/33... ). This indicates the need to assess the reliability of this instrument before further publications.

In the face of these challenges, the new 2020 assessment of patient safety practices, formerly called “self-assessment”, is now called “assessment” because its judicial process must consider data from self-assessment, revision by HS, and selection of hospitals for on-site verification. This new assessment changed the configuration of indicators after considering preliminary results of the study in discussions with the NHSS. In this sense, indicators that could not be proven remotely (indicators 4 and 5) were replaced by new indicators, revision criteria were adjusted to make the assessment more realistic (indicators 1 and 2), and proof of training was required for all protocol indicators. These adjustments enhance expectations for improving reliability of the initiative in upcoming years (BRASIL, 2020aBRASIL. Agência Nacional de Vigilância Sanitária. Instrutivo para a análise do formulário de avaliação das práticas de segurança do paciente - 2020. Brasília-DF: Anvisa , 2020a.).

Limitations of the study

Due to sample size, 95% confidence intervals were not calculated, which would certainly be inaccurate. Therefore, results should not be interpreted as evidence of reliability of the initiative but as pilot study (SATURNO HERNÁNDEZ; CARMEN SANTIAGO; ANTÓN BOTELLA, 2017SATURNO HERNÁNDEZ, P. J.; CARMEN SANTIAGO, M., ANTÓN BOTELLA, J. J. Análise da confiabilidade dos critérios: unidade temática 4. In: HERNÁNDEZ, P. J. S. et al. Atividades básicas para melhoria contínua: métodos e instrumentos para realizar o ciclo de melhoria: módulo II. SEDIS-UFRN, 2017. ). Precautions must also be taken when extrapolating data to other federation units. The study design, including ten hospitals randomly selected, refers only to the RN state. However, the study raised relevant hypotheses to be tested on a larger scale.

Future studies can also compare the reliability of public and private services, consider conflicts of interest of the private sector (BRASIL, 2011BRASIL. Conselho Nacional de Secretários de Saúde. Regulação em Saúde. Brasília: CONASS, 2011. Disponível em: Disponível em: https://www.conass.org.br/bibliotecav3/pdfs/colecao2011/livro_10.pdf Acesso em: 13 mai. 2019
https://www.conass.org.br/bibliotecav3/p... ), and assess accuracy indicators (e.g., positive and negative predictive values, sensitivity, and specificity) not calculated in this initial analysis.

Final contributions

Reliability of the Patient Safety Self-Assessment Practices, innovation in health surveillance of health services for monitoring risks, and sanitary regulation of the quality of health services is a topic that must be explored. Results showed the need for the HS to revise data provided by these services and presented reliability issues that should be considered in future evaluations. These results reinforce the common sense that regulators should not rely directly on information provided by the service when sanitary control actions involve self-inspection initiatives because, although this model supports sanitary adaptations toward good practices and legal requirements, HS action in the on-site verification/revision is still required. Furthermore, future cycles of evaluation and improvement of patient safety practices should analyze the reliability of results before disclosing the ranking of hospitals with high compliance with safety practices.⁸8R. C. D. Rodrigues: planning and design of the study, theoretical framework, data collection and analysis, discussion and writing of the manuscript. Z. A. da S. Gama: study planning and design, supervision and reviewing of data collection and analysis, relevant critical reviewing of the manuscript and writing. A. A. S. Souza: data collection and analysis and writing of the manuscript. H. T. Santana: relevant critical reviewing of the manuscript and writing. This research has not received funding. There is no conflict of interest by the authors.

References

Publication Dates

History