ABSTRACT

OBJECTIVE

To validate a set of indicators for monitoring the quality of surgical procedures in the Brazilian Unified Health System (SUS).

METHODS

Validation study developed in 5 stages: 1) literature review; 2) prioritization of indicators; 3) content validation of indicators by RAND/UCLA consensus method; 4) pilot study for reliability analysis; and 5) development of instruction for tabulation of outcome indicators for monitoring via official information systems.

RESULTS

From the literature review, 217 indicators of surgical quality were identified. The excluded indicators were: those based on scientific evidence lower than 1A, similar, specific, which corresponded to sentinel events; and those that did not apply to the SUS context. Twenty-six indicators with a high level of scientific evidence were submitted to expert consensus. Twenty-two indicators were validated, of which 14 process indicators and 8 outcome indicators with content validation index ≥80%. Of the validated process indicators, 6 were considered substantially reliable (Kappa coefficient between 0.6 and 0.8; p < 0.05) and 2 had almost perfect reliability (Kappa coefficient > 0.8, p < 0.05), when the inter-rater agreement was analyzed. One could measure and establish tabulation mechanism for TabWin for 7 outcome indicators.

CONCLUSION

The study contributes to the development of a set of potentially effective surgical indicators for monitoring the quality of care and patient safety in SUS hospital services.

Quality of Health Care; Quality Indicators in Health Care; Patient Safety; Surgical Operating Procedures

INTRODUCTION

The Brazilian Unified Health System (SUS) performs about five million surgeries annually, mostly elective surgical procedures¹1. Minisério da Saúde (BR). Banco de dados do Sistema Único de Saúde-DATASUS. Brasília, DF: Ministério da Saúde; 2021 [cited 2021 Oct 11]. Available from: http://www.datasus.gov.br
http://www.datasus.gov.br... . Such therapeutic resource has been increasingly regarded as an essential component of public health, its role growing in importance with the increase in life expectancy²2. Weiser TG, Haynes AB, Molina G, Lipsitz SR, Esquivel MM, Uribe-Leitz T, et al. Size and distribution of the global volume of surgery in 2012. Bull World Health Organ. 2016 Mar;94(3):201-209F. https://doi.org/10.2471/BLT.15.159293
https://doi.org/10.2471/BLT.15.159293... . However, little is known about the quality and safety of surgeries performed in SUS.

This is a crucial gap since, despite their benefits, surgeries also present risks to the patient and costs to the health system. Data shows 312.9 million surgeries were performed in 2012 worldwide²2. Weiser TG, Haynes AB, Molina G, Lipsitz SR, Esquivel MM, Uribe-Leitz T, et al. Size and distribution of the global volume of surgery in 2012. Bull World Health Organ. 2016 Mar;94(3):201-209F. https://doi.org/10.2471/BLT.15.159293
https://doi.org/10.2471/BLT.15.159293... , an increase of about 36.8% since the launch of the Second Global Challenge for Patient Safety, Safe Surgeries Saves Lives³3. World Health Organization & WHO Patient Safety. The Second Global Patient Safety Challenge: Safe Surgery Saves Lives. Geneva (CH): WHO; 2011 [cited 2021 Oct 11]. Available from: https://apps.who.int/iris/handle/10665/70080
https://apps.who.int/iris/handle/10665/7... .

The Ministry of Health, health sector regulatory agencies and non-governmental bodies have supported initiatives to improve the quality and safety of surgeries through actions related to the elaboration of public policies⁴4. Ministério da Saúde (BR). Portaria nº 529, de 1º de abril de 2013. Institui o Programa Nacional de Segurança do Paciente (PNSP). Brasília, DF: Ministério da Saúde; 2013 [cited 2021 Oct 11]. Available from: from:http://www.saude.mt.gov.br/upload/controle-infeccoes/pasta2/portariamsgm-n-529-de-01-04-2013.pdfBrasil
http://www.saude.mt.gov.br/upload/contro... , technical standards and regulations for inspection and monitoring purposes. However, there is still a lack of a standardized set of indicators for monitoring surgeries in SUS. Such monitoring is important since it enables quality improvement and provides learning to teams, in addition to enabling the development of regulatory capacity, being essential for a good clinical performance⁵5. World Health Organization; World Bank Group; Organization for Economic Co-operation and Development. Delivering quality health services. Geneva: WHO; 2018 [cited 2021 Oct 11]. Available from: https://apps.who.int/iris/bitstream/handle/10665/272465/9789241513906-eng.pdf?ua=1
https://apps.who.int/iris/bitstream/hand... ,⁶6. Pringle M, Wilson T, Grol R. Measuring “goodness”? in individuals and healthcare systems. BMJ. 2002;325(7366):704-7. http://doi:10.1136/bmj.325.7366.704.
http://doi:10.1136/bmj.325.7366.704... .

In the last decade, indicators have been developed to guide initiatives for improvement of quality in perioperative care⁷7. Chazapis M, Gilhooly D, Smith AF, Myles PS, Haller G, Grocott MP, et al. Perioperative structure and process quality and safety indicators: a systematic review. Br J Anaesth. 2018 Jan;120(1):51-66. https://doi.org/10.1016/j.bja.2017.10.001 PMID:29397138
https://doi.org/10.1016/j.bja.2017.10.00... ,⁸8. Haller G, Bampoe S, Cook T, Fleisher LA, Grocott MP, Neuman M, et al. Systematic review and consensus definitions for the Standardised Endpoints in Perioperative Medicine initiative: clinical indicators. Br J Anaesth. 2019 Aug;123(2):228-37. https://doi.org/10.1016/j.bja.2019.04.041
https://doi.org/10.1016/j.bja.2019.04.04... and to stimulate positive changes towards achieving quality at a reasonable cost⁹9. D’Innocenzo M, Adami NP, Cunha IC. O movimento pela qualidade nos serviços de saúde e enfermagem. Rev Bras Enferm. 2006;59(1):84-8. https://doi.org/10.1590/S0034-71672006000100016
https://doi.org/10.1590/S0034-7167200600... . These indicators are used as direct measures of the quality and safety of the care provided; however, they are still insufficient. Therefore, we are dealing with a scenario in which the existing indicators are not standardized and consolidated, nor periodically measured by the care network, leading to a void of important information and lack of comparability between existing information, negatively affecting the planning and quality management of care in the SUS.

Thus, this study aims to identify and validate a minimum set of process and outcome indicators that can be used to monitor the quality of surgical procedures in SUS.

METHODS

This study is part of the QualiCir Project, an intervention project aimed at improving the quality and safety of surgical procedures in the state of Rio Grande do Norte (RN), and is developed in partnership with the QualiSaúde Research Group of the Federal University of Rio Grande do Norte and the RN Public Health Secretariat.

This is a methodological study on the validation of perioperative quality indicators applicable to elective surgical procedures performed in SUS. The study was developed in 5 stages: 1) literature review; 2) selection of indicators for consensus; 3) content validation of indicators; 4) pilot study for reliability analysis; and 5) development of instructions for tabulation of outcome indicators.

Stage 1 - Literature review: A search was performed in PubMed and Google Scholar databases, looking for articles of current systematic reviews (< 5 years of publication). As search strategy, the keywords “quality indicators” and “surgical procedures” were included. Searches were also carried out on official State websites and documents, pursuing indicators developed by national organizations regarded as reference in the promotion of patient care and safety, so to obtain a list of potential indicators to be used to measure surgical quality in the Brazilian context. Indicators were selected from regulatory agencies in the health sector¹⁰10. Agência Nacional de Saúde Suplementar (BR). Consórcio de Indicadores de Qualidade Hospitalar: painel geral. Brasília, DF: ANS; 2014 [cited 2021 Oct 11]. Available from: https://www.gov.br/ans/ptbr/arquivos/assuntos/prestadores/qualiss-programa-de-qualificacao-dos-prestadores-de-servicos-de-saude-1/consorcio-de-indicadores-qualidade-hospitalar2-pdf
https://www.gov.br/ans/ptbr/arquivos/ass... ,¹¹11. Agência Nacional de Vigilância Sanitária. Critérios diagnósticos de infecção do sítio cirúrgico (ISC). Brasília, DF: ANVISA; 2017 [cited 2022 Mar 20]. Available from: http://portal.anvisa.gov.br/documents/33852/3507912/Caderno+2+-+Critérios+Diagnósticos+de+Infecção+Relacionada+à+Assistência+à+Saúde/7485b45a-074f-4b34-8868-61f1e5724501
http://portal.anvisa.gov.br/documents/33... , Patient Safety Indicators (ISEP-Brazil Project)¹²12. Gama ZA, Saturno-Hern?ndez PJ, Ribeiro DN, Freitas MR, Medeiros PJ, Batista AM, et al. Desenvolvimento e validação de indicadores de boas práticas de segurança do paciente: projeto ISEP-Brasil. Cad Saúde Pública. 2016 Sep;32(9):e00026215. https://doi.org/10.1590/0102-311X00026215
https://doi.org/10.1590/0102-311X0002621... , Health System Performance Assessment Project (PROADESS)¹³13. Fundaçãoo Oswaldo Cruz. PROADESS - Avaliação de Desempenho do sistema de saúde brasileiro: indicadores paramonitoramento: relatório final. Rio de Janeiro: Fiocruz; 2012. Available from: https://www.proadess.icict.fiocruz.br/Relatorio_Proadess_08-10-2012.pdf
https://www.proadess.icict.fiocruz.br/Re... , and the Collaborating Center for Quality and Patient Safety (PROQUALIS)¹⁴14. Gouvêa C, Travassos C, Caixeiro F, Carvalho LS, Pontes B. Desenvolvimento de indicadores de segurança para monitoramento do cuidado em hospitais brasileiros de pacientes agudos. Rio de Janeiro: Proqualis; 2015 [cited 2022 Mar 20]. Available from: https://proqualis.net/sites/proqualis.net/files/indicadores%20de%20qualidade.pdf
https://proqualis.net/sites/proqualis.ne... .

Stage 2 - Selection of indicators for consensus: Based on the indicators found in the previous step, those that had the following criteria were selected: a) aspects related to the entire surgical process; b) high scientific evidence (1A); c) able to evaluate the quality of surgical care in any hospital of the national health system; d) can be used to implement improvement measures based on their results. Indicators that were similar amongst themselves, sentinels, not applied to the SUS context, that evaluate a specific surgical procedure or patient group, with contradictory evidence, and indicators that present measurement difficulties (many components of measurements, unclear) were excluded.

Stage 3 - Content validation of the indicators: Validation was performed using the RAND/UCLA method¹⁵15. Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle R, Lazaro P, et al. The RAND/UCLA appropriateness method user’s manual. Santa Monica: Rand Corporation, 2001 [cited 2022 Mar 20]. Available from: https://www.rand.org/pubs/monograph_reports/MR1269.html
https://www.rand.org/pubs/monograph_repo... , which associates aspects of the Delphi and Nominal Group methods¹⁴14. Gouvêa C, Travassos C, Caixeiro F, Carvalho LS, Pontes B. Desenvolvimento de indicadores de segurança para monitoramento do cuidado em hospitais brasileiros de pacientes agudos. Rio de Janeiro: Proqualis; 2015 [cited 2022 Mar 20]. Available from: https://proqualis.net/sites/proqualis.net/files/indicadores%20de%20qualidade.pdf
https://proqualis.net/sites/proqualis.ne... and combines the observation of the available scientific evidence with the collective judgment of experts. The validation of indicators is done through a consensus opinion derived from a group, with aggregated individual opinions, which is an established approach for the development of health indicators⁵5. World Health Organization; World Bank Group; Organization for Economic Co-operation and Development. Delivering quality health services. Geneva: WHO; 2018 [cited 2021 Oct 11]. Available from: https://apps.who.int/iris/bitstream/handle/10665/272465/9789241513906-eng.pdf?ua=1
https://apps.who.int/iris/bitstream/hand... . The group of specialists consisted of eight surgeons and two nurses. Nine members of this group of specialists worked in public institutions in four different Brazilian states, and one was a Spanish surgeon who coordinated a similar study in his country.

Two rounds of consensus were established: the first occurred by completing the electronic questionnaire sent by email and the second was developed by web conferencing.

A questionnaire was developed using the Google Forms platform, based on similar studies¹²12. Gama ZA, Saturno-Hern?ndez PJ, Ribeiro DN, Freitas MR, Medeiros PJ, Batista AM, et al. Desenvolvimento e validação de indicadores de boas práticas de segurança do paciente: projeto ISEP-Brasil. Cad Saúde Pública. 2016 Sep;32(9):e00026215. https://doi.org/10.1590/0102-311X00026215
https://doi.org/10.1590/0102-311X0002621... ,¹⁴14. Gouvêa C, Travassos C, Caixeiro F, Carvalho LS, Pontes B. Desenvolvimento de indicadores de segurança para monitoramento do cuidado em hospitais brasileiros de pacientes agudos. Rio de Janeiro: Proqualis; 2015 [cited 2022 Mar 20]. Available from: https://proqualis.net/sites/proqualis.net/files/indicadores%20de%20qualidade.pdf
https://proqualis.net/sites/proqualis.ne... ,¹⁶16. Soria-Aledo V, Angel-Garcia D, Martinez-Nicolas I, Rebasa Cladera P, Cabezali Sanchez R, Pereira Garc?a LF. Desarrollo y estudio piloto de un conjunto esencial de indicadores para los servicios de cirugía general. Cir Esp. 2016 Nov;94(9):502-10. https://doi.org/10.1016/j.ciresp.2016.06.009
https://doi.org/10.1016/j.ciresp.2016.06... , containing five closed questions for each indicator, using a Likert-type scale for responses. The following criteria were used for the evaluation and selection of indicators: 1) Is the indicator clearly relevant?; 2) Does the indicator measure the quality of care or safety in surgical care?; 3) Can the indicator be modified with improvement interventions implemented by the hospital?; 4) Are the data for the indicator measurement possible to collect?; and 5) Is the wording of the indicator clear, with correct terminology and leaving no doubts?

Indicators that obtained a content validation index (CVI) greater than 80%¹⁷17. Yusoff MS. ABC of content validation and content validity index calculation. Educ Med J. 2019;11(2):49-54. https://doi.org/10.21315/eimj2019.11.2.6
https://doi.org/10.21315/eimj2019.11.2.6... in the five proposed items would be considered valid for the measurement of surgical quality. Indicators that did not reach this value in the first round were taken to the second round.

As a subsidy for the two rounds, an indicator form was developed containing the following information: title, measure, justification, indicator type, data source, numerator and denominator description, clarifications/definition of terms, limitations/exceptions, and bibliographic references.

Stage 4 - Pilot study for reliability analysis: For reliability analysis of process indicators, a pilot study was carried out in a hospital of the RN state health network. Three samples were established from the set of surgeries described in the Management System of the Table of Procedures (SIGTAP) of SUS. Sample 1 (A1): All procedures of the surgical procedures group, except the subgroups of minor surgeries and surgeries of the skin, subcutaneous tissue and mucosa, upper airway surgery, vision apparatus surgery, obstetric surgery and other surgeries; sample 2 (A2): Surgical procedures of the subgroup digestive tract surgeries (colon and rectum surgeries); sample 3 (A3): Surgical procedures of the osteomuscular apparatus subgroup surgeries (arthroscopy and knee prosthesis).

Collection was carried out by two independent evaluators, with previous experience in collecting data from medical records, in a cross-sectional manner, in samples of 30 medical records each, referring to elective surgeries occurred in 2020, selected systematically¹⁸18. Mokkink LB. COSMIN Risk of Bias checklist. Amsterdam: COSMIM; 2018 [cited 2022 Mar 20]. Available from: https://www.cosmin.nl/wp-content/uploads/COSMIN-RoB-checklist-V2-0-v17_rev3.pdf
https://www.cosmin.nl/wp-content/uploads... ,¹⁹19. Saturno-Hernandez P. Métodos y herramientas para la monitorización de la calidad. Cuernavaca: Instituto Nacional de Salud Pública; 2015 [cited 2022 Mar 20]. Available from: https://www.researchgate.net/publication/299486843_Metodos_y_herramientas_para_la_monitorizacion_de_la_calidad_en_servicios_de_salud
https://www.researchgate.net/publication... . The adequacy of indicators by sample type was established by consulting experts. Most of the process indicators were evaluated in sample A1, with the exception of the indicators “Timely removal of surgical nasogastric tubes” and “Early removal of bladder catheter”, which were evaluated in sample 2.

For the analysis of interobserver reliability, the Kappa index was calculated to identify the level of agreement according to the parameters established by Landis and Koch²⁰20. Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977 Jun;33(2):363-74. https://doi.org/10.2307/2529786
https://doi.org/10.2307/2529786... : poor agreement (Kappa < 0.00), mild agreement (0.00 ≤ Kappa ≤ 0.20), fair agreement (0.21 ≤ Kappa ≤ 0.40), moderate agreement (0.41 ≤ Kappa ≤ 0.60), substantial agreement (0.61 ≤ Kappa ≤ 0.80) and perfect agreement (0.81 ≤ 1.00).

Stage 5 - Identification of tabulation mechanism for result indicators so that they can be monitored via official information systems - The validated result indicators were analyzed for their possibility of monitoring through the use of data from official information systems, from the identification of tabulation mechanism for TabWin/DataSus with the Hospital Information System of SUS (SIH-SUS - Sistema de Informações Hospitalares do SUS) database.

The research was carried out under the approval of the Research Ethics Committee of the Federal University of Rio Grande do Norte (CEP-HUOL, CAAE: 39976920.6.0000.5292), following the ethical precepts in research with human beings, according to resolution CNS/MS 466/12.

RESULTS

217 quality or safety indicators related to surgical procedures, totaling 183 process indicators and 34 outcome indicators were found. The choice to use the content of systematic reviews as the main reference for the literature search was made to avoid the repetition of a recent study with similar objectives.

Of the 183 process indicators, 138 were excluded by the criterion of low scientific evidence (< 1A) (Figure 1). Although the level of evidence of the indicator “Use of safe surgery checklist” is not high, the researchers decided to keep this indicator in the study due to its regulation in Brazilian health services. Twelve indicators were excluded because they were considered similar, two because they were not applied to the sus, five because they were indicators applied to a very specific public or procedure, eight did not allow the development of improvement cycles and two were based on contradictory scientific evidence.

As for the outcome indicators, 10 indicators were excluded because they were considered similar, eight were very specific, two did not allow the development of improvement cycles and four were related to sentinel events. At the end of this trial, 16 process indicators and 10 outcome indicators were submitted to content validation with the group of experts. The selection flow of indicators can be seen in Figure 1.

In the first round, which was attended by 100% of the invited experts, validation questionnaires were sent by email and 26 indicators were presented to the group. In this round, the 13 indicators that received CVI greater than 80% were considered valid for measuring surgical quality within the SUS. The other 13 indicators, due to achieving CVI equal to or less than 80% in any of the evaluated criteria, were submitted to the second round of consensus. This step occurred through web conferencing and was attended by 80% of the invited experts. Discussions on indicators with CVI ≤ 80% took place at the time and, subsequently, a new evaluation was carried out, as can be seen in Chart 1.

At the end of the second round, four indicators received CVI ≤ 80% and were not considered valid: the indicator “Preoperative use of oral carbohydrates”, which presented CVI of 75% in the criterion related to the writing of the indicator; the indicator “Improved recovery” had CVI of 75% in the criteria related to the availability of data for measurement and clarity in the writing; the indicators “Post-surgical stroke” and “Unscheduled admission to an intensive care unit” obtained CVI of 75% in the criteria related to the availability of data and the possibility of modifying the indicator through improvement interventions. Thus, 22 indicators were considered valid for the measurement of quality in surgeries, of which 14 were process and 8 were outcome indicators. The data source, numerator and denominator of these indicators are described in Chart 2.

The qualification sheets of the validated indicators were reformulated according to suggestions of the experts, with the addition and reformulation of terms and concepts.

To analyze the reliability of the indicators, whose data source are the medical records, a retrospective pilot study was carried out at the Regional Hospital Mariano Coelho (HRMC), in Currais Novos/RN, between September and October 2021. The HRMC has 32 qualified surgical beds, and is a reference in the performance of elective surgical procedures for the health region in which the hospital is inserted.

Due to the HRMC qualification profile, it was not possible to collect the indicators “Postoperative discharge with postoperative evaluation, prophylaxis of venous thromboembolism and postoperative rehabilitation”, and “Record of pressure and time during controlled ischemia in surgery”. The search for another institution of the state hospital network that was qualified to perform orthopedic surgeries to evaluate these indicators was considered; however, this was not possible given the low number of orthopedic elective surgeries performed in 2020 due to the covid-19 pandemic, in addition to the lack of pneumatic tourniquet in the hospital institutions that make up the state network.

As for the reliability analysis, six indicators showed substantial reliability and two almost perfect reliability²⁰20. Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977 Jun;33(2):363-74. https://doi.org/10.2307/2529786
https://doi.org/10.2307/2529786... , as can be seen in Table 1. One could not measure the reliability for the process indicators “Control of normothermia in the perioperative period”, “Screening of postoperative delirium”, “Prophylaxis of adequate perioperative venous thromboembolism” and “Use of safe surgery checklist”, since the percentage of compliance for these indicators was 0% for both evaluators.

For outcome indicators, whose data source is SIH-SUS, it was observed that seven of the eight validated indicators can be monitored from the TabWin/DATASUS tabulator. Data are publicly accessible and available at https://datasus.saude.gov.br/transferencia-de-arquivos/.

It was not possible to perform tabulation for the indicator “Post-surgical readmission”. As this is a system that analyzes hospital production, it does not link hospitalizations to an individual user record, i.e., through the system one cannot identify how many times a single user was admitted to the hospital, nor is it possible to ascertain whether one admission would be related to the previous one.

An instruction was prepared to tabulate the result indicators for the TabWin/DATASUS application for teams that will collect data and monitor it. All results obtained with the other indicators can be seen in Table 2.

DISCUSSION

This study contributed to the development of a set of 22 indicators with a high level of evidence, which underwent a rigorous content validation process to enable the monitoring of the quality of surgical care within the SUS. These indicators can guide the management of institutions and of the hospital network as a whole, identifying weaknesses that must be addressed, aiming at providing safe care to the population. This is, therefore, an initial set of highly relevant indicators for monitoring and improving the quality of surgical care within the scope of SUS RN, with the possibility of being used by any other health service.

From the process indicators, one may evaluate all the steps and activities performed in the implementation of a treatment or care episode⁸8. Haller G, Bampoe S, Cook T, Fleisher LA, Grocott MP, Neuman M, et al. Systematic review and consensus definitions for the Standardised Endpoints in Perioperative Medicine initiative: clinical indicators. Br J Anaesth. 2019 Aug;123(2):228-37. https://doi.org/10.1016/j.bja.2019.04.041
https://doi.org/10.1016/j.bja.2019.04.04... . Thus, continuously monitoring these indicators enables one to identify weaknesses in the provision of care. According to Donabedian, process indicators are the only direct measure of quality, as the structure may not be used and outcomes may be due to factors other than good care²¹21. Donabedian A. The definition of quality and approaches to its assessment. Michigan: Anne Arbor; 1980..

Monitoring of the outcome indicators “Post-surgical mortality”, “Post-surgical readmission” and “Average length of stay with and without death” through the information system enables the measurement of the quality of an isolated health service, as well as benchmarking. That is, it enables the comparison of health services from the state hospital network and also at the national level, which strengthens information systems²²22. World Health Organization. Global patient safety action plan 2021-2030: towards eliminating avoidable harm in health care. Geneva: WHO; 2021 [cited [cited 2022 Mar 20] Available from: https://apps.who.int/iris/handle/10665/343477
https://apps.who.int/iris/handle/10665/3... .

The post-surgical mortality indicator is among the indicators proposed by the Lancet Commission²³23. Meara JG, Leather AJ, Hagander L, Alkire BC, Alonso N, Ameh EA, et al. Global Surgery 2030: evidence and solutions for achieving health, welfare, and economic development. Lancet. 2015 Aug;386(9993):569-624. https://doi.org/10.1016/S0140-6736(15)60160-X
https://doi.org/10.1016/S0140-6736(15)60... to assess surgical care. A similar study¹⁶16. Soria-Aledo V, Angel-Garcia D, Martinez-Nicolas I, Rebasa Cladera P, Cabezali Sanchez R, Pereira Garc?a LF. Desarrollo y estudio piloto de un conjunto esencial de indicadores para los servicios de cirugía general. Cir Esp. 2016 Nov;94(9):502-10. https://doi.org/10.1016/j.ciresp.2016.06.009
https://doi.org/10.1016/j.ciresp.2016.06... developed for the Spanish health system also pointed out the indicators: “Post-surgical readmission”, “Prophylaxis of venous thromboembolism”, “Adequate antibiotic prophylaxis” and “Surgical site infection” as valid indicators to assess surgical quality; however, these indicators are directed only to surgeries of the digestive tract.

Benchmarking has been used to seek opportunities for improvement and make comparisons of similar organizations¹⁶16. Soria-Aledo V, Angel-Garcia D, Martinez-Nicolas I, Rebasa Cladera P, Cabezali Sanchez R, Pereira Garc?a LF. Desarrollo y estudio piloto de un conjunto esencial de indicadores para los servicios de cirugía general. Cir Esp. 2016 Nov;94(9):502-10. https://doi.org/10.1016/j.ciresp.2016.06.009
https://doi.org/10.1016/j.ciresp.2016.06... ,²⁴24. Mello DM. Monitoramento das infecções de sítio cirúrgico no estado de São Paulo: seleção e implementação de indicadores [dissertação]. São Paulo: Universidade de São Paulo; 2013.. It has been listed as a strategy by the World Health Organization (WHO) in the Global Action Plan for Patient Safety 2021-2030²²22. World Health Organization. Global patient safety action plan 2021-2030: towards eliminating avoidable harm in health care. Geneva: WHO; 2021 [cited [cited 2022 Mar 20] Available from: https://apps.who.int/iris/handle/10665/343477
https://apps.who.int/iris/handle/10665/3... , and the development of “good” indicators is a success factor for benchmarking actions²⁵25. Wind A, Harten WH. Benchmarking specialty hospitals, a scoping review on theory and practice. BMC Health Serv Res. 2017 Apr;17(1):245. https://doi.org/10.1186/s12913-017-2154-y
https://doi.org/10.1186/s12913-017-2154-... .

In addition, 11 indicators could be measured with the available data sources (medical records and data from the official information system), of which 8 process indicators were evaluated in medical records and 3 outcome indicators were measured with SIH-SUS data, exploring the feasibility of using this system to evaluate the quality of surgical care. For the indicators “Screening for postoperative delirium”, “Use of safe surgery checklist” and “Prophylaxis of adequate perioperative venous thromboembolism”, one should institutionalize protocols related to these indicators, which signals an opportunity for improvement for the hospital where the pilot was developed.

The inter-rater reliability, tested by Kappa statistics for eight process indicators, found values that characterize a substantial and almost perfect degree of reliability, which reinforces the solidity of these indicators. The Kappa test is considered adequate to evaluate the reliability of inter-rater categorical and nominal variables, and is frequently used to evaluate the reliability in this type of study²⁰20. Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977 Jun;33(2):363-74. https://doi.org/10.2307/2529786
https://doi.org/10.2307/2529786... .

For the Surgical Site Infection (SSI) indicator, whose data sources may be medical records or system data, it was not possible to analyze the reliability, since the event was not observed in the medical records selected to compose the sample. Most SSIs occur, on average, four to six days after the procedure, and the average length of stay for the procedures included in the study was 1.5 days. Studies indicate that, in procedures in which the postoperative length of stay is short, SSI data, obtained only from hospitalized patients, do not reflect the actual occurrence of infection²⁶26. Oliveira AC, Martins MA, Martinho GH, Clemente WT, Lacerda RA. Estudo comparativo do diagnóstico da infecção do sítio cirúrgico durante e após a internação. Rev Saúde Pública. 2002 Dec;36(6):717-22. https://doi.org/10.1590/S0034-89102002000700009
https://doi.org/10.1590/S0034-8910200200... . There was a four-fold increase in SSI when post-discharge surveillance was performed²⁷27. Oliveira AC, Ciosak SI. Infecção de sítio cirúrgico em hospital universitário: vigilância pós-alta e fatores de risco. Rev Esc Enferm USP. 2007 Jun;41(2):258-63. https://doi.org/10.1590/S0080-62342007000200012
https://doi.org/10.1590/S0080-6234200700... , which leads one to the finding that patient’s medical record does not prove to be the best source of data for monitoring this indicator for the vast majority of procedures performed by the SUS.

For the outcome indicators “Complications related to anesthesia”, “Postoperative sepsis”, “Pulmonary edema or deep vein thrombosis”, measurement via the information system was not possible. The results were null, possibly due to underreporting of secondary events in the Hospital Admission Authorization (AIH) forms. A study on the reliability of AIH data in the country identified a high degree of underreporting of secondary diagnosis²⁸28. Escosteguy CC, Portela MC, Medronho RA, Vasconcellos MT. O Sistema de Informações Hospitalares e a assistência ao infarto agudo do miocárdio. Rev Saúde Pública. 2002 Aug;36(4):491-9. https://doi.org/10.1590/S0034-89102002000400016
https://doi.org/10.1590/S0034-8910200200... . The underreporting of secondary diagnosis in surgical admissions impacts the accuracy of measures calculated for these indicators, which is an opportunity for improvement for the health information system.

The Minimum Health Care Data Set (CMD), conceived in 2015, is a strategy assumed by managers of the three SUS management spheres to reduce fragmentation of information systems, and would replace the main health care information systems in the country. However, despite having been officially instituted by resolution of the Tripartite Intermanagerial Commission²⁹29. Ministério da Saíde (BR). Resolução nº 6, de 25 de agosto de 2016. Institui o Conjunto Mínimo de Dados da Atenção à Saúde e outras providências. Diário Oficial União, 9 set 2016., its implementation has not yet been completed. The CMD implementation would enable the use of administrative, clinical-administrative, and clinical data through a single document, in addition to enabling more specific analyzes, since it would relate the information to the identification of users through integration with the base of the National Health Card system. Despite the efforts and studies carried out in the field of patient safety, the ability to reduce risk, avoid harm, and improve health care safety is still hampered by the absence of high-quality information systems²²22. World Health Organization. Global patient safety action plan 2021-2030: towards eliminating avoidable harm in health care. Geneva: WHO; 2021 [cited [cited 2022 Mar 20] Available from: https://apps.who.int/iris/handle/10665/343477
https://apps.who.int/iris/handle/10665/3... .

The review of existing literature and consensus methods are increasingly used and recommended by the scientific community for this type of study¹⁶16. Soria-Aledo V, Angel-Garcia D, Martinez-Nicolas I, Rebasa Cladera P, Cabezali Sanchez R, Pereira Garc?a LF. Desarrollo y estudio piloto de un conjunto esencial de indicadores para los servicios de cirugía general. Cir Esp. 2016 Nov;94(9):502-10. https://doi.org/10.1016/j.ciresp.2016.06.009
https://doi.org/10.1016/j.ciresp.2016.06... ,³⁰30. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995 Aug;311(7001):376-80. https://doi.org/10.1136/bmj.311.7001.376
https://doi.org/10.1136/bmj.311.7001.376... . The use of the RAND/UCLA method to establish consensus, through the use of remote communication resources (internet), allowed to bring together qualified specialists from various regions of the country. The interest of experts in the studied area, associated with the observed consensus indexes, gave credibility to the results, as can be seen in other studies¹⁴14. Gouvêa C, Travassos C, Caixeiro F, Carvalho LS, Pontes B. Desenvolvimento de indicadores de segurança para monitoramento do cuidado em hospitais brasileiros de pacientes agudos. Rio de Janeiro: Proqualis; 2015 [cited 2022 Mar 20]. Available from: https://proqualis.net/sites/proqualis.net/files/indicadores%20de%20qualidade.pdf
https://proqualis.net/sites/proqualis.ne... ,³¹31. Sánchez Huerta JA. Diseño de indicadores del uso eficiente del quirófano en un hospital de segundo nivel. Cuernavaca: Instituto Nacional de Salud Pública; 2018 [cited 2022 Mar 20]. Available from: http://repositorio.insp.mx:8080/jspui/handle/20.500.12096/7137
http://repositorio.insp.mx:8080/jspui/ha... .

As limitations of this study, we can highlight the performance of the pilot study in a single hospital, whose care profile did not include surgical procedures of the musculoskeletal system, as well as the conduct of the pilot study in a pandemic period, which decreased the sample universe, due to the cancellation of elective surgeries throughout the hospital network. Other limitations, which may be the subject of further studies, are the non-assessment of structural indicators and the non-performance of the feasibility analysis for the collection of indicators.

CONCLUSION

This study contributed to the development of a set of quality indicators in the surgical sphere, which translates as an effective mechanism for measuring the performance and quality of care offered by the hospital service network of RN and Brazil. There are 22 indicators that were considered valid, with 8 process indicators considered reliable and seven result indicators, in which parameters were identified for tabulation using the official information systems. This set of indicators enables the documentation of quality of care, enables comparisons and benchmarking between health units, promotes the identification of priorities through the strengthening and optimization of monitoring strategies and improvements aimed at patient safety in SUS hospitals.

Therefore, this is an innovative proposal, compatible with the Brazilian reality, to guide public managers and researchers in the process of monitoring surgical quality.

REFERENCES

Publication Dates

History