ABSTRACT

OBJECTIVE

This study aims to assess the development and the validity analysis of the Assessment of Risk Management in Health Care Questionnaire (AGRASS).

METHODS

This is a validation study of a measurement instrument following the stages: 1) Development of conceptual model and items; 2) Formal multidisciplinary assessment; 3) Nominal group for validity analysis with national specialists; 4) Development of software and national pilot study in 62 Brazilian hospitals 5) Delphi for validity analysis with the users of the questionnaire. In stages 3 and 5, the items were judged based on face validity, content validity, and utility and viability, by a 1-7 Likert scale (cut-off point: median < 6). Accuracy and reliability of the questionnaire were analyzed with the Confirmatory Factor Analysis and the Cronbach’s alpha.

RESULTS

The initial version of the instrument (98 items) was adapted during stages 1 to 3 for the final version with 40 items, which were considered relevant, of adequate content, useful, and viable. The instrument has 2 dimensions and 9 subdimensions, and the items have closed-ended questions (yes or no). The software for the automatic collection and analysis generates indicators, tables, and automatic graphs for the assessed institution and aggregated data. The adjustment indices confirmed a bi-dimensional model composed of structure and process (X2/gl = 1.070, RMSEA ≤ 0.05 = 0.847, TLI = 0.972), with high reliability for the AGRASS Questionnaire (α = 0.94) and process dimension (α = 0.93), and adequate for the structural dimension (α = 0.70).

CONCLUSIONS

The AGRASS Questionnaire is a potentially useful instrument for the surveillance and monitoring of the risk management and patient safety in health services.

Patient Safety; Risk Management; Validation Studies

INTRODUCTION

The quality of health care is on the global health agenda, and patient safety is one of its critical components¹1. Scott KW, Jha AK. Putting quality on the global health agenda. N Engl J Med. 2014;371:3-5. https://doi.org/10.1056/NEJMp1402157
https://doi.org/10.1056/NEJMp1402157... . In order to improve the quality of health care by increasing the patient safety, health care systems are suggested to implement risk management practices²2. National Patient Safety Agency. Seven steps to patient safety: the full reference guide. London (UK): The Health Foundation; 2004 [citado 06 de novembro de 2019]. Disponível em: https://www.publichealth.hscni.net/sites/default/files/directorates/files/Seven%20steps%20to%20safety.pdf
https://www.publichealth.hscni.net/sites... and quality management/improvement methods applied to the health care safety⁵5. National Advisory Group on the Safety of Patients Group. A promise to learn: a commitment to act. Improving the safety of patients in England. London (UK): Department of Health; 2013 [citado06 de novembro 2019]. Disponível em: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/226703/Berwick_Report.pdf
https://assets.publishing.service.gov.uk... .

The risk management in health care is highlighted as one of the seven stages towards patient safety in the National Patient Safety Agency in the UK²2. National Patient Safety Agency. Seven steps to patient safety: the full reference guide. London (UK): The Health Foundation; 2004 [citado 06 de novembro de 2019]. Disponível em: https://www.publichealth.hscni.net/sites/default/files/directorates/files/Seven%20steps%20to%20safety.pdf
https://www.publichealth.hscni.net/sites... . The National Quality Forum of the United States also includes identifying and reducing risks and hazards as the fourth of its 34 evidence-based safe practices. In the Brazilian health policy context, risk management in health facilities is the first objective of the Programa Nacional de Segurança do Paciente (PNSP – National Patient Safety Program)⁴4. Ministério da Saúde (BR). Portaria Nº 529, de 1º de abril de 2013. Institui o Programa Nacional de Segurança do Paciente (PNSP). Diario Oficial União. 2 abr 2013 [citado 06 de novembro 2019.Disponível em: http://www.saude.mt.gov.br/upload/controle-infeccoes/pasta2/portaria-msgm-n-529-de-01-04-2013.pdf
http://www.saude.mt.gov.br/upload/contro... .

The relationship between patient safety and quality of care occurs in two approaches: in the first one, patient safety is considered a dimension of the quality of health care⁸8. Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy Press; 2001. , ⁹9. World Health Organization. Quality of care: a process for making strategic choices in health systems. Geneva: WHO; 2006 [citado 06 de novembro 2019]. Disponível em: http://www.who.int/management/quality/assurance/QualityCare_B.Def.pdf
http://www.who.int/management/quality/as... ; while in the second one, safety is considered an aspect of health care regardless of their quality¹⁰10. Emanuel L, Berwick D, Conway J, Combes J, Vincent C, Walton M. What exactly is patient safety? In: Henriksen K, Battles JB, Keyes MA, et al., editors. Advances in Patient Safety: new directions and alternative approaches. Vol. 1: Assessment. Rockville, MD: Agency for Healthcare Research and Quality; 2008 [citado 06 de novembro 2019]. Disponível em: https://www.ncbi.nlm.nih.gov/books/NBK43629/
https://www.ncbi.nlm.nih.gov/books/NBK43... . Overall, the World Health Organization⁶6. World Health Organization. Patient safety curriculum guide: multi-professional edition. Geneva: WHO; 2011 [citado 06 de novembro 2019]. Disponível em: http://apps.who.int/iris/bitstream/handle/10665/44641/9789241501958_eng.pdf;jsessionid=4D678ED42006DFE18915FFD5EB868EA6?sequence=1
http://apps.who.int/iris/bitstream/handl... (WHO) includes quality improvement methods as one of the 11 key topics of the multi-professional patient safety curriculum guide. In Brazil, the quality management focused on risk reduction is present in the regulation of health care⁷7. Gama ZAS, Saturno PJ. A segurança do paciente inserida na gestão da qualidade dos serviços de saúde. In: Ministério da Saúde (BR), Agência Nacional de Vigilância Sanitária. Assistência segura: uma reflexão teórica aplicada à prática. Brasília, DF: ANVISA; 2017 [citado 06 de novembro 2019]. p.29-40. (Série Segurança do Paciente e Qualidade em Serviços de Saúde). Disponível em: http://www.saude.pr.gov.br/arquivos/File/0SEGURANCA_DO_PACIENTE/Modulo_1AssistenciaSegura.pdf
http://www.saude.pr.gov.br/arquivos/File... , ¹¹11. Ministério da Saúde (BR), Agência Nacional de Vigilância Sanitária. Resolução da Diretoria Colegiada RDC Nº 63, de 25 de novembro de 2011. Dispõe sobre os Requisitos de Boas Práticas de Funcionamento para os Serviços de Saúde. Brasília, DF: ANVISA; 2011 [citado 06 de novembro 2019]. Disponível em:http://portal.anvisa.gov.br/documents/33880/2568070/rdc0063_25_11_2011.pdf/94c25b42-4a66-4162-ae9b-bf2b71337664
http://portal.anvisa.gov.br/documents/33... .

Considering the regulatory framework of the Brazilian health care, sanitary surveillance professionals are properly supported to oversee the implementation of quality management¹¹11. Ministério da Saúde (BR), Agência Nacional de Vigilância Sanitária. Resolução da Diretoria Colegiada RDC Nº 63, de 25 de novembro de 2011. Dispõe sobre os Requisitos de Boas Práticas de Funcionamento para os Serviços de Saúde. Brasília, DF: ANVISA; 2011 [citado 06 de novembro 2019]. Disponível em:http://portal.anvisa.gov.br/documents/33880/2568070/rdc0063_25_11_2011.pdf/94c25b42-4a66-4162-ae9b-bf2b71337664
http://portal.anvisa.gov.br/documents/33... and risk management in health care¹²12. Ministério da Saúde (BR), Agência Nacional de Vigilância Sanitária. Resolução da Diretoria Colegiada RDC Nº 36, de 25 de julho de 2013. Institui ações para a segurança do paciente em serviços de saúde e dá outras providências. Brasília, DF: ANVISA; 2013 [citado 06 de novembro 2019]. Disponível em: http://portal.anvisa.gov.br/documents/10181/2871504/RDC_36_2013_COMP.pdf/36d809a4-e5ed-4835-a375-3b3e93d74d5e
http://portal.anvisa.gov.br/documents/10... . The two management models can be used to improve safety. The external and internal sanitary inspectors and accountants need to have a wide and flexible insight to identify the implementation of principles and the achievement of objectives of the patient safety management, and not only a registry judgment of one model or another.

Many risk management and quality management models have been suggested for organizations in general¹³13. Gama ZAS, Saturno-Hernández PJ. Inspeção de boas práticas de gestão de riscos em serviços de saúde. Natal, RN: SEDIS-UFRN; 2017. , developing many variations to health care. However, few studies devoted themselves to validating instruments to assess the implementation of quality¹⁴14. Wagner C, Groene O, Thompson CA, Klazinga NS, Dersarkissian M, Arah OA, et al. Development and validation of an index to assess hospital quality management systems. Int J Qual Health Care. 2014;26 Suppl 1:16-26. https://doi.org/10.1093/intqhc/mzu021
https://doi.org/10.1093/intqhc/mzu021... and risks¹⁵15. Briner M, Kessler O, Pfeiffer Y, Wehner T, Manser T. Assessing hospitals’ clinical risk management: development of a monitoring instrument. BMC Health Serv Res. 2010;10:337. https://doi.org/10.1186/1472-6963-10-337
https://doi.org/10.1186/1472-6963-10-337... management activities, showing the need of the area, especially in Brazil, which has no validated instrument for its local reality.

In a partnership between the Agência Nacional de Vigilância Sanitária (ANVISA – National Sanitary Surveillance Agency), the Pan American Health Organization (PAHO), and two universities, a combined model was suggested based on the principles of risk management and the quality of patient safety¹⁴14. Wagner C, Groene O, Thompson CA, Klazinga NS, Dersarkissian M, Arah OA, et al. Development and validation of an index to assess hospital quality management systems. Int J Qual Health Care. 2014;26 Suppl 1:16-26. https://doi.org/10.1093/intqhc/mzu021
https://doi.org/10.1093/intqhc/mzu021... . Based on this model, the Assessment of Risk Management in Health Care (AGRASS) was established, aiming to coordinate external sanitary audits and inspections performed in the Brazilian context and assisting the self-assessment of health care. Considering this background, this study aims to assess the development and the validity analysis of the AGRASS Questionnaire.

METHOD

This is a validation study with a quality and quantity approach. The study was developed in partnership between ANVISA, PAHO, the Universidade Federal do Rio Grande do Norte (UFRN) in Brazil, and the Universidad de Murcia in Spain. This article describes the development and validation of the AGRASS Questionnaire in five stages, as shown in Figure 1 .

Stage 1 – Development of the Conceptual Model and Items

An original and combined conceptual model was developed to show the key structures and processes of patient safety management in health care. This model was based on: wide insight of the literature on national and international technical documents; risk and quality management; models to improve the quality and risk management in health care¹⁶16. Langley GJ, Nolan KM, Norman CL, Provost LP, Nolan TW. The improvement guide: a practical approach to enhancing organizational performance. New York, NY: Jossey-Bass; 1996. ; benchmarking risk management publications in European health care²⁰20. Ministerio de Sanidad y Consumo (ESP). Benchmarking de buenas prácticas en la gestión de riesgos y políticas de reordenación del gobierno clínico en el ámbito hospitalario. Madrid (ESP); 2008 [citado 06 de novembro 2019]. (Informes, Estudios e Investigaciones). Disponível em: https://www.mscbs.gob.es/organizacion/sns/planCalidadSNS/docs/BenchmarkingGestionRiesgosGobiernoClinico.pdf
https://www.mscbs.gob.es/organizacion/sn... ; international recommendations for patient safety practices that included risk management²2. National Patient Safety Agency. Seven steps to patient safety: the full reference guide. London (UK): The Health Foundation; 2004 [citado 06 de novembro de 2019]. Disponível em: https://www.publichealth.hscni.net/sites/default/files/directorates/files/Seven%20steps%20to%20safety.pdf
https://www.publichealth.hscni.net/sites... , ³3. National Quality Forum. Safe practices for better healthcare – 2010 update: a consensus report. Washington, DC; 2010. Disponível em: https://www.qualityforum.org/Publications/2010/04/Safe_Practices_for_Better_Healthcare_%E2%80%93_2010_Update.aspx
https://www.qualityforum.org/Publication... ; and Brazilian sanitary legislation⁴4. Ministério da Saúde (BR). Portaria Nº 529, de 1º de abril de 2013. Institui o Programa Nacional de Segurança do Paciente (PNSP). Diario Oficial União. 2 abr 2013 [citado 06 de novembro 2019.Disponível em: http://www.saude.mt.gov.br/upload/controle-infeccoes/pasta2/portaria-msgm-n-529-de-01-04-2013.pdf
http://www.saude.mt.gov.br/upload/contro... , ¹¹11. Ministério da Saúde (BR), Agência Nacional de Vigilância Sanitária. Resolução da Diretoria Colegiada RDC Nº 63, de 25 de novembro de 2011. Dispõe sobre os Requisitos de Boas Práticas de Funcionamento para os Serviços de Saúde. Brasília, DF: ANVISA; 2011 [citado 06 de novembro 2019]. Disponível em:http://portal.anvisa.gov.br/documents/33880/2568070/rdc0063_25_11_2011.pdf/94c25b42-4a66-4162-ae9b-bf2b71337664
http://portal.anvisa.gov.br/documents/33... , ¹²12. Ministério da Saúde (BR), Agência Nacional de Vigilância Sanitária. Resolução da Diretoria Colegiada RDC Nº 36, de 25 de julho de 2013. Institui ações para a segurança do paciente em serviços de saúde e dá outras providências. Brasília, DF: ANVISA; 2013 [citado 06 de novembro 2019]. Disponível em: http://portal.anvisa.gov.br/documents/10181/2871504/RDC_36_2013_COMP.pdf/36d809a4-e5ed-4835-a375-3b3e93d74d5e
http://portal.anvisa.gov.br/documents/10... . This stage was performed in 2016 and had the initial dimension consensus by two doctoral researchers, who are specialists in management and improvement of quality in health care, and a third professional, who is a quality manager in a hospital certified by the National Accreditation Organization (NAO). The details of the reviewed references and of the conceptual model are available in a previous publication¹³13. Gama ZAS, Saturno-Hernández PJ. Inspeção de boas práticas de gestão de riscos em serviços de saúde. Natal, RN: SEDIS-UFRN; 2017. .

Stage 2 – Proper Assessment

The basic version of the AGRASS Questionnaire was properly reviewed by a group of six experts. The method of analysis was qualitative, by detailed discussion of the questionnaire items, aiming to adjust it to the best systems of patient safety management and considering the Brazilian context. The method should also be concise and consistent with the requirements of the sanitary surveillance. For the saturation analysis and consensus, three meetings were needed, developing the second version of the AGRASS Questionnaire in July 2017.

Stage 3 – Validity Analysis with National Sphere Specialists

The second version was globally analyzed and reviewed item per item by ten ANVISA experts in August 2017. For the consensus, the nominal group technique²¹21. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311(7001):376-80. https://doi.org/10.1136/bmj.311.7001.376
https://doi.org/10.1136/bmj.311.7001.376... was used, with three voting moments (two of them were face-to-face and the other one by distance). The questionnaire items were judged according to four criteria related to face validity, content validity, utility, and feasibility. The criteria were formulated in the form of closed-ended questions on a Likert scale from 1 to 7, on which 1 was “totally disagree” and 7 “totally agree”. The questions were the following: “Is the item relevant to the risk management in health care?,” “Is the item related to the dimension it wants to measure?,” “Is the item feasible in the context of risk management in health care?,” and “Is the feedback information useful to detect improvement opportunities in the risk management in care health?.” Each expert could also include comments regarding each item. This stage was useful for removing items, adapting terms, and adding fundamental item clarifications, developing the third version of the AGRASS Questionnaire.

Stage 4 – Development of Software and National Pilot Study

With the product of the previous stage, the Laboratory of Technological Innovation in Health of UFRN developed the AGRASS System, which consists of two modules – mobile and web. The mobile module provides the questionnaire to be answered in health units, stores them, and sends the inserted responses to the web module. The web module automatically analyzes the data and provides individual or aggregate reports (sets of health care by region, state, municipality, and other groupings of institutions) with the report of the assessed health care and the indicators for the implementation of risk management, the latter showed in Table, Radar chart, and Pareto chart. The degree of risk management implementation in the health care assessed can be described according to individual items and group items. The items allow a positive response and are all equally weighted for the total estimate of risk management implementation in health care. For the descriptive analyses of group items, the percentage of positive responses is estimated, namely, the total positive responses in the group items regarding all items of the AGRASS Questionnaire or all items of the dimensions and subdimensions.

In possession of the electronic format of the AGRASS Questionnaire, a distance learning course was offered for 120 state and municipal professionals of the sanitary surveillance of all Brazilian federation, between September and December of 2017, based on the ad hoc didactic material that contains the AGRASS conceptual model. With the help and supervision of four researchers, the trained professionals applied the AGRASS Questionnaire in health care, which established the pilot study and the assessment of the offered course. The sample size was defined in a stratified manner and proportional to the amount of hospitals with intensive care unit (ICU) in each health care system of Brazil. The professionals made their choice conveniently seeking the largest services of their locality. Sixty-two hospitals participated in the pilot study. All of them had an ICU, because this type of service is considered a priority for sanitary surveillance, within the framework of the Plano Integrado para a Gestão Sanitária da Segurança do Paciente (Integrated Plan for Patient Safety Sanitary Management)²²22. Ministério da Saúde (BR), Agência Nacional de Vigilância Sanitária. Plano integrado para a gestão sanitária da segurança do paciente em serviços de saúde: monitoramento e investigação de eventos adversos e avaliação de práticas de segurança do paciente. Brasília, DF: ANVISA; 2015 [citado 06 de novembro 2019]. Disponível em: https://www20.anvisa.gov.br/segurancadopaciente/index.php/legislacao/item/plano-integrado-para-a-gestao-sanitaria-da-seguranca-do-paciente-em-servicos-de-saude
https://www20.anvisa.gov.br/segurancadop... .

Stage 5 – Validity Analysis with Instrument Users

Finishing the validation stages, the 74 professionals who applied the instrument were invited to participate in a validity analysis using the Delphi technique²¹21. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311(7001):376-80. https://doi.org/10.1136/bmj.311.7001.376
https://doi.org/10.1136/bmj.311.7001.376... . The examination was performed with an electronic validation form sent by e-mail, which included questions about the AGRASS Questionnaire using the same guidelines, criteria, and cut-off points of the validity analysis of stage 3. This examination occurred in February 2018, within 23 days, during which up to three reminders were sent to the non-respondents (10, 15, and 20 days after the examination).

Data analysis

The validity analysis of the questionnaire items had quantitative and qualitative components. The data analysis in stages 3 and 5 considered the median of each of the four criteria. The items that obtained a median equal to or greater than 6 in all criteria were included in the final version of the AGRASS Questionnaire. The items that had some criterion with a median lower than 6 were discussed by the respective groups, and may have undergone adjustments, being submitted to a retrial by the experts. If the item had any medians lower than 6 in the second vote, it was removed. As for the qualitative component, the comments recorded in the assessment form and in the discussion after the presentation of the results of each vote were considered, even if the criterion had reached the cut-off point.

For the reliability analysis, internal consistency was estimated using the Cronbach’s alpha (Appendix 1) for the complete AGRASS Questionnaire. The structure and process dimensions were estimated using the questionnaire responses of the pilot study.

The validity and the bi-dimensional model of the questionnaire were assessed with Confirmatory Factor Analysis (CFA). The structural equation modeling was used to consider the adjustment of the observed data to the structure and process dimensions of risk management in health care. The robust estimation method was applied by Weighted Least Square Mean and Variance Adjusted (WLSMV) with the MPlus v.7 software (Muthén & Muthén). The measures used to verify the adequacy of the model to the data were: (i) Chi-square Ratio/Degrees of Freedom (χ²/gl), (ii) Root Mean Square Error of Approximation (RMSEA), (iii) Tucker-Lewis Index (TLI), (iv) Composite Reliability (CC), and (v) Average Variance Extracted (AVE). The reference values considered for a good adjustment were χ²/gl < 3.0; RMSEA < 0.05; TLI > 0.95; CC ≥ 70, and VME ≥ 50. For the RMSEA, in an ideal situation, the lower value of the 90% confidence of interval (90%CI) includes or approaches zero or is not greater than 0.05; and the higher value is not very large, namely, it is lower than 0.08²³23. Hair JR, Black WC, Babin BJ, Anderson RE, Tatham RL, Sant’Anna MAG, et al. Análise multivariada de dados. 6.ed. Porto Alegre: Bookman; 2009. (Appendix 2).

Ethical aspects

This study was approved by the ethics committee in local research, with the opinion number 75662517.2.0000.5292, and it followed the requirements established for its realization.

RESULTS

Conceptual Model

The conceptual model developed in the first stage of the AGRASS Questionnaire regards the set of structures and processes that aim to constantly improve patient safety ( Figure 2 ). The dimension called “firm structure” consists of five subdimensions: awareness, accountability, ability, action, and safety culture. Additionally, the dimension called “key processes” consists of four subdimensions: risk identification, risk analysis and assessment, risk treatment , and risk communication. The processes joined by arrows represent the natural flow of integration between them.

The conceptual model aimed to present the practices of risk management in health care in a playful way and, for this, used an analogy with the art of a juggler ( Figure 2 ). In this figure, the juggler represents the person responsible for the patients’ safety in health care, as well as all other professionals involved. The balls balanced in the air represent the risk management processes, which must be fully present and integrated with each other. The firm soil and pleasant temperature, contextual factors that help the artist’s work, show the importance of the presence of leadership structures and systems and favorable organizational psychology regarding climate and safety culture.

Proper Assessment and Validity Reviews

The first version of the AGRASS Questionnaire developed in stage 1 included 98 items regarding the dimensions of the conceptual model and was submitted to proper assessment in stage 2. After the analysis by six experts, 50 items were removed and a second version of the AGRASS Questionnaire with 48 items was developed.

The second version was submitted, in stage 3, to the validity analysis by 10 ANVISA professionals, whose aspects are shown in Table 1 . After the first voting round, 12 items did not reach the cut-off point and underwent discussion before a second vote. After the second voting round, only two items obtained a median lower than 6 and were therefore removed. Three other items were removed, even reaching medians greater than or equal to 6, because of the agreement among the experts’ group on whether they were disposable questions. Four items merged into a single item and 26 of them underwent adjustments. Chart 1 shows the results of this stage.

The third version of the questionnaire had 40 items. An important contribution of this stage was the addition of the definition of each dimension and the descriptions about the items, to make the questionnaire clearer and more reliable. This version was used to develop the AGRASS System. Of the 120 sanitary surveillance technicians who enrolled in the course “Inspection of good risk management practices in health services,” 74 were approved and invited to participate in the second phase of the content validity analysis. Of these, 32 technicians signed an informed consent form and answered the validation study questionnaire. The participants’ description is shown in Table 1 . The group had members from all Brazilian regions (2 from the Midwest, 1 from the North, 7 from the Northeast, 17 from the Southeast, and 5 from the South). This group approved the relevance, usefulness, feasibility, and relationship with risk management of all 40 AGRASS Questionnaire items in the first vote, according to the results shown in Chart 1 .

The assessment measures of adjustments were used to verify the adequacy of the model to the data. It showed that the instrument is valid and appropriate to the two dimensions of structure and process, with < 3 chi-square ratio and degrees of freedom (X2/gl = 1.070). The RMSEA criterion reinforced the indication of good adjustment of the tested model (estimate = 0.037; 90%CI 0.000–0.057; probability of RMSEA ≤ 0.05 = 0.847). The TLI test (0.972), the composite reliability (0.931), and the extracted mean variance (0.563) also confirmed the structure of the two-factor model.

The standardized model shows that all items – except item 9, “The institution promoted its professionals training in the area (risk management, quality management, patient safety, etc.)” – presented significant factor loads (p < 0.05). However, as item 9 did not compromise the dimension model in general, it was kept in the questionnaire because of its attributed content validity by the experts.

Regarding the reliability analysis using Cronbach’s alpha, the AGRASS Questionnaire (40 items) had a result of 0.935 (high consistency), the structure dimension (12 items) had 0.704 (acceptable consistency), and the process dimension (28 items) had 0.931 (high consistency). The simulation of removal of items does not significantly change the internal consistency of the constructs assessed, showing that it is not necessary to remove or modify items from the final version.

AGRASS Questionnaire – Final Version

The AGRASS Questionnaire has 40 items grouped into two dimensions and nine subdimensions of risk management in health care: structure (12 items grouped into four subdimensions) and processes (28 items grouped into five subdimensions), as shown in Table 2 . The AGRASS Survey (https://doi.org/10.6084/m9.figshare.7058141.v1), the AGRASS System (https://agrass.lais.huol.ufrn.br), which contains links to the mobile module in the Apple Store and Google Play, and an example of automatic report produced by the system (https://doi.org/10.6084/m9.figshare.7045454.v1) are available online.

DISCUSSION

This study has a potential contribution to the improvement of surveillance and management of health care, as it provides a valid instrument for the assessment and monitoring of the organizational conditions that ensure the provision of health care insurance. After five stages of validation, the AGRASS Questionnaire was considered relevant, valid, useful, and feasible to measure the implementation of risk management in Brazilian health care. The questionnaire can be used to: assess health care regarding the implementation of risk management practices, guiding the planning of improvement interventions; review health care to measure improvement after performing interventions to implement risk management practices; and compare individual or aggregated health care (e.g. region, municipality, or state) regarding the level of risk management practices implementation. The instrument can be applied by external evaluators (sanitary inspectors, certifiers, accreditations and auditors, among others) or in self-assessment initiatives of health care systems.

Contributions to Patient Safety

The AGRASS Questionnaire meets an international and national need. Information systems are one of the six main strategic interventions that WHO stimulates for the quality of health care⁹9. World Health Organization. Quality of care: a process for making strategic choices in health systems. Geneva: WHO; 2006 [citado 06 de novembro 2019]. Disponível em: http://www.who.int/management/quality/assurance/QualityCare_B.Def.pdf
http://www.who.int/management/quality/as... and, particularly in the WHO Patient Safety Program, safety measurement is a priority objective²⁴24. World Health Organization. Patient safety: making health care safer. Geneva: WHO; 2017 [citado 06 de novembro 2019]. Disponível em: http://apps.who.int/iris/bitstream/handle/10665/255507/WHO-HIS-SDS-2017.11-eng.pdf?sequence=1&isAllowed=y’
http://apps.who.int/iris/bitstream/handl... . However, even well-developed countries in patient safety, like England, have been urged to improve patient safety information collection⁵5. National Advisory Group on the Safety of Patients Group. A promise to learn: a commitment to act. Improving the safety of patients in England. London (UK): Department of Health; 2013 [citado06 de novembro 2019]. Disponível em: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/226703/Berwick_Report.pdf
https://assets.publishing.service.gov.uk... . This improvement opportunity was also evident in an assessment of government decisions regarding patient safety in countries of the Organization for Economic Co-operation and Development (OECD), which showed that assessment and monitoring are not very advanced²⁵25. Hasegawa T, Fujita S. Patient Safety Policies: experiences, effects and priorities; lessons from OECD member states. Tokyo (JPN): Ministry of Health, Labour and Welfare; 2018 [citado 06 de novembro 2019]. Disponível em: https://www.mhlw.go.jp/file/06-Seisakujouhou-10800000-Iseikyoku/0000204013.pdf
https://www.mhlw.go.jp/file/06-Seisakujo... .

In Brazil, although the PNSP assessment and monitoring were planned²⁶26. Ministério da Saúde; Fundação Oswaldo Cruz; Agência Nacional de Vigilância Sanitária. Documento de referência para o Programa Nacional de Segurança do Paciente. Brasília, DF: Ministério da Saúde; 2014 [citado 06 de novembro 2019]. Disponível em: http://bvsms.saude.gov.br/bvs/publicacoes/documento_referencia_programa_nacional_seguranca.pdf
http://bvsms.saude.gov.br/bvs/publicacoe... , there is a wide scope for improvement in this aspect. The Sistema de Notificações para a Vigilância Sanitária (NOTIVISA – Sanitary Surveillance Notification System) and the annual assessment of patient safety practices implemented by ANVISA are important, but insufficient²⁶26. Ministério da Saúde; Fundação Oswaldo Cruz; Agência Nacional de Vigilância Sanitária. Documento de referência para o Programa Nacional de Segurança do Paciente. Brasília, DF: Ministério da Saúde; 2014 [citado 06 de novembro 2019]. Disponível em: http://bvsms.saude.gov.br/bvs/publicacoes/documento_referencia_programa_nacional_seguranca.pdf
http://bvsms.saude.gov.br/bvs/publicacoe... , ²⁷27. Ministério da Saúde, Agência Nacional de Vigilância Sanitária, Gerência Geral de Tecnologia em Serviços de Saúde. Relatório da autoavaliação nacional das práticas de segurança do paciente em serviços de saúde - 2017 – REVISADO. Brasília, DF; 2018 [citado 06 de novembro 2019]. Disponível em: http://portal.anvisa.gov.br/documents/33852/459495/Relat%C3%B3rio+com+os+resultados+da+Autoavalia%C3%A7%C3%A3o+2017/8c243765-5190-46d2-a9ee-52f53cda2294
http://portal.anvisa.gov.br/documents/33... . Thus, the AGRASS Questionnaire presents itself as an alternative instrument to produce information for decision-making in patients’ safety based on 40 items, which can be comparable to 38 simple indicators and 2 composite indicators (items 15 and 30). This new instrument can be used to measure the patient safety at the level on which it has a greater lack of assessment, which is its organizational component. The quality of care should be monitored from a systemic perspective, including the technical level of individual care, but also the organizational levels of the establishment of health and the health care¹⁸18. Saturno-Hernández PJ. Métodos y herramientas para la realización de ciclos de mejora de la calidad en servicios de salud. Cuernavaca (MEX): Instituto Nacional de Salud Pública; 2015. . Patient safety indicators have been proposed for all levels²⁸28. Gama ZAS, Saturno-Hernández PJ, Ribeiro DNC, Freitas MR, Medeiros PJ, Batista AM, et al. Desenvolvimento e validação de indicadores de boas práticas de segurança do paciente: Projeto ISEP-Brasil. Cad Saude Publica. 2016;32(9):e00026215. https://doi.org/10.1590/0102-311X00026215
https://doi.org/10.1590/0102-311X0002621... , ²⁹29. Gouvêa CSD, Travassos C. Indicadores de segurança do paciente para hospitais de pacientes agudos: revisão sistemática. Cad Saude Publica. 2010:26(6):1061-78. https://doi.org/10.1590/S0102-311X2010000600002
https://doi.org/10.1590/S0102-311X201000... , but few are based on organizational criteria. The measurement of safety should be complete, not only based on adverse results or events²⁸28. Gama ZAS, Saturno-Hernández PJ, Ribeiro DNC, Freitas MR, Medeiros PJ, Batista AM, et al. Desenvolvimento e validação de indicadores de boas práticas de segurança do paciente: Projeto ISEP-Brasil. Cad Saude Publica. 2016;32(9):e00026215. https://doi.org/10.1590/0102-311X00026215
https://doi.org/10.1590/0102-311X0002621... , ³⁰30. Vincent C, Burnett S, Carthey J. The measurement and monitoring of safety: drawing together academic evidence and practical experience to produce a framework for safety measurement and monitoring. London (UK): The Health Foundation; 2013 [citado 06 de novembro 2019]. Disponível em: https://www.hqsc.govt.nz/assets/Capability-Leadership/PR/2013-The-measurement-and-monitoring-of-safety-Health-Foundation-Dec-2016.pdf
https://www.hqsc.govt.nz/assets/Capabili... . Measuring the quality of care is not easy and requires balance.

The Validity of the AGRASS Questionnaire

The stages of experts’ validation enabled us to reach the objectives of developing and validating the content of the AGRASS Questionnaire. In addition to the extensive review of the literature, the final items were approved regarding the face validity, which is the logical and clear importance for risk management; content validity, which is the ability of each item to measure the expressed concept of each dimension and subdimension of the questionnaire; and the construct validity, which measures how much the items adjust to the dimensional structure of the questionnaire. Content validation seeks to value opinion makers and recognizes the experts’ important contribution²³23. Hair JR, Black WC, Babin BJ, Anderson RE, Tatham RL, Sant’Anna MAG, et al. Análise multivariada de dados. 6.ed. Porto Alegre: Bookman; 2009. . Regarding reliability, although it had acceptable internal consistency, it is emphasized that it depends on the evaluator’s training and on the correct application of the instrument, demanding and assessing the necessary vouchers. In addition, composite reliability values reinforce this aspect.

The name AGRASS Questionnaire emphasizes the risk management in health care, but it should be understood as an instrument for assessing patient safety management based on risk management and quality models. Regarding the size of the structure, this study highlights the structuring of leadership systems for patient safety and an environment defined by a favorable organizational culture. On the size of the risk management processes, those were included (identification, analysis and assessment, treatment, and communication) appear directly or indirectly in most risk and quality management models¹³13. Gama ZAS, Saturno-Hernández PJ. Inspeção de boas práticas de gestão de riscos em serviços de saúde. Natal, RN: SEDIS-UFRN; 2017. , ¹⁶16. Langley GJ, Nolan KM, Norman CL, Provost LP, Nolan TW. The improvement guide: a practical approach to enhancing organizational performance. New York, NY: Jossey-Bass; 1996. , ²⁸28. Gama ZAS, Saturno-Hernández PJ, Ribeiro DNC, Freitas MR, Medeiros PJ, Batista AM, et al. Desenvolvimento e validação de indicadores de boas práticas de segurança do paciente: Projeto ISEP-Brasil. Cad Saude Publica. 2016;32(9):e00026215. https://doi.org/10.1590/0102-311X00026215
https://doi.org/10.1590/0102-311X0002621... . Risk identification subdimension includes retrospective methods (such as indicators), in real-time methods (such as checklists), and prospective methods (such as the safety walk-around)³3. National Quality Forum. Safe practices for better healthcare – 2010 update: a consensus report. Washington, DC; 2010. Disponível em: https://www.qualityforum.org/Publications/2010/04/Safe_Practices_for_Better_Healthcare_%E2%80%93_2010_Update.aspx
https://www.qualityforum.org/Publication... , ¹³13. Gama ZAS, Saturno-Hernández PJ. Inspeção de boas práticas de gestão de riscos em serviços de saúde. Natal, RN: SEDIS-UFRN; 2017. . Another issue emphasized by the model is that processes should not be independent, but integrated with each other, showed in the subdimension “integration of processes.”

The fact that the AGRASS Questionnaire was originally developed for Brazilian health care favors its validity in the local context. Other instruments with similar objectives were developed abroad to evaluate the management of clinical risks¹⁵15. Briner M, Kessler O, Pfeiffer Y, Wehner T, Manser T. Assessing hospitals’ clinical risk management: development of a monitoring instrument. BMC Health Serv Res. 2010;10:337. https://doi.org/10.1186/1472-6963-10-337
https://doi.org/10.1186/1472-6963-10-337... and quality management¹⁴14. Wagner C, Groene O, Thompson CA, Klazinga NS, Dersarkissian M, Arah OA, et al. Development and validation of an index to assess hospital quality management systems. Int J Qual Health Care. 2014;26 Suppl 1:16-26. https://doi.org/10.1093/intqhc/mzu021
https://doi.org/10.1093/intqhc/mzu021... but were not transculturally adapted to Brazil. Although these instruments seem similar to the AGRASS at first, they have important differences in their conceptual model and items.

The AGRASS Electronic Assessment System

In order to help the periodic use of the AGRASS Questionnaire, the project sought to combine technological innovation to overcome usual barriers to the institutionalization of health care assessment processes in Brazil. As health care systems do not always have people with the time and skills to collect, analyze, and produce reports to guide decision-making, the AGRASS System helps these processes by providing a mobile application on Android and iOS systems. The application automatically provides service details and compliance tables regarding items, dimensions and subdimensions, as well as decision-making assessment charts. The web module provides the aggregation of individual assessments to measure the implementation in a network of services or geographic region. Reports are generated in .pdf files and databases in .xls files, remaining available for further analysis.

Limitations and Future Studies

The AGRASS Questionnaire does not intend to thoroughly describe all safety management practices, but those most internationally recommended and nationally essential. Although it can be used to assist the inspection of sanitary standards, the instrument is not a complete inspection script, as it merely guides the assessment of risk management in health care.

It is suggested to conduct future descriptive studies of the application of this instrument in different regions of the national context, in addition to analyses of factors associated with good or poor results of implementation of risk management in health care.

CONCLUSION

The AGRASS Questionnaire is an instrument considered valid after five stages of validation and potentially useful for the surveillance and measurement of the organizational structure for patient safety in Brazilian health care. The AGRASS System, composed of a mobile module and another web module, provides an efficient analysis of the implementation of risk management in health care, helping the institutionalization of the assessment focused on patient safety in health care and systems, which is a priority component and indispensable for the quality of health care.

REFERENCES

Publication Dates

History