Abstract:

This study aimed to reassess the psychometric properties of the Hospital Survey on Patient Safety Culture (HSOPSC) adapted to Portuguese for use in the Brazilian context. An observational, cross-sectional study was performed in a large, private, non-profit, acute care hospital, reference in patient safety, in a major Brazilian metropolis. Participants were selected from a non-probability sample of all eligible personnel in the various hospital departments invited to participate in the study. Reliability of the HSOPSC was assessed by estimating Cronbach’s alpha for each dimension. confirmatory factor analysis (CFA), a matrix of correlations between the dimensions, and exploratory structural equation modelling (ESEM) were used in exploratory and confirmatory analyses of the structural validity of the construct. The overall response rate was 18.7% (n = 1,439). Four dimensions (“overall perceptions of patient safety”; “staffing”; “teamwork across units”; and “non-punitive response to error”) returned problems of internal consistency. CFA returned acceptable fit with the original 12-dimension model. Correlations between the dimensions of the original 12-dimension model indicated discriminant validity problems, while residual variance was greater than 0.70 in 13 items. The ESEM of the original 12-dimension model returned good fit, with the following indices: CFI = 0.985; TLI = 0.968, and RMSEA = 0.026 (90%CI: 0.024-0.029). Although better than those of the first evaluation, the results obtained in this validity and reliability reassessment of the Brazilian version of the HSOPSC require further research.

Keywords:
Patient Safety; Organizational Culture; Health Care Surveys; Reproducibility of Results

Resumo:

O estudo teve como objetivo reavaliar as propriedades psicométricas da versão do Hospital Survey on Patient Safety Culture (HSOPSC) adaptada para o português, para uso no contexto brasileiro. Foi realizado um estudo observacional, transversal, em um hospital particular sem fins lucrativos, referência em segurança do paciente, localizado em uma metrópole brasileira. Os participantes foram selecionados a partir de uma amostra não-probabilística de todos os funcionários elegíveis de diversos departamentos hospitalares, convidados a participar no estudo. A confiabilidade do HSOPSC foi avaliada pela estimativa do alfa de Cronbach para cada dimensão. Foram usadas análise fatorial confirmatória (AFC), uma matriz de correlações entre as dimensões e modelagem de equações estruturais (MEE) na análise exploratória e confirmatória da validade estrutural do construto. Houve uma taxa global de resposta de 18,7% (n = 1.439). Quatro dimensões (“percepção geral de segurança do paciente”; “recursos humanos”; “trabalho de equipe entre unidades” e “resposta não-punitiva aos erros”) apresentaram problemas de consistência interna. A AFC mostrou um ajuste aceitável em relação ao modelo original de 12 dimensões. As correlações entre as dimensões do modelo original de 12 dimensões indicaram problemas de validade discriminante, enquanto a variância residual foi maior de 0,70 em 13 itens. A MEE do modelo original de 12 dimensões mostrou bom ajuste, com os seguintes índices: CFI = 0,985; TLI = 0,968 e RMSEA = 0,026 (IC90%: 0,024-0,029). Embora tenham superado a primeira avaliação, os resultados obtidos nesta reavaliação da validade e confiabilidade da versão do questionário traduzida e adaptada para o contexto brasileiro indicam a necessidade de mais estudos.

Palavras-chave:
Segurança do Paciente; Cultura Organizacional; Pesquisas sobre Serviços de Saúde; Reprodutibilidade dos Testes

Resumen:

El estudio tuvo como objetivo revisar las propiedades psicométricas de la versión del Hospital Survey on Patient Safety Culture (HSOPSC) adaptado al portugués, para su uso en el contexto brasileño. Se realizó un estudio transversal observacional, en un hospital grande con cuidados intensivos, privado, sin ánimo de lucro, punto de referencia en seguridad del paciente, situado en una de las mayores metrópolis brasileñas. Los participantes se seleccionaron de un muestreo no probabilístico entre todo el personal elegible de los diversos departamentos hospitalarios invitados a participar en el estudio. La fiabilidad del HSOPSC se evaluó estimando el alfa de Cronbach para cada dimensión. Se utilizaron tanto el análisis factorial confirmatorio (AFC), como la matriz de correlaciones entre las dimensiones y los modelos de ecuaciones estructurales exploratorios (ESEM) en el análisis exploratorio y confirmatorio para la validez estructural del constructo. El porcentaje general de respuesta fue de un 18,7% (n = 1,439). Hubo cuatro dimensiones (“percepciones generales de seguridad del paciente”; “dotación de personal”; “trabajo en equipo en las unidades”; y “respuesta al error no-punitiva”), problemas recurrentes de consistencia interna. El AFC mostró un ajuste aceptable con el modelo original de 12-dimensiones. Las correlaciones entre las dimensiones del modelo original de 12-dimensiones indicaron problemas de validez discriminante, mientras que varianza residual era mayor a 0,70 en 13 ítems. Los ESEM del modelo original de 12-dimensiones tuvieron un buen ajuste con los siguientes índices: CFI = 0,985; TLI = 0,968 y RMSEA = 0,026 (90%CI: 0,024-0,029). A pesar de que eran mejores que los de la primera evaluación, los resultados obtenidos en esta revisión de la validez y fiabilidad de la versión del cuestionario, traducido y adaptado al contexto brasileño, es necesario que haya más investigaciones al respecto sobre este asunto.

Palabras-clave:
Seguridad del Paciente; Cultura Organizacional; Encuestas de Atención de Salud; Reproducibilidad de los Resultados

Background

Safety culture comprises values, beliefs, and standards regarding what is important in an organization and what safety-related attitudes and behavior are valued, supported, and expected ¹1. Sorra J, Gray L, Streagle S, Famolaro T, Yount N, Behm J. AHRQ Hospital Survey on Patient Safety Culture: user's guide. Rockville: Agency for Healthcare Research and Quality; 2016. (AHRQ Publication, 15-0049-EF).. It is a multidimensional concept, defined in the health service context as the product of individual and group values, attitudes, perceptions, competences, and behavior patterns that determine the commitment to, and the style and proficiency of, an organization’s patient safety management ²2. Nieva VF, Sorra J. Safety culture assessment: a tool for improving patient safety in healthcare organizations. Qual Saf Health Care 2003; 12 Suppl 2:ii17-23..

In health care organizations, safety culture assessment is a key strategy in the world movement to improve health care quality and patient safety. There is evidence of the association between high safety culture scores and positive clinical outcomes, especially in hospitals, as a result of interventions to improve the quality of patient care ³3. Mardon RE, Khanna K, Sorra J, Dyer N, Famolaro T. Exploring relationships between hospital patient safety culture and adverse events. J Patient Saf 2010; 6:226-32.^,44. Di Cuccio MH. The relationship between patient safety culture and patient outcomes: a systematic review. J Patient Saf 2014; 22:11-8.^,55. Fan CJ, Pawlik TM, Daniels T, Vernon N, Banks K, Westby P, et al. Association of safety culture with surgical site infection outcomes. J Am Coll Surg 2016; 222:122-8..

Among the instruments developed to assess safety culture in hospitals, the Hospital Survey on Patient Safety Culture (HSOPSC) is one of the most widely used ¹1. Sorra J, Gray L, Streagle S, Famolaro T, Yount N, Behm J. AHRQ Hospital Survey on Patient Safety Culture: user's guide. Rockville: Agency for Healthcare Research and Quality; 2016. (AHRQ Publication, 15-0049-EF).^,66. Reis CT, Paiva SG, Sousa P. The patient safety culture: a systematic review by characteristics of Hospital Survey on Patient Safety Culture dimensions. Int J Qual Health Care 2018; 30:660-77.. Developed in American English by the Agency for Healthcare Research and Quality (AHRQ), this instrument contains 42 items and 12 dimensions and was conceived as a questionnaire to be self-administered, to ensure respondents’ anonymity. The theoretical model underpinning its development is fully documented by the AHRQ ¹1. Sorra J, Gray L, Streagle S, Famolaro T, Yount N, Behm J. AHRQ Hospital Survey on Patient Safety Culture: user's guide. Rockville: Agency for Healthcare Research and Quality; 2016. (AHRQ Publication, 15-0049-EF).. Since its introduction in 2004, the HSOPSC has been used in 93 countries and some 40 translated versions exist ⁷7. Agency for Healthcare Research and Quality. International use of SOPS. http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/pscintusers.html (accessed on 24/Nov/2017).
http://www.ahrq.gov/professionals/qualit... . In psychometric analysis, the dimensional structure of the original model of the instrument has been confirmed by several studies ⁸8. Olsen E. Reliability and validity of the Hospital Survey on Patient Safety Culture at a Norwegian hospital. In: Øvretveit J, Sousa PJ, editors. Quality and safety improvement research: methods and research practice from the International Quality Improvement Research Network (QIRN). Lisboa: Escola Nacional de Saúde Pública, Universidade Nova de Lisboa; 2008. p. 173-86.^,99. Chen IC, Li HH. Measuring patient safety culture in Taiwan using the Hospital Survey on Patient Safety Culture (HSOPSC). BMC Health Serv Res 2010; 10:152.^,1010. Ito S, Seto K, Kigawa M, Fujita S, Hasegawa T, Hasegawa T. Development and applicability of Hospital Survey on Patient Safety Culture (HSOPS) in Japan. BMC Health Serv Res 2011; 11:28.^,1111. Moghri J, Arab M, Saari AA, Nateqi E, Forooshani AR, Ghiasvand H, et al. The psychometric properties of the Farsi version of 'Hospital Survey on Patient Safety Culture' in Iran's hospitals. Iran J Public Health 2012; 42:80-6.^,1212. Robida A. Hospital Survey on Patiet Safety Culture in Slovenia: a psychometric evaluation. Int J Qual Health Care 2013; 25:469-75.^,1313. Eiras M, Escoval A, Grillo IM, Silva-Fortes C. The hospital survey on patient safety culture in Portuguese hospitals: instrument validity and reliability. Int J Health Care Qual Assur 2014; 27:111-22. and underwent alterations in others ¹⁴14. Smits M, Wagner C, Spreeuwenberg P, van der Wal G, Groenewegen PP. Measuring patient safety culture: an assessment of the clustering of responses at unit level and hospital level. Qual Saf Health Care 2009; 18:292-6.^,1515. Blegen MA, Gearhart S, O'Brien R, Sehgal NL, Alldredge BK. AHRQ's Hospital Survey on Patient Safety Culture: psychometric analyses. J Patient Saf 2009; 5:139-44.^,1616. Bodur S, Filiz E. Validity and reliability of Turkish version of 'Hospital Survey on Patient Safety Culture' and perception of patient safety in publichospitals in Turkey. BMC Health Serv Res 2010; 10:28..

In Brazil, the transcultural adaptation of the HSOPSC was previously studied ¹⁷17. Reis CT, Laguardia J, Martins M. Adaptação transcultural da versão brasileira do Hospital Survey on Patient Safety Culture: etapa inicial. Cad Saúde Pública 2012; 28:2199-210.^,1818. Reis CT, Laguardia J, Vasconcelos AGG, Martins M. Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture (HSOPSC): a pilot study. Cad Saúde Pública 2016; 32:e00115614. in a sample of 322 participants at two large (more than 149 beds), non-profit, public and private, acute-care general hospitals in two Brazilian states of the Southeast Region (one in Minas Gerais, with no history of quality improvement initiatives, and the other in the city of Rio de Janeiro, in preparation to submit to the hospital accreditation process). Confirmatory factor analysis (CFA) did not uphold the original 12-dimension, 42-item model of the instrument, but the exploratory nature of that study, the context in which the questionnaire was administered, and the sample may all have influenced that finding. Exploratory factor analysis returned a satisfactory 10-factor, 38-item model. However, the limitations of the study recommended caution when applying changes to the original model, pointing to a need to reassess the original 12-dimension, 42-item HSOPSC model adapted for use in the Brazilian context.

Considering the widespread use of the HSOPSC in diverse socioeconomic and cultural contexts and in comparative approaches, including organizations with more mature quality and patient safety cultures, this study aimed to reassess the reliability and validity of the original version of the HSOPSC adapted to Portuguese ¹⁷17. Reis CT, Laguardia J, Martins M. Adaptação transcultural da versão brasileira do Hospital Survey on Patient Safety Culture: etapa inicial. Cad Saúde Pública 2012; 28:2199-210. for use in the Brazilian context and administered to the personnel of a hospital that is a reference in the area of patient safety.

Methods

Study design and setting

This quantitative, observational, cross-sectional study was conducted at a large, private, non-profit, acute-care hospital in a major metropolis in Brazil’s Southeast Region. Since the mid-1990s, this facility has invested in quality and subscribed to certification and accreditation programes, prioritizing quality of care and especially patient safety.

Sample

This was a non-probability sample, comprising all the eligible personnel present or on duty at the hospital during the data collection period. The personnel of various departments of the hospital were invited to participate, and data were collected electronically from September 1st to November 1st, 2013. The inclusion criteria followed the recommendations of the instrument’s developers ¹1. Sorra J, Gray L, Streagle S, Famolaro T, Yount N, Behm J. AHRQ Hospital Survey on Patient Safety Culture: user's guide. Rockville: Agency for Healthcare Research and Quality; 2016. (AHRQ Publication, 15-0049-EF).: participants should work at least 20 hours per week at the hospital.

Participation in the study was voluntary and, after agreeing to participate in the research and with their anonymity guaranteed, participants completed the questionnaire electronically. As a strategy to maximize the percentage response rate, communication took place via intranet. This communication channel was chosen to foster awareness among health care personnel, leaders, and managers in the various hospital departments of the importance of the subject and the focus and objectives of the study, to, subsequently, allow the instrument to be completed online.

Data collection instrument

The instrument used was the HSOPSC translated into Portuguese and adapted for use in Brazil ¹⁷17. Reis CT, Laguardia J, Martins M. Adaptação transcultural da versão brasileira do Hospital Survey on Patient Safety Culture: etapa inicial. Cad Saúde Pública 2012; 28:2199-210.. The HSOPSC contains 12 dimensions and 42 items and takes about 15 minutes to complete. Seven dimensions are related to features of the hospital department or unit where the respondent works: (i) communication openness (3 items); (ii) feedback and communication about error (3 items); (iii) organizational learning and continuous improvement (3 items); (iv) supervisor/manager expectations and actions promoting patient safety (4 items); (v) non-punitive response to errors (3 items); (vi) teamwork within units (4 items); and (vii) staffing (4 items). Three dimensions measure hospital-wide safety culture: (i) management support for patient safety (3 items); (ii) handoffs & transitions (4 items); and (iii) teamwork across units (4 items). There are two other output dimensions: (i) overall perceptions of patient safety (4 items); and (ii) frequency of events reported (3 items).

The HSOPSC also contains questions about the respondent’s characteristics (sex, occupational category, education level, and time of employment at the hospital). Most items are answered on a five-point Likert-type scale reflecting level of agreement, from “strongly disagree” (1) to “strongly agree” (5), with a neutral category “neither agree nor disagree” (3). Other items are answered on a five-point frequency scale from “never” (1) to “always” (5) ¹1. Sorra J, Gray L, Streagle S, Famolaro T, Yount N, Behm J. AHRQ Hospital Survey on Patient Safety Culture: user's guide. Rockville: Agency for Healthcare Research and Quality; 2016. (AHRQ Publication, 15-0049-EF).. Most of the questionnaire items are written positively, but 15 are worded negatively and reverse coded so that the highest scores indicate positive responses regarding patient safety ¹1. Sorra J, Gray L, Streagle S, Famolaro T, Yount N, Behm J. AHRQ Hospital Survey on Patient Safety Culture: user's guide. Rockville: Agency for Healthcare Research and Quality; 2016. (AHRQ Publication, 15-0049-EF)..

Data analysis

The construct was examined for reliability and structural validity. The analyses used only complete questionnaires and no strategy was applied to treat for missing data. Reliability is the degree to which the results obtained from the HSOPSC in the study sample could be replicated. To compare the results with previous psychometric assessments ⁸8. Olsen E. Reliability and validity of the Hospital Survey on Patient Safety Culture at a Norwegian hospital. In: Øvretveit J, Sousa PJ, editors. Quality and safety improvement research: methods and research practice from the International Quality Improvement Research Network (QIRN). Lisboa: Escola Nacional de Saúde Pública, Universidade Nova de Lisboa; 2008. p. 173-86.^,1010. Ito S, Seto K, Kigawa M, Fujita S, Hasegawa T, Hasegawa T. Development and applicability of Hospital Survey on Patient Safety Culture (HSOPS) in Japan. BMC Health Serv Res 2011; 11:28.^,1212. Robida A. Hospital Survey on Patiet Safety Culture in Slovenia: a psychometric evaluation. Int J Qual Health Care 2013; 25:469-75.^,1313. Eiras M, Escoval A, Grillo IM, Silva-Fortes C. The hospital survey on patient safety culture in Portuguese hospitals: instrument validity and reliability. Int J Health Care Qual Assur 2014; 27:111-22.^,1818. Reis CT, Laguardia J, Vasconcelos AGG, Martins M. Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture (HSOPSC): a pilot study. Cad Saúde Pública 2016; 32:e00115614.^,1919. Andrade LEL, Melo LOM, Silva IG, Souza RM, Lima ALB, Freitas MR, et al. Adaptação e validação do Hospital Survey on Patient Safety Culture em versão brasileira eletrônica. Epidemiol Serv Saúde 2017; 26:455-68.^,2020. Sorra J, Nieva V. Hospital Survey on Patient Safety Culture. Rockville: Agency for Healthcare Research and Quality; 2004. (AHRQ Publication, 04-0041).^,2121. Smits M, Christiaans-Dingelhoff I, Wagner C, Wal Gv, Groenewegen PP. The psychometric properties of the 'Hospital Survey on Patient Safety Culture' in Dutch hospitals. BMC Health Serv Res 2008; 8:230.^,2222. Pfeiffer Y, Manser T. Development of the German version of the Hospital Survey on Patient Safety Culture: dimensionality and psychometric properties. Saf Sci 2010; 48:1452-62.^,2323. Waterson P, Griffiths P, Stride C, Murphy J, Hignett S. Psychometric properties of the Hospital Survey on Patient Safety Culture: findings from the UK. Qual Saf Health Care 2010; 19:2e.^,2424. Sarac C, Flin R, Mearns K, Jackson J. Hospital survey on patient safety culture: psychometrics analysis on a Scottish sample. BMJ Qual Saf 2011; 20:842-8.^,2525. Perneger TV, Staines A, Kundig F. Internal consistency, factor structure and construct validity of the French version the Hospital Survey on Patient Safety Culture. BMJ Qual Saf 2014; 23:389-97., the reliability of the HSOPSC was assessed through its internal consistency using the Cronbach’s alpha estimated for each dimension, with values above 0.6 considered acceptable ²⁰20. Sorra J, Nieva V. Hospital Survey on Patient Safety Culture. Rockville: Agency for Healthcare Research and Quality; 2004. (AHRQ Publication, 04-0041)..

In analyzing for structural validity, at the confirmatory and exploratory stages, CFA ²⁶26. Brown TA. Confirmatory factor analysis for applied research. New York: The Guilford Press; 2006., a correlation matrix between the dimensions, and exploratory structural equation modelling (ESEM) were used. The ESEM was chosen because of its ability to combine the characteristics of exploratory and confirmatory factor analysis, which makes it possible to explore the underlying factorial structure without the constraints imposed by CFA, in addition to offering CFA’s parameter and goodness-of-fit tests ²⁷27. Kelloay EK. Using Mplus for structural equation modeling: a researcher’s guide. 2nd Ed. Thousand Oaks: Sage Publications; 2015..

First of all, the dimensional structure of the original 12-dimension instrument was assessed by CFA. Since the dependent variables in the HSOPSC are ordinal (five-point, Likert-type response options), the weighted least squares mean and variance-adjusted (WLSMV) estimator was used. In CFA, model fit was evaluated by three indices: root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker-Lewis index (TLI). Absolute model fit was evaluated by RMSEA, in agreement with the use of the WLSMV estimator. RMSEA measures poor quality of fit by incorporating a penalty function to evaluate the lack of parsimony in the model’s degrees of freedom. The lower the RMSEA value, the better the fit: less than 0.05 is “good”, while values greater than 0.10 indicate “poor” fit and that the model should be rejected. The CFI and TLI indices, used as measures of incremental fit, enable the proposed model to be compared with an independent (null) model; the values of these indices range from 0 to 1, with values higher than 0.90 indicating an appropriate model ²⁶26. Brown TA. Confirmatory factor analysis for applied research. New York: The Guilford Press; 2006..

Subsequently, the dimensional structure was explored by ESEM, using the WLSMV estimator and oblique rotation (GEOMIN). Item factor loadings were also examined at this stage, with values from 0.35 to 0.50 considered acceptable; residual variance (uniqueness) above 0.70 was considered inadequate, as was the presence of cross-loading with a difference of more than 0.20 between loadings; in analyzing correlation between factors to determine discriminant validity, a value of 0.85 or higher was considered unsuitable ²⁸28. Reichenheim ME, Hokerberg YHM, Moraes CL. Assessing construct structural validity of epidemiological measurement tools: a seven-step roadmap. Cad Saúde Pública 2014; 30:927-39..

SPSS version 17.0 (https://www.ibm.com/br-pt/products/spss-statistics) was used to build the database, manage data, treat variables, perform descriptive statistical analyses and reliability calculations, and examine data factorability. Mplus version 7.0 (https://www.statmodel.com/) was used to calculate correlations between the dimensions, CFA, and ESEM.

This study was approved by the Ethics Research Committee of the Sergio Arouca National School of Public Health, Oswaldo Cruz Foundation (process n. 177/2011).

Results

Study sample and descriptive statistics

Of the 7,140 personnel invited to participate, 1,512 responded to the questionnaire. After exclusion of 28 questionnaires of participants who declared having no intention to collaborate and 45 records with more than 30% missing data, the final sample totalled 1,439 questionnaires (response rate = 18.7%).

The study sample comprised a young population, of which 82.7% (n = 1,082) were under 31 years old. Most respondents reported having completed higher education (99%; n = 1,422), being female (73.4%; n = 1,044), having direct contact with patients (89.8%; n = 1,320), and having worked at the hospital for more than three years (60.3%; n = 884).

The study participants were from various different hospital departments: intensive care unit (14.6%; n = 215); clinical surgery (10.7%; n = 157); internal medicine (7.6%; n = 112); emergency department (5.7%; n = 84); pharmacy (5.6%; n = 83); laboratory (5.6%; n = 82); rehabilitation (4.2%; n = 62); radiology (3.6%; n = 53); pediatrics (3.3%; n = 49); obstetrics (2.5%; n = 37); anesthesiology (0.5%; n = 8); psychiatry/mental health (0.3%; n = 4); and others.

The largest groups of participants, by occupational category, were nursing technicians (26.2%; n = 377) and nurses (20.3%; n = 292). Other categories of participant were: physical therapists/occupational therapists (9.1%; n = 131); administrative auxiliaries (6.5%; n = 93); pharmacists/biochemists/biologists (4.8%; n = 69); laboratory, pharmacy, x-ray, and electrocardiograph technicians (5.1%; n = 74); doctors (4.2%; n = 60); managers/director (3.2%; n = 46); nutritionists (1%; n = 14); psychologists (0.9%; n = 13); and others (18.6%; n = 268).

Internal consistency

The Cronbach’s alpha coefficients estimated for the original 12-factor model ranged from 0.41 (“overall perceptions of patient safety”) to 0.91 (“frequency of events reported”) (Table 1). Table 1 shows the results obtained in this study and in other international studies ⁸8. Olsen E. Reliability and validity of the Hospital Survey on Patient Safety Culture at a Norwegian hospital. In: Øvretveit J, Sousa PJ, editors. Quality and safety improvement research: methods and research practice from the International Quality Improvement Research Network (QIRN). Lisboa: Escola Nacional de Saúde Pública, Universidade Nova de Lisboa; 2008. p. 173-86.^,1010. Ito S, Seto K, Kigawa M, Fujita S, Hasegawa T, Hasegawa T. Development and applicability of Hospital Survey on Patient Safety Culture (HSOPS) in Japan. BMC Health Serv Res 2011; 11:28.^,1212. Robida A. Hospital Survey on Patiet Safety Culture in Slovenia: a psychometric evaluation. Int J Qual Health Care 2013; 25:469-75.^,1313. Eiras M, Escoval A, Grillo IM, Silva-Fortes C. The hospital survey on patient safety culture in Portuguese hospitals: instrument validity and reliability. Int J Health Care Qual Assur 2014; 27:111-22.^,1818. Reis CT, Laguardia J, Vasconcelos AGG, Martins M. Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture (HSOPSC): a pilot study. Cad Saúde Pública 2016; 32:e00115614.^,1919. Andrade LEL, Melo LOM, Silva IG, Souza RM, Lima ALB, Freitas MR, et al. Adaptação e validação do Hospital Survey on Patient Safety Culture em versão brasileira eletrônica. Epidemiol Serv Saúde 2017; 26:455-68.^,2020. Sorra J, Nieva V. Hospital Survey on Patient Safety Culture. Rockville: Agency for Healthcare Research and Quality; 2004. (AHRQ Publication, 04-0041).^,2121. Smits M, Christiaans-Dingelhoff I, Wagner C, Wal Gv, Groenewegen PP. The psychometric properties of the 'Hospital Survey on Patient Safety Culture' in Dutch hospitals. BMC Health Serv Res 2008; 8:230.^,2222. Pfeiffer Y, Manser T. Development of the German version of the Hospital Survey on Patient Safety Culture: dimensionality and psychometric properties. Saf Sci 2010; 48:1452-62.^,2323. Waterson P, Griffiths P, Stride C, Murphy J, Hignett S. Psychometric properties of the Hospital Survey on Patient Safety Culture: findings from the UK. Qual Saf Health Care 2010; 19:2e.^,2424. Sarac C, Flin R, Mearns K, Jackson J. Hospital survey on patient safety culture: psychometrics analysis on a Scottish sample. BMJ Qual Saf 2011; 20:842-8.^,2525. Perneger TV, Staines A, Kundig F. Internal consistency, factor structure and construct validity of the French version the Hospital Survey on Patient Safety Culture. BMJ Qual Saf 2014; 23:389-97.. In four dimensions, the alpha value was below 0.60 (“overall perceptions of patient safety” = 0.41; “staffing” = 0.46; “teamwork across units” = 0.45; “non-punitive response to error” = 0.54) (Table 1).

Confirmatory factor analysis

CFA showed the fit of the original 12-dimension model to range from acceptable (CFI = 0.92 and TLI = 0.91) to satisfactory (RMSEA = 0.04) (90%CI: 0.042-0.046) (Table 2). That result is similar to those of international studies ¹⁰10. Ito S, Seto K, Kigawa M, Fujita S, Hasegawa T, Hasegawa T. Development and applicability of Hospital Survey on Patient Safety Culture (HSOPS) in Japan. BMC Health Serv Res 2011; 11:28.^,1212. Robida A. Hospital Survey on Patiet Safety Culture in Slovenia: a psychometric evaluation. Int J Qual Health Care 2013; 25:469-75.^,1313. Eiras M, Escoval A, Grillo IM, Silva-Fortes C. The hospital survey on patient safety culture in Portuguese hospitals: instrument validity and reliability. Int J Health Care Qual Assur 2014; 27:111-22.^,1818. Reis CT, Laguardia J, Vasconcelos AGG, Martins M. Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture (HSOPSC): a pilot study. Cad Saúde Pública 2016; 32:e00115614.^,1919. Andrade LEL, Melo LOM, Silva IG, Souza RM, Lima ALB, Freitas MR, et al. Adaptação e validação do Hospital Survey on Patient Safety Culture em versão brasileira eletrônica. Epidemiol Serv Saúde 2017; 26:455-68.^,2020. Sorra J, Nieva V. Hospital Survey on Patient Safety Culture. Rockville: Agency for Healthcare Research and Quality; 2004. (AHRQ Publication, 04-0041).^,2121. Smits M, Christiaans-Dingelhoff I, Wagner C, Wal Gv, Groenewegen PP. The psychometric properties of the 'Hospital Survey on Patient Safety Culture' in Dutch hospitals. BMC Health Serv Res 2008; 8:230.^,2222. Pfeiffer Y, Manser T. Development of the German version of the Hospital Survey on Patient Safety Culture: dimensionality and psychometric properties. Saf Sci 2010; 48:1452-62.^,2323. Waterson P, Griffiths P, Stride C, Murphy J, Hignett S. Psychometric properties of the Hospital Survey on Patient Safety Culture: findings from the UK. Qual Saf Health Care 2010; 19:2e.^,2424. Sarac C, Flin R, Mearns K, Jackson J. Hospital survey on patient safety culture: psychometrics analysis on a Scottish sample. BMJ Qual Saf 2011; 20:842-8., but differs from the findings of a previous Brazilian study ¹⁸18. Reis CT, Laguardia J, Vasconcelos AGG, Martins M. Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture (HSOPSC): a pilot study. Cad Saúde Pública 2016; 32:e00115614. concerning the RMSEA value (Table 2).

Excessively high correlations (> 0.85) were observed between the dimensions “staffing” and “overall perceptions of patient safety” (0.89), between the dimensions “communication openness” and “feedback and communication about error” (0.94), and between the dimensions “handoffs and transitions” and “teamwork across units” (0.89) (Table 3). In five items (A5R, A7R, A10R, A16R and F10), loading was lower than 0.32. Residual variance was greater than 0.70 in 13 items (A5R, A7R, A8R, A10R, A11, A14R, A15, A16R, A17R, C1, F3R, F8 and F10), ranging from 0.71 to 1.0.

Regarding the high correlations observed, which occur often when true cross-loadings are constrained to be zero, such as in the CFA, a preliminary Bayesian second-order factor model was tested, assuming that a higher overall (safety culture) order might be influencing indirectly the manifest variables through the first order. The model was estimated with 10,000 total iterations, 5,000 burn-in and normal prior distribution for model parameters. Results showed that the model did not converge properly with convergence statistics = 1.0254, posterior predictive p-value = 0.0, and deviance information criteria = 8,739.53. Further Bayesian analysis must be performed in the future considering alternative hierarchical models.

Exploratory structural equation modelling

ESEM returned good fit for the original 12-dimension model. The indices - CFI = 0.99; TLI = 0.97, and RMSEA = 0.026 (90%CI: 0.024-0.029) - were better than those returned by CFA.

The factorial structure indicated items with low loading (A8R “Staff feel like their mistakes are held against them”); cross loading (A16R “Staff worry that mistakes they make are kept in their personnel file”; A17R “We have patient safety problems in this unit”; A18 “Our procedures and systems are good at preventing errors from happening”; B4R “My supervisor/manager overlooks patient safety problems that happen repeatedly”; F2R “Hospital units do not coordinate well with each other”; F6R “It is often unpleasant to work with staff from other hospital units”; F8 “The actions of hospital management show that patient safety is a top priority”); and low and cross loading (F3R “Things ‘fall between the cracks’ when transferring patients from one unit to another”) (Table 4).

Of the 12 dimensions, four - “teamwork within units,” “organizational learning,” “feedback and communication about error”, and “frequency of events reported” (outcome dimension) - loaded on all the items satisfactorily in the dimensional structure of the original HSOPSC model. The reverse items - A14R (“We work in ‘crisis mode’ trying to do too much too quickly”); A10R (“It is just by chance that more serious mistakes do not happen here”); and A8R (“Staff feel like their mistakes are held against them”) - were observed to load on the same factor, thus constituting a dimension, and two of whose items are related to the hospital department/unit and the other, to the outcome dimension (Table 4).

Table 5 shows the dimensional structure resulting from the ESEM for the study sample. Compared to the dimensional structure of the original model, the items of the dimensions “feedback and communication about error” and “communication openness” intersect with each other, as do those of “management support for patient safety” and “handoffs & transition”. Items of the dimensions “supervisor/manager expectations & actions promoting patient safety”, “management support for patient safety”, “communication openness”, “staffing”, and “non-punitive response to errors” loaded on other dimensions, producing dimensions comprising only two items, which is an undesirable configuration. On this criterion, the dimension “teamwork across units” displayed the most problematic factorial structure, with four items loading on three different dimensions.

Residual variance, although mostly borderline, was high in the following items: A11 (“When one area in this unit gets really busy, others help out”) (0.71) of the dimension “teamwork within units”; A8R (“Staff feel like their mistakes are held against them”) (0.83) of the dimension “non-punitive response to error”; A10R (“It is just by chance that more serious mistakes do not happen here”) (0.71) of the dimension “teamwork across units”; A7R (“We use more agency/temporary staff than is best for patient care”) (0.71) of the dimension “staffing”; and A5R (“Staff in this unit work longer hours than is best for patient care”) (0.71), also of the dimension “staffing”.

ESEM with 12 dimensions returned correlations that were not excessively high, indicating no problems of discriminant validity.

Discussion

The HSOPSC is widely used in countries with a diversity of socioeconomic and historical contexts regarding quality and patient safety. Considering the limitations of the previous study, which evaluated this instrument for reliability and validity in the Brazilian context ¹⁸18. Reis CT, Laguardia J, Vasconcelos AGG, Martins M. Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture (HSOPSC): a pilot study. Cad Saúde Pública 2016; 32:e00115614., its properties were reexamined here. The study sample considered here comprised 1,439 questionnaires (an 18.7% response rate), from which data was collected electronically, via intranet, at a hospital with a history and leading role in initiatives in the field of quality and patient safety. The study sample was much larger than in the previous study, which collected data using questionnaires on paper, but the response rate was lower, with the previous study obtaining a response rate of 38.5%. Response rates are important, because low rates can limit the generalization of the data on safety culture to the level of the hospital where the instrument is used. Comparative data from hospital surveys have shown higher response rates when the instrument is applied on paper ¹1. Sorra J, Gray L, Streagle S, Famolaro T, Yount N, Behm J. AHRQ Hospital Survey on Patient Safety Culture: user's guide. Rockville: Agency for Healthcare Research and Quality; 2016. (AHRQ Publication, 15-0049-EF).. Andrade et al. ¹⁹19. Andrade LEL, Melo LOM, Silva IG, Souza RM, Lima ALB, Freitas MR, et al. Adaptação e validação do Hospital Survey on Patient Safety Culture em versão brasileira eletrônica. Epidemiol Serv Saúde 2017; 26:455-68. adapted and validated an electronic Brazilian version of the HSOPSC and recommended addressing the low response rate from in-person administration by enabling completion on mobile electronic applications. However, the AHRQ suggests that participants may be concerned about the privacy of their responses and may not be receptive to completing the questionnaire at their workplace ¹1. Sorra J, Gray L, Streagle S, Famolaro T, Yount N, Behm J. AHRQ Hospital Survey on Patient Safety Culture: user's guide. Rockville: Agency for Healthcare Research and Quality; 2016. (AHRQ Publication, 15-0049-EF)..

Nursing personnel accounted for the largest percentage of study participants by profession, corroborating the findings of a recent systematic review of studies that have used the HSOPSC ⁶6. Reis CT, Paiva SG, Sousa P. The patient safety culture: a systematic review by characteristics of Hospital Survey on Patient Safety Culture dimensions. Int J Qual Health Care 2018; 30:660-77.. Awareness-building and encouragement strategies need to be directed to all occupational categories. Special attention is required, therefore, when assessing an organization’s overall safety culture.

Internal consistency

Four dimensions displayed problems of internal consistency (“overall perceptions of patient safety”; “staffing”; “teamwork across units”; “non-punitive response to error”). With the exception of the dimension “teamwork across units”, all dimensions had already returned inadequate results in the previous Brazilian study ¹⁸18. Reis CT, Laguardia J, Vasconcelos AGG, Martins M. Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture (HSOPSC): a pilot study. Cad Saúde Pública 2016; 32:e00115614.. In the validation of the electronic version for Brazil ¹⁹19. Andrade LEL, Melo LOM, Silva IG, Souza RM, Lima ALB, Freitas MR, et al. Adaptação e validação do Hospital Survey on Patient Safety Culture em versão brasileira eletrônica. Epidemiol Serv Saúde 2017; 26:455-68., three dimensions (“non-punitive response to error”, “staffing” and “communication openness”) returned values below 0.60. Unlike the international studies ⁸8. Olsen E. Reliability and validity of the Hospital Survey on Patient Safety Culture at a Norwegian hospital. In: Øvretveit J, Sousa PJ, editors. Quality and safety improvement research: methods and research practice from the International Quality Improvement Research Network (QIRN). Lisboa: Escola Nacional de Saúde Pública, Universidade Nova de Lisboa; 2008. p. 173-86.^,1010. Ito S, Seto K, Kigawa M, Fujita S, Hasegawa T, Hasegawa T. Development and applicability of Hospital Survey on Patient Safety Culture (HSOPS) in Japan. BMC Health Serv Res 2011; 11:28.^,1212. Robida A. Hospital Survey on Patiet Safety Culture in Slovenia: a psychometric evaluation. Int J Qual Health Care 2013; 25:469-75.^,2020. Sorra J, Nieva V. Hospital Survey on Patient Safety Culture. Rockville: Agency for Healthcare Research and Quality; 2004. (AHRQ Publication, 04-0041).^,2121. Smits M, Christiaans-Dingelhoff I, Wagner C, Wal Gv, Groenewegen PP. The psychometric properties of the 'Hospital Survey on Patient Safety Culture' in Dutch hospitals. BMC Health Serv Res 2008; 8:230.^,2222. Pfeiffer Y, Manser T. Development of the German version of the Hospital Survey on Patient Safety Culture: dimensionality and psychometric properties. Saf Sci 2010; 48:1452-62.^,2323. Waterson P, Griffiths P, Stride C, Murphy J, Hignett S. Psychometric properties of the Hospital Survey on Patient Safety Culture: findings from the UK. Qual Saf Health Care 2010; 19:2e.^,2424. Sarac C, Flin R, Mearns K, Jackson J. Hospital survey on patient safety culture: psychometrics analysis on a Scottish sample. BMJ Qual Saf 2011; 20:842-8.^,2525. Perneger TV, Staines A, Kundig F. Internal consistency, factor structure and construct validity of the French version the Hospital Survey on Patient Safety Culture. BMJ Qual Saf 2014; 23:389-97., the Brazilian studies ¹⁸18. Reis CT, Laguardia J, Vasconcelos AGG, Martins M. Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture (HSOPSC): a pilot study. Cad Saúde Pública 2016; 32:e00115614.^,1919. Andrade LEL, Melo LOM, Silva IG, Souza RM, Lima ALB, Freitas MR, et al. Adaptação e validação do Hospital Survey on Patient Safety Culture em versão brasileira eletrônica. Epidemiol Serv Saúde 2017; 26:455-68., the Portuguese study ¹³13. Eiras M, Escoval A, Grillo IM, Silva-Fortes C. The hospital survey on patient safety culture in Portuguese hospitals: instrument validity and reliability. Int J Health Care Qual Assur 2014; 27:111-22., and this study highlighted the dimension “non-punitive response to error” as showing internal consistency problems. The dimensions “overall perceptions of patient safety”, “staffing”, “teamwork across units”, and “non-punitive response to error” contain reverse items. Even though the strategy of interspersing reverse-phrased sentences is effective in reducing tendentious responses, the AHRQ states that HSOPSC users have reported difficulty in interpreting them and the agency plans to reduce the number of reverse items when the instrument is updated ²⁹29. Agency for Healthcare Research and Quality. Update on version 2.0 of the Hospital SOPS. https://www.ahrq.gov/sops/quality-patient-safety/patientsafetyculture/hospital/update/index.html (accessed on 16/Aug/2018).
https://www.ahrq.gov/sops/quality-patien... .

Confirmatory factorial analysis

The CFA results in this study were both better than those of the previous study ¹⁸18. Reis CT, Laguardia J, Vasconcelos AGG, Martins M. Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture (HSOPSC): a pilot study. Cad Saúde Pública 2016; 32:e00115614. and, unlike that study, similar to those of the international studies ¹²12. Robida A. Hospital Survey on Patiet Safety Culture in Slovenia: a psychometric evaluation. Int J Qual Health Care 2013; 25:469-75.^,2020. Sorra J, Nieva V. Hospital Survey on Patient Safety Culture. Rockville: Agency for Healthcare Research and Quality; 2004. (AHRQ Publication, 04-0041).^,2424. Sarac C, Flin R, Mearns K, Jackson J. Hospital survey on patient safety culture: psychometrics analysis on a Scottish sample. BMJ Qual Saf 2011; 20:842-8. in confirming a 12-factor model. Note that the study context may explain that finding. However, there were problems of relevance and reliability in certain items (residual variance was higher than 0.70 in 13 items) and of discriminant validity (with excessively high correlations between the dimensions “staffing” and “overall perceptions of patient safety”, between the dimensions “communication openness” and “feedback and communication about error”, and between the dimensions “handoffs and transitions” and “teamwork across units”). Five items (A5R, A7R, A10R, F10 and A16R) returned loading below 0.32.

Exploratory structural equation modelling

ESEM found very good model fit, with better indices than those returned by CFA. However, high residual variance in the behaviour of at least five items indicated irrelevance. The factorial structure showed weakness, suggesting a 12-dimension model, two of whose dimensions would comprise only two items each (which is undesirable), with problematical loading on eight items (A8R, A16R, A18, B4R, F2R, F6R, F8, and F3R), six of which are reverse items, which - as mentioned above - have led to problems of comprehension among users of the instrument in other contexts.

Note that, with the exception of this study and the previous validation study by Reis et al. ¹⁸18. Reis CT, Laguardia J, Vasconcelos AGG, Martins M. Reliability and validity of the Brazilian version of the Hospital Survey on Patient Safety Culture (HSOPSC): a pilot study. Cad Saúde Pública 2016; 32:e00115614., studies of versions of the HSOPSC adapted for use in Brazil did not present data on correlations between the dimensions.

The version translated and adapted for use in the Brazilian context ¹⁷17. Reis CT, Laguardia J, Martins M. Adaptação transcultural da versão brasileira do Hospital Survey on Patient Safety Culture: etapa inicial. Cad Saúde Pública 2012; 28:2199-210. did not attain the best results in all validity and reliability tests with the data obtained from this study sample, despite the fact that the universe of this study differed from the previous one in that the hospital in question has a proactive history in initiatives in the field of quality of care and patient safety. This leads the authors to question whether these findings resulted from the need to improve the translation and retest it or whether the psychometric problems are inherent to the instrument itself and not specific to the translation and adaptation of the version used in this study, as already inferred by other users of the HSOPSC ²⁵25. Perneger TV, Staines A, Kundig F. Internal consistency, factor structure and construct validity of the French version the Hospital Survey on Patient Safety Culture. BMJ Qual Saf 2014; 23:389-97.^,2929. Agency for Healthcare Research and Quality. Update on version 2.0 of the Hospital SOPS. https://www.ahrq.gov/sops/quality-patient-safety/patientsafetyculture/hospital/update/index.html (accessed on 16/Aug/2018).
https://www.ahrq.gov/sops/quality-patien... . Despite the latter question, certain points regarding the original version of the HSOPSC are worth stressing: (i) the instrument is applicable to clinical and non-clinical personnel in the hospital context, but does not offer the response option “not applicable/I don’t know”, which may create inappropriate responses on certain items, leading to inconsistencies in the assessment of the hospital’s culture; (ii) the HSOPSC focuses on non-punitive culture, which is probably consistent with the time at which the instrument was formulated, but there is now an emerging need to shift focus towards a culture of fairness that values learning rather than highlighting culpability, and thus favours understanding individual and system-related factors that contribute to the occurrence of safety incidents; and (iii) the HSOPSC is a very extensive instrument, containing many reverse items, which may make its questions not very clear to respondents.

However, subsequent analyses based on the ESEM results recommended excluding some nine items, which would result ultimately in dimensions comprising fewer than three items, which is inadvisable. Accordingly, considering the findings of the studies conducted in the Brazilian context, there is a need to examine the dimensional structure and the relevance of items.

The results reported here should be interpreted in light of the respondents, selected from a non-random sampling of a single hospital, with low response rates. Also, we were not able to examine if a possible selection bias might have occurred, since respondents were selected on a voluntary basis and we did not obtain data from non-respondents. These aspects may limit a generalization of our findings to other general hospitals in Brazil. Despite the limitations, this study aimed to test the instrument in a specific hospital context whose results could serve as reference for other studies that seek to implement safety strategies.

However, the larger purpose of measuring patient safety culture is to identify and strengthen measures to reduce unnecessary risks resulting from patient care. In that regard, the endeavour to establish an adaptation of the HSOPSC properly suited to the Brazilian context is important, because that will not only permit national and international comparisons, but can assist in monitoring recently introduced measures directed to fostering patient safety, which certainly may be more at risk in this conjuncture of financial crisis.

Conclusion

Although the results obtained from reassessing the validity and reliability of the version of the HSOPSC translated and adapted for use in the Brazilian context were better than those of the first assessment, they do need further investigation. Whether this is due to the translation process, the limits of this study, Brazilian particular characteristics and cultural diversity, or related to more general problems in the structure of the instrument cannot be ascertained here.

Acknowledgments

The study was supported by the Brazilian National Research Council (Edital Universal 14/2011, process n. 471764/2001-6; Bolsa PQ, process n. 306023/2016-4).

References

Publication Dates

History