Abstract

This paper aims to present an analysis of the Quality in Health and Patient Safety Training Program (CQSSP) results of the Sérgio Arouca National School of Public Health. This is a quantitative, qualitative, cross-sectional, and descriptive study based on the Kirkpatrick’s Evaluation Model. The findings provide us with a clearer picture of the success of the CQSSP pedagogical intent through a positive assessment for the Reaction, Learning, Behavioral, Results levels. From the analysis of the perception of the graduates, we observed that 82% of them started to fully consider the occurrence of an error in the provision of care, 68% incorporated the acquired knowledge into their professional practice, and 73% started to study and research themes related to Patient Safety (PS). We also conducted a survey on the number of Patient Safety Centers or equivalent bodies established or reformulated during or after the CQSSP. We also evaluated the extent of the contribution of the course in the implementation of Patient Safety protocols and actions in hospitals. The findings point to a significant contribution of the CQSSP to PS and the quality of hospital care.

Key words:
Patient safety; Health care quality; Educational evaluation; Institutional evaluation; Professional qualification

Introduction

Recent decades have witnessed a growing debate and actions to provide care that values quality and safety. The To Err is Human report, by the Institute of Medicine (IOM) of the United States, is the milestone for patient safety (PS)¹1 Trindade L, Lage MJ. A perspectiva histórica e principais desenvolvimentos da segurança do paciente. In: Sousa P, Mendes W, organizadores. Segurança do Paciente: conhecendo os riscos nas organizações de saúde. Rio de Janeiro: Editora Fiocruz; 2014.. This report indicated that about 100,000 people die each year from adverse events²2 Institute of Medicine (US) Committee on Quality of Health Care in America. Kohn LT, Corrigan JM, Donaldson MS, editors. To Err is Human: Building a Safer Health System. Washington (DC): National Academies Press (US); 2000. PMID:25077248.. The magnitude of PS-related issues has been described in several studies in the U.S., Australia, the United Kingdom, Portugal, and Brazil³3 Martins M. Qualidade do cuidado em saúde. In: Sousa P, Mendes W, organizadores. Segurança do Paciente: conhecendo os riscos nas organizações de saúde. Rio de Janeiro: Fiocruz; 2014. p. 25-38..

Several important global PS-oriented initiatives were developed, among which the World Health Organization (WHO) establishing the World Alliance for Patient Safety in 2004, which unfolded into a program⁴4 Reis CT, Martins M, Laguardia J. A segurança do paciente como dimensão da qualidade do cuidado de saúde: um olhar sobre a literatura. Cien Saude Coletiva 2013;18(7):2029-2036. whose lines of action were Education and Research initiatives, such as the creation of a specific taxonomy and campaigns called “Global Challenges” geared to curbing healthcare-related infections, improving safety in surgical care and reducing medication-related errors⁵5 Mendes WV, Noronha J. Qualidade de Serviços de Saúde no SUS - Relatório Final. Telma Ruth Pereira, coord. edit.; 2013.. In 2020, celebrating the World Patient Safety Day, the WHO launched the campaign “Keep health workers safe to keep patients safe”, drawing attention to the importance of PS professionals.

In Brazil, in 2013, the Ministry of Health (MS) launched the National Patient Safety Program (PNSP)⁶6 Brasil. Ministério da Saúde (MS). Portaria nº 529, de 1º de abril de 2013. Institui o Programa Nacional de Segurança do Paciente. Diário Oficial da União, Poder Executivo, Brasília, DF, n. 62, p. 43-44, 02 abr. 2013. Seção 1.. Then, the National Health Surveillance Agency (ANVISA) established actions for PS in health services⁷7 Brasil. Agência Nacional de Vigilância Sanitária (Anvisa). Resolução da Diretoria Colegiada nº 36, de 25 de julho de 2013. Institui ações para a segurança do paciente em serviços de saúde e dá outras providências. Diário Oficial da União, Brasília, DF, n. 143, p. 32-3, 26 jul. 2013. Seção 1.. The PNSP is structured into four lines of action: encouraging safe care practice; patient involved in their safety, the inclusion of the topic in Education; and increment to research⁸8 Brasil. Ministério da Saúde (MS). Fundação Oswaldo Cruz. Agência Nacional de Vigilância Sanitária. Documento Referência para o Programa Nacional de Segurança do Paciente. Brasília: MS; 2014..

Given the proposal to include PS in Education, in partnership with the New Lisbon University, the National School of Public Health (ENSP) of the Oswaldo Cruz Foundation (FIOCRUZ) developed the International Quality Patient Health and Safety Course (CQSSP) to develop skills and scientific knowledge on care quality and PS.

As a public policy strategy, recognizing the leading role of this initiative and the relevance of its evaluation, we employed an evaluation model of training processes used in Training, Development, and Education (TD&E). TD&E is a system integrated by subsystems, where evaluation is the subsystem responsible for providing information and feedback and improving the entire system⁹9 Abbad GS, Mourão L, Meneses PPM, Zerbini T, Borges-Andrade JE, Vilas-Boas R. Medidas de avaliação em treinamento, desenvolvimento e educação: ferramentas para gestão de pessoas. Porto Alegre: Artmed; 2012..

Donald Kirkpatrick’s Four-Level Assessment Model has been widely accepted in many types of organizations since 1959. By proposing to assess the levels of Reaction, Learning, Behavior, and Results, it allows a comprehensive look at the training process and the results achieved¹⁰10 Kirkpatrick DL, Kirkpatrick JD. Como avaliar programas de treinamento de equipes: os quatro níveis. Tradução José Henrique Lamensdorf. Rio de Janeiro: Senac Rio; 2010.. Furthermore, given the theoretical-practical character of the quality of care and PS outlined in the CQSSP, Kirkpatrick’s model was presented as a potential evaluative tool suitable for dimensioning the bridge between training and practice. Shortell and Richardson (1978) argue that a program is an “intervention as an organized response to health problems, achieving goals, to improve the health of a population”¹¹11 Caldas BM. Implementação do Programa Nacional de Segurança do Paciente em hospitais públicos: uma avaliação qualitativa [tese]. Rio de Janeiro: Fundação Oswaldo Cruz / Escola Nacional de Saúde Pública Sergio Arouca; 2017.. Admitting the CQSSP as a program within the PNSP, its evaluation is outlined as evaluative research, that is, a “procedure that consists of making an ex-post judgment of an intervention using scientific methods”¹²12 Hartz ZMA, organizador. Avaliação em Saúde: dos modelos conceituais à prática na análise da implantação de programas.Rio de Janeiro: Editora Fiocruz; 1997., seeking to know, thus, the effects of the CQSSP in the transformation of reality and its impact on health care.

The study adds relevance as a retrospective evaluative approach, considering the context of the COVID-19 pandemic that affected the living conditions of populations and impacted health organizations. This context changed priorities and imposed the centrality of actions in the field of health. Old and new risks related to the care quality and PS emerge. Thus, courses with this theme, in the distance modality, anchored in a proposed intervention to improve the care provided, can be a strategy to achieve the SDGs, contributing, above all, to the objective 3 that proposes “To ensure access to quality health care and promote well-being for all, at all ages”¹³13 Nações Unidas Brasil. Os objetivos de Desenvolvimento Sustentável no Brasil. [acessado 2020 out 28]. Disponível em: https://brasil.un.org/pt-br/sdgs.
https://brasil.un.org/pt-br/sdgs... .

This study aimed to evaluate the quality in health and patient safety specialization course (CQSSP) offered by ENSP based on Kirkpatrick’s model, considering aspects of training, learning, practice, and production of results. The two guiding questions were: (i) As an intervention program, has the CQSSP produced results in health organizations? (ii) What characteristics were different in this intervention as a training process offered in the Distance Learning (DL) modality?

Methods

This is a cross-sectional study with a descriptive quantitative approach that involved qualitative elements. We analyzed data from an electronic questionnaire applied to CQSSP graduates and information available on the course platform in the quantitative approach. The qualitative approach was based on interviews with actors involved in the development and implementation of the course and on the review of the course documentation to support the construction of the electronic questionnaire.

According to the evaluation model applied, the training process must be evaluated at four levels: Reaction, Learning, Behavior, and Results. As it is a theoretical model classified as generic⁹9 Abbad GS, Mourão L, Meneses PPM, Zerbini T, Borges-Andrade JE, Vilas-Boas R. Medidas de avaliação em treinamento, desenvolvimento e educação: ferramentas para gestão de pessoas. Porto Alegre: Artmed; 2012., it allowed using multiple techniques and research instruments. The organization of information for each level is shown in Chart 1. The items that made up each category were measured using a five-point Likert-type scale.

The CQSSP is a lato sensu specialization course offered by ENSP, in the distance/semi-classroom modality, with a workload of 408 hours, requiring a minimum weekly dedication of eight hours from the participating professionals. The first edition, the object of this analysis, was held between September 2014 and September 2015. One thousand fifty spots were offered (1,000 students from Brazil, 25 from Portugal, and 25 from African Countries of Portuguese Official Language (PALOPS) to a target audience of higher education professionals working in hospitals with 200 or more beds.

The interviews carried out with the general coordinator of the course, the pedagogical coordinator, a learning advisor, and a tutor were based on pre-established roadmaps, which sought to respect the specificity of the type of inclusion of each stakeholder.

The document review was based on CQSSP records that refer to the virtual platform, the course monitoring form (FAC), the course project, the course evaluations, the student notebook, and the course completion papers (TCC) elaborated as PS improvement plans. An extraction roadmap created for this purpose was applied for collecting documentary data.

The interviews and document review considered the following CQSSP aspects: organization and planning (background, need, objectives, and thematic content); structuring and implementation (coordination, selection of tutors, selection of participants, physical structure, and teaching resources); evaluation (institutional evaluation of learning and institutional evaluation of the course and results); and results linked to the course objectives and those in the work plan.

A questionnaire was elaborated from the previous steps and applied to a convenience sample consisting of 52 students from the course in the state of Minas Gerais, selected for the CQSSP based on the indication of the directions of the hospitals located in that state. The state of Minas Gerais was selected because of its more stable situation in the face of the economic, social, and political crises affecting the country when the study was developed, which could considerably influence the results.

The semi-structured questionnaire sent to CQSSP graduates included students’ reactions, behavioral changes, and results from the training process. It was prepared in digital form, using Google Forms’ platform, and sent via e-mail to students in 2017. An initial prototype of the questionnaire was applied to two graduate students, not included in the sample, to assess its clarity, scope, and time taken to reply. Searching the platform, we collected data on the performance of students in the different CQSSP modules and the course completion work. This information was used for the assessment at the Learning level of the model.

We used content analysis as a technique adapted to meet the proposal of this study in the qualitative approach of the research, which involved interviews and document review. We pre-explored course-related documents and proceeded with a free-floating reading of these texts and the interviews, following steps proposed by Campos¹⁴14 Campos CJ. Método de análise de conteúdo: ferramenta para a análise de dados qualitativos no campo da saúde. Rev Bras Enf 2004; 57(5):611-614.. Then, the units of analysis were selected and categorized based on the study’s theoretical framework. This theoretical foundation aimed to provide intrinsic validity to the result of the analysis. The units of analysis were grouped as categories or subcategories by frequency and implicit relevance.

In the qualitative approach, the information obtained with the application of the questionnaire and collected on the course platform was analyzed using descriptive statistics. Quantitative information was stratified by categorical variables: professional category, seniority in the profession, management position, participation in Patient Safety Centers (NSPs), and facility’s management type.

ENSP’s Research Ethics Committee approved the research project under Opinion.

Results

The CQSSP had particularities regarding the motivations and context in which it was developed, its structure, assumptions, and pedagogical instruments. It is aligned with global initiatives and was created to find solutions to PS-related issues in Brazil, Portugal, and the PALOPs, and associated with the interest of the Schools of Public Health in Brazil and Portugal in expanding their partnership.

The scientific production on PS and the experience accumulated by the two schools, an already existing discipline in the stricto sensu ENSP course, and a PS course at the Portuguese school, were facilitating elements for the establishment of the theoretical and conceptual bases of the CQSSP that formed the syllabus, culminating in the development of specific teaching material (Chart 2). The content was divided into three large blocks: (i) theoretical and historical notion of PS, (ii) care practice issues sensitive to problems in care quality and PS, and (iii) measures and actions to manage PS issues in organizations.

The CQSSP was structured on a clear line of thought to change the PS culture in hospitals: assuming error as a possibility and a human condition and not blaming professionals as the main guidelines for PS-oriented actions. CQSSP’s methodological assumptions are built on the competence theory, involving an interactive and continuous pedagogical work, articulating theory and practice in constructing new knowledge. Thus, the reality experienced by the student was brought into the context, enhancing learning and skills development. The method included learning tools that facilitated the transfer to practice. Some methodological specificities of the CQSSP stand out:

Student/team concept/strategy as a learning unit: formed by four professionals from the same hospital, who studied and worked together, sharing learning and work.

Active methodologies: theory and practice articulation;

Construction of a PS plan by the students: it was a singularity of the CQSSP;

Pedagogical mediation: the tutor was the “mediator” who guided the student in constructing knowledge, valuing his practice and experience;

Teaching material: the course was supported by books, the theoretical basis of the course;

Procedural Assessment: assessment that concerned the student’s trajectory within the course, considering his commitment and dedication much more than a finished product of knowledge;

As a facilitator of the training process, technology is committed to enhancing learning resources and reaching the most significant number of students and health organizations. It also aimed to promote interaction between these stakeholders, responsible educational institutions, and health organizations.

Based on their perceptions and experiences in the CQSSP, the assessment made by the key actors conducting the training process was quite positive, highlighting the theme of interest to health professionals, the teaching material, the structure, teacher support, and the student/team strategy as facilitators of the training process. The political and economic crises that affected the hospitals, the difficulty of bringing the management of institutions into the course, the asymmetric training of professionals, the methodological difficulties due to teaching/learning method, as it is a new process, the criteria for appointing discretionary participants from management not always including professionals most suitable for the course, and the workload of students that could conflict with the necessary dedication were pointed out as obstacles to the training process.

The evaluation of students/alumni was based on a questionnaire built per the Kirkpatrick’s Four-Level Model and included 34 students (65%) of the total of 52 graduates from Minas Gerais, with computed losses of 35%. Forty-six students completed the course, to whom 45 questionnaires were effectively applied and from which 34 responses were obtained, corresponding to 76% of the total.

The profile of the 46 students who graduated from the CQSSP corresponded to the guidelines of the course coordination, as most were nurses (44%), pharmacists (26%), and a smaller percentage of doctors (15%). The participation of other professional categories (15%) indicated that the PS theme is essentially multidisciplinary, as disseminated by the WHO. Concerning the 34 students who responded to the questionnaire, the share of students who had never participated in the PS responsible bodies (6%) and the high proportion of students (35%) who participated but later left the NSPs or equivalent bodies (IEs) caused a stir.

The profile of respondents primarily consisted of experienced professionals with advanced seniority in the institution: 44% of participants had 11-20 years seniority, 29% had more than 20 years, and only 24% had 6-10 years. A relevant piece of information was the significant number of graduates in management positions: 71% held management positions before taking the course, and 62% continued to hold management positions at the time of the survey.

The Reaction level evaluation established evaluative categories, units of analysis included in these categories, and evaluative dimensions related to the units of analysis. In the evaluative category Teaching Material and Program Content, the reaction of respondents was quite positive for their units of analysis and evaluative dimensions (Table 1).

Graduates reacted positively concerning the evaluative category “Activities”. In the Activity Requested in the Virtual Learning Environment (AVA) unit of analysis, the highest disapproval rates were in the dimensions Interest and Stimulus to search for new references. Six percent of students partially disagreed with the assertion in both dimensions. The Forums and Chats analysis unit had the lowest approval, and the highest percentage of disapproval was in the dimension that investigated the students’ opinion about this type of activity to promote interaction between the hospital staff, where 3% totally disagreed with the statement, 15% partially disagreed, and 6% neither disagreed nor agreed (Table 1).

The students’ reaction was positive in evaluating the Tutoring System analysis category in the Tutoring and Pedagogical Mediation unit of analysis. However, in the evaluative dimensions that questioned whether the tutor used good strategies to facilitate student learning, 15% maintained the neutral position and 3% partially disagreed; in the dimension in which it was asked whether the tutor helped students to build their knowledge instead of giving ready-made answers, 9% remained neutral, and 3% partially disagreed; and, if the tutor guided the construction of the TCC well, 12% remained neutral, and 3% partially disagreed (Table 1).

Regarding the Learning Support Structure category, the percentage of adverse reaction fell on the quality of the data network (internet connection), where 3% totally disagreed that it was of good quality, 6% partially disagreed, and 3% neither disagreed nor agreed with the statement (Table 1).

In the analysis category Factors That Could Impact Students’ Participation in the Course, the most relevant data was that 18% partially disagreed that they were available to fulfill the minimum course load of 8 weekly hours (Table 1).

The Learning level inferred that the CQSSP achieved excellent results. The individual performance of students in the three Learning Units (LUs), added to the individual grade of the TCC, indicated that the mean of graduate students who achieved grade A (excellent) corresponded to 69%, 23.9% achieved grade B (good), and 7.1% achieved grade C (fair); there was no record of failure (Table 1). Learning by professional category did not vary significantly, and student/staff performance did not change because of the geographic location of the hospitals where they worked.

At the Behavior level, the research indicated that the great motto of the CQSSP, graduates valuing error as an opportunity to improve the quality of care, obtained the highest percentages of total (82%) and partial (15%) agreement. Concerning the opportunity to apply what they learned, most students said they had the opportunity to apply the knowledge and skills acquired, which achieved 59% total agreement and 35% partial agreement. Furthermore, 68% of professionals totally agreed, and 26% partially agreed that they started to incorporate course learning into their professional practice. These data show changes in attitudes in professional practice. (Table 1).

The Results level addressed the contribution of the CQSSP in the implementation of PS actions in hospitals. We observed that all established NSPs or IEs, most fully operational (70%). Nurses were the predominant professional category in 100%, pharmacists in 90%, and physicians in 80% of NSPs or IEs. Aside from health professionals, the most frequent category was the administrator, found in 40% of NSPs. However, the dedication of professionals to this body was still problematic: 80% of the hospitals had only a partial dedication to the workload, and only 10% had exclusively dedicated professionals. Most of the NSPs or IEs were linked to senior management, or the general management (30%), or the clinical/technical management (50%). Half only met with such bodies, when necessary, regarding the articulation of NSPs or IEs with higher bodies. In 40% of the hospitals, these meetings were periodic, and the situation was more critical in 10%, with infrequent meetings. As for CQSSP’s contribution, we noticed that most NSPs or IEs were already established before the course (70%). However, 20% were formed during or after the course, and 10% were reformulated during or after the course.

The notification system was in operation in 90% of hospitals, mostly under NSPs or IEs. Thirty percent of hospitals implemented the system during or after taking the course, besides 20% having been improved in the same period. Overall, in 80% of hospitals, notifications followed ANVISA guidelines. The data indicated that 40% of NSPs or IEs devoted more than 50% of their time to notifications, and 20% used between 20% to 50% of their time to notifications. Concerning safety culture, essential to building safe care systems, we observed that 40% of students/teams applied a questionnaire to measure the safety culture.

The patient identification protocol was fully implemented in 50% of the hospitals; it was implemented but not monitored in 30%, and it was partially implemented in 20% (Table 2). Regarding the contribution of the CQSSP to the implementation of this protocol, 50% of the hospitals recognized a partial contribution, and 10% a total contribution (Table 2).

The hand hygiene protocol was fully implemented in 60% of the hospitals and implemented and not monitored in 40% (Table 2). Regarding the relationship between learning and implementation, the total agreement rate was 20%, and 10% was partial. Half of the hospitals did not recognize a relationship between learning and the implementation of the protocol, as it was already implemented before the completion of the CQSSP, and 20% disagree with the existence of this relationship (Table 1).

The pressure ulcer prevention protocol was planned and not implemented in 30% of the hospitals and was not planned in 10% (Table 2). Thirty percent of students partially recognized the relationship between CQSSP learning and the implementation of this protocol (Table 1).

The implementation of the prevention of falls protocol was fully completed in 50% of the hospitals; it was partially implemented in 10% and was implemented and not monitored in 20%. Only 20% of hospitals planned and did not implement this protocol (Table 2). The relationship between learning the CQSSP and implementing this protocol was highly marked in the opinion of students/teams: 30% totally agreed, and 40% partially agreed, against 20% disagreement about the existence of this relationship (Table 1).

The safety protocol in the prescription, use, and administration of medications was fully implemented in 60% of the hospitals it was planned and not implemented in 20%. Only 20% of hospitals did not plan the protocol (Table 2). There was also significant recognition of the relationship between learning the CQSSP and implementing the protocol, where 30% fully agreed, and 30% partially agreed with this relationship. Disagreement was found in 20% of the total (Table 1).

The Safe Surgery protocol was fully implemented in 40% of hospitals. Thirty percent partially implemented it, and 20% implemented it and did not monitor it. This protocol was not planned in only 10% of hospitals (Table 2). In the students/teams’ opinion, the relationship between learning the CQSSP and implementing the Safe Surgery protocol was positive, and 20% agreed, and 40% partially agreed. Only 10% indicated that this item was not applicable because the protocol was already implemented before the CQSSP (Table 1).

Continuing and permanent education actions, systematic actions with formal policies and programs, covering all professional categories existed in only 30% of hospitals. Regarding the recognition of the influence of the CQSSP on continuing and permanent education actions, 70% of students/teams said that although actions already existed before the course, these were intensified during or after the CQSSP, and they were implemented by the influence of the CQSSP in 10% (Table 1).

Concerning the contributing factors that affected the implementation of PS actions, in the students’ perception of the hospital infrastructure, 9% totally disagreed, and 41% partially disagreed that this favored the provision of safe care. In evaluating the influence of external contributing factors (political, social, and economic) on the implementation of PS actions, 62% fully agreed, and 17% partially agreed with the influence of these external factors.

In the hospitals surveyed, most graduate students felt the support of management (32% partial agreement and 38% total agreement). Regarding meetings between NSPs or IEs with higher levels of management, 40% stated that they participated periodically, and 50% participated only when necessary. However, 32% of the students fully agreed, and 44% partially agreed that NSPs or IEs participated in the hospitals’ deliberations; partial disagreement was 12%, and total disagreement was 6%.

Regarding the contribution of the communication factor between NSPs or IEs and other professionals, it was noteworthy that 32% of graduates totally disagreed, and 12% partially disagreed that this factor contributed to the implementation of PS actions (Table 3).

Discussion

Evaluating implies inferring a value judgment from the measurement of the properties of an object⁹9 Abbad GS, Mourão L, Meneses PPM, Zerbini T, Borges-Andrade JE, Vilas-Boas R. Medidas de avaliação em treinamento, desenvolvimento e educação: ferramentas para gestão de pessoas. Porto Alegre: Artmed; 2012.. A fundamental point to be highlighted is that the evaluation of the CQSSP is related to both the training process and its results.

At the Reaction level, it can be inferred that the CQSSP opened horizons for sound learning, achieving a positive evaluation from the graduates. However, the lowest approval percentages in the Forums and Chats analysis unit caused a stir, a finding that deserves attention since the chats and forums in distance learning are essential tools promoting group interaction. Also noteworthy is the evaluation of the Strategies that the Tutor Used to Facilitate Student Learning, the Use of Resources Available in the Course to Establish the Relationship between Theory and Practice, and the Tutor’s Guidance for the Construction of the TCC. Since the percentages of disapproval and neutrality were higher, these units of analysis should also be an essential point of attention.

Regarding the assessment at the Learning level, Kirkpatrick and Kirkpatrick¹⁰10 Kirkpatrick DL, Kirkpatrick JD. Como avaliar programas de treinamento de equipes: os quatro níveis. Tradução José Henrique Lamensdorf. Rio de Janeiro: Senac Rio; 2010. state that measuring learning must determine the knowledge acquired, skills developed, or attitudes modified, since at least one of these points must have been achieved for there to be a behavioral change. However, reaching any one of them does not essentially imply behavioral change. Associating the concepts of learning in psychology and common language, citing Michaelis (2002) and Pozo (2002), Abbad et al.⁹9 Abbad GS, Mourão L, Meneses PPM, Zerbini T, Borges-Andrade JE, Vilas-Boas R. Medidas de avaliação em treinamento, desenvolvimento e educação: ferramentas para gestão de pessoas. Porto Alegre: Artmed; 2012. highlighted that learning results from associative and constructive processes and, as such, allow the “acquisition of different types learning outcomes, from the learning of facts, behaviors, to social, verbal, conceptual, and procedural learning” that will ultimately result in “changes in knowledge structures and creation of new solutions to problems”, which in the transfer to work, learning will involve the acquisition, retention, and generalization of knowledge⁹9 Abbad GS, Mourão L, Meneses PPM, Zerbini T, Borges-Andrade JE, Vilas-Boas R. Medidas de avaliação em treinamento, desenvolvimento e educação: ferramentas para gestão de pessoas. Porto Alegre: Artmed; 2012.. The CQSSP was seemingly successful in its endeavor concerning learning, an essential condition for changes in the levels of behavior and results, pointed by the good performance of students in the course modules and the TCCs.

According to Kirkpatrick and Kirkpatrick¹⁰10 Kirkpatrick DL, Kirkpatrick JD. Como avaliar programas de treinamento de equipes: os quatro níveis. Tradução José Henrique Lamensdorf. Rio de Janeiro: Senac Rio; 2010., behavior has to do with the share of knowledge, skills, and attitudes transferred to work. It is worth emphasizing that, although it is stated that the behavior in the position does not make sense for distance courses, as they are not usually offered by an organization different from the one with which students have employment relationships, preferring to use the term transfer of training in the research, the term “behavior” was kept, seeking fidelity to the evaluation model used. However, the educational institution is not the same work organization as the graduates.

Furthermore, the Behavior level was one of the most critical points of the assessment. First, because it was drawn from what was called “Profile of graduates” and represented the success or failure of the training process and its objectives. Otherwise, it was necessary to carry out the desired changes in practice, which turned into results.

According to the students’ positive self-assessment, the learning was transferred to their work practice. It is worth noting that professionals involved in care should be deeply committed to these issues to promote PS, the quality of care, and boost the safety culture in organizations, as the safety culture is a product of values, attitudes, perceptions, competences, and individual and group behavior patterns⁴4 Reis CT, Martins M, Laguardia J. A segurança do paciente como dimensão da qualidade do cuidado de saúde: um olhar sobre a literatura. Cien Saude Coletiva 2013;18(7):2029-2036..

Kirkpatrick and Kirkpatrick¹⁰10 Kirkpatrick DL, Kirkpatrick JD. Como avaliar programas de treinamento de equipes: os quatro níveis. Tradução José Henrique Lamensdorf. Rio de Janeiro: Senac Rio; 2010. consider the Results level the most challenging part of the assessment, given the difficulty of relating these directly to the training process. It is not always possible to obtain concrete evidence that the result stems from the formative process, but it is possible to obtain evidence. Three years into the CQSSP, we can consider a reasonable time to evaluate the implementation of the PS actions described in the hospitals’ Patient Safety Plans.

NSPs or IEs were established and fully operational, or more or less operational, in all participating hospitals, in the latter case a minority. The NSPs play a crucial role in the implementation and articulation of PS actions and could transform into learning and organizational changes the individual demands and experiences of professionals in the care process, as they analyze and act on health incidents while creating mechanisms to convert problems and solutions into organizational guidelines and policies. Some positive evidence points to a portion of CQSSP’s contribution since part of these bodies were established or reformulated during the course. Most was linked to senior management, following the recommendation by ANVISA⁷7 Brasil. Agência Nacional de Vigilância Sanitária (Anvisa). Resolução da Diretoria Colegiada nº 36, de 25 de julho de 2013. Institui ações para a segurança do paciente em serviços de saúde e dá outras providências. Diário Oficial da União, Brasília, DF, n. 143, p. 32-3, 26 jul. 2013. Seção 1. that the NSP or IE coordinator should have a seat in the deliberative bodies. The closer to decision levels, the more strategic and widespread can become the PS actions in the organization.

However, the dedication of professionals to the NSP or IEs is still problematic. In most hospitals, the center’s professionals dedicated to only part of the workload; only a small portion had exclusively dedicated professionals, which can occur for two reasons: the low number of professionals available or the NSP is not yet seen with due importance. Regardless of the reasons, this situation can generate work overload, and the professional will not be divided between several tasks without the risk of compromising work quality.

The approach of NSPs or IEs with other sectors proved to be quite shy, which can hinder the spread of PS actions and the dissemination of the safety culture. NSPs or IEs are closer to related areas such as hospital infection, pharmacy and therapeutics, death review, and medical record review committees, but slight approximation with other hospital sectors.

The Notification System was in operation in most hospitals, mainly under the responsibility of NSPs or IEs. However, regarding the contribution of the CQSSP to the implementation of the notification system, it is worth mentioning that 40% declared not to recognize any contribution from the course. The coordination was concerned that the NSPs or IEs would occupy most of their time with the notification system when the importance of participating in internal policies was recognized, and thus promoting PS as a priority. Data analysis indicated that the desired situation did not materialize.

The findings were inconclusive regarding the contribution of the CQSSP to change the patient safety culture, given the design of this study, and most hospitals were not concerned with measuring their safety culture through the application of questionnaires.

At the Results level, the evaluation of the CQSSP for the category Implementation of PS Protocols favors the conclusion that the course collaborated, to a greater or lesser degree, in the adoption of these protocols. Given the information collected, this contribution becomes evident. Implementing the safety protocol in the prescription, use, and administration of medications showed one of the best results.

Regarding continuing and permanent education actions, which are fundamental to promote PS in hospitals, disseminating knowledge and strengthening a culture that values PS and the quality of care, the findings showed that, while these actions were in place in the hospitals studied, a greater systematization was required, with formal policies and programs that cover all professional categories. This situation exists in a small part of the surveyed hospitals.

In the perception of graduate students, half of the hospitals surveyed did not offer an infrastructure that favored the provision of safe care. Another extremely relevant factor, Support from the directors, was felt by most of the graduate students. Management committed to PS issues gives legitimacy to NSPs and reinforces PS importance. Political, economic, social, and cultural external factors also act on organizations. The political and economic crisis in the country in recent years has affected the implementation of PS actions in hospitals. The frequent meetings between NSPs or IEs and higher levels is a sensitive issue. However, low systematization does not seem to affect the participation of NSPs or IEs in the hospitals’ deliberations. A contributing factor, communication between NSPs or IEs and other professionals, significantly influenced the implementation of PS actions, a relevant factor, considering that culture change involves awareness and open dialogue with all professionals.

Some limitations of the study were the short implementation time, the distance from the geographic location of the sample, and the adaptation of the model to achieve the research objectives since it was performed after the course completion.

Final considerations

This study highlights the importance and potential of specialization courses that combine the academic axes and the perspective of transformation in practice in the distance learning modality. Based on this assessment, we can affirm that the CQSSP was successful. By analyzing it, from the moment of its conception to the results described, we could measure the reach of its objectives as a training process and program that aimed to contribute to improved quality of care and PS. Having graduated 855 students, distributed over 50 classes in the national territory, it provided the graduates with supporting elements that transformed the practice’s actions. These actions are the foundation for a cultural turning-point in organizations regarding the difficult task of promoting safe and better-quality care. However, continuity is central to the sustainability of these advances.

The adaptation and application of the model are also considered a significant contribution, which can represent the genesis of a methodological approach for evaluating the results of courses with the same essence or similarity to the CQSSP.

After the edition of the first experience, analyzed in this study, this proposal gave rise to three subsequent calls, however, with a specific focus and by demand outside the ENSP. Thus, in 2016, the Patient Safety Specialization Course was offered to Professionals from the Urgent and Emergency Care Network. The Patient Safety Improvement Course in Maternity Hospitals was developed in 2017. Both were offered on demand by the Ministry of Health. In 2019, the CQSSP was carried out in partnership with ANVISA, aimed at professionals from public and charitable hospitals, with more than 100 beds, from the State Health Secretariats, ANVISA, and the Ministry of Health. Thus, the CQSSP remains in the ENSP’s EAD catalog, but it is carried out on-demand and through partnerships with other bodies, always aimed at strengthening the PS within the scope of the Unified Health System.

Acknowledgments

We are grateful to the National Council for Scientific and Technological Development (CNPq) for their support.

References

Publication Dates

History