ABSTRACT

Objectives.

The aim of this initiative was to assess whether a novel training program – Understanding Stigma and Strengthening Cognitive Behavioral Interpersonal Skills – could improve primary health care providers’ confidence in the quality of mental health care they provide in the Caribbean setting by using the Plan-Do-Study-Act rapid cycle for learning improvement.

Methods.

We conducted a prospective observational study of the impact of this training program. The training was refined during three cycles: first, the relevance of the program for practice improvement in the Caribbean was assessed. Second, pilot training of 15 local providers was conducted to adapt the program to the culture and context. Third, the course was launched in fall 2021 with 96 primary care providers. Pre- and post-program outcomes were assessed by surveys, including providers’ confidence in the quality of the mental health care they provided, changes in stigma among the providers and their use of and comfort with the tools. This paper describes an evaluation of the results of cycle 3, the official launch.

Results.

A total of 81 participants completed the program. The program improved primary care providers’ confidence in the quality of mental health care that they provided to people with lived experience of mental health disorders, and it reduced providers’ stigmatization of people with mental health disorders.

Conclusions.

The program’s quality improvement model achieved its goals in enhancing health care providers’ confidence in the quality of the mental health care they provided in the Caribbean context; the program provides effective tools to support the work and it helped to empower and engage clients.

Keywords
Primary health care; quality improvement; mental health; Caribbean Region; social stigma

RESUMEN

Objetivos.

La finalidad de esta iniciativa fue determinar si un nuevo programa de capacitación, titulado Understanding Stigma and Strengthening Cognitive Behavioral Interpersonal Skills (Comprender la estigmatización y fortalecer las competencias interpersonales cognitivas y conductuales), podría mejorar la confianza de los prestadores de atención primaria de salud en la calidad de la atención de salud mental que proporcionan en los países del Caribe, mediante el uso del ciclo rápido Planificación-Realización-Estudio-Acción para la mejora del aprendizaje.

Métodos.

Realizamos un estudio de observación prospectivo sobre el impacto de este programa de capacitación. La formación se perfeccionó a lo largo de tres ciclos: en primer lugar, se evaluó la pertinencia del programa para la mejora de las prácticas en el Caribe; en segundo lugar, se llevó a cabo una capacitación piloto de 15 prestadores locales para adaptar el programa a la cultura y el contexto; en tercer lugar, en el otoño del 2021 se puso en marcha el curso con 96 prestadores de atención primaria. Los resultados anteriores y posteriores al programa se evaluaron mediante encuestas que incluían la confianza de los prestadores en la calidad de la atención de salud mental que proporcionaban, los cambios aparecidos en los prestadores en cuanto a la estigmatización, y su uso y conocimiento de las herramientas. En este artículo se describe una evaluación de los resultados del ciclo 3, correspondiente a la puesta en marcha oficial.

Resultados.

Un total de 81 participantes completaron el programa. El programa mejoró la confianza de los prestadores de atención primaria en la calidad de la atención de salud mental que proporcionaban a las personas que presentaban trastornos de la salud mental y redujo la estigmatización de las personas con este tipo de trastornos por parte de los prestadores de la atención.

Conclusiones.

El modelo de mejora de la calidad del programa logró sus objetivos en cuanto a mejorar la confianza de los prestadores de atención de salud en la calidad de la atención de salud mental que proporcionan en los países del Caribe; el programa aporta herramientas eficaces para sustentar esta tarea y ayudó a empoderar e involucrar a las personas usuarias.

Palabras clave
Atención primaria de salud; mejoramiento de la calidad; salud mental; Región del Caribe; estigma social

RESUMO

Objetivos.

O objetivo da iniciativa foi avaliar se um novo programa de capacitação, Compreensão do estigma e fortalecimento das competências cognitivas e comportamentais interpessoais, seria capaz de aumentar a confiança dos profissionais de atenção primária à saúde na qualidade da atenção à saúde mental que oferecem a pacientes no Caribe, utilizando o ciclo rápido Plan-Do-Study-Act (planejar, fazer, avaliar e agir) para melhorar o aprendizado.

Métodos.

Este foi um estudo observacional prospectivo sobre o impacto desse programa de capacitação. A capacitação foi aperfeiçoada ao longo de três ciclos. Inicialmente, avaliou-se a relevância do programa para o aprimoramento da prática no Caribe. A seguir, foi realizada uma capacitação-piloto de 15 profissionais locais para adaptar o programa à cultura e ao contexto. No terceiro ciclo, realizado no outono de 2021, o curso foi lançado, com a participação de 96 profissionais de atenção primária. Os resultados antes e depois do programa foram avaliados por meio de questionários, que incluíam a confiança dos provedores na qualidade da atenção à saúde mental oferecida, mudanças no estigma entre os profissionais e a utilização das ferramentas e o conforto dos profissionais em usá-las. Este documento apresenta a avaliação dos resultados do ciclo 3, o lançamento oficial do curso.

Resultados.

Oitenta e um participantes completaram o programa. O programa melhorou a confiança dos profissionais de atenção primária na qualidade dos cuidados de saúde mental que ofereciam às pessoas com experiência vivida relacionada a problemas de saúde mental. Além disso, reduziu a estigmatização das pessoas com problemas de saúde mental pelos profissionais de saúde.

Conclusão.

O modelo de melhoria da qualidade do programa atingiu suas metas de aumentar a confiança dos prestadores de serviços de saúde na qualidade dos serviços de saúde mental que prestavam no Caribe; o programa fornece ferramentas efetivas de apoio ao trabalho e ajudou a empoderar e envolver os clientes.

Palavras-chave
Atenção primária à saúde; melhoria de qualidade; saúde mental; Região do Caribe; estigma social

In 2019, in the World Health Organization’s (WHO’s) Region of the Americas, the burden of mental health disorders was estimated to be between 1 516 and 2 235/100 000 disability-adjusted life years (¹1. Pan American Health Organization. The burden of mental disorders [Internet]. Washington (DC): Pan American Health Organization; 2021 [cited 2022 Jun 18]. Available from: https://www.paho.org/en/noncommunicable-diseases-and-mental-health/enlace-data-portal-noncommunicable-diseases-mental-2
https://www.paho.org/en/noncommunicable-... ). Disorders due to the use of alcohol and other psychoactive substances, such as illegal drugs and medically prescribed psychotropics, are rising and have considerable social impact on Member States. Despite this, only a small proportion of people receive adequate treatment (²2. Borges G, Aguilar-Gaxiola S, Andrade LH, Benjet C, Cia AH, Kessler R, et al. Twelve-month mental health service use in six countries of the Americas: a regional report from the World Mental Health Surveys. Epidemiol Psychiatr Sci. 2019;29:e53.). In the Region, it is estimated that 71% of persons requiring psychosocial support do not access services (³3. Kohn R, Ali AA, Puac-Polanco V, Figueroa C, López-Soto V, Morgan K, et al. Mental health in the Americas: an overview of the treatment gap. Rev Panam Salud Publica. 2018;42:e165. http://iris.paho.org/xmlui/handle/123456789/49540
http://iris.paho.org/xmlui/handle/123456... ). Differences in accessing treatment for mental health disorders between high-income and low- and middle-income countries extend to all types and levels of care. Among adults with severe or moderate affective, anxiety or substance use disorders, the mean gap among those who need treatment and those who have treatment is 47.2% in North America and 77.9% in Latin America and the Caribbean (³3. Kohn R, Ali AA, Puac-Polanco V, Figueroa C, López-Soto V, Morgan K, et al. Mental health in the Americas: an overview of the treatment gap. Rev Panam Salud Publica. 2018;42:e165. http://iris.paho.org/xmlui/handle/123456789/49540
http://iris.paho.org/xmlui/handle/123456... , ⁴4. Thornicroft G, Chatterji S, Evans-Lacko S, Gruber M, Sampson N, Aguilar-Gaxiola S, et al. Undertreatment of people with major depressive disorder in 21 countries. Br J Psychiatry. 2017;210(2):119–24.).

The treatment gap for these mental health conditions has steadily increased as countries within the Region restructured their health systems in response to the COVID-19 pandemic (⁵5. Pan American Health Organization. The impact of COVID-19 on mental, neurological and substance use services in the Americas: results of a rapid assessment, November 2020. Washington (DC): Pan American Health Organization; 2020. https://iris.paho.org/handle/10665.2/52999
https://iris.paho.org/handle/10665.2/529... ). In a study by the World Economic Forum, approximately 45% of adults stated that within 1 year of the pandemic, their emotional and mental health had deteriorated (⁶6. Tausch A, Souza RO, Viciana CM, Cayetano C, Barbosa J, Hennis AJ. Strengthening mental health responses to COVID-19 in the Americas: a health policy analysis and recommendations. Lancet Reg Health Americas. 2022;5:100118.). Reducing the treatment gap and the 12-month prevalence of mental disorders are the primary modifiable factors in diminishing the regional burden of these disorders and their societal impact (⁷7. Orozco R, Vigo D, Benjet C, Borges G, Aguilar-Gaxiola S, Andrade LH, et al. Barriers to treatment for mental disorders in six countries of the Americas: a regional report from the World Mental Health Surveys. J Affect Disord. 2022;303:273-85.).

Empowering health workers, decision-makers and the public with trusted knowledge about mental health is critical to protecting people with lived experience of mental health disorders and providing essential and effective services to those at risk. The Pan American Health Organization’s (PAHO’s) Plan of Action on Mental Health 2015–2020 promotes the development of a community model, with new services and alternatives that aim to offer comprehensive and continual care for people with mental health disorders and make it possible to integrate mental health services into primary health care (⁸8. Pan American Health Organization. Plan of action on mental health 2015–2020. Washington (DC): Pan American Health Organization; 2015. https://www.paho.org/en/documents/plan-action-mental-health-2015-2020
https://www.paho.org/en/documents/plan-a... ). Under this Plan, people with mental health problems can receive prompt, evidence-based, person-centered care at facilities where they have established trust with the providers (⁹9. Mickus M, Colenda CC, Hogan AJ. Knowledge of mental health benefits and preferences for type of mental health providers among the general public. Psychiatr Serv. 2000;51(2):199–202.).

Although numerous training programs for providing mental health care are available, research has shown that most do not necessarily translate into changes in practice patterns (¹⁰10. Sikorski C, Luppa M, König HH, van den Bussche H, Riedel-Heller SG. Does GP training in depression care affect patient outcome? A systematic review and meta-analysis. BMC Health Serv Res. 2012;12:10., ¹¹11. Kalet AL, Gillespie CC, Schwartz MD, Holmboe ES, Ark TK, Jay M, et al. New measures to establish the evidence base for medical education: identifying educationally sensitive patient outcomes. Acad Med. 2010;85(5):844–51.). The core barriers to changing practices in mental health care are the need to strengthen providers’ capacity through knowledge and skills and address factors that impede motivation to change.

Health care providers’ stigmatizing of mental health disorders is another important barrier to treatment in the Caribbean (¹²12. Mascayano F, Tapia T, Schilling S, Alvarado R, Tapia E, Lips W, et al. Stigma toward mental illness in Latin America and the Caribbean: a systematic review. Braz J Psychiatry. 2016;38(1):73–85.). Providers’ lack of awareness of their own prejudices and an incomplete understanding of how important they are in a client’s recovery process can be harmful and a barrier to people seeking help, treatment and recovery (¹³13. Horsfall J, Cleary M, Hunt GE. Stigma in mental health: clients and professionals. Issues Ment Health Nurs. 2010;31(7):450–5.).

A lack of skills and confidence also contribute to stigma. Providers may have a sense of helplessness, anxiety or of not knowing what to do, or a lack of belief that health care providers can realistically help clients recover, thus reinforcing a provider’s preference for clinical distancing (¹⁴14. Brown L, Macintyre K, Trujillo L. Interventions to reduce HIV/AIDS stigma: what have we learned? AIDS Educ Prev. 2003;15(1):49–69., ¹⁵15. Lauria-Horner B. Stigma reduction for health care workers. In: Dobson KS, Stuart H, editors. The stigma of mental illness: models and methods of stigma reduction. Oxford: Oxford University Press; 2021. p. 161–78.). WHO has concluded that training programs for primary health care personnel are urgently needed. The implications are significant; approximately 70% of people with a lived experience of mental health disorders respond positively when treated in a dignified manner, and they will play an active role in their treatment process. For this reason, a central target of policy- and decision-makers involved in redesigning mental health care is to address stigma among providers (⁸8. Pan American Health Organization. Plan of action on mental health 2015–2020. Washington (DC): Pan American Health Organization; 2015. https://www.paho.org/en/documents/plan-action-mental-health-2015-2020
https://www.paho.org/en/documents/plan-a... , ¹⁶16. World Health Organization. Investing in mental health: evidence for action. Geneva: World Health Organization; 2013. https://apps.who.int/iris/handle/10665/87232
https://apps.who.int/iris/handle/10665/8... ).

To this end, PAHO, in partnership with the Mental Health Commission of Canada, delivered an evidence-based virtual training program – known as Understanding Stigma and Strengthening Cognitive Behavioral Interpersonal Skills (US-CBIS) – to English-speaking primary health care practitioners from Caribbean countries. The course contributes towards achieving the targets of WHO’s Comprehensive Mental Health Action Plan 2013–2020, which aims for universal health coverage of mental health, neurological and substance use conditions by ensuring provision of comprehensive, integrated and responsive mental health and social care services in community-based settings (¹⁷17. World Health Organization. Comprehensive mental health action plan 2013–2030. Geneva: World Health Organization; 2021. https://apps.who.int/iris/handle/10665/345301
https://apps.who.int/iris/handle/10665/3... ).

The program includes a stigma-reduction component as well as a nonpharmacological skills-building component to treat mild to moderate depression and other psychological conditions. It complements training in WHO’s Mental Health Gap Action Programme (¹⁸18. World Health Organization. Enhancing mental health pre-service training with the mhGAP intervention guide: experiences and lessons learned. Geneva: World Health Organization; 2020. https://apps.who.int/iris/handle/10665/333970
https://apps.who.int/iris/handle/10665/3... ) by providing practical tools and strategies to achieve goals in essential care and practice that promote effective communication skills, respect and dignity, and functioning in daily activities and community life (i.e. using a holistic approach to assess a client’s challenges and strengths, uncover ineffective character traits, and introduce self-management strategies).

The training was delivered using the Plan-Do-Study-Act (PDSA) rapid cycle for learning improvement, reflecting best practices. The Accreditation Council for Graduate Medical Education emphasizes integrating quality improvement (QI) components into the clinical learning environment of medical professionals (¹⁹19. Nasca T, Philibert I, Brigham T, Flynn T. The next GME accreditation system – rationale and benefits. N Engl J Med. 2012;366:1051-6.). The US-CBIS training process differs from podium-style training in that it allows for between-session in-office practice time to incorporate learning, study, learn, tailor and implement small practice changes that can lead to improvements in client care. Practitioners provide regular feedback that helps shape the content and methodology of the training in terms of culture, context and emerging needs. Thus, capacity building interventions such as this training program are key to restructuring mental health services at the community level (²⁰20. Pan American Health Organization. Deinstitutionalization of psychiatric care in Latin America and the Caribbean. Washington (DC): Pan American Health Organization; 2021. https://iris.paho.org/handle/10665.2/53207
https://iris.paho.org/handle/10665.2/532... ).

An evaluation of the Adult Mental Health Practice Support Program – which uses CBIS as its core component and a PDSA delivery model – was shown to be effective in reducing stigma and improving depression scores (²¹21. Beaulieu T, Patten S, Knaak S, Weinerman R, Campbell H, Lauria-Horner B. Impact of skill-based approaches in reducing stigma in primary care physicians: results from a double-blind, parallel-cluster, randomized controlled trial. Can J Psychiatry. 2017;62(5):327–35., ²²22. Lauria-Horner B, Beaulieu T, Knaak S, Weinerman R, Campbell H, Patten S. Controlled trial of the impact of a BC adult mental health practice support program (AMHPSP) on primary health care professionals’ management of depression. BMC Fam Pract. 2018;19(1):183.). The Adult Mental Health Practice Support Program is more complex to reproduce in a new setting as it requires a higher investment of funds than is required to implement the US-CBIS program, ministerial approvals and a redesign of mental health services (e.g. compensation for health care providers who attend training and specific telephone-based coaching services) (²³23. Canadian Mental Health Association. BounceBack: reclaim your health [Internet]. Ontario: Canadian Mental Health Association; 2023 [cited 2020 June 13]. Available from: https://bouncebackontario.ca/
https://bouncebackontario.ca/... ). Our program includes the same CBIS component with the addition of an online stigma-reduction component (²⁴24. Centre for Addiction and Mental Health. Understanding Stigma: a CAMH self-directed course [Internet]. Toronto: Centre for Addiction and Mental Health; 2023 [cited 2020 July 12]. Available from: https://www.camh.ca/en/education/continuing-education-programs-and-courses/continuing-education-directory/understanding-stigma
https://www.camh.ca/en/education/continu... ) adapted for the Region of the Americas. Due to the heterogeneity of health systems in the Region of the Americas, and differences among mental health care providers and financial support, we set out to design a program with a higher likelihood of successful implementation and to test its impact as compared with similar QI models. To our knowledge, mental health care training programs with the PDSA rapid cycle for learning improvement have not been used in the Caribbean.

The aim of the initiative was to assess whether a novel training program – US-CBIS – could improve primary health care providers’ confidence in the quality of mental health care they provide in the Caribbean setting by using the PDSA rapid cycle for learning improvement.

METHODS

We conducted a prospective observational study of all participants who registered for the training. Volunteer primary care practitioners from English-speaking Caribbean countries with ≥ 2 years’ experience in health care who were interested in enhancing the mental health care they provided, and who were supported by the Ministry of Health or other supervising body, were eligible to participate.

Quality improvement process

The collaboration between PAHO and the Mental Health Commission of Canada was crucial to carrying out this initiative. According to Plsek (²⁵25. Plsek PE. Complexity and the adoption of innovation in health care. Washington (DC): National Institute for Health Care Management Foundation, National Committee for Quality Health Care; 2003.) early discussions with local leaders to identify key contextual differences can guide us in adapting ideas and program content to promote the adoption and sustainability of innovations in health care.

The initiative consisted of three PDSA cycles (Table 1). In cycle 1, we determined the relevancy of the program’s goals to improving practice in the Caribbean. In cycle 2, the pilot phase of the project, the purpose was twofold: to train local trainers and to refine the program for cultural and contextual integration of learning. Fourteen primary health care providers from diverse cultures, fields and backgrounds were enrolled and trained. Five champion tutors emerged to support participants in the official launch for course delivery in fall 2021. The third cycle was the official launch of the fall 2021 course. This paper describes an evaluation of the results of cycle 3, the official launch.

Intervention: the training program

The program combines two complementary, evidence-based stigma-reduction programs. The first is Understanding Stigma, which is composed of three self-directed modules; it is an online course originally designed in 2007 by the Ontario Central Local Health Integrated Network and adapted by the Mental Health Commission of Canada for self-directed, web-based delivery. It is currently available through the Centre for Addiction and Mental Health at the University of Toronto. It was designed to help professionals recognize their own stigmatizing attitudes and behaviors, their impacts, how they present in primary care and to understand practical approaches to address some of the associated challenges (²⁴24. Centre for Addiction and Mental Health. Understanding Stigma: a CAMH self-directed course [Internet]. Toronto: Centre for Addiction and Mental Health; 2023 [cited 2020 July 12]. Available from: https://www.camh.ca/en/education/continuing-education-programs-and-courses/continuing-education-directory/understanding-stigma
https://www.camh.ca/en/education/continu... ). The program uses contact-based education – that is, videos of personal stories and perspectives – as a core teaching element along with educational and action-oriented components. The program was adapted for this initiative with the permission of the Centre for Addiction and Mental Health.

The second program is the CBIS, which is composed of four tutor-led modules that are virtually delivered; this is the complementary nonpharmacological skills-building stigma-reduction component (¹⁴14. Brown L, Macintyre K, Trujillo L. Interventions to reduce HIV/AIDS stigma: what have we learned? AIDS Educ Prev. 2003;15(1):49–69., ²¹21. Beaulieu T, Patten S, Knaak S, Weinerman R, Campbell H, Lauria-Horner B. Impact of skill-based approaches in reducing stigma in primary care physicians: results from a double-blind, parallel-cluster, randomized controlled trial. Can J Psychiatry. 2017;62(5):327–35.) designed to increase providers’ confidence to help clients recover, thus reducing a preference for clinical distancing. Practitioners learn an organized approach that moves from an assessment of problems to an assessment of strengths to the development of an action plan. Flow charts help practitioners navigate through user-friendly cognitive behavioral therapy self-management options for clients, which are key for providing effective management of chronic illness (²⁶26. Turner A, Anderson JK, Wallace LM, Bourne C. An evaluation of a self-management program for patients with long-term conditions. Patient Educ Couns. 2015;98(2):213–9., ²⁷27. Theeboom T, Beersma B, Vianen AEM. Does coaching work? A meta-analysis on the effects of coaching on individual level outcomes in an organizational context. J Posit Psychol. 2014;9:1-18.).

Modules were delivered in 3-hour sessions with intervals for in-office practice time to help link meaningful learning to practical performance. The program increases communication skills and enhances engagement with clients through joint decision-making, responsibility, choice, trust and empowerment. People with lived experience of mental health disorders are true partners in their recovery efforts.

The academic coordinator (BL-H) delivered both the train-the-trainers course and the official launch of the fall 2021 course for primary care providers from English-speaking Caribbean countries to ensure program fidelity and standardization. A facilitator’s manual and instruction documents were created prior to the train-the trainers course and updated for the official launch. The academic coordinator discussed course content, activities and tasks with the five trained tutors. The five tutors and the academic coordinator (six tutors total) provided regular synchronous and asynchronous feedback, support and guidance to participants for the fall 2021 course. Participants in the official launch, in turn, shared their opinions about the course.

Quality improvement measures

After each training session and practice period, participants provided feedback through open-ended surveys on course content, challenges, success implementing the tools or strategies, other experiences and the perceived impact on their practice. Common themes were categorized within the four queries described in the Results section.

We used an iterative, consensus-driven approach to refine the program by capturing practitioners’ authentic experiences over the course of the training.

Quantitative evaluation

Identical surveys were administered pre-and post-training. The main outcomes used to assess the impact of the QI initiative were (i) provider confidence in the quality of mental health care they could provide, (ii) changes in their stigmatization of people with mental health disorders and (iii) use of or comfort with the new tools and skills.

To evaluate changes in providers’ stigmatizing attitudes, we used the 15-item Opening Minds Stigma Scale for Health Care Providers (OMS-HC), a validated self-reported questionnaire that measure the attitudes and behavioral intentions of health care providers towards people with mental health disorders. Also, by grouping certain questions from the scale together, the OMS-HC measures three important dimensions of stigma: attitudes towards people with mental health disorders, providers’ willingness to disclose their own mental health needs and seek help, and preference for social distance from people with a mental health disorder (²⁸28. Modgill G, Patten SB, Knaak S, Kassam A, Szeto ACH. Opening Minds Stigma Scale for Health Care Providers (OMS-HC): examination of psychometric properties and responsiveness. BMC Psychiatr. 2014;14:120.). Items are rated on a 5-point scale from strongly agree to strongly disagree.

To assess participants’ confidence, respondents answered a series of questions about their confidence in assessing mental health, and diagnosing and treating mental health disorders using a four-point scale, with scores ranging from 1 (not at all confident) to 4 (very confident). Participants also provided information about whether they used any of the new tools for mental health management presented in the program by responding yes or no. As part of the same survey, participants also reported on their clients’ perception of the strategies, using a five-point scale with ratings from very negative to very positive.

Comfort in using the new tools and skills was assessed on a six-point scale, with scores ranging from 1 (not at all) to 6 (very comfortable). An evaluation working group consisting of researchers, program developers and facilitators had developed the survey items as part of a previous training evaluation that included the CBIS module (²⁹29. Weinerman R, Campbell H, Miller M, Stretch J, Kallstrom L, Kadlec H, et al. Improving mental healthcare by primary care physicians in British Columbia. Healthc Q. 2011;14(1):36–8.). These have continued to be used in similar trainings as they have shown good face validity, reliability and responsiveness to change (²¹21. Beaulieu T, Patten S, Knaak S, Weinerman R, Campbell H, Lauria-Horner B. Impact of skill-based approaches in reducing stigma in primary care physicians: results from a double-blind, parallel-cluster, randomized controlled trial. Can J Psychiatry. 2017;62(5):327–35.).

Research Ethics Board approval was not sought, as evaluations for the purpose of improving practice quality within educational requirements do not constitute research. The confidentiality of participants was maintained through safeguards such as using unique and anonymous identification codes for participants and ensuring that databases contained only deidentified information.

Analysis

A consultant independent from the operations of the study (SK) conducted the analysis. The sample included participants who successfully passed the course and participants completing program activities but who did not pass.¹1Because survey data were collected anonymously, a subanalysis could not be completed comparing those who completed and passed and those who did not. From this, 54 matched tests were compared for analysis. This represents an overall response rate of 78.3% (54/69) (³⁰30. Fincham JE. Response rates and responsiveness for surveys, standards, and the journal. Am J Pharm Educ. 2008;72(2):43.).

For the qualitative data, thematic analyses were performed using the steps outlined by Braun and Clarke (³¹31. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101.). Qualitative responses provided in the end-of-course survey were triangulated with qualitative feedback collected at the end of each module to verify the completeness of coverage of the main themes. For all quantitative measures, data were examined for normal distribution using quantile-quantile plots and found to be normally distributed. Results describe average changes in mean scores pre- and post-program, using paired t-tests at the 95% confidence level. The magnitude of change was calculated using Cohen’s d (³²32. Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013;4:863.).

RESULTS

Participants

Figure 1 illustrates the progression from the original number of participants (n = 96) to the final number (n = 81). A total of 96 participants were assigned to 6 tutor groups (5 trained tutors and the academic coordinator) Six tutors provided regular synchronous and asynchronous feedback, support and guidance over the course of the training for 81 active participants from 14 countries: Antigua and Barbuda, Argentina, Bahamas, Belize, Cayman Islands, Grenada, Guyana, Jamaica, Montserrat, Saint Kitts and Nevis, Saint Lucia, Suriname, Trinidad and Tobago, and Turks and Caicos. Of these, 60 participants passed the course with a grade above 80%.

Table 2 describes all participants who registered for the program (n = 96) and those who completed the program and the pre- and post-program surveys (n = 54). A between-groups comparison revealed one significant difference among participants: those who completed the program and both surveys were slightly older than those who registered but either did not complete both surveys or did not complete the course (mean age: 40.7 years versus 38.7 years; P = 0.009).

Post-session surveys: common themes

In general, participants liked the delivery model (i.e. self-learning and tutor-led modules). Contact-based education increased awareness of the challenges affecting people with lived experience of mental health disorders and how prejudice and discrimination within the health care setting impact their lives. The tutor-led sessions and intramodular practice activities helped participants understand the course concepts prior to applying the tools and strategies within a clinical setting. Themes emerging in response to queries are outlined below.

What impact has the course had on your practice generally?

A total of 57 responses were received for this question. All indicated that the course had a positive impact. Comments centered on developing a greater understanding and awareness of stigmatization and how to reduce it, as well as on improvements in providers’ skills and their confidence to successfully support and plan care for people living with mental health disorders. Some comments are presented here.

How successful have you been in implementing the tools and client self-management materials in your practice?

A total of 39 of the 56 (70%) participants who responded to this query indicated they were somewhat or very successful in implementing the tools and strategies during the action periods. One made the following comments.

Altogether 8 (14%) respondents had not yet been successful in integrating one of the tools, the Diagnostic Assessment Interview, but were still planning to incorporate it. And 9 participants (16%) indicated they had not used the tool because clients who required a comprehensive mental health screening had not consulted the primary care provider during the course of the training.

What barriers have you confronted?

Among the 42 respondents the top reported barrier was time constraints (¹⁵15. Lauria-Horner B. Stigma reduction for health care workers. In: Dobson KS, Stuart H, editors. The stigma of mental illness: models and methods of stigma reduction. Oxford: Oxford University Press; 2021. p. 161–78., 36%). Of this group, the majority reported difficulties balancing coursework with personal responsibilities or limited time to use the tools in a busy practice during the challenges of the COVID-19 pandemic. Other barriers included variables associated with patients, such as illiteracy, hesitancy, missing appointments or not completing tasks (¹¹11. Kalet AL, Gillespie CC, Schwartz MD, Holmboe ES, Ark TK, Jay M, et al. New measures to establish the evidence base for medical education: identifying educationally sensitive patient outcomes. Acad Med. 2010;85(5):844–51., 26%); lack of colleague or staff buy-in (⁸8. Pan American Health Organization. Plan of action on mental health 2015–2020. Washington (DC): Pan American Health Organization; 2015. https://www.paho.org/en/documents/plan-action-mental-health-2015-2020
https://www.paho.org/en/documents/plan-a... , 19%); and difficulty in getting printed materials to patients (⁵5. Pan American Health Organization. The impact of COVID-19 on mental, neurological and substance use services in the Americas: results of a rapid assessment, November 2020. Washington (DC): Pan American Health Organization; 2020. https://iris.paho.org/handle/10665.2/52999
https://iris.paho.org/handle/10665.2/529... , 12%).

Do you have any other comments about your experiences with the program?

Of the 42 participants who responded, the majority (37, 88%) used the space to provide positive feedback.

Five participants (12%) provided recommendations to enhance the course. A couple of examples are given here.

Quantitative evaluation

Changes in stigmatization were evaluated using the OMS-HC. A statistically significant improvement in scores with a medium effect size was observed on the total OMS-HC scale, as well as two of the subscales (i.e. the preference for social distance and practitioners’ own willingness to disclose or seek help for a mental health disorder (Table 3).

Measures of confidence, comfort and familiarity.

Significant improvements were observed on all measures of confidence and comfort between the pre- and post-program surveys (Table 4). For changes in confidence in the overall quality of mental health care they provided to clients, the magnitude of improvement was in the medium effect range (Cohen’s d = −0.67). The magnitude of improvement was in the large effect range (Cohen’s d range: −0.92 to 1.26) for a number of categories: confidence in screening for anxiety, assessing patients’ problems and strengths (Cohen’s d = −0.92), developing systematized care plans, engaging mental health patients in a range of interventions (Cohen’s d = 1.26), knowledge of nonpharmacological interventions, and knowledge of regional mental health resources.

We also grouped the percentages of participants who reported being very or somewhat confident about various aspects of assessment, diagnosis and treatment after training. Important areas in which there were changes between pre- and post-program included engaging clients in a range of interventions (changed from 59.3% [32/54] of participants to 98.1% [53/54]); assessing clients’ problems and strengths (changed from 83.3% [45/54] to 100% [54/54]), in particular in this area, 68.5% (37/54) reported being very confident after training compared with 29.6% (16/54) before training; and the overall quality of mental health care they provided (changed from 85.2% [46/54] to 98.1% [53/54]) (data not shown).

Post-training, the vast majority of participants were using nearly all the tools introduced in the program (range: 77.5% to 87.0%, depending on the tool). Participants’ average comfort scores increased significantly for all tools, with large effect sizes (Cohen’s d range: 1.34 to 3.16, depending on the tool). Finally, participants who indicated that clients’ perceptions of CBIS tools were positive or very positive increased from 0% at baseline to 89% (41/46) post-training (data not shown).

Correlation between confidence and changes in stigmatization.

A regression analysis showed that improvements in confidence were predictive of reductions in stigmatizing behaviors and attitudes (R² = 0.10, F_1,52 = 6.08, P = 0.02).

DISCUSSION

Research suggests that primary care providers who offer mental health care have a sense of helplessness that often contributes to feelings of anxiety and greater clinical distance, and these are likely to translate into stigmatization. Thus, including practical skills training in stigma-reduction programs is important. However, providing targeted knowledge and information about stigma awareness is not enough. It is essential to challenge status quo podium-style training programs by reflecting on reasons that impede providers’ motivations to change.

The Accreditation Council for Graduate Medical Education recommends integrating a QI component into medical professionals’ clinical learning as an integral impetus to facilitate knowledge translation. The results of our study support this recommendation. Significant improvements in confidence after the program were observed on all measures assessing providers’ engagement with clients on a range of interventions. Most participants were using nearly all of the tools included in the program and showed significant improvements in comfort with their use and confidence to successfully support and plan care for people with mental health disorders. Practitioners strongly supported the practitioner–client shared responsibility model for recovery, suggesting that the tools empower and engage the clients in their care. Finally, we saw significant improvements in overall stigmatization correlated with improvements in confidence, willingness of health care providers to disclose their own mental health issues and their preferences for social distancing.

Results from this evaluation are encouraging when compared with other versions of this training in other places. For example, participants who attended the CBIS program in Canada that did not include the Understanding Stigma module had smaller reductions in their stigma scores (small effect range, Cohen’s d = 0.33) compared with participants in the training reported in this paper (medium effect range, Cohen’s d = 0.48) (²¹21. Beaulieu T, Patten S, Knaak S, Weinerman R, Campbell H, Lauria-Horner B. Impact of skill-based approaches in reducing stigma in primary care physicians: results from a double-blind, parallel-cluster, randomized controlled trial. Can J Psychiatry. 2017;62(5):327–35.). This may be an indication that including the Understanding Stigma module in skills-based training is important to achieve maximum stigma reduction.

Overall improvements in providers’ confidence in managing patients with mental health conditions and comfort using the various tools taught were in the large effect size range (i.e. Cohen’s d of ≥ 0.80) for evaluations of both. As the Canadian cohort had in-person training, this may be an encouraging finding to support other research into whether the efficacy is similar for virtual and in-person models (³³33. Vaona A, Banzi R, Kwag KH, Rigon G, Cereda D, Pecoraro V, et al. E-learning for health professionals. Cochrane Database System Rev. 2018;(8):CD011736.).

A 3-day US-CBIS workshop for health care practitioners in Suriname had similar results to ours in terms of stigma reduction (Cohen’s d = 0.65 for Suriname and 0.48 for our study). Due to the small number of surveys returned for this training at the 6-month follow up (n = 12), results across programs could not be compared. Encouragingly, the 6-month results from the Suriname program showed that improvements in stigmatization scores were maintained (³⁴34. Knaak S, Lauria-Horner B. Delivery of the Understanding Stigma and CBIS Programs in Suriname: evaluation results: pre to follow up. Ottawa: Mental Health Commission of Canada; 2019.). However, confidence in the management and use of the skills for mental health conditions were not assessed, but we believe improvements in these areas likely would be smaller because of the workshop-style delivery, which does not allow providers to practice and gain confidence in the skills in a real-world environment between learning modules.

To maximize sustainability and reproducibility, we created standardized training and process documents (e.g. content, instructions), and the Understanding Stigma component has been translated into Spanish and is available through the PAHO Virtual Campus for Public Health. Additionally, five tutors were trained through two revised iterations of the program and they are able to act as trainers in their country. We also delivered a booster session 6 months after training to aid in learning retention for all trainers and participants. Version 2.0 of the program was delivered in fall 2022, and results of assessments were similar to those for version 1.0. This suggests that the program is reproducible as long as there is adequate training and standardized materials and processes.

A strength of our model is that practitioners used the tools as they saw fit in a real-world environment, within the constraints of busy practices. Therefore, this design lends weight to the program’s impact on practitioners’ confidence in the quality of mental health care that they provide (³⁵35. Brundiers K, Wiek A, Redman C. Real-world learning opportunities in sustainability: from classroom into the real world. Int J Sustain High Educ. 2010;11:308–24., ³⁶36. Camm AJ, Fox KAA. Strengths and weaknesses of ‘real-world’ studies involving non-vitamin K antagonist oral anticoagulants. Open Heart. 2018 Apr 1;5(1):e000788.).

Strengths and limitations

A strength of the project was its high survey response rate and using matched pre- and post-program data. While it cannot be known if those who completed both surveys had notably different experiences than those who did not, the consistency in participant characteristics between the full sample (n = 96) and those who completed the program and both surveys (n = 54) – and the high response rate (78.3%) – provides reassurance that the risk of nonresponse bias was low.

There are some limitations. As participants were selected by their country’s Ministry of Health, our results may have been influenced by selection bias. Furthermore, participants who registered for this training were volunteers. Their inherent interest in mental health could be greater than those who chose not to participate (i.e. reflecting self-selection bias). Also, a low response to the 6-month follow-up survey prevented assessment of longer-term outcomes.

Conclusions

The US-CBIS QI model achieved its goals by enhancing health care professionals’ confidence in the quality of mental health care that they provided to their clients in the Caribbean, providing effective tools to support the work and helping to empower and engage clients.

Disclaimer.

Authors hold sole responsibility for the views expressed in the manuscript, which may not necessarily reflect the opinion or policy of the Revista Panamericana de Salud Pública/Pan American Journal of Public Health or those of the Pan American Health Organization.

Acknowledgements.

The authors acknowledge the original version of the US-CBIS program, which led to the development of this virtual academic course. The authors also thank the course tutors: Dr. Katija Khan (Trinidad), Dr. Asha Pemberton (Trinidad), Mrs. Mosa Hutson (Guyana), Mrs. Djavila Ho (Jamaica) and Ms. Valeria Florez (Peru) who were stellar in their roles and in championing future stigma-reduction efforts in the Region of the Americas. We would also like to acknowledge Mr. Carlos Leon, specialist in pedagogical management and virtual learning environments of the Virtual Campus for Public Health, for his considerable work in developing the course platform.

REFERENCES

Publication Dates

History