ABSTRACT

Objective.

Health policymakers in the Caribbean face challenges with research use in decision-making. Although copious approaches to strengthen evidence-informed policy can be found in the literature, these strategies should be applied and evaluated in specific settings. We developed a theory of change for strengthening research uptake in health policy, and the interventions were implemented as the Evidence Informed Decision Making Network of the Caribbean (EvIDeNCe). We assessed the model's logic and evaluated whether the expected outcome was achieved.

Methods.

The model was mapped in three stages: problem identification; goal determination; and backward linking of interventions. Beneficiaries were surveyed to assess the design logic and to evaluate the main outcome.

Results.

A total of 137 respondents completed evaluation questionnaires. The inclusion of evidence briefs, stakeholder dialogues, a research database, and training programs for policymakers in the model was validated. Respondents also reported their intention to act on research evidence to which they were exposed. After respondents had participated in stakeholder dialogues, the mean intention-to-use score was 6.4 on a scale of 1 (strongly disagree) to 7 (strongly agree), and 6.3 on the same scale, after exposure to training.

Conclusions.

This work provides initial validation of EvIDeNCe as a consolidated strategy to strengthen the application of research in policy in the Caribbean. To our knowledge, it is the first study to develop and apply a comprehensive model of this type to the Caribbean. The findings support results from similar initiatives in other countries, but additional work is needed to evaluate the overall impact of the initiative.

Keywords
Translational medical research; health policy; health systems; West Indies

RESUMEN

Objetivo.

Los responsables de las políticas de salud en el Caribe afrontan retos con respecto al uso de la investigación en la toma de decisiones. Si bien en la bibliografía especializada pueden encontrarse estrategias muy diversas para fortalecer las políticas fundamentadas en la evidencia, dichas estrategias deben aplicarse y evaluarse en entornos específicos. Elaboramos una teoría del cambio para fortalecer la incorporación de la investigación en las políticas de salud, y las intervenciones se ejecutaron en la Red de Toma de Decisiones Basadas en Evidencia del Caribe (conocida como EvIDeNCe). Evaluamos la lógica del modelo y también si se alcanzó el resultado previsto.

Métodos.

El modelo se proyectó en tres etapas: determinación de los problemas, establecimiento de las metas y vinculación retrospectiva de las intervenciones. Se realizaron encuestas a los beneficiarios para evaluar tanto la lógica de diseño como el resultado principal.

Resultados.

Un total de 137 encuestados respondió los cuestionarios de evaluación. Se validó la inclusión en el modelo de sinopsis de datos científicos, diálogos entre los interesados directos, una base de datos de investigación y programas de capacitación para los responsables de las políticas. Los encuestados también informaron sobre su intención de actuar en respuesta a los resultados de la investigación a los que estuvieron expuestos. Después de que los encuestados participaron en los diálogos entre interesados directos, la puntuación media de la intención de usar la investigación fue de 6,4 en una escala de 1 (totalmente en desacuerdo) a 7 (totalmente de acuerdo), y después de exponerse a una capacitación, de 6,3 en la misma escala.

Conclusiones.

Este trabajo arroja una validación inicial de EvIDeNCe como una estrategia consolidada para fortalecer la aplicación de la investigación en las políticas en el Caribe. Hasta donde sabemos, es el primer estudio que ha elaborado y aplicado un modelo integral de este tipo en el Caribe. Los resultados apoyan las observaciones de iniciativas similares en otros países, pero hace falta seguir trabajando para evaluar las repercusiones más amplias de la iniciativa.

Palabras clave
Investigación en medicina traslacional; política de salud; sistemas de salud; Indias Occidentales

RESUMO

Objetivo.

Os responsáveis pelas políticas de saúde no Caribe enfrentam desafios no uso de pesquisa na tomada de decisão. Apesar de a literatura oferecer uma abundância de enfoques para reforçar políticas informadas por evidências, essas estratégias devem ser aplicadas e avaliadas em contextos específicos. Foi formulada uma teoria de mudança para reforçar a incorporação de pesquisa em políticas de saúde, sendo implementadas intervenções como a Rede de tomada de decisão informada por evidências do Caribe (EvIDeNCe). Foi avaliada a lógica do modelo e analisado se o resultado esperado foi alcançado.

Métodos.

O modelo foi traçado em três estágios: identificação de problemas, determinação de metas e ligação regressiva de intervenções. Foi aplicada uma pesquisa aos beneficiários para avaliar a lógica do design e analisar o resultado principal.

Resultados.

Ao todo, 137 participantes responderam os questionários de avaliação. Foi validada a inclusão no modelo de informes de evidências, diálogos de interessados diretos, uma base de dados de pesquisa e programas de formação profissional para os responsáveis pelas políticas. Os participantes também informaram a intenção de proceder segundo as evidências de pesquisas às quais tiveram conhecimento. Após tomarem parte em diálogos de interessados diretos, a pontuação média de intenção de uso dos participantes foi 6,4 (em uma escala de 1 [discordo totalmente] a 7 [concordo plenamente]) e 6,3 após o treinamento.

Conclusões.

Este estudo proporciona a validação inicial da rede EvIDeNCe como uma estratégia consolidada para reforçar a aplicação de pesquisa em políticas no Caribe. Ao que nos consta, trata-se do primeiro estudo a formular e aplicar um modelo completo deste tipo no Caribe. Os achados ajudam a respaldar os resultados de iniciativas semelhantes em outros países, porém outros estudos são necessários para avaliar o impacto geral da iniciativa.

Palavras-chave
Pesquisa médica translacional; política de saúde; sistemas de saúde; Índias Ocidentais

Evidence-informed health policy aims to utilize the best available research evidence to inform policy-making (¹1. Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 1: What is evidence-informed policymaking? Health Res Policy Syst. 2009;7 Suppl 1:S1.). This approach gained increased attention globally after publication of the World Health Organization's seminal report on strengthening health research systems in 2004 (²2. World Health Organization. World report on knowledge for better health: strengthening health research systems. Geneva: WHO; 2004.). The real-world practice of evidence-informed policy-making is, however, less than optimal, with many reasons advanced for persistence of the research-policy gap in health systems (³3. Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014 Jan 3;14:2.–⁷7. van de Goor I, Hämäläinen R-M, Syed A, Juel Lau C, Sandu P, Spitters H, et al. Determinants of evidence use in public health policy making: results from a study across six EU countries. Health Policy. 2017 Mar;121(3):273–81.). In the Caribbean region, these explanations include poor access to published literature; limited capacity for knowledge translation and dissemination; and incongruity between the type of research that is produced and the type required for decision-making (⁸8. Caribbean Health Research Council. Translating health research to action: a toolkit for the Caribbean. Available from: http://www.chrc-caribbean.org/Portals/0/Downloads/Publications/Research/Health%20Research%20Policy/Health%20Research%20Policy%20for%20the%20Caribbean.pdf Accessed on 13 October 2013.
http://www.chrc-caribbean.org/Portals/0/... –¹⁰10. Greaves DE. Evidence-based management of Caribbean health systems: barriers and opportunities. Int J Health Gov. 2017 Jun 5;22(2):104–17.).

Many approaches have been suggested for strengthening the application of research in health decision-making, with interventions having varied effectiveness, depending on the context in which they are applied (¹¹11. Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS One. 2011;6(7):e21704.–¹⁴14. Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy. a systematic review. PLoS One. 2013;8(10):e77404.). Models to guide country selection of strategies also exist. For example, one makes a distinction among four types of efforts: 1) those by research users to pull for evidence; 2) those made by intermediary groups to push evidence towards users; 3) efforts to facilitate user pull; and 4) efforts to engage the producers and users of research in collaborative activities (¹⁵15. Lavis JN, Lomas J, Hamid M, Sewankambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ. 2006 Aug;84(8):620–8.). This model has been implemented as an integrated strategy in settings such as Canada (¹⁶16. McMaster University. McMaster Health Forum. Our services. Available from: https://www.mcmasterforum.org/lets-collaborate/our-services Accessed on 26 December 2017.
https://www.mcmasterforum.org/lets-colla... ), Europe (¹⁷17. World Health Organization Regional Office for Europe. EVIPNet Europe: annual report 2016. Copenhagen: WHO Regional Office for Europe; 2017. Available from: http://www.euro.who.int/__data/assets/pdf_file/0007/339667/final_EVIPNet-Europe_annual-report_2016_eng.pdf?ua=1 Accessed on 26 December 2017.
http://www.euro.who.int/__data/assets/pd... ), Africa (¹⁸18. Berman J, Mitambo C, Matanje-Mwagomba B, Khan S, Kachimanga C, Wroe E, et al. Building a knowledge translation platform in Malawi to support evidence-informed health policy. Health Res Policy Syst. 2015 Dec;13(1):73.–²⁰20. Ongolo-Zogo P, Lavis JN, Tomson G, Sewankambo NK. Initiatives supporting evidence informed health system policymaking in Cameroon and Uganda: a comparative historical case study. BMC Health Serv Res. 2014 Nov 29;14:612.), and Latin America (²¹21. Mansilla C, Herrera C, Basagoitia A, Pantoja T. The Evidence-Informed Policy Network in Chile: lessons learned from a year of coordinated efforts. Rev Panam Salud Publica. 2017;41(e36).). Johnson and Lavis (²²22. Johnson N, Lavis JN. Procedures manual for evaluating knowledge-translation platforms in low- and middle-income countries. Overview. Available from: https://www.mcmasterforum.org/evaluate-innovations/impact-lab/ktpe/procedures Accessed on 22 December 2017.
https://www.mcmasterforum.org/evaluate-i... ) developed a common logical framework for evaluating implementation of this strategy in low- and middle-income countries. They noted, however, that more needs to be known about the interrelationships in the framework, specifically how different activities and outputs can be combined, and how the framework can be applied in different jurisdictional and institutional contexts. This paper seeks to address this knowledge gap.

The Caribbean Public Health Agency (CARPHA) is a regional public health institution serving 24 Member States in the Caribbean. Public policymakers are the primary clients of CARPHA. One of the challenges faced by these policymakers is research utilization in policy. A framework to explain the mechanisms of change expected to result from implementation of interventions to address this challenge was needed. According to Breuer and colleagues (²³23. Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016 May 6;11:63.), logical frameworks are limited in their ability to explain how various components of a program work together in a causal manner to achieve impact. Theory of change (ToC), on the other hand, is a more flexible format. ToC describes how a program, through a logical sequence of intermediate outcomes, can bring about specific long-term outcomes. The pathways through which interventions work to achieve a desired impact are made explicit. ToC also allows for depiction of multiple pathways and feedback loops, which is more reflective of the complexities involved in real-world settings (23, 24). The objectives of the study reported in this paper were: i) to develop a ToC to narrow the research-policy gap in the Caribbean; ii) to assess its design logic; and iii) to evaluate the extent to which the main expected outcome was achieved.

METHODS

Development of the theory of change

Using the approach suggested by the World Bank (²⁵25. International Bank for Reconstruction and Development/World Bank. Designing a results framework for achieving results: a how-to guide. Washington D.C.: International Bank for Reconstruction and Development/World Bank; 2012.) and Morra and Rist (²⁶26. Morra Imas LG, Rist R. The road to results: designing and conducting effective development evaluations. Available from: http://elibrary.worldbank.org/doi/book/10.1596/978-0-8213-7891-5 Accessed on 4 July 2017.
http://elibrary.worldbank.org/doi/book/1... ), we constructed the ToC using the following steps: i) identification of long-term goals; ii) backward mapping and connecting the preconditions to achieve goals; and iii) identifying interventions to generate desired change. Prior to goal determination, we also included a problem identification step, to ensure that the model's expected results were directly linked to the problems faced by Caribbean policymakers.

To identify long-term goals and map preconditions, we searched for published and gray literature that investigated application of evidence in decision-making in the Caribbean. Two pertinent studies were identified (⁸8. Caribbean Health Research Council. Translating health research to action: a toolkit for the Caribbean. Available from: http://www.chrc-caribbean.org/Portals/0/Downloads/Publications/Research/Health%20Research%20Policy/Health%20Research%20Policy%20for%20the%20Caribbean.pdf Accessed on 13 October 2013.
http://www.chrc-caribbean.org/Portals/0/... , ⁹9. Bennett NR, Cumberbatch C, Francis DK. There are challenges in conducting systematic reviews in developing countries: the Jamaican experience. J Clin Epidemiol. 2015 Sep;68(9):1095–8.). Seven challenges were common across these studies: 1) access to research was not systematic or widespread; 2) the dissemination of research findings to users was less than ideal; 3) the type of research that was being conducted was not the type needed to inform decisions; 4) national planning and policy-making processes provided limited avenues for the systematic review of evidence; 5) there was little interaction between producers and users of research; 6) research had a low standing in the list of factors deemed important to decisionmakers; and 7) policymakers generally lacked expertise to apply research in decision-making. We therefore posited that three broad changes were needed to address these shortcomings: i) relevant evidence should be made available to policymakers when needed; ii) the skills of policymakers to apply evidence in the policy-making process should be enhanced; and iii) policymakers and other knowledge users who could inform health system changes should be well equipped to make, or to advocate for, these changes. We further hypothesized that these adjustments would stimulate knowledge users to act on the available evidence. Interventions that could generate these outcomes were then identified.

Selection of interventions to generate desired change was informed by analysis of published literature on the topic. We found that providing research databases was a recognized strategy to facilitate user pull, as these resources assisted policymakers to efficiently find research and increased the likelihood of research use (²⁷27. Lavis JN, Wilson MG, Moat KA, Hammill AC, Boyko JA, Grimshaw JM, et al. Developing and refining the methods for a ‘one-stop shop’ for research evidence about health systems. Health Res Policy Syst. 2015 Dec;13(1):10.–²⁹29. Makkar SR, Gilham F, Williamson A, Bisset K. Usage of an online tool to help policymakers better engage with research: Web CIPHER. Implement Sci. 2015 Apr 23;10:56.). A promising push strategy to answer policymakers’ questions was summarizing evidence and highlighting messages from systematic reviews (28, 30, 31). We found that deliberative dialogues were used to foster knowledge exchange between policymakers and researchers (30, 32–34). Communities of practice (¹⁸18. Berman J, Mitambo C, Matanje-Mwagomba B, Khan S, Kachimanga C, Wroe E, et al. Building a knowledge translation platform in Malawi to support evidence-informed health policy. Health Res Policy Syst. 2015 Dec;13(1):73., ³⁵35. Kothari A, Boyko JA, Conklin J, Stolee P, Sibbald SL. Communities of practice for supporting health systems change: a missed opportunity. Health Res Policy Syst. 2015 Jul 25;13:33.) and capacity- building programs (³⁶36. Champagne F, Lemieux-Charles L, Duranceau M-F, MacKean G, Reay T. Organizational impact of evidence-informed decision making training initiatives: a case study comparison of two approaches. Implement Sci. 2014 May 2;9:53.–³⁸38. Brennan SE, McKenzie JE, Turner T, Redman S, Makkar S, Williamson A, et al. Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers’ capacity to engage with and use research. Health Res Policy Syst. 2017 Jan 17;15(1):1.) were also used in several settings to support users in their pull for research. Based on these findings, the following five interventions were identified as appropriate for inclusion in the model: i) an electronic library of clinical, population health, and health systems research (the Evidence Informed Decision Making Network of the Caribbean (EvIDeNCe) portal); ii) preparation and dissemination of research syntheses (evidence briefs); iii) deliberative dialogues using the best available research evidence (stakeholder dialogues); iv) CARPHA-sponsored training courses for policymakers; and v) a community of practice for evidence-informed policy.

Assessment of design logic and evaluation of expected outcome

Four of the interventions in the initiative were implemented by CARPHA during 2014–2017. Due to resource constraints, start-up of the community of practice was not realized.

We surveyed beneficiaries exposed to each intervention, to collect data about the plausibility of the hypothesized relationships in the model. We also evaluated whether two interventions (stakeholder dialogues and training) achieved the main anticipated outcome, that is, the intention to act on presented evidence.

For the EvIDeNCe portal survey, beneficiaries were purposefully selected from CARPHA's stakeholders who reflected the portal's main target audiences (i.e., policymakers at ministries of health, CARPHA staff, professionals in health-related civil society organizations, and researchers). They were sent a link to an online survey, which included questions about the value of the portal, the usefulness of its search features in assisting users to build a search relevant to their needs, and users’ perception of its ability to improve job efficiency and effectiveness. Feedback was measured on a 5-point Likert scale. Development, pilot testing, and refinement of the data collection instrument was done under guidance of the EvIDeNCe portal project team, which consisted of technical staff from the Pan American Health Organization/World Health Organization (PAHO/WHO), the University of the West Indies, and CARPHA.

For surveys on evidence briefs and stakeholder dialogues, we followed the methodology used by Moat et al. (³⁰30. Moat KA, Lavis JN, Clancy SJ, El-Jardali F, Pantoja T. Evidence briefs and deliberative dialogues: perceptions and intentions to act on what was learnt. Bull World Health Organ. 2014 Jan 1;92(1):20–8.) and Lavis et al. (³⁹39. Lavis JN, Boyko JA, Gauvin F-P. Evaluating deliberative dialogues focussed on healthy public policy. BMC Public Health. 2014 Dec;14(1):1287.). Three stakeholder dialogues were convened using evidence briefs that were circulated to participants approximately two weeks in advance of the dialogue. Each dialogue included 17 to 19 participants, chosen from policy, research, and other stakeholder communities. A steering committee comprised of policymakers, researchers, and other stakeholders selected the participants based on two criteria. The first was that they would be able to articulate views and experiences about the issue under discussion, while constructively engaging with others. The second was that they would be able to champion, within their respective constituencies, actions to address the policy challenges.

All participants attending stakeholder dialogues received evidence briefs and evidence brief evaluation questionnaires prior to the dialogue. They also received stakeholder dialogue evaluation questionnaires at the end of each deliberation. We used the instruments developed and tested by Moat et al. (³⁰30. Moat KA, Lavis JN, Clancy SJ, El-Jardali F, Pantoja T. Evidence briefs and deliberative dialogues: perceptions and intentions to act on what was learnt. Bull World Health Organ. 2014 Jan 1;92(1):20–8.) and Lavis et al. (³⁹39. Lavis JN, Boyko JA, Gauvin F-P. Evaluating deliberative dialogues focussed on healthy public policy. BMC Public Health. 2014 Dec;14(1):1287.). Three types of questions were included: i) how well beneficiaries felt briefs or dialogues achieved their intended purpose; ii) how helpful the beneficiaries found the design characteristics of these two mechanisms; and iii) a question about beneficiaries’ intention to use the evidence presented. All responses were measured on a 7-point Likert scale.

For the training component, CARPHA Member States were invited to nominate individuals to participate in a combined online and face-to-face training workshop. Eleven individuals whose main responsibility was health policy and planning were nominated. All beneficiaries of the training received evaluation questionnaires that rated: i) their overall assessment of the course; ii) whether the course enhanced their skills in finding and using research evidence; and iii) their intention to use the type of research presented during the training. Responses were measured on a 7-point Likert scale.

Data analysis

In total, 164 respondents received evaluation questionnaires: 49 evidence briefs; 50 EvIDeNCe portal; 11 training evaluation; and 54 stakeholder dialogue. The respective response rates were 85.7%, 72.0%, 100.0%, and 88.9%.

We calculated the mean scores on the Likert scales for all questions. To assess the model's design logic, we analyzed responses to questions about the intended objectives of each intervention. Since the EvIDeNCe portal was expected to increase availability of relevant evidence, we analyzed mean scores for five questions about the usefulness of the portal's search features in helping to build a suitable search. Evidence briefs were expected to increase availability of relevant evidence, and to contribute to stakeholder dialogues. Beneficiaries’ feedback about how well the briefs achieved this purpose was examined. We also considered views about the design features of the briefs. Similarly, stakeholder dialogues were expected to equip participants to make or to advocate for evidence-informed change. How well the dialogue supported a full discussion of issues was considered, along with feedback about the dialogue's design features. Finally, the training intervention was intended to enhance participants’ knowledge and skills. Responses to three questions about finding and applying evidence in policy-making were investigated.

The key expected outcome from implementation of the EvIDeNCe initiative was the intention to act. We evaluated whether this was achieved after participation in stakeholder dialogues and exposure to training, by examining the responses to the “intention to use” question. For stakeholder dialogues, we also compared the mean intention-to-use scores, for public policymakers working in government and in regional agencies.

RESULTS

The ToC diagram (Figure 1) shows the relationships among inputs, activities, and outputs for each intervention, and across interventions. These interventions were collectively implemented by CARPHA, under the umbrella of the EvIDeNCe. Table 1 summarizes the survey results.

All interventions were assessed in relation to their intended objectives. Evidence briefs achieved the purpose as a product to present available research evidence, and for use in stakeholder dialogues. The mean score was 6.6 on a scale of 1 (fail) to 7 (achieved). These briefs summarized findings from systematic reviews and Caribbean research, contextualized the evidence by outlining the political and socioeconomic relevance, and included considerations for implementation to guide users. These characteristics were among the highest rated by the beneficiaries, in terms of their helpfulness. Stakeholder dialogues, when combined with evidence briefs, achieved the purpose as a mechanism to support a full discussion of policy issues. The mean score was 6.6 on a scale of 1 (fail) to 7 (achieved). Providing an opportunity for frank deliberation and engaging a facilitator to assist with the deliberation were prominent features contributing to the usefulness of this mechanism. Beneficiaries agreed that the EvIDeNCe portal would enhance job effectiveness and efficiency. Three key features of the portal, designed to assist beneficiaries to build a relevant search, received the highest ratings: i) the ability to search for research in different languages spoken in the Caribbean; ii) the availability of full text; and iii) the ability to narrow a search by Caribbean country of interest. Policymakers also agreed that the training course enhanced knowledge of sources of research evidence, with a mean score of 6.7 on scale of 1 (strongly disagree) to 7 (strongly agree). The course also enhanced skills in searching for research, with a mean score of 6.3 on the same scale.

Beneficiaries’ intention to act on presented research was a key expected outcome. Stakeholder dialogues resulted in the production of three outputs (a dialogue summary, a topic overview, and a commitment-to-act form), which included clear statements of the participants’ intention to use the evidence in policy briefings, policy decisions, or advocacy. (These qualitative findings are not reported here.) The intention to act was also reported quantitatively. Overall, the mean response for all dialogue participants was 6.4 on a scale of 1 (strongly disagree) to 7 (strongly agree). Participants who were policymakers in government or in regional agencies reported a slightly higher preference, with a mean score of 6.6 for both groups. Beneficiaries of the training course also signaled their intention to act on the research evidence to which they were exposed, with an overall mean score of 6.3 on a scale of 1 (strongly disagree) to 7 (strongly agree).

DISCUSSION

This study has shown that the barriers faced with research utilization in decision-making in the Caribbean can be precisely addressed through a portfolio of interventions incorporated into a single cohesive program. The logical pathway linking each intervention to the desired change in decision-making was demonstrated. Surveys of beneficiaries validated the inclusion of four of the interventions in the model: i) EvIDeNCe portal research database; ii) evidence briefs; iii) stakeholder dialogues; and iv) training for policymakers. Two interventions (stakeholder dialogues using evidence briefs, and training) achieved the key desired outcome, namely, the intention to act on available research.

The design characteristics of the included interventions were instrumental. Policymakers need context-specific evidence to support their decision-making. The EvIDeNCe portal database allowed users to focus a search on a specific Caribbean country or group of countries and a particular language spoken in the region. These features were viewed by beneficiaries as helpful. Evidence briefs presented evidence on important topics while also considering the political and socioeconomic environment. This, along with the opportunity to hold a frank off-the-record deliberation, was also viewed by beneficiaries as very useful.

The findings of this study are similar to those conducted in other countries. Evaluations of evidence briefs and deliberative dialogues used in Africa (²⁰20. Ongolo-Zogo P, Lavis JN, Tomson G, Sewankambo NK. Initiatives supporting evidence informed health system policymaking in Cameroon and Uganda: a comparative historical case study. BMC Health Serv Res. 2014 Nov 29;14:612., ³⁰30. Moat KA, Lavis JN, Clancy SJ, El-Jardali F, Pantoja T. Evidence briefs and deliberative dialogues: perceptions and intentions to act on what was learnt. Bull World Health Organ. 2014 Jan 1;92(1):20–8.) and Canada (³⁹39. Lavis JN, Boyko JA, Gauvin F-P. Evaluating deliberative dialogues focussed on healthy public policy. BMC Public Health. 2014 Dec;14(1):1287.) have shown that participants respond in a similar manner when asked about the usefulness of these mechanisms. The integrated approach adopted by the EvIDeNCe initiative is also in alignment with knowledge translation platforms (KTPs), such as the Evidence-Informed Policy Network (EVIPNet), which is implemented in Africa, Europe, and Latin America. The activities included in the EvIDeNCe initiative are similar to those undertaken by EVIPNet teams (20, 21, 40), with the exception of a Rapid Response Service, which was not included in EvIDeNCe. Two main institutional differences can also be observed. Firstly, KTPs are usually hosted in universities, hospitals, or ministries of health (⁴¹41. El-Jardali F, Lavis J, Moat K, Pantoja T, Ataya N. Capturing lessons learned from evidence-to-policy initiatives through structured reflection. Health Res Policy Syst. 2014 Jan 17;12:2.). The EvIDeNCe initiative is, however, located in a regional public health agency. Secondly, the work of KTPs tends to be country-specific, while CARPHA's EvIDeNCe addresses issues relevant to the health systems across an entire region. It would therefore be important to understand, through future research, how this difference in governance arrangement and activity scope impacts operations and sustainability.

Our study has several strengths. It is the first to develop and apply a comprehensive framework for development and evaluation of interventions to strengthen the use of evidence in decision-making in the Caribbean region. It contributes theoretical and empirical knowledge on the activities and outputs that can be combined to support evidence-informed decision-making in this geographical and institutional context. It links the literature about the specific problems faced in the Caribbean region to program design. By using ToC instead of the logical framework, it depicts how the required change process can be logically brought about through the interrelated elements of the program.

Despite these strengths, the limitations of the work should also be acknowledged. Firstly, the study is a formative assessment that signals the change process possible, using the strategies identified. We did not evaluate all outcomes included in the model. Although the intention to use evidence in decision-making is an important and measurable outcome, some attempt should be made to determine whether the other expected outcomes were also achieved. This would include determining whether the portal actually increased evidence availability, and whether policymakers and stakeholders were indeed better equipped to make evidence-informed decisions. We did not evaluate the consolidated initiative by using a composite data collection instrument. Therefore, until this further evaluative work is done, we are unable to draw conclusions about the overall impact. Secondly, the model rests upon several critical assumptions. For example, we have assumed the absence of access barriers to electronic resources. The validity of these assumptions would have to be verified.

Conclusion

This work developed a theory of change to narrow the research-policy gap in the Caribbean. The model's design logic was validated by beneficiaries, who gave positive ratings to the overall purpose and usefulness of four of the included interventions. The ability of two of these interventions (stakeholder dialogues using evidence briefs, and capacity-building workshops) to contribute to the main desired outcome was also confirmed.

Policy recommendations

While the results are promising, consideration could be given to incorporating a Rapid Response Service in the EvIDeNCe model. Such a service would provide research syntheses to policymakers, at their request, in a relatively short time frame. It was reported in Chile that this service is widely used and highly valued by ministry of health officials (²¹21. Mansilla C, Herrera C, Basagoitia A, Pantoja T. The Evidence-Informed Policy Network in Chile: lessons learned from a year of coordinated efforts. Rev Panam Salud Publica. 2017;41(e36).). However, because the EvIDeNCe initiative serves a region and not a country, the operation of such a service would have to be carefully managed and the required resources made available. One possible approach could be to build the capacity of country teams to support implementation.

A consolidated indicator framework should also be developed to monitor the reality of progress in research utilization in policy. We recommend a high-level framework for use in the Caribbean (Table 2), although it could be applied elsewhere. The indicators in this framework span the four strategic areas included in the ToC model. Monitoring them would require collection of baseline data and intermittent surveys of policymakers. This framework can be implemented by the Caribbean Community Secretariat, as part of the monitoring and evaluation process for the Caribbean Cooperation in Health (CCH). CCH facilitates coordinated action and collaboration among partners for health in the region, and that initiative clearly articulates the desire to enhance regional capacity to generate and use quality evidence for effective policy and programming.

Acknowledgments and funding.

The EvIDeNCe initiative was developed with the support of the McMaster Health Forum, which provided the methodological approach for evidence briefs and stakeholder dialogues; conducted training on finding and using research evidence; designed the survey instruments for evaluating these components; and developed the scoping document for the EvIDeNCe portal. PAHO/WHO provided financial support for stakeholder dialogues and completed the developmental work for the portal. The University of the West Indies MedCarib interface supports uploading of published and gray literature. We also thank the students from McMaster University who assisted with data entry and analysis and the graphic work presented here. We also acknowledge all the persons who reviewed and provided comments on earlier versions of the manuscript.

REFERENCES

Publication Dates

History