ABSTRACT

This report contributes towards the monitoring and evaluation of the Pan American Health Organization/World Health Organization (PAHO/WHO) Policy on Research for Health (CD49/10), specifically, how it has been used by Member States and how it has influenced their policies, strategies, research, and relevant stakeholders. This cross-sectional study examined the frequency, method of utilization, region of interest or “scope,” and rationale behind citing the Policy in PAHO/WHO documents, databases, government websites, and internal documents. The extent of utilization was measured through an ordinal scale; the country or region of interest was systematically extracted. Of the 993 documents that were identified through the search strategy, 95 met the inclusion criteria. There was a significant relationship between type of document and scope (χ² = 69.5; P < 0.001), with web pages and scientific articles covering the Americas more frequently mentioning the Policy. The Policy was most often used at the country level in government documents and at the regional level in webpages and journal articles.

Although the Policy has been utilized by several countries, many countries have yet to integrate it with their national health research initiatives. Additional research should focus on understanding why Policy utilization differs among countries. To promote better cohesion across sectors and levels of governance, researchers and policymakers should seize opportunities to integrate the Policy with the research process, research governance, and policy development. The protocol developed for this study can be applied to similar analyses of other PAHO/WHO policies to gain a greater understanding of their influence.

Keywords
Health policy; health services research; World Health Organization; Pan American Health Organization; Americas

RESUMEN

El presente informe contribuye al seguimiento y la evaluación de la Política de investigación para la salud de la Organización Panamericana de la Salud/Organización Mundial de la Salud (OPS/OMS) (documento CD49/10), específicamente, para determinar cómo han empleado dicha política los Estados Miembros y cómo ha influido en sus políticas, estrategias e investigación y en los interesados directos pertinentes. En este estudio transversal se examinaron la frecuencia, el método de utilización, la región de interes o “alcance” y la justificación mencionados al citar la política en los documentos de la OPS/OMS, las bases de datos, los sitios web gubernamentales y los documentos internos. El grado de utilización se midió con una escala ordinal; se extrajeron sistemáticamente el país o la región de interes. De los 993 documentos que se encontraron mediante la estrategia de búsqueda, 95 cumplían con los criterios de inclusión. Hubo una relación significativa entre el tipo de documento y el alcance (χ² = 69,5; P < 0.001); las páginas web y los artículos científicos relativos a la Región de las Américas fueron los que más mencionaron la política. La política se usó con mayor frecuencia a nivel de país en los documentos gubernamentales, y a nivel regional, en las páginas web y los artículos de revistas.

Si bien varios países han utilizado la política, muchos otros todavía no la han integrado en sus iniciativas nacionales de investigación en salud. Las futuras investigaciones deben centrarse en comprender por qué el uso de la política varía entre los países. Para fomentar una mejor cohesión entre los distintos sectores y niveles de gobernanza, los investigadores y los responsables de formular políticas deben aprovechar las oportunidades de integrar la política en el proceso de investigación, la gobernanza de la investigación y la formulación de políticas. El protocolo elaborado para el presente estudio puede aplicarse a análisis similares de otras políticas de la OPS/OMS, a fin de comprender mejor su influencia.

Palabras clave
Política de Salud; investigación en servicios de salud; Organización Mundial de la Salud; Organización Panamericana de la Salud; Américas

RESUMO

Este informe visa contribuir com o monitoramento e avaliação da política de pesquisa para a saúde da Organização Pan-Americana da Saúde/Organização Mundial da Saúde (OPAS/OMS) (CD49/10), em particular, como ela vem sendo aplicada pelos Estados Membros e como tem influenciado políticas, estratégias, pesquisa e interessados diretos relevantes. Trata-se de um estudo transversal que examinou a frequência e forma de aplicação da política, região de interesse ou “alcance” e justificação para citar a política em documentos da OPAS/OMS, bancos de dados, sites do governo e documentos internos. O nível de aplicação foi medido em uma escala ordinal e o país ou a região de interesse foram obtidos com a coleta sistemática. Dos 993 documentos identificados com a estratégia de busca, 95 satisfizeram os criterios de inclusão. Observou-se uma relação significativa entre o tipo de documento e o alcance (χ² = 69,5; P < 0,001) e maior ocorrência de menção da política em sites e artigos científicos relacionados às Americas. A política foi mais comumente aplicada ao nível nacional em documentos do governo e ao nível regional em sites e artigos de periódicos.

Apesar de vários países terem aplicado a política de pesquisa para a saúde, muitos ainda precisam integrá-la às próprias iniciativas nacionais de pesquisa em saúde. Outros estudos devem ser realizados para investigar por que a aplicação da política é distinta entre os países. Para melhorar a coesão entre os diversos setores e níveis de governança, os pesquisadores e os responsáveis por políticas devem aproveitar as oportunidades para integrar esta política ao processo de pesquisa, governança em pesquisa e formulação de políticas. O protocolo preparado para este estudo pode ser empregado em análises semelhantes de outras políticas da OPAS/OMS para um entendimento mais aprofundado da influência que elas têm.

Palavras-chave
Política de Saúde; Pesquisa sobre Serviços de Saúde; Organização Mundial da Saúde; Organização Pan-Americana da Saúde; Américas

The Pan American Health Organization (PAHO) was founded in 1902. It is the Regional Office of the World Health Organization (WHO) for the Americas and has a long trajectory of health policies and strategies for the Region (¹1. Pan American Health Organization. Key facts about PAHO. 2015. Available from: www.paho.org/hq/index.php?option=com_content&view=article&id=92&Itemid=40697&lang=en Accessed 19 June 2015.
www.paho.org/hq/index.php?option=com_con... ). The PAHO/WHO Policy on Research for Health (the “Policy”) was approved by the 49^th Directing Council, 61^st Session of the Regional Committee on 30 September 2009 and marks a milestone in the field (²2. Pan American Health Organization. Policy on research for health. Proceedings of the 49th Directing Council of PAHO, 61st Session of the Regional Committee of the WHO for the Americas, 28 September - 2 October 2009. Washington, DC: PAHO; 2009. (Document CD49/10; Resolution CD49.R10)). The Policy guides technical cooperation, provides a basis for strategy development in the Americas, and aligns national, regional, and international research policy objectives. It promotes strong research governance and the integration of research with health care, policy, and prevention. Its six objectives are:

1. to promote the generation of relevant, ethical, and quality research;

2. to strengthen research governance and promote the definition of research agendas;

3. to improve competencies of and support for human resources involved in research;

4. to seek efficiencies and enhanced impact and appropriation of research through effective and strategic alliances, collaboration, and the building of public trust and engagement in research;

5. to foster best practices and enhanced standards for research; and

6. to promote the dissemination and utilization of research findings (²2. Pan American Health Organization. Policy on research for health. Proceedings of the 49th Directing Council of PAHO, 61st Session of the Regional Committee of the WHO for the Americas, 28 September - 2 October 2009. Washington, DC: PAHO; 2009. (Document CD49/10; Resolution CD49.R10)).

These objectives highlight the primary focus of achieving research for health: to promote best practice research at all levels in order to improve human health. Research for health is a crucial component of WHO's primary objective: to obtain the highest level of health possible for all. Research for health is critical in promoting development, improving global health, and fostering greater innovation (³3. World Health Organization The WHO strategy on research for health. Proceedings of the 63rd World Health Assembly, 21 May 2010. Geneva: WHO; 2012. (Document A63/22). Available from: www.paho.org/hq/index.php?option=com_content&view=article&id=10324%3A2014-who-strategy-on-research-for-health&catid=7532%3Apolicy&Itemid=41202&lang=en Accessed 28 February 2017.
www.paho.org/hq/index.php?option=com_con... ). Overarching guiding values of this PAHO/WHO Policy include equity, solidarity, respect, integrity, and excellence. This Policy and the WHO Strategy on Research for Health include the five main areas of research: problem assessment; understanding causes; identifying solutions; translation to policy; and evaluation (³3. World Health Organization The WHO strategy on research for health. Proceedings of the 63rd World Health Assembly, 21 May 2010. Geneva: WHO; 2012. (Document A63/22). Available from: www.paho.org/hq/index.php?option=com_content&view=article&id=10324%3A2014-who-strategy-on-research-for-health&catid=7532%3Apolicy&Itemid=41202&lang=en Accessed 28 February 2017.
www.paho.org/hq/index.php?option=com_con... ).

The implementation of this Policy, as dictated by the Health Agenda for the Americas (²2. Pan American Health Organization. Policy on research for health. Proceedings of the 49th Directing Council of PAHO, 61st Session of the Regional Committee of the WHO for the Americas, 28 September - 2 October 2009. Washington, DC: PAHO; 2009. (Document CD49/10; Resolution CD49.R10)), requires monitoring and evaluation activities. The Policy was created nine years ago and a true impact analysis among PAHO Member States remains incomplete. An analysis of the influence, spread, and use of PAHO/WHO policies and strategies is critical to understanding their impact. Not only would such an analysis aid in understanding the utilization of specific plans and alignment with policies, it can also help identify gaps in policy usage, provide insight for future practices, and promote better coordination and consistency across different levels of governance and stakeholders.

As far as the authors have been able to ascertain, there are no existing protocols for evaluating the impact of PAHO/WHO policy utilization nor has there been any analysis of how, where, and to what extent the Policy has been utilized. Therefore, this reported aimed to understand the impact of the Policy on Research for Health by identifying its use in electronic and available printed documents published by PAHO, its Member States, and/or stakeholders. Specifically, the report sought to analyze utilization of the Policy in technical and policy documents by classifying the frequency location, and extent of its use, as well as identifying gaps in policy application.

MATERIALS AND METHODS

This was a cross-sectional study that examined the frequency method, location, user profile, and rationale for utilization of the Policy on Research for Health by PAHO, its Member States, and stakeholders, such as universities, development/research partners, PAHO/WHO Collaborating Centers, Pan American Centers, and PAHO Country Offices. A systematic search was conducted from 5 - 29 June 2015 using a protocol based on existing systematic review guidelines and recommendations, including PRISMA (⁴4. Jones K. Doing a literature review in health. In: Poyner A, Milman E, eds. Researching health: qualitative, quantitative, and mixed methods. London: SAGE; 2013. pp. 42 - 64.

5. Ruiz M, Morillo L. Clinical epidemiology: applied clinical research. Bogotá: Medica Panamericana; 2004.

6. Liberati A, Altman, D, Tetzlaff, J, Mulrow, C, Gotzsche, P. Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLOS Med. 2009;6(7):e1000100. doi:10.1371/journal.pmed.1000100
https://doi.org/10.1371/journal.pmed.100... - ⁷7. Abalos E, Carroli G, Mackey ME, Bergel E. Critical appraisal of systematic reviews. Geneva: World Health Organization; 2001. (WHO/RHR/01.6)).

Data collection

The search included PAHO/WHO documents, databases, government websites, internal documents, and grey literature, as well as the following electronic databases: IRIS (iris.paho.org); Health Systems Evidence (healthsystemsevidence.org); EVIPNet (global.evipnet.org); EQUATOR Network (equator-network.org); BIREME (regional.bvsalud.org); LILACS (lilacs.bvsalud.org); PUBMed (nlm.nih.gov); SciELO (sielo.org); Research Gate (researchgate.net); OAS (oas.org); Web of Knowledge (webofknowledge.com); and Global Health Library (globalhealthlibrary.net). Search engines were utilized to locate webpages that met the search criteria. Citation network analysis was performed to identify related, eligible articles from among the references of published literature. Additionally national and local government websites provided online by PAHO/WHO were systematically searched. Grey literature was reviewed to identify documents not published on government websites or in peer-reviewed journals. PAHO advisors also provided grey literature, including press releases, published presentations, institutional repositories, and travel reports.

Inclusion criteria were as follows: published in a PAHO/WHO official language (English, French, Portuguese, and Spanish); produced between the Policy's adoption date (September 2009) and the time of the study search (June 2015); contained a keyword and PAHO identifier (“PAHO” or “AMRO” or “Pan American Health Organization” or name of the specific strategy/strategy code); and cited the Policy Additional data on documents that did not meet the inclusion criteria were not collected.

Taxonomy in Medical Subject headings (MeSH), standard terms in the literature, abbreviations, translations, and cultural spelling differences were included as keywords. The final list of search terms was reviewed by an expert in the field (LGC) and comprised the following:

Variables

Variables were “article type,” “extent of utilization rating,” “country or region of interest/scope,” and date published. Categories for “article type” included PAHO/WHO documents found in the IRIS database, articles found in electronic databases, general Internet search, official country websites, and internal documents. Categories for “region of interest/scope” included global, Americas/Regional (grouped as Latin America, Latin America and the Caribbean, and PAHO Member States), and individual countries. While the study focused on PAHO Member States, any Policy utilization by other WHO Regions was noted.

Extent of utilization rating. To measure the extent of utilization, an ordinal scale from 1 - 4 was created and validated through a pilot of 15 articles identified in a general Internet search. Two reviewers rated the articles with 100% agreement. A higher rating on the scale indicated a relatively greater amount of influence by the Policy on the document. The scale was defined as follows:

1. Policy citation without explanation;

2. citation and explanation of the Policy;

3. application of the Policy for the purpose of the article; and

4. the Policy was the basis for the document.

A PAHO content expert was consulted if necessary when assigning ratings.

Data analyses

Resulting data were systematically extracted and organized using Microsoft Excel™ (Microsoft Corp., Redmond, Washington, United States) and following PRISMA methodology as applicable (⁶6. Liberati A, Altman, D, Tetzlaff, J, Mulrow, C, Gotzsche, P. Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLOS Med. 2009;6(7):e1000100. doi:10.1371/journal.pmed.1000100
https://doi.org/10.1371/journal.pmed.100... ). Any duplicates were manually identified and eliminated. The complete file is available from the authors upon request.

The scope, frequency and the extent of utilization for each citation were assessed. Potential gaps in utilization were identified based on low frequency of Policy citation by individual countries. Chi-square tests were conducted to identify any significant relationship between article type and rating; article type and scope; and rating and scope.

RESULTS

Of 993 documents that were identified through the search strategy and screened, 95 met the inclusion criteria. The “scope” was categorized as: global; Americas; or country level. The “type of document” was categorized as: official PAHO/WHO documents; webpages; database/journal articles; government; or internal documents. Among the documents categorized as “internal,” the Policy was most often promoted at a conference or in a presentation.

Including internal documents, 43 citations were found relating to 17 PAHO Member Countries, as well as Denmark, Italy Spain, and the United Kingdom. Excluding internal documents, 16 citations were found for 10 countries: Argentina, Brazil, Colombia, Cuba, Dominican Republic, Ecuador, Honduras, Peru, Uruguay and Paraguay

As shown in Table 1, the types of documents most frequently found at the country level were internal (n = 27) and government (n = 7); while the types of documents most frequently observed at the Regional level were web pages (n = 24), journal articles (n = 12), and PAHO/WHO documents (n = 10). Overall, the Policy was most frequently cited in web pages (n = 30) and internal documents (n = 28), and least cited in PAHO/WHO documents (n = 12) and in government documents (n = 7).

Table 2 shows the results from government websites. Of 30 functioning websites produced by PAHO Member States in an official PAHO language, 5 included articles that were relevant to the search: Colombia, Dominican Republic, Ecuador, Panama, and Peru.

The chi-squared tests performed to examine the relationships between the different variables showed a significant association between type of document and scope (χ² = 69.5; P < 0.01). On the country level, the Policy was most often found an internal document (n = 27). The Policy was least frequently cited on a global level (n = 5). The relationship was also significant between type of document and rating (χ² = 35.2; P < 0.01). More specifically the article category with the highest average rating or greatest application of the Policy was internal, followed by web pages, government web pages/articles, and journal articles. The highest ratings were most often seen in internal documents (rating 3, n = 15; rating 4, n = 8).

DISCUSSION

While a number of countries in the Americas have cited the Policy, the remaining countries represent potential gaps and should be considered opportunities for Policy promotion and implementation.

The significant relationship between document type and scope showed that the Policy has had relatively more influence at the country level and in government and internal documents. On the Regional (Americas) level, the Policy has been more influential in general web pages, scientific papers, and PAHO/WHO related publications.

Regarding the significant relationship between document type and rating, low rankings often occurred in journal articles. Researchers frequently cited the Policy in the background section, without an explanation or application of the Policy i.e., a rating of 1; these findings do not demonstrate effective policy utilization. Higher ratings (3 and 4) most often occurred in internal documents, such as presentations on the Policy or promotion of the Policy at conferences. We could not find significant differences in rankings among government website citations because there were only two or fewer articles within each ranking group.

Countries that have implemented national policies on research for health and serve as examples for other countries include the Dominican Republic and Paraguay The National Policy of Research for Health of the Dominican Republic (⁸8. Ministry of Public Health of the Dominican Republic, Vice Minister of Planning and Development of National Health Research. National policy of research for health. Santo Domingo: Ministry of Public Health; 2014. Available from: www.msp.gob.do/dinisa/documentos/Pol-dinisa.pdf Accessed 19 June 2015.
www.msp.gob.do/dinisa/documentos/Pol-din... ) was a collaborative effort involving PAHO/WHO, the Ministry of Health, local universities, and other collaborators (⁸8. Ministry of Public Health of the Dominican Republic, Vice Minister of Planning and Development of National Health Research. National policy of research for health. Santo Domingo: Ministry of Public Health; 2014. Available from: www.msp.gob.do/dinisa/documentos/Pol-dinisa.pdf Accessed 19 June 2015.
www.msp.gob.do/dinisa/documentos/Pol-din... ). The National Policy on Research for Health of Paraguay was a collaborative effort involving PAHO/WHO, the Ministry of Public Health, local universities, and other collaborators (⁹9. Ministry of Public Health of Paraguay National policy of research for health. Asunción: PAHO/WHO; 2010. Available from: www.cepep.org.py/archivos/POLITICA_INVEST_SALUD_070710.pdf Accessed 22 June 2015.
www.cepep.org.py/archivos/POLITICA_INVES... ). One of the guiding frameworks for Paraguay's national policy was the PAHO/WHO Policy on Research for Health (⁹9. Ministry of Public Health of Paraguay National policy of research for health. Asunción: PAHO/WHO; 2010. Available from: www.cepep.org.py/archivos/POLITICA_INVEST_SALUD_070710.pdf Accessed 22 June 2015.
www.cepep.org.py/archivos/POLITICA_INVES... ). Furthermore, Peru has integrated the Policy on Research for Health into the call for the National Research Priorities for 2015 - 2021 (¹⁰10. National Institute of Health of Peru. National research priorities, 2015 - 2021. Lima: Ministry of Health; 2015. Available from: www.ins.gob.pe/portal/jerarquia/2/1141/prioridades-nacionales-de-investigacion-2015-2021/jer.1141 Accessed 23 June 2015.
www.ins.gob.pe/portal/jerarquia/2/1141/p... ).

Of the 30 functioning websites produced by PAHO Member States at the time of the study, only 4 contained articles that met the inclusion criteria. The majority of country-level policy citations were found through internal documents, government websites, and journal articles. Several countries cited the Policy on Research for Health, 17 including internal documents and 10 excluding internal documents. A number of countries have national policies on health research that do not cite the Policy and thus, were not included in this analysis.

There were a number of countries that did not mention the Policy in any type of document. These countries are potential areas of opportunity for Policy promotion and utilization. Countries that have only a few articles on the Policy should still be considered for further cooperation. Parallel to the Country-Focused Cooperation Initiative (¹¹11. Pan American Health Organization. Country-focused cooperation and national health development. Proceedings of the 46th Directing Council, 57th Session of the Regional Committee, 26 - 30 September 2005. Washington, DC: PAHO; 2005. (Document CD46/19)), which calls for PAHO/WHO to encourage a shift towards country-focused strategies, there is an evident need to further focus on country-specific applications of the Policy.

This study used a systematic protocol to examine utilization of PAHO/WHO policies and strategies in various types of documents. This protocol has been employed to assess other PAHO/WHO policies or strategies, such as the PAHO/WHO Strategy and Plan of Action on eHealth for the Americas, and can continue to be useful for this purpose. It can also serve as a systematic way to identify areas where stronger ties are needed among policy Member States, and research. This protocol may be used as an entry point to identify gaps and encourage further research to form a complete picture of policy utilization.

Limitations

This study was limited to published online material and accessible documentation. If countries utilized the Policy in another manner and/or had not updated their web pages, we did not have access to this information and it would not have been included.

Risk of bias includes only having one reviewer to identify articles that met inclusion criteria to rate the extent of utilization. To mitigate this bias, the scale was validated with two reviewers prior to the study. The scale allows for a high-level rating and does not capture in-depth qualitative information on the impact of the Policy. Documents obtained from experts were also susceptible to reporting or publication bias (⁷7. Abalos E, Carroli G, Mackey ME, Bergel E. Critical appraisal of systematic reviews. Geneva: World Health Organization; 2001. (WHO/RHR/01.6)). Thus, results are reported both with and without internal documents.

Conclusions

The PAHO/WHO Policy on Research for Health was most frequently cited on web pages, internal documents, and databases/journal articles, and most frequently at the regional (Americas) or country level. The countries that did not cite the Policy should be the focus of further Policy integration.

The most effective actions should be taken to equitably benefit the people in PAHO Member States. Specifically we recommend:

• Working with each individual country especially any country that has not utilized the Policy, to integrate it with national institutions, strategies, and/or policies, and to possibly prioritize national actions within the Policy framework.

• Promoting utilization of policies by researchers at all stages of the investigative process so that their work may best contribute within a relevant, contextual, policy framework.

• Encouraging countries, researchers, and relevant stakeholders to not just cite policies, but to strategically apply them to strengthen research for health.

• Conducting future research to examine why policy utilization and uptake differs among different countries.

In conclusion, we recommend not simply citing PAHO/WHO policies and strategies, but rather promoting their country-specific application and technical support.

Acknowledgements

The authors wish to thank Marcelo D'Agostino for serving as advisor to the project and protocol design, as well as editor; Eliane Pereira Dos Santos for providing support on search strategies and resources; David Novillo for his support on search strategies; Michele Gilbert for providing support with data analysis design and editing; and Claudia Frankfurter, Rita Lechuga, and Julia Rodriguez for serving as editors.

REFERENCES

Publication Dates

History