ABSTRACT

The improvement of health in the twenty-first century is inextricably linked to research for health. In response to growing international appeal to address regional health needs, the Pan American Health Organization (PAHO) and its Member States approved the Policy on Research for Health (CD49/10) in 2009. This document represents the flagship regional policy on research for health and outlines how health systems and services in the region can be strengthened through research. It has been implemented by the two components of PAHO -the Member States and the Pan American Sanitary Bureau. The policy contained a specific directive mandating PAHO to report on its implementation, development of subsequent strategies, and action plans targeting its governing bodies. The Americas are the first World Health Organization (WHO) region to issue a regional Policy on Research for Health, which was harmonized with WHO's Strategy on Research for Health, approved in 2010. Attending to the recommendations issued by PAHO's Advisory Committee on Health Research and WHO's Advisory Committee on Health Research, the PAHO Department of Knowledge Management, Bioethics and Research set out to advance the assessment of the implementation of the Policy on Research for Health through the creation of a monitoring and evaluation Scorecard. Indicators relevant to the Policy on Research for Health objectives were mapped from the Compendium of Impact and Outcome Indicators, with new indicators created. A practical framework based on available indicator data was proposed to generate a baseline policy assessment and incorporate a means of incrementally enhancing the measurements. In this case study, we outline the iterations of the PAHO Policy on Research for Health Scorecard, as well as the lessons learned throughout the development process that may be a valuable guide for health research entities monitoring and evaluating the progress of their own policies.

Keywords:
Pan American Health Organization; Health policy; Evaluation studies

RESUMEN

La mejora de la salud en el siglo XXI está inextricablemente ligada a la investigación sanitaria. En respuesta a la llamada internacional creciente, de cara a abordar las necesidades sanitarias regionales, la Organización Panamericana de la Salud (OPS) y sus Estados miembros aprobaron en 2009 laPolítica sobre Investigación Sanitaria(CD49/10). Este documento constituye la política regional insignia sobre investigación sanitaria, y destaca cómo pueden reforzarse los sistemas y servicios sanitarios en la región a través de la investigación. Ha sido implementado por parte de los dos componentes de la OPS: los Estados miembros y la Agencia Sanitaria Panamericana. La política contenía una directiva específica, que encomendaba a la OPS la realización de un informe sobre su implementación, desarrollo y políticas subsiguientes, al igual que los planes de acción dirigidos a sus órganos directivos. El continente americano es la primera región de la Organización Mundial de la Salud (OMS) que establece una política regional sobre investigación sanitaria, armonizada con laEstrategia sobre investigación sanitariade la OMS, aprobada en 2010. Atendiendo a las recomendaciones emitidas por el Comité Asesor sobre investigación sanitaria de la OPS y el Comité Asesor sobre investigación sanitaria de la OMS, el Departamento de Gestión del Conocimiento, Bioética e Investigación de la OPS propuso avanzar en la evaluación de la implementación de la política de investigación sanitaria a través de la creación de un cuadro de mando de supervisión y evaluación. Los indicadores relevantes de los objetivos de la política sobre investigación sanitaria fueron pareados a partir de los indicadores de Compendio del Impacto y Resultados, con los nuevos indicadores. Se propuso un marco práctico basado en los datos indicadores disponibles, para generar una evaluación de la política basal e incorporar un medio de mejorar las medidas gradualmente. En este estudio de caso destacamos las iteraciones del cuadro de mando de la política sobre investigación sanitaria de la OPS, así como las lecciones aprendidas a lo largo del proceso de desarrollo, que podrían constituir una guía valiosa para las entidades de investigación sanitaria de cara a supervisar y evaluar el progreso de sus propias políticas.

Palabras clave:
Organización Panamericana de la Salud; Política sanitaria; Estudios de evaluación

Background

“Research for health” has been a core activity of the Pan American Health Organization (PAHO) since its establishment in 1902. “Research”, defined as a systematic process of evidence generation in response to a specified hypothesis or challenge, is critical to fulfilling PAHO's role in improving health and health equity in the Americas.¹1. Commission on Health Research Development. Health Research. New York, NY: Oxford University Press; 1990. ²2. Pan American Health Organization. 49th Directing Council 61st Session of the Regional Committee of WHO for the Americas. Policy on research for health: document CD49/10. Washington, DC; 2009. Available from: http://www2.paho.org/hq/dmdocuments/2009/CD49-10-e.pdf
http://www2.paho.org/hq/dmdocuments/2009... “For health” encapsulates the philosophy that the generation of evidence driving health decision-making is an inter-sectoral, multi-disciplinary activity.²2. Pan American Health Organization. 49th Directing Council 61st Session of the Regional Committee of WHO for the Americas. Policy on research for health: document CD49/10. Washington, DC; 2009. Available from: http://www2.paho.org/hq/dmdocuments/2009/CD49-10-e.pdf
http://www2.paho.org/hq/dmdocuments/2009... ³3. World Health Organization. The Bamako call to action: research for health. World Health Organization. Geneva, Switzerland; 2008. Available from: http://www.who.int/rpc/news/BAMAKOCALLTOACTIONFinalNov24.pdf
http://www.who.int/rpc/news/BAMAKOCALLTO... High-quality research for health produces information that can guide PAHO and its Member States in making evidence-informed decision and promoting health equity in the Americas.²2. Pan American Health Organization. 49th Directing Council 61st Session of the Regional Committee of WHO for the Americas. Policy on research for health: document CD49/10. Washington, DC; 2009. Available from: http://www2.paho.org/hq/dmdocuments/2009/CD49-10-e.pdf
http://www2.paho.org/hq/dmdocuments/2009...

PAHO's Policy on Research for Health (Policy), adopted by its Member States at the 49^th Directing Council in 2009, represents an international recognition that research for health is crucial in achieving health equity and represents a core function of the Organization. The Policy is a result of iterative consultations with the World Health Organization (WHO), PAHO Advisory Committee on Health Research (ACHR), as well as external and internal stakeholders, including WHO Collaborating Centers, research institutions, and consumer representatives and others who participated in public consultations during its development process. The Policy also outlines how high-quality research can strengthen health systems and services in the PAHO region through six objectives: (1) promotion of research generation, (2) research governance strengthening, (3) human resource competency development, (4) establishment of effective and strategic alliances, (5) adoption of best research practices, and (6) research finding dissemination and utilization.⁴4. Pan American Health Organization. Resolution CD49/10. Washington, DC; 2009. Available from: http://new.paho.org/hq/pdmdocuments/2009/CD49-R10-Eng.pdf
http://new.paho.org/hq/pdmdocuments/2009...

The Policy features a specific directive that mandates PAHO to report to the Governing Bodies on its implementation alongside the development of subsequent strategies and action plans. PAHO reinforced this mandate at its 44^th ACHR meeting in 2012 by issuing a recommendation to develop a Scorecard to assess implementation progress of the policy across the Americas.⁵5. Advisory Committee on Health Research. Advisory Committee on Health Research 44th Meeting. Barcelona, Spain; 2011. The Department of Knowledge Management, Bioethics and Research of PAHO took it upon itself to devise such a tool. In this commentary, we present our experiences and lessons learned in the development of the “Policy on Research for Health Scorecard” (Scorecard) as a tool to monitor the implementation of PAHO's Policy on Research for Health CD 49/10.

Development of the Scorecard

A Scorecard leveraging PAHO's existing administrative evaluation tools was designed by the Department of Knowledge Management, Bioethics and Research of the Pan American Health Organization. This tool was derived from a total of three iterations. We initially conducted a survey of 45 managers of the Pan American Sanitary Bureau (PASB), the Secretariat of PAHO, to assess how managers perceive and support the implementation of the Policy. In this survey, 44 percent (20/45) of the PASB managers participated and from their responses, we identified two priority areas of concern: adherence to research registration at PASB and the need for higher compliance with standards and processes for guideline formulation and the review of research protocols. The full results of this survey are reported elsewhere.⁶6. Cuervo L, Soelling S, Hutchinson R. Policy on research for health implementation survey (CD49/10). Washington, DC; 2014. Given the limited response rate of the survey and the need to assess Policy progress in Member States, we opted to pursue a more comprehensive and objective methodology of assessing the Policy implementation.

In the first iteration we identified indicators for each of the recommended actions in the Policy, which resulted in a long list of over 60 questions that would require of workshops in each country to be completed. Recognizing the limited feasibility of the implementation of such workshops and the need for annual maintenance, this option was shelved.

In the second iteration, we reviewed the Policy, the Report of the 45^th ACHR of the Pan American Health Organization, the Strategic Plan of the Pan American Health Organization 2014-2019, the Compendium of Impact and Outcome Indicators, and the WHO Strategy on Research for Health. We identified 36 potential indicators to assess adherence and compliance to the Policy.⁴4. Pan American Health Organization. Resolution CD49/10. Washington, DC; 2009. Available from: http://new.paho.org/hq/pdmdocuments/2009/CD49-R10-Eng.pdf
http://new.paho.org/hq/pdmdocuments/2009... ⁷7. Advisory Committee on Health Research. 45th session of PAHO's Advisory Committee on Health Research. Washington, DC; 2012.

8. PAH Organization. Strategic plan of the Pan American Health Organization 2014-2019. Washington, DC; 2014.

9. Pan American Health Organization. Compendium of impact & outcome indicators. Washington, DC; 2014. Available from: http://www.paho.org/hq/index.php?option=com_docman&tas=doc_download&gid=23129&Itemid=270&lang=en.
http://www.paho.org/hq/index.php?option=... ^-1010. World Health Organization. The WHO Strategy on Research for Health. Geneva, Switzerland; 2012. Available from: http://www.who.int/phi/WHO_Strategy_on_research_for_health.pdf
http://www.who.int/phi/WHO_Strategy_on_r... This methodology required workshops in Member States to enable indicator measurement and was deemed highly resource-intensive out of keeping with long-term sustainability.

In preparation for the third and final iteration of the Scorecard, we conducted a scan of PAHO's 2014-2015 Compendium of Impact and Outcome Indicators, as well as the Strategic Plan of the Pan American Health Organization 2014-2019 and found nine indicators that met policy objectives that were being routinely assessed through PAHO's Performance Monitoring and Assessment semi-annual assessments and end-of-biennium reviews.⁸8. PAH Organization. Strategic plan of the Pan American Health Organization 2014-2019. Washington, DC; 2014. ⁹9. Pan American Health Organization. Compendium of impact & outcome indicators. Washington, DC; 2014. Available from: http://www.paho.org/hq/index.php?option=com_docman&tas=doc_download&gid=23129&Itemid=270&lang=en.
http://www.paho.org/hq/index.php?option=... A review of other repositories of health policy evaluation indicators such as the European Core Health Indicators was conducted, but no further policy indicators that were relevant for CD49/10 were identified.¹¹11. Public Health Evaluation and Impact Assessment Consortium. Evaluation of the use and impact of the European community health indicators. 2013. Available from: http://ec.europa.eu/health/indicators/docs/echi report v20131031.pdf.
http://ec.europa.eu/health/indicators/do... By extracting existing indicators in the current internal monitoring system utilized by PAHO, the need for a new data collection methodology was eliminated as proxy indicators in the biannual periodic evaluations were already being conducted within the countries. We accessed the data behind the indicators to populate a preliminary Scorecard based on the existing indicators and proposed novel indicators to address gaps in the coverage of the policy. In leveraging routinely collected data, implementation of the Scorecard did thus not require substantial investment of time or personnel.

Scorecard implementation

To assess of the implementation of PAHO's Policy on Research for Health, we designed an initial Scorecard comprising of 29 indicators (see Appendix 1 online). A summary of the major Scorecard components is featured in Table 1. We divided the indicators into six main categories (quality, governance, human resources, partnerships/alliances, standards, dissemination/impact), reflecting the six core objectives of the Policy. For example, the “Governance” policy objective is captured by indicators assessing countries and territories with functional mechanisms for governance of health research, implementation of regional policy on Research for Health, and implementation of regional knowledge management strategy, amongst others.

Of the 45 policy sub-objectives, we were able to find 29 data sources within the Compendium of Impact and Outcome Indicators that mapped onto our indicators. Some policy sub-objectives did not have any applicable indicators available. A total of three new indicators/outputs were therefore proposed to be added into the Compendium to fulfill the core of the Policy assessment in future assessments: delivery of research reports by the PASB, budget allocation to research for health, and monitoring of scholars in health research. Our approach was informed by scanning the literature for implementation of other health research policies. The World Health Organization's Monitoring and Evaluation of Mental Health Policies and Plan advised that a combination of quantitative and qualitative data be employed and recommended use of data that may be already available.¹²12. World Health Organization. Monitoring and evaluation of mental health policies and plan. Geneva; 2007. Available from: http://www.who.int/mentalhealth/policy/services/14-monitoring_evaluation_HKprinter.pdf
http://www.who.int/mentalhealth/policy/s...

The Research Promotion and Development team of PAHO's Office of Knowledge Management, Bioethics, and Research, piloted the Scorecard in 2015. The purpose of this pilot test was to collect data to evaluate each of the proposed indicators based on three parameters: feasibility, relevance, and convenience. We evaluated feasibility in terms of how easily accessible data was collected for specific indicators. Relevance was assessed based on whether the objectives being captured were reflective of the key components of the Policy. Convenience was determined from the ease and time required for data access.

We populated the Scorecard using previously gathered information from the internal PAHO monitoring system from 2014-2015. We calculated adherence to each objective based on whether data was reported or not. A Member State was labelled to be adherent to an objective if at least 50 percent of sub-objectives were met. Based on our initial population of the framework, we generated percentages of completed reporting per member, as well as a heat map outlining adherence rates. The results of the Scorecard will be used by PAHO initially, and may be subsequently published in the future following collection of feedback and associated Scorecard revisions. Once institutionalized and operationalized, data collected biennially based on the selected indicators may provide relevant stakeholders with an assessment of Policy implementation progress, as well as the factors hampering progress in a timely systemic fashion post-policy adoption.

According to the database maintained on Health Research Web, active policies on research for health have been listed in 16 countries and a region: Antigua and Barbuda (2016), Bahamas (2010), Belize (2014), Bermuda (2014), Brazil (2015), Cayman Islands (2012), Chile (2011), Dominican Republic (2014), El Salvador (2017), Jamaica (2015), Mexico(2013), Panama (2016), Paraguay (2016, State Policy), San Kitts and Nevis (2011), Suriname (2011), United States of America (2013) and 18 CARICOM countries. Active agendas on research have been listed by 8 countries and two sub-regions: Chile (2009), Guatemala (2014), Guyana (2013), Honduras (2015), Mexico (2017) Panama (2016), Peru (2016), and Uruguay (2017). The CARICOM countries share a Caribbean Research Agenda (2011), and the COMISCA Countries share the Health Plan for Central America and the Dominican Republic, and have reactivated their Commission on Health Research in 2017.¹³13. Pan American Health Organization. Health Research Web. Available from: https://www.healthresearchweb.org
https://www.healthresearchweb.org... Engagement and commitment in health research is evidently taking place around the region of the Americas, and it is thus important to capture the activity taking place and monitor its development in concordance with Policy on Research for Health. Towards the end of the Scorecard development process, the WHO released a series of indicators and data developed in limited consultation with regional offices as part of the Global Observatory on Health Research and Development.¹⁴14. World Health Organization. Global Health Observatory Data. 2018. Available from: http://www.who.int/gho/en/
http://www.who.int/gho/en... These indicators are valuable additions to the literature, and in collaboration with the WHO, PAHO is exploring ways to promote their use and appropriation by Member States into the Scorecard, in keeping with the proposal of the 46^th Session of the ACHR to use such complementary data for policy evaluations.¹⁵15. Pan American Health Organization. 46th Session of the Advisory Committee on Health Research. Washington, DC; 2017.

Limitations

There were limitations to the development of this Scorecard. Firstly, there is a need for standardization in reporting mannerisms across Member States. Data collection may vary amongst Members, but we proceeded with this approach due to its ability to leverage existing systems and data. Furthermore, some policy sub-objectives are challenging to assess given the difficulty of attaining certain data. We have begun to remedy this by proposing new indicators to add to the Compendium as an initial step, with the intention of eventually building more indicators into the Compendium in order to be able to systematically and routinely collect data from members. Lastly, given the lack of available indicators for all sub-objectives, some areas of policy CD 49/10 are not able to be assessed. We nonetheless believe the development of this preliminary Scorecard is an important step in the assessment of the Policy that will pave the path forward for future systematic and comprehensive policy evaluation and assessment.

Recommendations

Overall, PAHO, Member States, ministries of health, and academic centers will likely benefit from renewing their commitment to the realization of the Policy on Research for Health and reviving their efforts towards bringing about progress in the Americas. Looking to the future, a two-fold approach will be undertaken by the ACHR Secretariat and entities involved in promoting research for health at PAHO. Increased awareness on the Policy on Health for Research and its objectives will be promoted. It is very likely that in a very fast-paced political environment subject to personnel turnover, the policy has fallen off the radar of many PAHO leaders and Member State delegates. Having a continued assessment of progress with the Policy will provide actionable data to guide its implementation. The ACHR Secretariat at PAHO will send out an update on the policy to all governing bodies, as well as suggestions on how to advance and update the policy as the landscape evolves. This Secretariat will also ask country offices to contact actors central to the realization of policy objectives such as academic institutional leaders, health department personnel, as well as science and technology departments to ask them to bolster their initiatives towards strengthening research for health.

Secondly, the entities responsible for promoting and developing research within the Pan American Sanitary Bureau should support country offices to provide details on the indicators within the Scorecard. Increasing the robustness of the Scorecard will include validation of the tool with primary data from the member states. Efforts will be principally targeted towards attaining full reporting across all members, and towards fine-tuning the instruments. Initiatives fulfilling objectives of the Policy may very well be underway in many countries, but without the comprehensive and timely reporting of country offices, PAHO will not be able to assess what areas under its jurisdiction it should continue to support and where significant attention and resources ought to be directed to fulfill the policy mandates. Reinforcement of reporting mandates, consultation with the Governing Bodies over evolving indicators, as well as dedication of appropriate human resources towards the completion of the monitoring and evaluation system is likely to help improve the net quantity and quality of indicator reporting. Uptake from the organizational leadership will be required to champion these endeavors.

In order to institutionalize the Scorecard, support from PAHO's executive administration will be sought out. We will continue to refine the existing indicators and progressively propose the introduction of new indicators that will make the assessment of the policy a corporate product built around the existing administrative tools and systematic across all relevant entities. The Scorecard will be completed on a biennial basis, with milestones documented on a biannual schedule, using data provided by the members. The ACHR will then provide high-level commentary on the results each time the scorecard filled out. The Scorecard will be populated by professionals assessing the Policy implementation within the Secretariat for PASB and the ACHR, and then subsequently circulated for review. Feedback from the ACHR members will be aggregated and then re-directed to the respective members working in coordination with the unit of Program Budget and of Country and Sub-Regional Coordination.

It is important to seek mechanisms to promote harmonization and coordination among WHO regions (e.g. global and regional Advisory Committees on Health Research) that identify strategies seeking the appropriation and participation of Member States and WHO regional offices in the WHO Global Observatory on Research for Health.

Conclusions

The PAHO Department of Knowledge Management, Bioethics and Research designed a practical way to use available institutional indicators as a means of assessing the implementation of the Policy on Research for Health (CD49/10). Our experiences are likely to be relevant to other WHO regional offices and research entities that are dealing with health and engaged in measuring the progress of their own mandates, policies or strategies and likely to encounter similar journeys in striving for feasible policy monitoring and evaluation. The task of monitoring and evaluating the implementation of a policy is a complex phenomenon yet critical to the policy process. There is a delicate balance between comprehensiveness, accuracy, and feasibility throughout the policy evaluation process that is to be striven for.

Acknowledgements

We would like to thank members of the Advisory Committee on Health Research who invaluably participated in the consultation process on the Scorecard. We would like to thank Dr. John Lavis for his guidance and supervision, Ms. Rachel Hutchinson for her contributions to the earlier versions of the Scorecard, as well as Mr. Rony Maza at the Pan American Health Organization for his support with the Compendium of Impact and Outcome Indicators.

Appendix A   Supplementary data

Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.gaceta.2018.04.015.

References

Publication Dates

History