Strengthening health research capacity in low- and middle-income countries (LMIC) is a recognized way to advance health and development. Numerous approaches to strengthening capacity exist at different levels, including training for individuals, improving research systems within institutions, and international collaborations among national health research agencies.¹1 Bates I, Boyd A, Smith H, Cole DC. A practical and systematic approach to organisational capacity strengthening for research in the health sector in Africa. Health Res Policy Syst. 2014;12(1):11. doi: http://dx.doi.org/10.1186/1478-4505-12-11 PMID: 24581148
https://doi.org/10.1186/1478-4505-12-11... Systematic evidence on the effectiveness of different approaches remains limited, as their complexity and diversity make monitoring and evaluation difficult. Assessing returns on investments on research capacity strengthening has been challenging, since many funders have given low priority to rigorous monitoring and evaluation.

In 2008 funding agencies came together as the ESSENCE on Health Research initiative. This group realized that there were too many different forms of monitoring and evaluation for health research capacity strengthening. It wanted to reduce the administrative burden on funding recipients, to enable learning and improvement among stakeholders and to demonstrate the impact of research capacity strengthening projects. In 2011, ESSENCE published a Planning, Monitoring and Evaluation Framework as a guide for its member funding agencies and grantees. The framework encouraged the sharing of lessons about evaluations of these projects.

We were interested in understanding different approaches to health research capacity strengthening and analysed evaluation reports of health interventions funded by ESSENCE members and other agencies. This perspective is based on analyses of the funders’ reports.²2 Bates I, Boyd A, Aslanyan G, Cole DC. Tackling the tensions in evaluating capacity strengthening for health research in low- and middle-income countries. Health Policy Plan. 2014;2014:1–11. doi: http://dx.doi.org/10.1093/heapol/czu016 PMID: 24717984
https://doi.org/10.1093/heapol/czu016... ^–44 Cole DC, Boyd A, Aslanyan G, Bates I. Indicators for tracking programmes to strengthen health research capacity in lower- and middle-income countries: a qualitative synthesis. Health Res Policy Syst. 2014;12(1):17. doi: http://dx.doi.org/10.1186/1478-4505-12-17 PMID: 24725961
https://doi.org/10.1186/1478-4505-12-17...

Addressing tensions

We found that a critical tension facing funders and policy-makers is the degree to which recipients of funding should be involved in the evaluation of their own projects. Some funders perceive that an externally-led evaluation better demonstrates accountability and value for money. Funders noted that some recipients had limited involvement in – or capacity for – evaluations, lacking critical skills such as the ability to set testable goals and measurable targets.

However, when recipients participate in evaluations from conception to completion, they have a greater sense of ownership over the project and are more likely to learn from their work. They are also able to make ongoing improvements to the project in the context of developmental or empowerment-type evaluations and can subsequently implement recommendations leading to long-term change. Compared to external evaluators, recipients usually have greater in-depth knowledge about the project, the context and other stakeholders. This knowledge is vital for solving problems. Recipients who are involved in the evaluation can help the rest of the team detect and correct problems early on, communicate decisions and act on results. Engaging other stakeholders – such as service users, community members, health practitioners and policy-makers – is helpful for setting realistic goals, meeting local priorities and addressing resource issues. This requires extensive participation and hence more resources.

We recommend that funders and recipients jointly agree on the purpose and process of the evaluation, and help stakeholders to fulfil their roles in evaluations of research capacity strengthening. Among their portfolios of projects and programmes, funders can up-front commission evaluations, participate in them and subsequently use evaluation results. Funding recipients can design monitoring and evaluation processes and generate data. Evaluation teams including external evaluators and recipients are ideally composed of people with technical, organisational and interpersonal skills. Their understanding of the complexity of research capacity strengthening can enrich the evaluations they undertake.²2 Bates I, Boyd A, Aslanyan G, Cole DC. Tackling the tensions in evaluating capacity strengthening for health research in low- and middle-income countries. Health Policy Plan. 2014;2014:1–11. doi: http://dx.doi.org/10.1093/heapol/czu016 PMID: 24717984
https://doi.org/10.1093/heapol/czu016...

Theory-based frameworks

Capacity strengthening is a complex process with a long intervention pathway. Applying a theory of change can help to define the pathway and inform the evaluation.⁵5 Bertone MP, Meessen B, Clarysse G, Hercot D, Kelley A, Kafando Y, et al. Assessing communities of practice in health policy: a conceptual framework as a first step towards empirical research. Health Res Policy Syst. 2013;11(1):39. doi: http://dx.doi.org/10.1186/1478-4505-11-39 PMID: 24139662
https://doi.org/10.1186/1478-4505-11-39... It can help to identify impact trajectories, strengthen evaluation rigour, foster assessment of generalizability to other contexts, and guide influence on policy and practice.

Theory-driven evaluative thinking can lead to more useful evaluations, as stakeholders try to identify underlying assumptions and rationale for their actions. However, most evaluations do not have the time and resources to incorporate theory-informed indicators of impact and sustainability, or to collect data against these indicators. This means missed opportunities to enhance knowledge and learning among funders and funding recipients about how to improve planning, monitoring and evaluation of research capacity initiatives.

A key recommendation from our research is for projects to develop comprehensive, prospective systems for research capacity evaluation. Funders and policy-makers could develop supporting guidance, tools or training for such evaluations and allocate adequate funding to evaluations.³3 Boyd A, Cole DC, Cho DB, Aslanyan G, Bates I. Frameworks for evaluating health research capacity strengthening: a qualitative study. Health Res Policy Syst. 2013;11(1):46. doi: http://dx.doi.org/10.1186/1478-4505-11-46 PMID: 24330628
https://doi.org/10.1186/1478-4505-11-46...

Indicators

A systematic analysis of assumptions, preconditions and measurement options associated with an evaluation can make monitoring processes easier. Although a few common measurable, reliable indicators may be feasible, most projects will need some indicators tailored to the context and the project objectives. Baseline measurement of selected indicators would enable a prospective, rather than the much less rigorous, retrospective approach.

It is important to maintain flexibility and revisit indicators as the research capacity project proceeds.¹1 Bates I, Boyd A, Smith H, Cole DC. A practical and systematic approach to organisational capacity strengthening for research in the health sector in Africa. Health Res Policy Syst. 2014;12(1):11. doi: http://dx.doi.org/10.1186/1478-4505-12-11 PMID: 24581148
https://doi.org/10.1186/1478-4505-12-11... Evaluations should distinguish between the provincial-national research environment, the international-global environment and research networks. This would help facilitate greater clarity of relationships in pathways of change and consistency in cross-case comparisons.⁴4 Cole DC, Boyd A, Aslanyan G, Bates I. Indicators for tracking programmes to strengthen health research capacity in lower- and middle-income countries: a qualitative synthesis. Health Res Policy Syst. 2014;12(1):17. doi: http://dx.doi.org/10.1186/1478-4505-12-17 PMID: 24725961
https://doi.org/10.1186/1478-4505-12-17...

Evaluation as a learning tool

There are different elements to evaluation as a learning tool. Broader lessons can be shared among the health research capacity community (Box 1). Skills learnt by individuals during evaluation of projects can enhance their knowledge and ability to plan, fund, or implement better research capacity strengthening programmes.

Donor governments often encourage research capacity strengthening initiatives to garner additional resources, so that the initiatives become self-sustaining. Developmental or empowerment evaluations can contribute to sustainability through the learning achieved. Evaluators are well positioned to facilitate discussions among stakeholders and promote empowerment by enhancing stakeholders’ skills to address tensions, negotiate approaches to problems and come to decisions. Funders could also support the development of a community of practice to share lessons and experiences of research capacity strengthening and to consider joint ways forward.⁵5 Bertone MP, Meessen B, Clarysse G, Hercot D, Kelley A, Kafando Y, et al. Assessing communities of practice in health policy: a conceptual framework as a first step towards empirical research. Health Res Policy Syst. 2013;11(1):39. doi: http://dx.doi.org/10.1186/1478-4505-11-39 PMID: 24139662
https://doi.org/10.1186/1478-4505-11-39...

Conclusion

Funders and policy-makers aiming to harmonize evaluation approaches for health research capacity strengthening initiatives must successfully manage underlying tensions to move forward. These include the degree of stakeholder participation, the right balance of quantitative and qualitative data, the promotion of learning while gathering information and an emphasis on long-term, as well as short-term, gains. A deeper analysis of health research capacity strengthening projects, using consistent and multiple methods would enable learning to be shared and transferred. It would also relieve funding recipients of the burden of multiple reporting, consistent with aid effectiveness principles, and potentially enable funders to better demonstrate impact and value for money.

Competing interests:

References

Publication Dates

History