POLICY & PRACTICE

Increasing access to health workers in underserved areas: a conceptual framework for measuring results

Améliorer l'accès au personnel de santé dans les zones mal desservies : cadre conceptuel pour la mesure des résultats

Ampliar el acceso a los trabajadores sanitarios en las zonas subatendidas: marco conceptual para medir los resultados

Luis Huicho^I; Marjolein Dieleman^II; James Campbell^III; Laurence Codjia^IV; Dina Balabanova^V; Gilles Dussault^VI; Carmen Dolea^VII,*

^IDepartment of Paediatrics, Universidad Peruana Cayetano Heredia, Lima, Peru
^IIKIT Development, Policy and Practice, Royal Tropical Institute, Amsterdam, Netherlands
^IIIInstituto de Cooperación Social, Integrare, Barcelona, Spain
^IVGlobal Health Workforce Alliance, Geneva, Switzerland
^VLondon School of Hygiene and Tropical Medicine, London, England
^VIIntituto de Higiene e Medecina Tropical, Universidade Nova de Lisboa, Lisboa, Portugal
^VIIDepartment of Human Resources for Health, World Health Organization, 20 avenue Appia, 1211 Geneva 27, Switzerland

ABSTRACT

Many countries have developed strategies to attract and retain qualified health workers in underserved areas, but there is only scarce and weak evidence on their successes or failures. It is difficult to compare lessons and measure results from the few evaluations that are available. Evaluation faces several challenges, including the heterogeneity of the terminology, the complexity of the interventions, the difficulty of assessing the influence of contextual factors, the lack of baseline information, and the need for multimethod and multi-disciplinary approaches for monitoring and evaluation. Moreover, the social, political and economic context in which interventions are designed and implemented is rarely considered in monitoring and evaluating interventions for human resources for health. This paper proposes a conceptual framework that offers a model for monitoring and evaluation of retention interventions taking into account such challenges. The conceptual framework is based on a systems approach and aims to guide the thinking in evaluating an intervention to increase access to health workers in underserved areas, from its design phase through to its results. It also aims to guide the monitoring of interventions through the routine collection of a set of indicators, applicable to the specific context. It suggests that a comprehensive approach needs to be used for the design, implementation, monitoring, evaluation and review of the interventions. The framework is not intended to be prescriptive and can be applied flexibly to each country context. It promotes the use of a common understanding on how attraction and retention interventions work, using a systems perspective.

RÉSUMÉ

De nombreux pays ont mis au point des stratégies pour attirer et retenir le personnel médical qualifié dans les zones mal desservies, mais les preuves du succès ou de l'échec de ces stratégies sont rares et faibles. Il est difficile de comparer les enseignements tirés et de mesurer les résultats à partir des quelques évaluations disponibles. L'évaluation de ces stratégies se heurte en effet à plusieurs difficultés, dont l'hétérogénéité de la terminologie, la complexité des interventions, la difficulté d'évaluer l'influence des facteurs contextuels, le manque de données de référence et la nécessité de démarches empruntant à plusieurs méthodologies et à plusieurs disciplines pour le suivi et l'évaluation. En outre, le contexte social, politique et économique dans lequel les interventions sont conçues et mises en œuvre est rarement pris en compte dans le suivi et l'évaluation des interventions en faveur des ressources humaines pour la santé. Le présent article présente un cadre conceptuel proposant un modèle pour le suivi et l'évaluation des interventions visant à retenir la main d'œuvre qui tient compte de ces difficultés. Ce cadre conceptuel repose sur une démarche systématique et a pour objectif de guider la réflexion dans l'évaluation d'une intervention pour améliorer l'accès au personnel de santé dans des zones mal desservies, depuis sa phase de conception jusqu'à ses résultats. Il vise aussi à guider le suivi des interventions par le recueil systématique d'une série d'indicateurs, applicable au contexte considéré. Il introduit l'idée qu'une approche globale est nécessaire pour la conception, la mise en œuvre, le suivi, l'évaluation et le bilan des interventions. Il n'est pas destiné à être prescriptif et s'applique de manière flexible à chaque contexte national. Il favorise l'utilisation d'une interprétation commune des modalités d'action des interventions pour attirer et retenir la main-d'œuvre, en recourant à une perspective systématique.

RESUMEN

Muchos países han desarrollado estrategias para atraer y conservar a personal sanitario cualificado en zonas insuficientemente atendidas, pero la evidencia sobre el éxito o fracaso de esas iniciativas es escasa y poco robusta. Es difícil comparar las enseñanzas extraídas y cuantificar los resultados de las escasas evaluaciones disponibles. Las medidas de evaluación tropiezan con varios problemas, en particular la heterogeneidad de la terminología, la complejidad de las intervenciones, las dificultades para evaluar la influencia de factores contextuales, la falta de información basal, y la necesidad de aplicar enfoques multimetódicos y multidisciplinarios en la vigilancia y la evaluación. Por otra parte, en la vigilancia y evaluación de las intervenciones relacionadas con los recursos humanos para la salud rara vez se tiene en cuenta el contexto social, político y económico en el que se diseñan y aplican esas intervenciones. En este artículo se propone un marco conceptual que brinda un modelo de vigilancia y evaluación de las intervenciones de promoción de la permanencia que sí tiene en cuenta esos factores. Dicho marco está basado en un enfoque sistémico y pretende servir de orientación a la hora de evaluar una intervención de ampliación del acceso al personal sanitario en las zonas subatendidas, desde su fase de diseño hasta sus resultados. También tiene por objeto orientar la vigilancia de las intervenciones mediante la recopilación sistemática de un conjunto de indicadores, aplicables al contexto específico. Se sugiere que es necesario aplicar un enfoque omnicomprensivo al diseño, ejecución, vigilancia, evaluación y examen de las intervenciones. El marco no pretende ser prescriptivo y se puede aplicar con flexibilidad en el contexto de cada país. Promueve el uso de una perspectiva común sobre el funcionamiento de las intervenciones de fomento de la captación y la permanencia, aplicando un enfoque sistémico.

Introduction

Health workers' willingness to practise in underserved areas, such as rural, remote or poor areas, is a recognized challenge in achieving equitable access to health services. Many countries have developed strategies to attract and retain qualified health workers in these areas. But the evidence on the successes or failures of such interventions is scarce and weak, and it is difficult to compare lessons and measure results from the few evaluations that are available.^{1 - 3}

There has been significant progress in generating common understanding and debate on ways to evaluate the impact of development interventions, and on using consistent monitoring and evaluation terminology, such as "outputs", "outcomes" and "impact".^4-6 However, this has yet to be applied to the evaluation of human resources for health interventions and, specifically, to those seeking to increase access to health workers in underserved areas. There is an urgent need to achieve an agreement and strengthen the evidence that would underpin sound policy recommendations in this area.

Seeking to address this gap, this paper proposes a conceptual framework to guide managers, policy-makers and evaluators in the assessment of interventions to increase access to health workers in underserved areas. The framework aims to support all stages of policy development. It suggests a logical sequence that can be followed when deciding on any intervention that addresses attraction and retention issues and assists in formulating key questions to be answered when designing, monitoring and evaluating such interventions. It includes examples of indicators to inform the process, which can be adapted to a specific context. A comprehensive consideration of indicators to monitor the health workforce in general is offered elsewhere.⁷

Challenges in evaluation

Policy- and decision-makers need to know whether interventions work or not, why they work and in which context. Therefore it is important to have information about the effects of the interventions, but also about the factors that made the intervention succeed or fail (the questions dealing with "when, why, how, and in what circumstances such interventions work well or fail to work"). ⁸

Evaluating interventions to improve human resources for health is complex for different reasons. First, relating these interventions to health status is very difficult due to broad socioeconomic, cultural, political and health systems factors that influence health. For instance, although improved health outcomes, such as reduction in maternal mortality, are directly correlated with increased availability of health workers, it is difficult to attribute the improvement directly to a certain health workforce intervention. ^{9 , 10} Illustrative country case studies showed that Afghanistan and Ethiopia have implemented comprehensive health sector strategies with multiple co-existing interventions, like the recruitment of community health workers, which may have promoted access to life-saving services, thus improving health status. ¹¹

Another problem is that many evaluations lack a baseline against which to assess the results, particularly in countries with a major health worker deficit,^3,10,12 as well as a specific intervention logic that clarifies the expectations of the intervention designers (S Kane, B Gerretsen, R Scherpbier, M Dal Poz, MA Dieleman, unpublished data, 2009). ¹³ Moreover, the social, political and economic context in which interventions are designed and implemented is rarely considered in monitoring and evaluation of human resource interventions.^14,15

Thus, the main challenges which evaluators face are related to the multidimensional nature of interventions, and the difficulty of assessing the influence of contextual factors. To be able to assess such interventions in complex systems, there is a need for multimethod and multidisciplinary monitoring and evaluation approaches, inclusive of all relevant stakeholders. ¹⁶ Another challenge is generatinglessons across regions and countries to inform the global health agenda and promote a meaningful dialogue at both national and international level. There is clear need for a framework that offers a model for monitoring and evaluation of health worker retention interventions taking into account such challenges.

Conceptual framework

The proposed framework is based on a systems approach and differentiates between "inputs", "outputs", "outcomes" and"impact", with regards to the results of interventions to attract and retain health workers in underserved areas. It builds on the proposed common framework for monitoring performance and evaluating progress in the scale-up for better health,¹⁷ and proposes indicators to measure progress in implementing various strategies, allowing users to determine what works or not, and to explore contextual factors influencing their success or failure. The framework has two aims: (i) to guide the thinking in evaluating an intervention to increase access to health workers in underserved areas, from its inception/design phase through to its results, by suggesting key questions about the relevance, efficacy, efficiency, effectiveness and sustainability of the intervention; and (ii) to guide the monitoring of interventions, through a focus on a routine collection of a set of indicators, applicable to the specific context.

We used the following definitions for evaluation and monitoring:⁴

We define "evaluation" as the systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability.

We define "monitoring" as a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds.

Later in this paper, we examine the application of the framework to the stages of policy development: design, implementation, monitoring and evaluation (Fig. 1). The sequence in design and implementation phases does not necessarily imply a linear policy process, as monitoring and evaluation may help identify a need for new interventions. They are, in effect, part of an ongoing policy cycle.

Design

The evaluation of the design stage should identify the extent to which the chosen intervention was relevant and adequate to the need, the expectations of the population and the health system context, and the reasons for selecting a particular option. ¹⁸ By relevance, we mean the extent to which the objectives and the elements of the intervention are consistent with beneficiaries' requirements, country needs, global priorities and partners' and donors' policies. ⁴ The relevance issue can and should also be addressed retrospectively; and then the question is whether the objectives of an intervention or its design were and are still appropriate given changed circumstances. ⁴

Questions that an evaluator should ask are: did the intervention respond to a documented need, was it appropriate for the given context and how was it selected? Is the choice of the intervention based on evidence or at least on robust arguments?

The assessment of the situation analysis should check if the problem of limited access to health workers has been clearly described. Is it due to an insufficient number of students willing to work in underserved areas, or an insufficient production of health workers, or were there other reasons? The evaluation needs to assess if a health labour market analysis has been conducted, to identify whether the lack of access to health workers in underserved areas was a matter of insufficient demand, insufficient supply, or insufficient remuneration and incentives, and unattractive working conditions. ¹⁹

The situation analysis should also identify whether the elements in place to support the implementation of the intervention were assessed and if a feasibility analysis preceded the choice of the intervention. Strategies to increase access to health workers in underserved areas require participation of stakeholders from different sectors. Therefore, questions to be asked include: were mechanisms for multi-stakeholder engagement in place and used, were regulatory frameworks analysed before the intervention, were the intervention costs calculated and considered affordable (availability of funds), was the organizational capacity to support its implementation analysed? At this stage, the Human Resources for Health Action Framework is a recognized tool that can be used to guide the situation analysis (Fig. 2). ²¹ It comprises health system dimensions such as policy, finance, education, partnership, leadership and human resources management systems.

At the end of this phase, a strategy can be proposed to address the perceived need of lack of access to health workers in underserved areas and of maldistribution of health workers. Such a strategy can be in one or more of the following thematic areas: education, regulatory, financial, and/or management and support, described elsewhere. ³

Implementation

When the implementation of an intervention is evaluated, questions might include: did the activities take place according to plan, did any changes occur in implementation capacity or in the context that influenced the implementation of the intervention components? Did any changes occur regarding the involvement and commitment of stakeholders? Which ones and why? Was any action undertaken to adapt the intervention to the changes? What was done and why? Or why was no action taken? Monitoring the progress in implementation can be informed by the systematic collection and analysis of a set of key indicators, which are further discussed in this paper.

Results

Intervention results can be evaluated at different levels (Fig. 1): outputs, intermediate outcomes, outcomes and impact. Questions to be asked include: did the intervention achieve its intended results at the level of outputs, outcomes, or impact? Is the intervention sustainable? What were the contextual factors (political, economic, social, or organizational) that have influenced the results?

The level of "outputs" is the direct effect, attributable to the intervention, such as increased attractiveness of the areas/or the profession, improved recruitment and deployment, and increased retention of health workers in those areas.

The level of intermediate "outcomes" is that of improved health workforce performance, the dimensions of which have been described in The world health report 2006 as "availability, competence, responsiveness and productivity".¹⁰ Some attraction and retention interventions can have a direct effect on these dimensions too. For example, changes in rural curricula can improve competencies to address rural health problems and performance-based financial incentives can improve productivity. However, often improved health workforce performance cannot be attributed to retention interventions alone, as there are many other factors that contribute, such as education strategies or health sector management reforms.

Finally, the level of "impact" is understood as the effect on health outcomes and an improved health status, and other unexpected effects on underserved communities (e.g. attraction/retention of other professionals or services). This cannot be directly attributed to retention interventions, as the simple availability of health workers cannot directly improve health outcomes in the absence of effective components such as drug supply, functioning facilities and good clinical practices. ⁹ But a well performing health workforce is a significant component of a strong health system that, together with other social determinants, can lead to improved health status. ²²

Evaluations of any attraction and retention intervention need to consider the contextual factors that may explain why and how such interventions have achieved the intended outputs, outcomes and impact, i.e. the social, economic and political context. This includes a stakeholder analysis including their respective power bases and interests, as well as an analysis of the capacity of the health sector to implement. Evaluations should use both qualitative and quantitative methods to measure outputs and impacts, but also to analyse formal and informal policy processes accompanying the design and implementation of interventions. Ideally, to find out if an attraction and retention intervention works or not, a cluster randomized controlled trial with an adequate control group would be needed, but this may not be feasible or ethical in the "real life" policy scenario. Moreover, causal chains are often complex in public health interventions, making results of randomized controlled trials subject to effect modification in different populations. ²⁰ Alternative observational methods such as before-after and plausibility designs can also be considered, taking care to document and control the contextual factors appropriately.^20,23 Studies with adequacy and plausibility designs can improve both the internal and external validity of randomized controlled trials, and plausibility studies are often the only feasible option for getting valid evidence of impact when attempting the evaluation of large-scale interventions. ²³

Monitoring

The framework depicted in Fig. 1 proposes a tool to guide the monitoring of interventions to increase access to health workers in underserved areas. At each stage of the implementation process, indicators are proposed to assess whether the implementation is on track.

Indicators to monitoring at the design stage should be those that will allow for a detailed description of the stocks and flows of health workers across sectors, age and sex categories, geographical areas and professions. Detailed description of these indicators, as well as data sources for health workforce monitoring, are described elsewhere. ⁷

When monitoring the implementation of interventions, systematic data collection is required on the planned activities, by constantly reviewing the production capacity, as well as the organization and management capacity. Monitoring implementation also requires tracking changes in context and stakeholders, so as to be able to adapt the interventions when required.

Regarding the monitoring of results of the intervention, at the level of "outputs" it is important to monitor whether the dimensions of "attractiveness", "engagement/recruitment", "deployment"and "retention" are achieved. Indicators to monitor progress on these dimensions can include, for example, changes in the intentions of students or health workers to relocate, reside in or leave underserved areas (attractiveness). The available funded positions in underserved areas can be an indicator of effective engagement, whereas the number of health workers recruited and changes in vacancy rates can be indicators of effective deployment and recruitment. Finally, the effects of an intervention on retention can be measured by duration in post, turnover, absenteeism, stability index or even survival rates.^24,25

At the level of "outcomes", it is proposed to monitor the progress on improved health workforce performance and health systems performance, through dimensions and indicators proposed by The world health report 2006 (Table 1). ¹⁰ As said before, assessing the longer-term impact of human resource interventions on health outcomes should be done within the broader context of factors influencing health status.

The effectiveness of attraction and retention interventions can only be measured at the level of outputs and, to a certain extent, at the level of outcomes. In addition, implementing human resource retention interventions can take many years before any results can be seen, for example in the case of creating new schools in rural areas or rolling out financial incentive schemes. Strong human resource information systems, linked to national health information systems, as well as clearly defined and agreed metrics or indicators are required to monitor the progress of such interventions in terms of increased attractiveness, improved recruitment and duration of stay, increased availability, improved competence, productivity and responsiveness. Table 2 makes an attempt towards advancing a set of key questions and indicators to be used when evaluating and monitoring retention interventions for human resources for health.

The conceptual framework presented here needs to be extensively piloted and will be refined through further inputs from different sources. There are ongoing plans for applying it in several countries that are in the process of designing and implementing attraction and retention strategies in underserved areas. Efforts will also be made to make sure the evaluation tool is user-friendly. For the sake of argument, a hypothetical example of its potential use is presented in Box 1.

The way forward

This framework proposes a common approach to facilitate the evaluation and monitoring of interventions to increase access to health workers in underserved areas. It suggests that a comprehensive approach needs to be used for the design, implementation, monitoring, evaluation and review of such interventions. The framework is not intended to be prescriptive and can be applied flexibly to each country context. It is hoped that it will promote the use of a common understanding/logic on how attraction and retention interventions work, using a systems perspective. It starts with a common set of indicators, which enable comparison between various cases, and facilitates reviews of published studies. As such it can be used to monitor and evaluate interventions, either using a method-based or a theory-based evaluation approach with a specific set of indicators.

Funding: This work was funded by the World Health Organization, Health Systems and Services, Department of Human Resources for Health, Health Workforce Migration and Retention Unit, Geneva, Switzerland. Additional support was received from the WHO Collaborating Centre on Human Resources for Health at the Royal Tropical Institute (KIT),The Netherlands.

Competing interests: None declared.

References

1. Grobler L, Marais BJ, Mabunda SA, Marindi PN, Reuter H, Volmink J. Interventions for increasing the proportion of health professionals practising in rural and other underserved areas. Cochrane Database Syst Rev 2009;1:CD005314. PMID: 19160251         

2. Wilson NW, Couper ID, De Vries E, Reid S, Fish T, Marais BJ. A critical review of interventions to redress the inequitable distribution of healthcare professionals to rural and remote areas. Rural Remote Health 2009;9:1060. PMID: 19530891         

3. Dolea C, Stormont L, Braichet J-M. Evaluated strategies to increase attraction and retention of health workers in remote and rural areas. Bull World Health Organ 2010;88:379-385.         

4. Development Assistance Committee. Glossary of key terms in evaluation and results-based management. Paris: Organisation for Economic Co-operation and Development; 2002. Available from: http://www.oecd.org/dataoecd/29/21/2754804.pdf [accessed on 15 February 2010]          .

5. Leeuw F, Vaessen J. Impact evaluations and development: NONIE guidance on impact evaluation. Washington, DC: The Network of Networks on Impact Evaluation; 2009. Available from: http://siteresources.worldbank.org/EXTOED/Resources/nonie_guidance.pdf [accessed 15 February 2010]          .

6. White H. Theory-based impact evaluation: principles and practice. New Delhi: International Initiative for Impact Evaluation (3ie); 2009.         

7. Dal Poz M, Gupta N, Quain E, Soucat ALB. Handbook on monitoring and evaluation of human resources for health - with special applications for low- and middle-income countries. Geneva: World Health Organization; 2009.         

8. Mills A, Gilson L, Hanson K, Palmer N, Lagarde M. What do we mean by rigorous health-systems research? Lancet 2008;372:1527-9. doi:10.1016/S0140-6736(08)61633-5 PMID:18984174        

9. Anand S, Bärnighausen T. Human resources and health outcomes: cross-country econometric study. Lancet 2004;364:1603-9. doi:10.1016/S0140 6736(04)17313-3 PMID:15519630        

10. The world health report 2006: working together for health. Geneva: World Health Organization; 2006.         

11. Peters DH, El-Saharty S, Siadat B, Janovsky K, Vujicic M, editors. Improving health service delivery in developing countries: from evidence to action. Washington, DC: The World Bank; 2009.         

12. Terms of reference for an independent evaluation of the impact of Malawi's Emergency Human Resources Programme. Lilongwe: Department for International Development (UK); 2009.         

13. Dieleman M, Gerretsen B, van der Wilt GJ. Human resource management interventions to improve health workers' performance in low and middle income countries: a realist review. Health Res Policy Syst 2009;7:7. doi: 10.1186/1478-4505-7-7 PMID:19374734        

14. Dieleman MA, Kane S, Zwanikken P, Gerretsen B. Identifying patterns in retention intervention studies: revisiting the evidence from a realist perspective. Geneva: World Health Organization; 2009.         

15. Fogarty L, de Vries D, Reavely E, Bunch E. Indicators for evaluating human resources for health capacity-building. In: Evaluation 2009: context and evaluation, American Evaluation Association, Orlando, 11-14 November 2009.         

16. Bristow G, Farrington J, Shaw J, Richardson T. Developing an evaluation framework for crosscutting policy goals: the Accessibility Policy Assessment Tool. Environ Plan A 2009;41:48-62. doi:10.1068/a4092        

17. Monitoring and Evaluation Working Group of the International Health Partnership and related initiatives (IHP+). Monitoring performance and evaluating progress in the scale-up for better health: a proposed common framework. Geneva: World Health Organization; 2009. Available from: http://www.internationalhealthpartnership.net//CMS_files/documents/a_proposed_common_framework_EN.pdf [accessed 15 February 2010]          .

18. Codjia L, Jabot F, Dubois H. Evaluation du programme d'appui à la médicalisation des aires de santé rurales au Mali. Geneva: World Health Organization; 2008. In French.         

19. Lemiere C, Herbst C, Jahanshani N, Smith A, Soucat A. Reducing geographical imbalances of health worker distribution: what works, what does not and why? A labour market perspective of urban-rural health disparities, policy solution and experiences in sub-Saharan Africa. Washington, DC: The World Bank; 2009.         

20. Habicht J-P, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol 1999;28:10-8. doi:10.1093/ije/28.1.10 PMID:10195658        

21. Human Resources for Health (HRH) Action Framework [internet site]. Available from: http://www.capacityproject.org/framework/ [accessed 15 February 2010]          .

22. Everybody's business: strengthening health systems to improve health outcomes. Geneva: World Health Organization; 2007. Available from: http://www.who.int/healthsystems/strategy/everybodys_business.pdf [accessed 15 February 2010]          .

23. Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health 2004;94:400-5. doi:10.2105/AJPH.94.3.400 PMID:14998803        

24. Buchan J. The benefits of AHP workforce stability. Edinburgh: Scottish Executive Health Department; 2005.         

25. Rabinowitz HK, Diamond JJ, Markham FW, Rabinowitz C. Long-term retention of graduates from a program to increase the supply of rural family physicians. Acad Med 2005;80:728-32. doi:10.1097/00001888-200508000-00004 PMID:16043525        

(Submitted: 5 October 2009 - Revised version received: 19 November 2009 - Accepted: 5 January 2010 )

* Correspondence to Carmen Dolea (e-mail: doleac@who.int).