POLICY & PRACTICE
Evaluated strategies to increase attraction and retention of health workers in remote and rural areas
Évaluation de stratégies pour attirer davantage et retenir le personnel médical dans les zones rurale ou reculées
Evaluación de estrategias de aumento de la captación y permanencia de trabajadores sanitarios en zonas remotas y rurales
Carmen Dolea*; Laura Stormont; Jean-Marc Braichet
Department of Human Resources for Health, World Health Organization, 20 avenue Appia, 1211 Geneva 27, Switzerland
ABSTRACT
The lack of health workers in remote and rural areas is a worldwide concern. Many countries have proposed and implemented interventions to address this issue, but very little is known about the effectiveness of such interventions and their sustainability in the long run. This paper provides an analysis of the effectiveness of interventions to attract and retain health workers in remote and rural areas from an impact evaluation perspective. It reports on a literature review of studies that have conducted evaluations of such interventions. It presents a synthesis of the indicators and methods used to measure the effects of rural retention interventions against several policy dimensions such as: attractiveness of rural or remote areas, deployment/recruitment, retention, and health workforce and health systems performance. It also discusses the quality of the current evidence on evaluation studies and emphasizes the need for more thorough evaluations to support policy-makers in developing, implementing and evaluating effective interventions to increase availability of health workers in underserved areas and ultimately contribute to reaching the United Nations' Millennium Development Goals.
RÉSUMÉ
Le manque de personnel de santé dans les zones rurales ou reculées est un problème mondial. De nombreux pays ont proposé et mis en œuvre des interventions pour y faire face, mais on sait peu de chose de l'efficacité et de la durabilité à long terme de ces interventions. Le présent article analyse l'efficacité des interventions pour attirer et retenir le personnel médical dans les régions rurales ou reculées à travers une évaluation de leur impact. Il indique, d'après une revue de la littérature, les études ayant donné lieu à une évaluation de ces interventions. Il présente une synthèse des indicateurs et des méthodes employés pour mesurer les effets des interventions en faveur de la rétention des agents en milieu rural selon plusieurs dimensions politiques telles que : l'attractivité des zones rurales ou reculées, le déploiement/recrutement, le maintien et les performances de la main-d'œuvre médicale, ainsi que les performances du système de santé. Il examine également la qualité des preuves actuellement disponibles à partir des études d'évaluation et souligne la nécessité d'évaluations plus approfondies pour aider les décideurs politiques à développer, mettre en œuvre et évaluer des interventions permettant d'accroître efficacement la disponibilité du personnel médical dans les zones mal desservies et en fin de compte de contribuer à la réalisation des objectifs du Millénaire pour le développement des Nations Unies.
RESUMEN
La falta de personal sanitario en zonas remotas y rurales es motivo de preocupación en todo el mundo. Muchos países han propuesto y emprendido intervenciones para abordar este problema, pero es muy poco lo que se sabe sobre la efectividad de esas intervenciones y su sostenibilidad a largo plazo. Este artículo ofrece un análisis de la eficacia de las intervenciones destinadas a atraer y fidelizar a los trabajadores sanitarios en las zonas remotas y rurales, aplicando para ello una perspectiva de evaluación del impacto. Se informa de una revisión bibliográfica de los estudios en que se han realizado evaluaciones de ese tipo de intervenciones. Se presenta una síntesis de los indicadores y los métodos utilizados para medir los efectos de las intervenciones de fidelización rural relacionándolos con varias dimensiones de las políticas aplicadas, como por ejemplo el atractivo de las zonas rurales o remotas, la distribución y contratación, la permanencia, y el desempeño de la fuerza laboral sanitaria y de los sistemas de salud. También se analiza la calidad de la evidencia disponible sobre los estudios de evaluación y se hace hincapié en la necesidad de realizar evaluaciones más detalladas para ayudar a los formuladores de políticas a desarrollar, implementar y evaluar intervenciones eficaces orientadas a aumentar la disponibilidad de personal sanitario en zonas subatendidas y contribuir a la larga a que se alcancen los Objetivos de Desarrollo del Milenio de las Naciones Unidas.
Introduction
The availability of well trained and motivated health workers in underserved areas will improve access to essential health services to achieve the health-related United Nations' Millennium Development Goals within the framework of a primary health care renewal.1-3 Yet there are stark imbalances in the geographical distribution of health workers, both in developed and developing countries. Approximately one half of the world's population lives in rural areas but these areas are served by only 38% of the total nursing workforce and by less than 25% of the total physicians' workforce.2 At the country level, imbalances in the distribution of health workers are even more prominent.4, 5
In recent years, there has been increased interest from both researchers and policy-makers to identify and implement effective solutions to address the shortages of health workers in remote and rural areas.6-11 In response to this increased interest and perceived need, the World Health Organization has recently launched a programme of work on "Increasing access to health workers in remote and rural areas through improved retention." The programme aims to expand the knowledge base in this domain and to provide evidence-based global recommendations to address this problem, while at the same time to provide technical cooperation to countries that need to address this problem.12
As part of this programme of work, this paper builds on and expands earlier work on assessing the evidence on effectiveness of interventions to increase access to health workers in rural and remote areas.13 This paper expands the original search to focus mainly on studies that evaluated such interventions, and attempts to analyse the impact of such interventions on certain dimensions of health workforce and health systems performance. It also discusses the quality of the evidence from evaluation studies and identifies the evidence gaps in this domain. It is expected that evaluations would give policy-makers additional information with regards to the effectiveness and applicability of various interventions in their own context.14
Conceptual framework
The analysis in this paper is based on the assumption that the final result of having health workers in remote and rural areas depends on two inter-linked aspects8: (i) the factors that influence the decision or choice of health workers to relocate to, stay in or leave those areas, and (ii) the extent to which health system policies and interventions respond to these factors. These responses are usually grouped into four main categories: education, regulatory, financial and personal and professional support interventions.6-10
There is a wealth of descriptive literature highlighting the extent of geographical imbalances and deficits in health personnel in rural and remote areas,4, 5 or reporting on the factors that influence health workers' preferences or choices with regards to practicing in remote and rural areas.15, 16There are also studies that describe or recommend potential interventions, without analysing the effects of these interventions.17-19 However, there is very limited research on comprehensive evaluations of specific retention strategies.We set out to conduct a review of this type of study with the aim of further informing the methodology in conducting evaluations of rural health workforce retention strategies.
Methods
We conducted an extensive review of the literature that reported on evaluations of interventions to increase the availability of health workers in remote and rural areas. Electronic searches were conducted in August and September 2009 in PubMed, the Cochrane database, Embase® and LILACS. Reference lists of the retrieved studies were also searched to complement the final list of articles. Further evidence was gathered from experts in the field of human resources for health, and from grey literature, through searches in Google, the Human Resources for Health Global Resource Centre and various web sites of government ministries.
We used the following subject headings and text words and combination thereof: "doctors", "nurses", "midwives", "midlevel health workers", "community health workers", "health managers", "laboratory technicians", "health worker", "health professional", "human resources for health", "health workforce","health technician", "clinical engineer", "health teams", "physician", in combination with: "rural", "remote", "underserved", "rural/urban imbalances", "maldistribution"; "retention", "recruitment", "retention strategy", "retention scheme"; "financial incentive", "monetary incentive", "non-financial incentive", "non-monetary incentive", "allowances", "salaries", "benefits"; "compulsory service", "bonding scheme"; "rural pipeline", "professional development", "professional support", "telemedicine"; "vacancy rates", "motivation", "patient satisfaction", "utilization of services", "duration in service"; and "evaluation", "impact", "program result". No language restrictions were placed on the search.
We included articles that were published between 1995 and September 2009 that reported on the results/effects of an intervention to increase availability of health workers in rural or remote areas, from both developed and developing countries, and covering all types of health workers, and including a clear description of the study design and methods used. We excluded studies that described the issue of misdistribution, studies reporting on surveys of factors influencing choices of practice, without a direct link to a specific intervention, and studies that were only describing a potential intervention. We also excluded news, editorials, policy briefs and commentaries as they did not report on a specific evaluation.
A total of 14 746 studies were retrieved from the electronic searches. Two reviewers screened all titles and abstracts found in the search. The reviewers independently judged the titles and abstracts according to the above specified inclusion and exclusion criteria. All titles and abstracts judged as eligible for this global review were then retrieved and thoroughly reviewed in their entirety. Upon full review of the texts, further studies were deemed as not eligible for the inclusion criteria and were discarded (Fig. 1 provides the search diagram and reasons for exclusion). Both reviewers were in agreement over the final selection of studies included in the global review.
Results
Twenty-seven studies met our inclusion and exclusion criteria and were included in the final analysis. The results reported by these studies are presented against a framework for monitoring and evaluating retention interventions. 20 This framework proposes four dimensions on which various policy interventions can have a direct effect: attractiveness of rural/remote areas forstudents and/orhealth works, recruitment/deployment, retention, and health workforce or health system performance. These dimensions follow logic by which it is expected that the creation of "demand" (attractiveness) will be followed by recruitment and retention, which in turn will have a positive effect on improving the availability of health workers, and the quality of health services, thus improving population health outcomes in the long term. For each of these dimensions, specific indicators were reported. A summary of the indicators, methods and results reported in the evaluation studies included in our review is included (available at: http://www.who.int/hrh/resources/evaluation_tables). In addition, for each study included in the review we also specified the category and the subtype of intervention, i.e. education, regulatory, financial, or professional and personal support.
Attractiveness
Twelve studies reported on interventions that have attracted students towards working in rural and remote areas, the majority of them being education interventions. Studies from developed countries have consistently shown that health professionals from a rural background are more likely to practise in rural areas, clinical rotations in a rural setting may influence medical students' subsequent decision to work in an underserved area, and appropriate educational preparation for rural service, including adapting curricula to include rural health issues, creates more interest to work in these areas.21-24Additionally, several studies consistently showed that rurally oriented medical education programmes influenced subsequent choices of graduates to practise in rural areas21, 23, 25-28 (Appendix A, available at: http://www.who.int/hrh/resources/evaluation_tables).
Intentions to relocate to, stay in or leave rural areas have also been influenced by other categories of interventions. One study on financial incentives in South Africa found that the allowance had positively influenced the future plans of health workers, particularly nurses, to work in a rural area.29 Positive changes in the intentions to stay in rural areas were also reported in a continuing professional development programme,30 a programme using financial incentives for long-serving rural doctors in Australia,31 a personal support programme for rural general practitioners in Australia,32 a programme providing rural clinical experiences to medical students, and a scholarship scheme for medical students, the latter two in the United States of America (USA)33, 34
Recruitment
Seven studies have reported the effects of interventions on the number of graduates or health workers actually recruited in underserved areas following their implementation. Two multifaceted educational programmes in the USA have consistently shown a steady increase in the number of their graduates recruited to work in rural areas.26-28, 35, 36 A bonding scheme for post-graduate students in Australia also showed an increase in the percentage of workers practising in rural areas.37 For other types of interventions, the size of effect, expressed by the proportion or number of workers recruited in rural areas as a result of the scheme, is relatively small. For example, a compulsory service in South Africa reported less than 25% of physicians recruited to rural facilities,38 and a financial incentive scheme in the Niger, targeted at doctors, pharmacists and dental surgeons, reported an increase of 42-44% in recruited workers.39 Finally, a programme aiming to support physicians setting up their practice in rural areas in Mali has reported that more than 100 young doctors were recruited in those areas over a 10-year period40 (Appendix B, available at: http://www.who.int/hrh/resources/evaluation_tables).
Retention
Various indicatorswere used to measure effects on retention: length of service, proportion of health workers staying in rural areas, survival rates, turn-over rates and settlement rates. Two studies reported on length of service, one showing a mean duration of 15 months for physicians engaged in a network of rural academic family practice, and the other reporting an average of 4 years in the "medicalization of rural areas" programme in Mali.40, 41Other studies reported only on the proportion of health workers remaining in the rural areas. This varied from a 20% retention rate in a bonding scheme in USA, through to 86% for a financial incentives programme in Australia.31, 42 A more comprehensive analysis was provided by Matsumoto et al., who reported on the effects of a rurally located medical school coupled with a compulsory service scheme in Japan. Almost 70% of the graduates of Jichi medical school remained in their home prefectures for at least 6 years after their obligatory service.43 Finally, another way of measuring retention was to determine the "survival rates" of health workers, using the Kaplan-Meyer method for survival analysis. For example, Rabinowitz et al. found that more than two-thirds of graduates from the Physician Shortage Area Programme in the USA remained in family medicine in the same rural area after 11-16 years28 (Appendix C, available at: http://www.who.int/hrh/resources/evaluation_tables).
Impact
Ten studies have reported on the effects of rural retention interventions on the performance of health workers, or on health services and communities. Two studies reported that certain competencies of health workers improved as a result of retention interventions,30, 38 while improved job satisfaction was reported in five evaluations of retention strategies.32, 34, 39, 44, 45 Finally, four studies reported on the effects of rural retention interventions on service delivery and communities, in terms of improved quality of care, reduced referrals and reduced waiting times33, 38, 46, 47 (Appendix D, available at: http://www.who.int/hrh/resources/evaluation_tables).
It was noted that most of the interventions had multiple effects on the continuum from attraction through to recruitment, retention and, finally, health workforce or health systems performance. Fig. 2 depicts the mapping of the interventions included in our review against these proposed dimensions, and the indicators that were reported by these studies. This is an important finding, as policy-makers need to be clear from the start about the expected results of the chosen intervention and should be able to make choices on the relative importance of the various expected results. Knowing the expected results is also key for monitoring and evaluating these interventions.
Discussion
We assessed the methodological quality of the studies included in our review using traditional criteria for evidence-based clinical medicine: study design, consistency, precision, directness, likelihood of publication bias and the magnitude of effect.48
Study design
We found no randomized controlled trials analysing the effects of retention interventions in rural or remote areas. Five longitudinal cohort studies reported on the effects of multifaceted education programmes26-28, 49 and a financial incentive scheme.29 Three retrospective cohort studies reported on the effects of a medical education programme,33 a loan repayment scheme50, and a compulsory service programme.43 Five studies used a before-and-after design: two looked at clinical rural placements,23, 25 and one in each of the compulsory service,32financial incentives39 and personal or professional support programmes.44 Only one study used a control group in comparing the results of a financial incentives programme between rural physicians enrolled in the scheme and those who were not.34 The remaining twelve studies used a crosssectional observational design and did not report any baseline against which to compare the observed changes.
Reporting results
In terms of methods used, the majority of the studies employed mixed methods, using both quantitative and qualitative methods to describe the complexity of the effects of such interventions. However, many studies relied upon surveys and questionnaires, for which inferences may be difficult to make, and did not always account for design, biases and sampling. Interestingly, only one study used census data to analyse the location of health workers before and after the intervention.39
There was great variability in the reported outcomes, specifically on retention, with some studies considering retention rates as the number or the proportion of health workers remaining in the area, while others providing more comprehensive measures that accounted for both the number of workers and the duration of their stay in months or years. For some studies, there was also a misalignment of outcomes reported in relation to the retention scheme they were evaluating. For example, Fisher et al. found that community health aides had longer retention rates than doctors or nurses, but concede that it was not possible in their study to link these outcomes to the telemedicine intervention they originally wished to evaluate.49 The magnitude of effect therefore is difficult to judge because of these inconsistencies in reporting across the studies.
Publication bias
Our findings confirm other observations,6, 8-10 which showed a skewed geographical distribution of studies that analysed the effectiveness of rural retention interventions. Most evidence comes from high-income countries, such as Australia, Canada, Japan, New Zealand and the USA, with very few studies originating from developing countries in Africa, Latin America or south-eastern Asia, and no evaluations from the eastern Mediterranean region. This is also interesting when compared with our previous findings on the regional representations of descriptive research, where many studies have been conducted in sub-Saharan Africa describing the extent of the problem or the factors influencing choices and preferences of location.13 Also, there is a wealth of anecdotal evidence from many regions, which may not get published due to language barriers. There is a clear need to expand evaluation and operational research efforts in medium- and low-income countries from these regions.
Directness and relevance
The final result of having health workers in remote and rural areas is dependent upon both the factors that influence health workers preferences for rural or remote areas, and the health system's response to these factors.8 In other words, the intervention has to respond to the factors that health workers value in choosing to work in these areas.
The findings showed that rural retention interventions are rarely implemented as a result of an analysis of the preferences or choices of health workers to practise in these areas. Two exceptions are worth mentioning: Dunbabin et al. provide a comprehensive situation analysis before moving on to evaluate the policy response of postgraduate medical placements in rural areas in Australia;37 and in the Niger, a study analysing the motivations of health workers to work in rural areas preceded the development of a financial allowance scheme, which was subsequently evaluated after two years of implementation.39 Given their importance in the design and implementation of retention strategies, it is critical to have a very good understanding of these factors. More refined research methods are needed to identify these factors and to gauge their relative weight in the final decision of health workers.51
Content gaps
The majority of evaluations were conducted for educational programmes and, to a certain extent, regulatory interventions such as compulsory service or bonding schemes. Only four studies were found in both the financial incentives and support programmes categories. This gap is crucial for two reasons. First, policy-makers are inclined to adopt financial incentives programmes almost as a "first-aid" measure when confronted with shortages of health workers in rural or remote areas, without enough knowledge about the effectiveness of such interventions or about their sustainability in the long run. Second, there is an apparent reluctance to adopt professional and personal support measures, even though these factors consistently top the surveys analysing choices and preferences for work in these areas.8 It is important to have an in-depth understanding of the effects of such interventions if policy-makers are to adequately address the expectations of health workers in rural areas.
Finally, a significant gap is that almost all studies have evaluated programmes targeted at physicians or graduates of medical schools. Only three studies reported on other categories of health workers: two rural clinical placements programmes for pharmacy students in New Zealand and nursing students in Australia, and one programme on financial incentives for doctors, pharmacists and dental surgeons in the Niger.23, 25, 39This gap needs to be urgently addressed by the research community, as rural practice requires all categories of health workers and, more importantly, effective health teams to deliver good quality services.
Conclusion
We focused our analysis on studies that have evaluated the impact of policy interventions to increase availability of health workers in rural areas, because evaluation studies can provide useful insights for policy-makers who are faced with health-worker shortages in those areas.14 This allowed us to confirm previous findings on the effectiveness of such interventions6, 8-10 but, more importantly, to provide a clear account of the outcome measures, methods and indicators used, to better inform future evaluation research.
We identified certain gaps in the published evidence on evaluation research in this field: there is an overall bias towards physician-targeted programmes, and towards developed English-speaking countries. More efforts should be made to evaluate programmes targeted at other types of health workers and, in particular, health teams and programmes from developing countries. With regards to the latter, efforts should also be made to strengthen human resources information systems so that sound analysis can be conducted.
In terms of scope, the majority of evaluations were for education programmes, with little information about financial incentive programmes, which are frequently used by policy-makers, and about professional and personal support programmes, which are most valued by health workers.
In terms of relevance, there is frequently a lack of coherence between the proposed retention strategy and the factors that matter for health workers in their choice for location. A situation analysis, including a survey of factors that influence choices of location of health workers in rural areas, should be mandatory as a basis for selecting the most appropriate category of intervention. Finally, in terms of methodology, using a control group is a powerful way to demonstrate the results of a certain intervention; therefore comparative studies, as well as cohort studies or controlled experimental studies should be increasingly encouraged and appropriately funded.
The added value of this study is to provide a clear and systematic account of methodological challenges and research gaps in conducting evaluation research in the field of rural health workforce retention. It is hoped that this will help future evaluations in choosing the study design and the methods appropriate for this type of research. However, although assessing the methodological quality of such studies is necessary, we acknowledge that more needs to be considered by policy-makers when deciding the type of interventions to use in their own context. While traditional criteria to assess the methodological quality have provenuseful in clinical medicine,when used for public health and health systems interventions they risk yielding only low quality or very low quality evidence, which by itself can deter policy-makers to take any action on these interventions. Therefore, additional information needs to be provided to policy-makers with regards to the relevance of the interventions to certain contexts and situations and the mechanisms by which certain interventions have worked in some contexts but not in others.52 Better informing the choice of appropriate interventions to address the geographical imbalances of health workers, and assessing progress on implementing these interventions, will inevitably contribute to the broader goals of universal coverage and to the achievement of the health-related Millennium Development Goals. 
Acknowledgements
We thank members of theWorld Health Organization's core expert group on "Increasing access to health workers in remote and rural areas through improved retention" and Mario Dal Poz of the Health Workforce Information and Governance Team, Department of Human Resources for Health, World Health Organization, for comments on earlier drafts of this paper.
Funding: Financial support for literature searches in languages other than English was provided by the Health Workforce Migration and Retention Unit, Department of Human Resources for Health, Health Systems and Services, World Health Organization, Geneva.
Competing interests: None declared.
References
1. Travis P, Bennett S, Haines A, Pang T, Bhutta Z, Hyder AA et al. Overcoming health-systems constraints to achieve the Millennium Development Goals. Lancet 2004;364:900-6. doi:10.1016/S0140-6736(04)16987-0 PMID:15351199
2. World health report 2006: working together for health. Geneva: World Health Organization; 2006.
3. World health report 2008: primary health care, now more than ever. Geneva: World Health Organization; 2006.
4. Zurn P, Dal Poz MR, Stilwell B, Adams O. Imbalance in the health workforce. Hum Resour Health 2004;2:13. doi:10.1186/1478-4491-2-13 PMID:15377382
5. Dussault G, Franceschini MC. Not enough there, too many here: understanding geographical imbalances in the distribution of the health workforce. Hum Resour Health 2006;4:12. doi:10.1186/1478-4491-4-12 PMID:16729892
6. Grobler L, Marais BJ, Mabunda SA, Marindi PN, Reuter H, Volmink J. Interventions for increasing the proportion of health professionals practising in rural and other underserved areas. Cochrane Database Syst Rev 2009;1:CD005314. PMID:19160251
7. Henderson LN, Tulloch J. Incentives for retaining and motivating health workers in Pacific and Asian countries. Hum Resour Health 2008;6:18. doi:10.1186/1478-4491-6-18 PMID:18793436
8. Lehmann U, Dieleman M, MartineauT. Staffing remote rural areas in middleand low-income countries: a literature review of attraction and retention. BMC Health Serv Res 2008;8:19. doi:10.1186/1472-6963-8-19 PMID:18215313
9. Wilson NW, Couper ID, De Vries E, Reid S, Fish T, Marais BJ. A critical review of interventions to redress the inequitable distribution of healthcare professionals to rural and remote areas. Rural Remote Health 2009;9:1060. PMID:19530891
10. Bärnighausen T, Bloom DE. Financial incentives for return of service in underserved areas: a systematic review. BMC Health Serv Res 2009;9:86. doi:10.1186/1472-6963-9-86 PMID:19480656
11. Global Health Workforce Alliance. The Kampala Declaration and Agenda for Global Action. Geneva:World Health Organization; 2008. Available from: http://www.who.int/workforcealliance/Kampala%20Declaration%20and%20Agenda%20web%20file.%20FINAL.pdf [accessed 19 March 2010] .
12. WHO global recommendations for the retention of health workers [Internet site]. Geneva: World Health Organization; 2009. Available from: http://www.who.int/hrh/migration/retention/en/index.html [accessed 19 March 2010] .
13. Dolea C. Stormont L, Shaw D, Zurn P, Braichet JM. Increasing access to health workers in remote and rural areas through improved retention. Geneva: World Health Organization; 2009. Available from: http://www.who.int/hrh/migration/background_paper.pdf [accessed 19 March 2010] .
14. Mills A, Gilson L, Hanson K, Palmer N, Lagarde M. What do we mean by rigorous health-systems research? Lancet 2008;372:1527-9. doi:10.1016/S0140-6736(08)61633-5 PMID:18984174
15. Martineau T, Lehmann U, Matwa P, Kathyola J, Storey K. Factors affecting retention of different groups of rural health workers in Malawi and Eastern Cape Province, South Africa. Geneva: World Health Organization Alliance for Health Policy and Systems Research; 2006.
16. Dieleman M, Cuong PV, Anh LV, Martineau T. Identifying factors for job motivation of rural health workers in North Viet Nam. Hum Resour Health 2003;1:10. doi:10.1186/1478-4491-1-10 PMID:14613527
17. Furth R. Zambia pilot study of performance-based incentives. Bethesda, MD: USAID; 2006.
18. Bourgueil Y, Mousquès J, Tajahmadi A. Comment améliorer la répartition géographique des professionnels de santé? Les enseignements de la litérature internationale et des mesures adoptées en France [in French]. Paris: Institut de recherche et documentation en économie de la santé; 2006.
19. Yumkella F. Retention: health workforce issues and response actions in low resource settings [Capacity project resource paper]. Chapel Hill, NC: IntraHealth International; 2005. Available from: http://www.capacityproject.org/images/stories/files/retention_paper_long050823.pdf [accessed 19 March 2010] .
20. Huicho L, Dieleman M, Campbell J, Codjia L, Balabanova D, Dussault G et al. Increasing access to health workers in underserved areas: a conceptual framework for measuring results. Bull World Health Organ 2010;88:357-363.
21. Halaas GW, Zink T, Finstad D, Bolin K, Center B. Recruitment and retention of rural physicians: outcomes from the rural physician associate program of Minnesota. J Rural Health 2008;24:345-52. doi:10.1111/j.1748 0361.2008.00180.x PMID:19007388
22. Woloschuk W, Tarrant M. Does a rural educational experience influence students' likelihood of rural practice? Impact of student background and gender. Med Educ 2002;36:241-7. doi:10.1046/j.1365 2923.2002.01143.x PMID:11879514
23. Capstick S, Beresford R, Gray A. Rural pharmacy in New Zealand: effects of a compulsory externship on student perspectives and implications for workforce shortage. Aust J Rural Health 2008;16:150-5. doi:10.1111/j.1440-1584.2008.00965.x PMID:18471185
24. Mathews M, Rourke JTB, Park A. The contribution of Memorial University's medical school to rural physician supply. Can J Rural Med 2008;13:15-21. PMID:18208648
25. Courtney M, Edwards H, Smith S, Finlayson K. The impact of rural clinical placement on student nurses' employment intentions. Collegian 2002;9:12-8. PMID:11893112
26. Rabinowitz HK, Diamond JJ, Markham FW, Hazelwood CE. A program to increase the number of family physicians in rural and underserved areas: impact after 22 years. JAMA 1999;281:255-60. doi:10.1001/jama.281.3.255 PMID:9918481
27. Rabinowitz HK, Diamond JJ, Markham FW, Paynter NP. Critical factors for designing programs to increase the supply and retention of rural primary care physicians. JAMA 2001;286:1041-8. doi:10.1001/jama.286.9.1041 PMID:11559288
28. Rabinowitz HK, Diamond JJ, Markham FW, Rabinowitz C. Long-term retention of graduates from a program to increase the supply of rural family physicians. Acad Med 2005;80:728-32. doi:10.1097/00001888-200508000-00004 PMID:16043525
29. Reid S. Monitoring the effect of the new rural allowance for health professionals[Research project report]. Durban: Health Systems Trust; 2004. Available from: http://healthlink.org.za/uploads/files/rural_allowance.pdf [accessed 19 March 2010] .
30. White CD, Willett K, Mitchell C, Constantine S. Making a difference: education and training retains and supports rural and remote doctors in Queensland. Rural Remote Health 2007;7:700. PMID:17430081
31. Gibbon P, Hales J. Review of the rural retention program [final report]. St Peters, S.A.: Health Outcomes International; 2006. Available from: http://www.health.gov.au/internet/main/publishing.nsf/Content/751B9B296D05A4C8CA25741E0079E487/$File/review.pdf [accessed 19 March 2010] .
32. Gardiner M, Sexton R, Kearns H, Marshall K. Impact of support initiatives on retaining rural general practitioners. Aust J Rural Health 2006;14:196-201. doi:10.1111/j.1440-1584.2006.00808.xPMID:17032295
33. Smucny J, Beatty P, Grant W, Dennison T, Wolff LT. An evaluation of the Rural Medical Education Program of the State University of New York Upstate Medical University 1990-2003. Acad Med 2005;80:733-8. doi:10.1097/00001888-200508000-00006 PMID:16043527
34. Jackson J, Shannon CK, Pathman DE, Mason E, Nemitz JW. A comparative assessment of West Virginia's financial incentive programs for rural physicians. J Rural Health 2003;19(Suppl):329-39. PMID:14526516
35. Pacheco M, Weiss D, Vaillant K, Bachofer S, Garrett B, Dodson WH 3rd et al. The impact on rural New Mexico of a family medicine residency. Acad Med 2005;80:739-44. doi:10.1097/00001888-200508000-00007 PMID:16043528
36. Cosgrove EM, Harrison GL, Kalishman S, Kersting KE, Romero-Leggott V, Timm C et al. Addressing physician shortages in New Mexico through a combined BA/MD program. Acad Med 2007;82:1152-7. doi:10.1097/ACM.0b013e318159cf06 PMID:18046118
37. Dunbabin JS, McEwin K, Cameron I. Postgraduate medical placements in rural areas: their impact on the rural medical workforce. Rural Remote Health 2006;6:481. PMID:16594864
38. Reid SJ. Compulsory community service for doctors in South Africa - an evaluation of the first year. S Afr Med J 2001;91:329-36. PMID:11402906
39. Impact des mesures d'incitation financière accordées aux médecins, pharmaciens et chirurgiens dentistes [in French]. Niyamé: Niger Ministry of Health; 2008.
40. Codjia L, Jabot F, Dubois H. Evaluation du programme d'appui à la médicalisation des aires de santé rurale au Mali [in French]. Geneva: World Health Organization; 2009.Available online at: http://www.who.int/hrh/resources/case_mali/fr/index.html (accessed 12 April 2010).
41. Wilkinson D, Symon B, Newbury J, Marley JE. Positive impact of rural academic family practices on rural medical recruitment and retention in South Australia. Aust J Rural Health 2001;9:29-33. doi:10.1046/j.1440 1584.2001.00293.x PMID:11703264
42. Cullen TJ, Hart LG, Whitcomb ME, Rosenblatt RA. The National Health Service Corps: rural physician service and retention. J Am Board Fam Pract 1997;10:272-9. PMID:9228622
43. Matsumoto M, Inoue K, Kajii E. Long-term effect of the home prefecture recruiting scheme of Jichi Medical University, Japan. Rural Remote Health 2008;8:930. PMID:18643712
44. Cavender A, Albán M. Compulsory medical service in Ecuador: the physician's perspective. Soc Sci Med 1998;47:1937-46. doi:10.1016/S0277 9536(98)00335-9 PMID:10075237
45. Janes R, Dowell A. New Zealand Rural General Practitioners 1999 Survey- Part 3: rural general practitioners speak out. N Z Med J 2004;117:U815. PMID:15107884
46. Omole OB, Marincowitz G, Ogunbanjo GA. Perceptions of hospital managers regarding the impact of doctors' community service.South African family practice 2005;47:55-59.
47. Como del Corral MJ, Luquin PA, Guevara JC, Movilla AO, Torres GT, Garcia JL. Utility of a thematic network in primary health care: a controlled interventional study in a rural area. Hum Resour Health 2005;3:1-7. doi:10.1186/1478 4491-3-1 PMID:15659241
48. Mangham LJ, Hanson K, McPake B. How to do (or not to do)... Designing a discrete choice experiment for application in a low-income country. Health Policy Plan 2009;24:151-8. doi:10.1093/heapol/czn047 PMID:19112071
49. Fisher DG, Pearce FW, Statz DJ, Wood MM. Employment retention of health care providers in frontier areas of Alaska. Int J Circumpolar Health 2003;62:423-35. PMID:14964768
50. Lapolla M, Brandt EN Jr, Barker A, Ryan L. State public policy: the impacts of Oklahoma's physician incentive programs. J Okla State Med Assoc 2004;97:190-4. PMID:15212104
51. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P et al.; GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924-6. doi:10.1136/bmj.39489.470347.AD PMID:18436948
52. Dieleman M, Kane S, Zwanikken P, Gerretsen B. Realist review and synthesis on retention studies for health workers in rural and remote areas. Geneva: World Health Organization; 2009. Available from: http://www.who.int/hrh/resources/realist_review [accessed 26 March 2010] .
(Submitted: 8 October 2009 - Revised version received: 8 March 2010 - Accepted: 9 March 2010)
* Correspondence to Carmen Dolea (e-mail: doleac@who.int).