POLICY & PRACTICE

Meeting global health challenges through operational research and management science

Relever les défis de la santé mondiale par la recherche opérationnelle et à la science de gestion

Cumplimiento de los retos sanitarios globales a través de la investigación operativa y las ciencias administrativas

Geoff Royston^*

Consultant, Burton Leonard, Harrogate, HG3 3SJ, England

ABSTRACT

This paper considers how operational research and management science can improve the design of health systems and the delivery of health care, particularly in low-resource settings. It identifies some gaps in the way operational research is typically used in global health and proposes steps to bridge them. It then outlines some analytical tools of operational research and management science and illustrates how their use can inform some typical design and delivery challenges in global health. The paper concludes by considering factors that will increase and improve the contribution of operational research and management science to global health.

RÉSUMÉ

Ce document examine comment la recherche opérationnelle et la science de gestion peuvent améliorer la conception des systèmes de santé et la prestation de soins de santé, en particulier dans les milieux à faibles ressources. Il identifie certaines lacunes dans la façon dont la recherche opérationnelle est généralement utilisée dans la santé mondiale et propose des mesures pour y remédier. Il décrit ensuite certains outils d'analyse de la recherche opérationnelle et de la science de gestion et montre comment leur utilisation peut influencer certains défis typiques en termes de conception et de prestation dans la santé mondiale. L'article conclut en étudiant les facteurs qui accroîtront et amélioreront la contribution de la recherche opérationnelle et de la science de gestion à la santé mondiale.

RESUMEN

Este artículo valora la medida en la que la investigación operativa y las ciencias administrativas podrían mejorar el diseño de los sistemas sanitarios y el suministro de atención sanitaria, especialmente en entornos de escasos recursos. También identifica algunas lagunas en la manera en la que se suele utilizar la investigación operativa en la sanidad global y propone los pasos que deben darse para cubrir dichas carencias. Además, se perfilan algunas herramientas analíticas de investigación operativa y ciencias administrativas y se muestra cómo su uso puede servir de apoyo en algunos diseños típicos y en retos de suministro de la sanidad global. El artículo concluye considerando los factores que incrementarán y mejorarán la contribución de la investigación operativa y las ciencias administrativas a la sanidad global.

Introduction

The historic international health conference at Alma-Ata in 1978 first identified the need for research on "operation, control and evaluation problems" in primary health care.¹ Since then, successive reports from the World Health Organization (WHO) have expressed the need for improvements in organization and delivery and have called for research, both on particular interventions and on entire health systems.² Penetrating the "fog of delivery" clearly presents a major global health research challenge.

Operational research (also known as operations research) and management science have made major contributions in improving organization and delivery in many fields of human activity. Operational research originated in the military arena with the design of an integrated information and control system for the British air force in World War II, work which was estimated to have doubled the efficacy of its fighter command. Since then its use has spread, often providing big returns on investment. Two recent examples are: the world's largest logistics company redesigned its overnight delivery network which was estimated to yield savings of more than 270 million United States dollars (US$) and a global automobile manufacturer streamlined its prototype vehicle testing, saving US$ 250 mil lion annually. These examples and more are available at: http:// www.scienceofbetter.org.

However, operational research and management science are underused in the health field, certainly in global health. For example, the Global Fund to Fight AIDS, TB and Malaria allows 5–10% of each grant for monitoring, evaluation and operations research. However, recent estimates are that projects only budget an average of 3% for operational research and actually spend considerably less.³

Missing links

Despite the low priority that global health has given operational research, some valuable work has been and is being done. Some examples include: a 32-country programme on primary-care operations research established in 1981 by the US Agency for International Development;⁴ a sustained operational research effort over several decades underpinning the development of a global strategy on tuberculosis control;⁵ and an established body of operational research around HIV/AIDS.⁶

In global health, operational research has an extremely broad interpretation. The term is used for almost any type of improvement-oriented investigation into a programme's operations. Where management science generally uses systems modelling and related analytical techniques, operational research in global health does not use these tools sufficiently. Almost two decades ago a review of this field noted this gap and stated that many operational research studies in global health "do not carry the full flavour of operational research".⁷ With some exceptions, such as for HIV/AIDS where there has been a good deal of operational research modelling work,^8,9 that gap clearly remains, particularly for neglected tropical diseases.¹⁰ For example, guides on operational research published by WHO and the Global Fund to Fight AIDS, TB and Malaria focus mainly on the use of interviews, questionnaires and observations, with less emphasis on experimentation and no coverage of modelling or other analytical methods.¹¹ Stronger links between the practical and analytical approaches would bridge this gap.

Strengthening the use of management science in global health would also improve communication between operational research workers in global health and development. A recent review of operational research in developing countries provides useful detail.¹²

Analytical tools

Management scientists use a range of analytical tools, from quantitative prediction and optimization techniques to qualitative problem structuring and solution search approaches (Fig. 1).¹³ The tools listed on the top left side of the figure are predominantly qualitative and are typically used with groups of people in participative sessions while those on the bottom right are predominantly quantitative and are typically used by individual analysts in desk-based work. For example, behavioural simulation involves groups of stakeholders acting out a system change or problem situation to gain insights – in a risk-free environment – into likely consequences of their actions. Discrete event and agent simulation seeks to gain such insights rapidly by using computer modelling to represent key system components (agents), their interactions and resulting changes (discrete events).

Qualitative approaches are most likely to be of use at the early stages of analysis, or when time and/or data are in short supply or when the problems are ill-defined. Several problem-focused qualitative analytical tools, such as root cause analysis or influence diagramming (which could be termed "system mapping" approaches), combine analytical rigour with simplicity and transparency. They are particularly useful where it may be difficult or inappropriate to use more quantitative methods.

Some of the more quantitative or computational approaches can be useful, even with limited data, as their key outputs are often determined more by the structure and logic of a situation than precise values of parameters. One such approach is "system dynamics" modelling, a more aggregate modelling approach than discrete event or agent simulation and one that pays particular attention to feedback effects. A noteworthy example of this is the system dynamics epidemiological model used in the global campaign to eradicate polio. The model incorporated a feedback loop that represented how observed impacts of interventions would influence subsequent interventions. This informed a debate on the best approach by showing that eradication was a more effective and less costly long-term strategy than control as the latter approach would not prevent regular major flare-ups of the disease.¹⁴

These tools are not mutually exclusive. For example, workers on pandemic influenza have used tools including: brainstorming-based SWOT (strengths, weaknesses, opportunities and threats) analysis,¹⁵ behavioural simulation exercises, ¹⁶ scenario analysis,¹⁷ system dynamics modelling¹⁸ and various combinations of simulation and mathematical modelling.¹⁹

Nor should modelling be divorced from experiment; they can make a powerful combination with the strengths of one compensating for the limitations of the other. For example, given some basic experimental data, modelling can rapidly explore the effect of varying testing intervals in screening programmes rather than waiting years for field tests on each option.

Note that the word "system" appears twice at the very centre of Fig. 1. While some aspects of health-care delivery are essentially logistical, and so amenable to the use of traditional analytic methods, many health issues are complex and require systemic analysis,²⁰ as is being increasingly recognized. For instance in a recent WHO report,²¹ which stated "systems thinking has a huge and untapped potential, first in deciphering the complexity of an entire health system, and then in applying this understanding to design and evaluate interventions". Systems thinking has several important differences from the more usual "linear" perspective, some of which are summarized in Table 1.

Global health challenges

We now look at some key generic challenges that arise in global health (Fig. 2) and consider how operational research and management science can contribute to each.

Identifying problems

Operational research methods are useful for the systematic identification of problems and the search for potential solutions. Structured approaches to identifying options, such as the strategic choice approach²³ or systematic creativity approaches such as the Russian-invented methodology TRIZ (translated as "theory of inventive problem solving"),²⁴ have great potential for use in low-resource settings.²⁵ New approaches are important for global health because strategies and programmes must be designed to deal with both current and future challenges – from the global spread of disease to the impact of climate change. This can sometimes take us beyond traditional forecasting methodologies to the use of scenario analysis and other futures thinking methods. There has been a fair amount of such work in the health field and its methods have been adopted in high-profile initiatives such as by the Government Office for Science in the United Kingdom of Great Britain and Northern Ireland. However, little appears to have been done on global health with the exception of scenario analyses on pandemic influenza¹⁷ and on AIDS in Africa up to the year 2025.²⁶

With an expected increase in extreme events linked to climate change, disaster planning and humanitarian logistics are becoming even more important.²⁷ Decision support tools specifically designed for wide-scale emergency situations should play an increasing role.²⁸ Developing experience in this sort of logistics analysis will have application in global health both in and beyond emergency situations.

Choosing interventions

Choosing appropriate interventions is clearly a crucial step. Effectiveness, safety, cost and equity should all be considered and researchers will be familiar with standard text-book methods for assessing these. But, in reality, assessments are rarely straightforward. It is often more effective and efficient to combine interventions, or to treat more than one disease at a time. For example, the integration of treatments for different diseases into cost-saving combined "packages". Finding the best combinations and delivery methods is a major research exercise in its own right. Modelling different intervention strategies before roll-out is now ubiquitous in many industries but is less common in health care.²⁹ Combining this approach with the necessary field and pilot studies should speed up and focus efforts. Modelling work has been done on ways to reduce maternal mortality³⁰ and in cervical cancer screening in low-resource settings.³¹

Introducing new interventions

Innovation is a key issue in global health. For instance telemedicine is increasingly being used in high-income countries, and beginning to be applied in middle- and low-income countries, enabled by the rapid global spread of mobile phone networks. However, many such applications in developing countries are not evaluated so there is a clear role for operational research to assess them and to improve their design and delivery.³²

Mobile phones also allow much easier collection and collation of data for operational and related research. For example, an analysis of mobile phone data in Zanzibar (in the United Republic of Tanzania) showed movement patterns to regions with high levels of malaria. Combining this information with mathematical models of disease transmission suggested ways to improve malaria control.³³

The contribution that operational research and management science can make to design and delivery is not restricted to high technology. Oral rehydration therapy is a "low-tech–lowcost–high-impact" innovation in which operational research was used to explore ways it could be administered at low cost using readily available ingredients by lay people, with an escalation pathway to treatment by health-care professionals when necessary.³⁴

Scaling-up

Small-scale projects generally need considerable modifications to work on a larger scale. For example, we need to understand better what happens when whole countries are treated with drugs or vaccines. Such upscaling includes a need to forecast future demand, to decide the location and size of facilities and to set staffing levels. Classic operational research techniques such as simulation modelling can be used in locating services, managing the pharmaceutical supply chain and developing the health care workforce. One paper³⁵ presents models for efficient and equitable location of community health facilities in rural areas while another³⁶ describes the development of a simulation model of task allocation to reduce pressures on physicians at HIV clinics in Rwanda.

Integrating into health systems

Successes in global health programmes often result from synergistic interactions between individual, community and national actors rather than from any single "magic bullet". We need a greater focus on how interventions should be used in a complex behavioural environment, to better capture the dynamics of social networks and to understand how complex systems can adapt positively to change. This is a task where operational research and management science tools can be useful, as demonstrated by systems analysis of programmes for cervical cancer prevention³⁷ or agent simulation modelling of spread of HIV in villages.³⁸

Health systems are of course embedded in wider systems. Modelling can be a powerful tool in considering the sustainability of health interventions in the context of the wider environment³⁹ and in designing systems that affect health, such as waste management.⁴⁰

Evaluation

One of the greatest challenges for global health is the measurement and evaluation of performance of projects and programmes (Fig. 2). Recent evaluation has shown that the results of the United Nations Children's Fund (UNICEF) accelerated child survival and development programme fell far short of original claims,⁴¹ a study that prompted an accompanying editorial in the Lancet that stated: "Evaluation must now become the top priority in global health".

Standard control trial approaches to evaluation are sometimes feasible and appropriate but often a more flexible systems-oriented approach is required, together with modelling (for example, for assessing screening programmes⁴² or to help assess the effectiveness of preventive interventions).⁴³ Decision tree modelling can give rapid insights into the operational effectiveness and costeffectiveness of procedures⁴⁴ and programmes⁴⁵ and a variety of operational research and management science tools have assisted evaluative work on broader global health issues.⁴⁶

Ingredients for success

A first requirement for success is the availability of appropriate research and analysis skills and resources, which is clearly a challenge particularly in developing countries. This can be ameliorated by training people within the health service who can lead and champion operational research, such as the programme launched by the International Union Against Tuberculosis and Lung Disease.⁴⁷ Further steps might include extending the scope of the international consortium to strengthen health research capacity in Africa to include a full range of operational research and management science activities.

A second requirement is the effective deployment of capacity in operational research and management science. While there is an important role for people with these skills in developed countries to contribute to global health endeavours, more work is needed at the local level. Two recent articles^48,49 discussed factors for and barriers against successful operational research in low-income countries. While they share some features of operational research in health with countries such as the United Kingdom of Great Britain and Northern Ireland,⁵⁰ some features are specific to low-income settings. Success factors that are particularly important in such settings include involving appropriate people, using accessible methods and ensuring effective communication of results (Box 1).

Attention to these factors should help reduce implementation problems. Developing stronger links with those active in practical quality improvement work in health care, such as the Institute for Health care Improvement,⁵¹ could also assist.

Conclusion

Achieving major improvements in global health will require some fundamental changes,⁵² including advances in research and analysis of organization and delivery of health care. Operational research and management science approaches can inform a range of important design and delivery issues, but need to pay more attention to success factors and to draw on a broader range of analytical methods, with more interchange with wider operational research work. With greater capability in this field, operational research and management science can play a significant role in global health.

Acknowledgements

The author thanks Dan Colley, Nigel Crisp, Don Enarson, Richard Feachem, Andrew Green, Anthony Harries, David Lalloo, Eva Lee, Barnett Parker, Jonathan Rosenhead, Clive Smee, Colin Thun-hurst, Martin Utley and Rony Zachariah.

Competing interests: None declared.

References

1. Primary health care: report of the international conference on primary health care. Alma-Ata USSR, 6–12 September 1978. Geneva: World Health Organization; 1978.         

2. Report of the task force on health systems research. Geneva: World Health Organization; 2005.         

3. Korenromp E, Komatsu R, Katz I, Katz I, Xueref S, Low-Beer D, et al. Operational research on HIV/AIDS, tuberculosis and malaria control in Global Fund-supported programmes: Round 1-6 grants. In: Proceedings of the 5th European conference on tropical medicine and international health, Amsterdam, 24–28 May 2007.         

4. Reynolds J. Introduction: operations research in primary health care. Socioecon Plann Sci 1987;21:73–7. doi:10.1016/0038-0121(87)90016-4        

5. Nunn P, Harries A, Godfrey-Faussett P, Gupta R, Maher D, Raviglione M. The research agenda for improving health policy, systems performance and service delivery for tuberculosis control: a WHO perspective. Bull World Health Organ 2002;80:471–6. PMID:12132005        

6. HIV testing, treatment and prevention: generic tools for operational research. Geneva: World Health Organization; 2009.         

7. Datta S. Applications of OR in health in developing countries: a review. Soc Sci Med 1993;37:1441–50. doi:10.1016/0277-9536(93)90178-7 PMID:8303328        

8. Harper PR, Shahani AK. A decision support system for care of HIV and AIDS patients in India. Eur J Oper Res 2003;147:187–97. doi:10.1016/S0377-2217(02)00270-9        

9. Rauner MS, Brailsford SC, Flessa S. Use of discrete-event simulation to evaluate strategies for prevention of mother to child transmission of HIV in developing countries. J Oper Res Soc 2005;56:222–33. doi:10.1057/palgrave.jors.2601884        

10. Kealey A, Smith R. Neglected tropical diseases: infection, modeling and control. J Health Care Poor Underserved 2010;21:53–69. doi:10.1353/hpu.0.0270 PMID:20173255        

11. The framework for operations and implementation research in health and disease control programmes. Geneva: World Health Organization; 2008.         

12. White L, Smith H, Currie C. OR in developing countries: a review. European J Opl Res 2011;208:1–11. doi:10.1016/j.ejor.2010.02.015        

13. Pidd M. Tools for thinking: modelling in management science, 2nd edition. Chichester: John Wiley & Sons; 2003.         

14. Thompson KM, Duintjer Tebbens RJ. Using system dynamics to develop policies that matter: global management of poliomyelitis and beyond. Syst Dyn Rev 2008;24:433–49. doi:10.1002/sdr.419        

15. Uscher-Pines L, Barnett DJ, Sapsin JW, Bishai DM, Balicer RD. A systematic analysis of influenza vaccine shortage policies. Public Health 2008;122:183–91. doi:10.1016/j.puhe.2007.06.005 PMID:17825858        

16. Kaiser R, Ciotti M, Thinus G, Simpson J. Common ground: a pandemic influenza simulation exercise for the European Union, 23–25 November 2005. Euro Surveill. 2005;10: pii=2860. Available from: http://www.eurosurveillance.org/ViewArticle.aspx?ArticleId=2860 [accessed 15 August 2011].         

17. van Genugten MLL, Heijnen M-LA, Jager JC. Pandemic influenza and health-care demand in the Netherlands: scenario analysis. Emerg Infect Dis 2003;9:531–8. PMID:12737735        

18. Pruyt E, Hamarat C. The influenza A (H1N1) pandemic: an exploratory system dynamics approach. In: 18th International Conference of the System Dynamics Society, Seoul, 25–29 July 2010.         

19. Ferguson NM, Cummings DAT, Fraser C, Cajka JC, Cooley PC, Burke DS. Strategies for mitigating an influenza pandemic. Nature 2006;442:448–52. doi:10.1038/nature04795 PMID:16642006        

20. Trochim WM, Cabrera DA, Milstein B, Gallagher RS, Leischow SJ. Practical challenges of systems thinking and modeling in public health. Am J Public Health 2006;96:538–46. doi:10.2105/AJPH.2005.066001 PMID:16449581        

21. de Savigny D, Adam T, editors. Systems thinking for health systems strengthening. Geneva: World Health Organization; 2009.         

22. Richmond B. "The "thinking" in systems thinking:Seven essential skills. Pegasus Communications 2010.         

23. Friend J, Hickling A. Planning under pressure, 3rd edition. Oxford: Elsevier Butterworth-Heinemann; 2005.         

24. Salamatov Y. TRIZ: the right solution at the right time: a guide to innovative problem solving. Hattem: Insytec BV; 1999.         

25. Thunhurst C, Barker C. Using problem structuring methods in strategic planning. Health Policy Plan 1999;14:127–34. doi:10.1093/heapol/14.2.127 PMID:10538716        

26. AIDS in Africa: three scenarios to 2025. Geneva: Joint UN Programme on HIV/AIDS; 2004.         

27. Van Wassenhove LN. Blackett Memorial Lecture: humanitarian aid logistics: supply chain management in high gear. J Oper Res Soc 2006;57:475–89. doi:10.1057/palgrave.jors.2602125        

28. Lee EK, Maheshwary S, Mason J, Glisson W. Decision support system for mass dispensing of medications for infectious disease outbreaks and bioterrorist attacks. Ann. Oper. Res. Computing and Optimization in Medicine and Life Sciences 2006;148:25–53.         

29. Young T, Brailsford S, Connell C, Davies R, Harper P, Klein JH. Using industrial processes to improve patient care. BMJ 2004;328:162–4. doi:10.1136/bmj.328.7432.162 PMID:14726351        

30. Pagel C, Lewycka S, Colbourn T, Mwansambo C, Meguid T, Chiudzu G et al. Estimation of potential effects of improved community-based drug provision, to augment health facility-strengthening, on maternal mortality due to post-partum haemorrhage and sepsis in sub-Saharan Africa: an equity-effectiveness model. Lancet 2009;374:1441–8. doi:10.1016/S0140-6736(09)61566-X PMID:19783291        

31. Kim JJ, Salomon JA, Weinstein MC, Goldie SJ. Packaging health services when resources are limited: the example of a cervical cancer screening visit. PLoS Med 2006;3:e434. doi:10.1371/journal.pmed.0030434 PMID:17105337        

32. Royston G, Halsall J, Halsall D, Braithwaite C. Operational research for informed innovation: NHS Direct as a case study in the design, implementation and evaluation of a new public service. J Oper Res Soc 2003;54:1022–8. doi:10.1057/palgrave.jors.2601617        

33. Tatem AJ, Qiu Y, Smith DL, Sabot O, Ali AS, Moonen B. The use of mobile phone data for the estimation of the travel patterns and imported Plasmodium falciparum rates among Zanzibar residents. Malar J 2009;8:287. doi:10.1186/1475-2875-8-287 PMID:20003266        

34. Sengupta PG, Mondal SK, Ghosh S, Gupta DN, Sikder SN, Sircar BK. Review on development and community implementation of oral rehydration therapy. Indian J Public Health 1994;38:50–7. PMID:7835996        

35. Smith HK, Harper PR, Potts CN, Thyle A. Planning sustainable community health schemes in rural areas of developing countries. Eur J Oper Res 2009;193:768–77. doi:10.1016/j.ejor.2007.07.031        

36. Xiong W, Hupert N, Hollingsworth EB, O'Brien ME, Fast J, Rodriguez WR. Can modeling of HIV treatment processes improve outcomes? Capitalizing on an operations research approach to the global pandemic. BMC Health Serv Res 2008;8:166. doi:10.1186/1472-6963-8-166 PMID:18680594        

37. Suba EJ, Murphy SK, Donnelly AD, Furia LM, Huynh ML, Raab SS. Systems analysis of real-world obstacles to successful cervical cancer prevention in developing countries. Am J Public Health 2006;96:480–7.          doi:10.2105/AJPH.2004.061606 PMID:16449592

38. Alam SJ, Meyer R, Norling E. A model for HIV spread in a South African village. In: Multi-Agent Based Simulation IX: International Workshop (MABS 2008), Estoril, Portugal, 12–13 May 2008. Revised selected papers 2009; 5269:33-35.         

39. Gruen RL, Elliott JH, Nolan ML, Lawton PD, Parkhill A, McLaren CJ et al. Sustainability science: an integrated approach for health-programme planning. Lancet 2008;372:1579–89. doi:10.1016/S0140-6736(08)61659-1 PMID:18984192        

40. Brent AC, Rogers DEC, Ramabitsa-Siimane TSM, Rohwer MB. Application of the analytical hierarchy process to establish healthcare waste management systems that minimize infection risks in developing countries. Eur J Opl Res 2007;181:403–24. doi:10.1016/j.ejor.2006.06.015        

41. Bryce J, Gilroy K, Jones G, Hazel E, Black RE, Victora CG. The Accelerated Child Survival and Development programme in west Africa: a retrospective evaluation. Lancet 2010;375:572–82. doi:10.1016/S0140-6736(09)62060-2 PMID:20071020        

42. Sherlaw-Johnson C, Gallivan S, Jenkins D. Evaluating cervical cancer screening programmes for developing countries. Int J Cancer 1997;72:210–6. doi:10.1002/(SICI)1097-0215(19970717)72:2<210::AID-IJC2>3.0.CO;2-U PMID:9219822        

43. van Vliet C, Meester EL, Korenromp EL, Singer B, Bakker R, Habbema JD. Focusing strategies of condom use against HIV in different behavioural settings: an evaluation based on a simulation model. Bull World Health Organ 2001;79:442–54. PMID:11417040        

44. Lubell Y, Reyburn H, Mbakilwa H, Mwangi R, Chonya S, Whitty CJ et al. The impact of response to the results of diagnostic tests for malaria: cost–benefit analysis. BMJ 2008;336:202–5. doi:10.1136/bmj.39395.696065.47 PMID:18199700        

45. Bachmann MO. Cost effectiveness of community-based therapeutic care for children with severe acute malnutrition in Zambia: decision tree model. Cost Eff Resour Alloc 2009;7:151–60. doi:10.1186/1478-7547-7-2 PMID:19146668        

46. Ahsan MK, Bartlema J. Monitoring healthcare performance by analytic hierarchy process: a developing country perspective. Int Trans Oper Res 2004;11:465–78. doi:10.1111/j.1475-3995.2004.00470.x        

47. Other technical courses: operational research. Paris: International Union Against Tuberculosis and Lung Disease; 2011. Available from: http://www.theunion.org/index.php/en/courses/other-technical-courses-/item/689-operational-research [accessed 30 June 2011].         

48. Zachariah R, Harries AD, Ishikawa N, Rieder HL, Bissell K, Laserson K et al. Operational research in low-income countries: what, why, and how? Lancet Infect Dis 2009;9:711–7. doi:10.1016/S1473-3099(09)70229-4 PMID:19850229        

49. Zachariah R, Ford N, Draguez B, Yun O, Reid T. Conducting operational research within a nongovernmental organization: the example of Médecins Sans Frontières. International Health 2010;2:1–8. doi:10.1016/j.inhe.2009.12.008        

50. Royston G. One hundred years of operational research in health – UK 1948-2048. J Opl Res Soc 2009;60:S169–79. doi:10.1057/jors.2009.14        

51. Berwick DM. The science of improvement. JAMA 2008;299:1182–4. doi:10.1001/jama.299.10.1182 PMID:18334694        

52. Crisp N. Turning the world upside down: the search for global health in the 21st century. London: Royal Society of Medicine Press; 2010.         

(Submitted: 19 January 2011 – Revised version received: 26 April 2011 – Accepted: 23 May 2011 – Published online: 5 July 2011)

* Correspondence to Geoff Royston (e-mail: geoff.royston@gmail.com)