POLICY AND PRATICE
Knowledge for better health a conceptual framework and foundation for health research systems
La connaissance au service de la santé – cadre conceptuel et base des systèmes de recherche en santé
Aprovechar los conocimientos para mejorar la salud: marco conceptual y base para los sistemas de investigación sanitaria
Tikki PangI, 1; Ritu SadanaII; Steve HanneyIII; Zulfiqar A. BhuttaIV; Adnan A. HyderV; Jonathon SimonVI
IDirector, Research Policy & Cooperation, World Health Organization, Geneva, Switzerland (email: pangt@who.int)
IIScientist, Research Policy & Cooperation, World Health Organization, Geneva, Switzerland
IIIResearch Fellow, Health Economics Research Group, Brunel University, England
IVProfessor, Department of Pediatrics, Aga Khan University, Karachi, Pakistan
VAssistant Professor and Leon Robertson Faculty Development Chair, Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, USA
VIDirector, Center for International Health and Development, Boston University School of Public Health, USA
ABSTRACT
Health research generates knowledge that can be utilized to improve health system performance and, ultimately, health and health equity. We propose a conceptual framework for health research systems (HRSs) that defines their boundaries, components, goals, and functions. The framework adopts a systems perspective towards HRSs and serves as a foundation for constructing a practical approach to describe and analyse HRSs. The analysis of HRSs should, in turn, provide a better understanding of how research contributes to gains in health and health equity. In this framework, the intrinsic goals of the HRS are the advancement of scientific knowledge and the utilization of knowledge to improve health and health equity. Its four principal functions are stewardship, financing, creating and sustaining resources, and producing and using research. The framework, as it is applied in consultation with countries, will provide countries and donor agencies with relevant inputs to policies and strategies for strengthening HRSs and using knowledge for better health.
Keywords: Health services research/organization and administration; Knowledge; Information management; Models, Theoretical; Review literature; Meta-analysis (source: MeSH, NLM).
RÉSUMÉ
La recherche en santé génère des données qui peuvent être utilisées pour rendre les systèmes de santé plus performants et pour, en fin de compte, améliorer la santé et l'équité en matière de santé. Nous proposons un cadre conceptuel pour les systèmes de recherche en santé dans lequel sont définis les limites, les composantes, les objectifs et les fonctions de ces systèmes. Le cadre adopte une perspective systémique et sert à mettre au point une approche pratique visant à décrire et analyser ces systèmes de recherche. L'analyse de ces systèmes devrait permettre de mieux comprendre comment la recherche contribue à faire avancer la santé et l'équité en matière de santé. Le cadre propose que les systèmes de recherche en santé aient deux objectifs, l'amélioration des connaissances scientifiques et l'utilisation de ces connaissances pour améliorer la santé et l'équité en matière de santé, et quatre fonctions principales, l'administration, le financement, la génération et le maintien de ressources ainsi que la production et l'utilisation de recherches. Le cadre, de par son application en consultation avec les pays, fournira aux pays et aux organismes donateurs les éléments nécessaires aux politiques et aux stratégies pour renforcer les systèmes de recherche en santé et mettre les connaissances au service de la santé.
Mots clés: Recherche en santé publique/organisation et administration; Connaissance; Gestion information; Modèle théorique; Revue de la littérature; Méta-analyse (source: MeSH, INSERM).
RESUMEN
Las investigaciones sanitarias generan conocimientos que pueden utilizarse para mejorar el desempeño de los sistemas de salud y, en último término, la salud y la equidad sanitaria. Proponemos aquí un marco conceptual para los sistemas de investigación sanitaria (SIS) que define sus límites, componentes, metas y funciones. El marco adopta una perspectiva de sistemas ante los SIS y sirve de base para desarrollar un método práctico de descripción y análisis de esos sistemas. El análisis de los SIS debería a su vez permitirnos comprender mejor cuáles son los mecanismos por los que las investigaciones propician mejoras de la salud y de la equidad sanitaria. En este marco, las metas intrínsecas de los SIS son el avance de los conocimientos científicos y la utilización de dichos conocimientos para mejorar la salud y la equidad en ese terreno. Sus cuatro principales funciones son la rectoría, el financiamiento, la creación y el sostenimiento de los recursos, y la realización y aplicación de investigaciones. El marco, aplicado en consulta con los países, proporcionará a éstos y a los organismos donantes datos de interés para las políticas y las estrategias con miras a fortalecer los SIS y asegurar que los conocimientos se apliquen y redunden en mejoras de la salud.
Palabras clave: Investigación sobre servicios de salud/organización y administración; Conocimiento; Gerencia de la información; Modelos teóricos; Literatura de revisión; Meta-análisis (fuente: DeCS, BIREME).
Introduction
Knowledge produced by health research, if disseminated widely, is a global public good. Knowledge contributes to the policies, activities, and performance of health systems (1–3), and to the improvement of individuals' and populations' health. Using existing knowledge adapted to local conditions is particularly crucial in achieving the Millennium Development Goals (4). To achieve these and other health-related goals, a well-functioning health system must be able to access and utilize research-based knowledge and the products of research (5). It should also be part of the global effort to generate new knowledge to address the problems of tomorrow.
A systems perspective will enable research stakeholders to improve their understanding and implementation of a national health research system (HRS) in order to improve health outcomes and health equity. The Commission on Macroeconomics and Health recently reiterated the need for a substantial increase in health research investments globally (6). These and other efforts (7, 8) highlight the important role of research and scientific knowledge (9) in addressing the diseases and conditions that afflict people, particularly in the developing world.
Why the need for a "systems" perspective? Health research is too often a fragmented, competitive, highly specialized, sectoral activity where researchers within scientific disciplines often work in isolation from other disciplines. There is often little effective communication and consultation between these producers of research and the end-users (i.e. the decision- and policy-makers, health professionals, consumers, and the public). Arguably, there is a need for a rational framework that values both the production and use of research, and a platform for effective communication and interaction between all the players and stakeholders in health research (10–13). This requires a look at the endeavour from a more integrated and coordinated, systems perspective.
The landmark Commission on Health Research for Development (14) noted that "research is a system involving people, institutions and processes", yet relatively few formal attempts to articulate and define this system's boundaries, goals and functions, either for descriptive or analytical purposes, have been put forward. The present paper proposes a definition for HRSs and describes the goals and functions of a "model" HRS. We then discuss selected issues with regard to the functions. The objective is that the conceptual framework will guide the development of further operational work within the recently launched Health Research System Analysis (HRSA) initiative of WHO. It is intended that the processes, sets of tools, descriptions, and analyses resulting from this work will provide useful information to identify potential policy options to strengthen national HRSs, facilitate cross-national sharing of experiences, use scientific knowledge to improve actions within health systems, and ultimately improve health and health equity. The HRSA initiative is also part of WHO's strategy to help Member States achieve Millennium Development Goals.
Methods
The conceptual framework presented here was developed on the basis of a comprehensive literature review and an extensive consultation process, which began in January 2001. These consultations involved many experts in the field, interested organizations, a wide range of researchers and representatives from different countries, and individuals and institutions working on strengthening HRSs. More than 100 individuals from at least 40 countries have actively contributed to the process during 10 consultations and forums held between March 2001 and June 2003.
Definitions, boundaries, and context of HRSs
Health research has been broadly defined as "the generation of new knowledge using the scientific method to identify and deal with health problems" (14). A "system" has been defined as a group of elements operating together to achieve a common goal (15). We define a HRS as: the people, institutions, and activities whose primary purpose in relation to research is to generate high-quality knowledge that can be used to promote, restore, and/or maintain the health status of populations; it should include the mechanisms adopted to encourage the utilization of research. The definition includes all actorsinvolved in knowledge generation, research synthesis, and using research results in the public and private sectors. The paper focuses primarily on exploring national HRSs rather than private sector, global or subnational contexts. Nevertheless, some of the analysis is relevant for the private sector within national boundaries.
A HRS in a country exists at the intersection of two, larger, complex systems the health system and the research system (Fig. 1); this subset of the two systems captures the production of health-related knowledge which, when used appropriately, can contribute to the improvement of health. "Health" research involves many different types of research including biomedical, clinical, epidemiological, health systems and policies research, socioeconomic and behavioural research contributions, as well as ongoing programme evaluations, surveillance and operational research activities embedded within health systems. It also includes research not usually considered to be health related for example, engineering studies to improve car or road safety or economic research leading to policy changes that affect poverty.
Strengthening the connections and relevance of the HRS for the health system is a key challenge. This is particularly so given that the transitions from research to policy to actions and eventually to health improvements are non-linear processes. These transitions are also influenced by important factors outside of the system such as professional and public values and expectations and the sociopolitical milieu.
Goals of the HRS
Our framework proposes that the HRS has two complementary, intrinsic goals: the advancement of scientific knowledge and the utilization of knowledge to improve health and health equity. Although health research may produce many other benefits (16), the intrinsic goals of health research proposed here are primary to HRSs.
Functions of the HRS
The framework proposes that the functions of an effective HRS include stewardship, financing, creating and sustaining resources, and producing and using research. Each function is defined by several key operational components (Table 1).
Stewardship
Drawing from the World health report 2000 – Health systems: improving performance (17), we define stewardship as being concerned with oversightof the entire HRS. As with the stewardship function of health systems, stewardship of HRSs is intended to cover both the public and private sectors. Inevitably, however, it has most relevance for the public sector. Although stewardship is usually primarily the responsibility of government, the involvement of representative forums of key stakeholders and players, such as national health research councils or professional associations, is an increasingly common strategy. We propose that stewardship within the HRS should include four components (Table 1): definition and articulation of a vision for a national HRS; identification of appropriate health research priorities and coordination of adherence to them; setting and monitoring of ethical standards for health research and research partnerships; and monitoring and evaluation of the HRS itself.
Financing: securing and allocating funds
The financing of HRSs through securing of research funds and allocating these accountably is a central function of the HRSs (Table 1), as it is for health systems (17). In addition to securing funds from the private and public sectors, both national and external, effective mechanisms for disbursement and subsequent monitoring must be established. An efficient, transparent, and peer-review-based process should be at the core of this function. To reach the goals of HRSs, the allocation of resources should reflect the overall vision and agreed scientific priorities of the national bodies that are responsible for articulating a research agenda, to the extent that the stakeholders can reach a consensus, even if policy-making and actions are almost never guided solely by technical evidence.
Creating and sustaining resources
Creating and sustaining human and physical resources for health research remains as a central issue (18, 19). This capacity strengthening function is not only concerned with bringing new researchers and institutions into the system, but also to further develop and sustain the existent human and physical capacity to conduct, absorb, and utilize health research (16, 20). In addition to good physical facilities to conduct health research, a favourable and conducive enabling environment for research must exist, together with an attractive career structure to entice and retain the most talented individuals. This includes good research management, availability of funding for research, opportunities to present and openly discuss research data, and, in an increasingly "wired" world, rapid access to current research information. A crucial issue is how countries can sustain human resources for national health research, where the already limited numbers of health researchers is further reduced through internal or external migration or "brain drain" (21).
Producing, synthesizing, and utilizing research
The production of valid research disseminated in scientific publications in peer-reviewed and grey (non-peer-reviewed) literature, policy publications, reports, books or discussion papers, etc. is a major part of the fourth function of the HRS. The publication of research findings is considered to be the primary output of the research process (16). However, such outputs and the resulting knowledge need not come from original research only but may also be a result of adapting existing knowledge to local conditions or from research syntheses.
Research can be utilized in two main ways: first, for developing new tools (drugs, vaccines, devices and other applications) to improve health; and second, for translating, communicating, and promoting the utilization of research to inform health policies, strategies, and practices, particularly within health systems. Research can also be used to educate the population and change public opinions and practices. It is generally agreedthat a wide gap exists between current health systems and the needs that health systems should address, one major cause being the inability to synthesize existing research outputs and apply existing knowledgetowards improving interventions and the performance of health systems.
Discussion
Stewardship
The central question is how to obtain the best "bundle" of knowledge, and the resulting gains in health, out of the investments devoted to health research (1). Where good stewardship exists, health research and its utilization have apparently flourished (22, 23). Identifying appropriate health research priorities (3, 24) and coordinating adherence to these are crucial to the efficient functioning of a HRS. Having the ability to link the allocation of funding to national priorities is essential. The stronger national HRSs will also be able to influence multilateral and private sector research flows towards high-priority needs. Finding the appropriate balance among various research disciplines, between commissioned or policy-driven and investigator-driven research, and selecting research investments among competing priorities are major challenges for the HRSs stewards.
Stewardship in relation to ethics (25, 26) is particularly important, as both developed and developing countries face considerable ethical, legal, economic, and social challenges, which have been highlighted in attempts to deal with recent scientific advances in genomics and biotechnology (27). The differential potential to capture the benefits of genomics and other new technologies adds to the myriad of existing inequities between developed and developing countries. Given that much of the knowledge generated by research is "owned" by the private sector and is largely market-driven in terms of applications, the people and organizationswho guide the HRSs need to be vigilant in ensuring that the outputs and outcomes benefit all. This is equally true in providing a vision for the use and benefits of clinical trials in developing countries, the ethics of patenting DNA sequences, and intellectual property rights issues in general. In this regard, stewardship must be fostered in order to provide appropriate incentives to prevent both market and state failures.
Finally, stewardship includes evaluating and monitoring the production and use of health research and resulting knowledge. Most current evaluations of research focus on easily measurable outputs and outcomes such as number of publications. Measuring the impacts on improved policies, better programmes, and improved health outcomes are much harder to accomplish. In addition, although there have been attempts at quantifying the exceptional economic returns on investments in medical research (28, 29), further improvements in methods and scope would be desirable. The ability to improve the efficiency of an HRS will depend on measuring and monitoring the functions, components, and progress towards achieving the intrinsic goals.
Financing
Global financing of health research remains deeply problematic. The Global Forum for Health Research estimates that less than 10% of global resources allocated for health research and development is spent on studying 90% of the world's health problems (7). The Commission on Health Research for Development (14) recommended in 1990 that developing countries should invest at least 2% of national health expenditures in research and research-capacity strengthening and argued that such investment is one of the most powerful, cost-effective, and sustainable means of advancing health and development. Although data on funding levels for health-related research and development is available to some degree for high-income countries (30), there is little or fragmented information from low- and middle-income countries.
Another crucial issue is the key role of the for-profit private sector. The biopharmaceutical industry and medical devices companies spent approximately 42% of the US$ 73 billion investment in health research and development in 1998 (7). The vast majority of this was targeted to health problems of the developed world: a recent report indicated that only 16 out of 1393 new drugs marketed between 1975 and 1999 were for treating tropical diseases that occur inthe developing world (31). The challenge is how to redirect some of this spending to research and development that is related to neglected diseases and conditions of the developing world.
Creating and sustaining resources
The "brain drain" of health researchers from the developing to the developed countries is considered to be an important issue for health research (21); for example, around 23 000 academic professionals emigrate from Africa annually. As well as a country having research expertise, having research capacity assists that country in learning, adapting, and benefiting from research conducted elsewhere (16). Such assets could be weakened by the brain drain. However, a charged debate has emerged as to whether the brain drain phenomenon is a loss or a benefit to developing countries (32). Brain drain could benefit developing countries, for example, when emigrating researchers maintain contact and provide valuable technical and material assistance to their home institutions. Brain drain also occurs between high-income countries and within low-income countries (21) a phenomenon referred to as "internal brain drain", where qualified researchers are lost to other, non-research sectors. Some recent ideas to address the brain drain issue include the creation of attractive schemes to persuade scientists of developing countries to remain in their home countries (33) or to return if they have emigrated (34), or the creation of properly structured research partnerships (35). There is a general consensus that attractive institutions, and career structures and pathways, are essential to retain and expand a national health researcher cadre.
Producing and using research
The process of conducting research studies and utilizing knowledge is a highly complex one (36, 37). One glaring symptom of the current weakness of HRSs across countries is that the research process and the policy process tend to exist in different worlds (11, 12, 38), with the result that research often has a limited impact on policy. Researchers and decision-makers tend to interact only around the "products" of their processes for example, the results of a study for the researcher and a set of priorities for the decision-maker. Clearly, more attention needs to be given to establishing and maintaining ongoing links between the two worlds (12)and, as noted previously, taking stock of the non-linearity of the research–policy–practice processes.
Improved communication among the researchers, policy-makers, and consumers requires new approaches. Far greater use is being made of systematic reviews (39) that attempt to distill and synthesize the vast amount of research results in a manner that will help to inform researchers, policy-makers, practitioners, and members of the public. These research syntheses are important because of the cumulative nature of science, the knowledge "explosion", and haphazard and biased publication or access to research. The utilization of research is also influenced by access to information resources and Internet connectivity to facilitate networking. Many researcherswelcome initiatives like HINARI (Health InterNetwork Access to Research Initiative) (40), which allows free full-text access to more than 1500 journals to developing country researchers or access to e-journals that are freely accessible through the Internet.
Gauging the impact of research on practice is clearly an important facet of the evidence base on knowledge utilization. Some developing approaches link bibliometric analysis with the use of research findings in clinical guidelines (41). An important review (42) has identified numerous studies investigating interventions aimed at changing the behaviours of health care professionals so that they were more in line with the latest research evidence. The impact of research on public opinion is receiving increasing attention (43, 44), with particular interest in novel communication and dissemination approaches as a means of increasing public awareness and understanding, as well as the relevance of research results. Active two-way engagement between the community and the HRS is seen to be increasingly important, with some communities involved in guiding research priorities, thereby helping to ensure the effective application of and even participation in the conduct of research.
Next steps
The framework proposed is a first attempt at systematically identifying the existence and attributes of HRSs in countries. The process of conceptual exploration has been intense and inclusive, with special emphasis on low- and middle-income countries. Participants across countries have postulated that without effective stewardship, sufficient financing, and adequate human and physical resources, the challenge to produce and use scientifically valid research may be insurmountable. Any analyses and eventual strategies to strengthen HRSs should therefore address all functions and goals rather than focus narrowly on research outputs.
The conceptual framework is not intended as a blueprint for the precise organization of HRSs: it is expected that the framework will require some modification within each country. This framework presents a set of unifying ideas that need to be evaluated, and further operational development needs to be pursued and validated, based on experiences from many countries. This operational process, development of new methods and testing in a diverse set of national contexts are currently under way with representatives from some 20 developing and developed countries around the world. Further deliberations and the results from extensive testing will improve the conceptual and technical basis of the framework, and inform the planned important WHO report in 2004 addressing health research and knowledge for better health.
Concluding remarks
Governments and donor agencies are increasingly interested in evaluating the costs and benefits of their investments in health research. The productivity and efficiency of these investments are strongly influenced by the HRSs in which individual scientists operate, even if research results are ultimately global public goods. The concept of a national HRS is therefore receiving renewed attention (13). A conceptual framework and overall typology of HRSs will serve as a foundation to base operational descriptions and analyses on a wide array of current issues and projected challenges. Such analyses will, in turn, allow benchmarking, identification of best practices and lessons to be learnt, within and across countries, as a means of enhancing the functions of the HRS and better achieve its goals. Developing countries in particular will hopefully benefit from more efficient and effective management of a national HRS and the crafting of rational health research policies. Besides overall contributions to society, the knowledge generated by research needs to be translated into better health systems and into improving the health of populations and reducing global inequities in health. 
Acknowledgements
We gratefully acknowledge the contributions of numerous individuals in the development of the ideas presented in this paper. In particular, we thank (in alphabetical order) Miguel Gonzalez Block, Jacques Bury, Martin Buxton, Nina Castillo, Somsak Chunharas, Sally Davies, David Evans, Phyllis Freeman, Jonathan Grant, Marian Jacobs, Yongyuth Kachondham, Shyama Kuruvilla, Mary Ann Lansang, Grant Lewison, Sarah MacFarlane, Abdelhay Mechbal, Ophelia Mendoza, Christopher Murray, David Okello, Alberto Pellegrini, Alan Pettigrew, Abha Saxena, Sacha Sidjanski, Chitr Sitthi-Amorn, Tessa Tan Torres, Peter Tugwell, Bedirhan Üstün, Jimmy Volmink, and Adik Wibowo. We thank everyone for their support and faith in our attempt to do something that is meaningful and potentially useful, but extremely difficult.
Funding: We gratefully acknowledge financial support for this work from the Rockefeller Foundation (USA) and the Swedish International Development Agency (SIDA/Sarec).
Conflicts of interest: none declared.
References
1. Investing in health research for development. Report of the Ad Hoc Committee on Health Research Relating to Future Intervention Options. Geneva: World Health Organization; 1996.
2. Pardes H, Manton KG, Lander ES, Tolley HD, Ullian AD, Palmer H. Effects of medical research on health care and the economy. Science 1999;283:36-7.
3. The Working Group on Priority Setting. Priority setting for health research: lessons from developing countries. Health Policy & Planning 2000;15:130-6.
4. Millennium Development Goals. World Health Organization. Available from: URL: http://www.who.int/mdg/en/ (accessed on 20 February 2003).
5. Kaul I, Faust M. Global public goods and health: taking the agenda forward. Bulletin of the World Health Organization 2001;79:869-74.
6. Macroeconomics and health: investing in health for economic development. Report of the Commission on Macroeconomics and Health. Geneva: World Health Organization; 2001.
7. The 10/90 report on health research, 2001–2002. Geneva: Global Forum for Health Research; 2002.
8. Altman LK. Gates gives $200 million to aid poor nations. New York Times 2003; 27 January, 16.
9. Lee JW. Science and the health of the poor. Bulletin of the World Health Organization 2003; 81:473.
10. Frenk J. Balancing relevance and excellence: organizational responses to link research with decision making. Social Science and Medicine 1992;35:1397-404.
11. Hanney S, Gonzalez-Block M, Buxton M, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Research Policy and Systems 2003;1:2. Available from: URL: http://www.health-policy-systems.com/content/1/1/2 (accessed on 10 October 2003).
12. Lomas J. Connecting research and policy. ISUMA 2000;1:140-4.
13. National health research systems. Report of an international workshop. Geneva: World Health Organization; 2001.
14. Health research-essential link to equity in development. Report of the Commission on Health Research for Development. New York: Oxford University Press; 1990.
15. Forrester JW. Principles of systems. Williston: Pegasus Communications; 1990.
16. Buxton M, Hanney S. How can payback from health services research be assessed? Journal of Health Services Research and Policy 1996;1:25-43.
17. The World health report 2000 – Health systems: improving performance. Geneva: World Health Organization; 2000.
18. Nchinda T. Research capacity strengthening in the South. Social Science and Medicine 2002; 54:1699-711.
19. White F. Capacity-building for health research in developing countries: a manager's approach. Pan American Journal of Public Health 2002;12:165-72.
20. Dialogue on research and policy interlinks in public health. Training and Research in Public Health Dialogue Series No. 4. Copenhagen: WHO Regional Office for Europe; 1996.
21. Pang T, Lansang MA, Haines A. Brain drain and health professionals. BMJ 2002;324:499-500.
22. Stiglitz J. Scan globally, reinvent locally: knowledge infrastructure and the localization of knowledge. Keynote address, first Global Development Network conference. Bonn, Germany; 1999.
23. For 80 cents more even a tiny health budget, if spent well, can make a difference. The Economist 2002; 17 August, 20-2.
24. Remme JF , Blas E, Chitsulo L, Desjeux PM, Engers HD, Kanyok TP, et al. Strategic emphases for tropical diseases research: a TDR perspective. Trends in Parasitology 2002;18:421-6.
25. The ethics of research related to healthcare in developing countries. London: Nuffield Council on Bioethics; 2002.
26. Ethical and policy issues in international research: clinical trials in developing countries. Bethesda: National Bioethics Advisory Commission; 2001.
27. Genomics and World Health. Report of the Advisory Committee on Health Research. Geneva: World Health Organization; 2002.
28. Murphy K, Topel R, editors. Measuring the gains from medical research: an economic approach. Chicago; Chicago University Press; 2003.
29. Rosenberg LE. Exceptional economic returns on investments in medical research. Medical Journal of Australia 2002;177:368-71.
30. Measuring expenditure on health-related R & D. Paris: Organization for Economic Co-operation and Development (OECD); 2001.
31. Trouiller P, Olliaro P, Torreele E, Orbinski J, Laing R, Ford N. Drug development for neglected diseases: a deficient market and a public health policy failure. Lancet 2002;359:2188-93.
32. Outward bound. The Economist 2002; 28 September, 29-32.
33. Plugging the brain drain. Nature 2002;417:683.
34. The reverse brain drain project. National Science and Technology Development Agency, Thailand. Available from: URL: http://rbd.nstda.or.th (accessed on 24 February 2003).
35. Heller PS, Mills A. The brain drain health workers here and there. International Herald Tribune 2002; 25 July, 8.
36. Yach D, Dick J. Implementation of research: the key to closing the gap between public health knowledge and action. Community Health Association of South Africa Journal of Comprehensive Health 1991; 2:15-20.
37. Garner P, Kale R, Dickson R, Dans T, Salinas R. Implementing research findings in developing countries. BMJ 1998;317:531-5.
38. Trostle JF, Bronfman M, Langer A. How do researchers influence decision makers? Case studies of Mexican policies. Health Policy and Planning 1999;14:103-14.
39. Systematic reviews in health care. Egger M, Smith GD, Altman DG, editors. London: BMJ Books; 2001.
40. Aronson B. WHO's health InterNetwork Access to Research Initiative (HINARI). Health Information and Libraries Journal 2002;19:164-5.
41. Grant J, Cottrell R, Cluzeau F, Fawcett G. Evaluating payback on biomedical research from papers cited in clinical guidelines: applied bibliometric study. BMJ 2000;320:1107-11.
42. Getting evidence into practice. Effective Health Care 1999;5:1-16.
43. Irwin A. Constructing the scientific citizen: science and democracy in the biosciences. Public Understanding of Science 2001;10:1-18.
44. Leshner AI. Public engagement with science. Science 2003:299:977.
1 Correspondence should be sent to this author.